1# Run a V2 membership change that adds two voters at once and auto-leaves the
2# joint configuration. (This is the same as specifying an explicit transition
3# since more than one change is being made atomically).
4
5# Bootstrap n1.
6add-nodes 1 voters=(1) index=2
7----
8INFO 1 switched to configuration voters=(1)
9INFO 1 became follower at term 0
10INFO newRaft 1 [peers: [1], term: 0, commit: 2, applied: 2, lastindex: 2, lastterm: 1]
11
12campaign 1
13----
14INFO 1 is starting a new election at term 0
15INFO 1 became candidate at term 1
16INFO 1 received MsgVoteResp from 1 at term 1
17INFO 1 became leader at term 1
18
19propose-conf-change 1 transition=auto
20v2 v3
21----
22ok
23
24# Add two "empty" nodes to the cluster, n2 and n3.
25add-nodes 2
26----
27INFO 2 switched to configuration voters=()
28INFO 2 became follower at term 0
29INFO newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
30INFO 3 switched to configuration voters=()
31INFO 3 became follower at term 0
32INFO newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
33
34# n1 immediately gets to commit & apply the conf change using only itself. We see that
35# it starts transitioning out of that joint configuration (though we will only see that
36# proposal in the next ready handling loop, when it is emitted). We also see that this
37# is using joint consensus, which it has to since we're carrying out two additions at
38# once.
39process-ready 1
40----
41Ready MustSync=true:
42Lead:1 State:StateLeader
43HardState Term:1 Vote:1 Commit:4
44Entries:
451/3 EntryNormal ""
461/4 EntryConfChangeV2 v2 v3
47CommittedEntries:
481/3 EntryNormal ""
491/4 EntryConfChangeV2 v2 v3
50INFO 1 switched to configuration voters=(1 2 3)&&(1) autoleave
51INFO initiating automatic transition out of joint configuration voters=(1 2 3)&&(1) autoleave
52
53# n1 immediately probes n2 and n3.
54stabilize 1
55----
56> 1 handling Ready
57 Ready MustSync=true:
58 Entries:
59 1/5 EntryConfChangeV2
60 Messages:
61 1->2 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
62 1->3 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
63
64# First, play out the whole interaction between n1 and n2. We see n1's probe to
65# n2 get rejected (since n2 needs a snapshot); the snapshot is delivered at which
66# point n2 switches to the correct config, and n1 catches it up. This notably
67# includes the empty conf change which gets committed and applied by both and
68# which transitions them out of their joint configuration into the final one (1 2 3).
69stabilize 1 2
70----
71> 2 receiving messages
72 1->2 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
73 INFO 2 [term: 0] received a MsgApp message with higher term from 1 [term: 1]
74 INFO 2 became follower at term 1
75 DEBUG 2 [logterm: 0, index: 3] rejected MsgApp [logterm: 1, index: 3] from 1
76> 2 handling Ready
77 Ready MustSync=true:
78 Lead:1 State:StateFollower
79 HardState Term:1 Commit:0
80 Messages:
81 2->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
82> 1 receiving messages
83 2->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
84 DEBUG 1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 2 for index 3
85 DEBUG 1 decreased progress of 2 to [StateProbe match=0 next=1]
86 DEBUG 1 [firstindex: 3, commit: 4] sent snapshot[index: 4, term: 1] to 2 [StateProbe match=0 next=1]
87 DEBUG 1 paused sending replication messages to 2 [StateSnapshot match=0 next=1 paused pendingSnap=4]
88> 1 handling Ready
89 Ready MustSync=false:
90 Messages:
91 1->2 MsgSnap Term:1 Log:0/0 Snapshot: Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
92> 2 receiving messages
93 1->2 MsgSnap Term:1 Log:0/0 Snapshot: Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
94 INFO log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 4, term: 1]
95 INFO 2 switched to configuration voters=(1 2 3)&&(1) autoleave
96 INFO 2 [commit: 4, lastindex: 4, lastterm: 1] restored snapshot [index: 4, term: 1]
97 INFO 2 [commit: 4] restored snapshot [index: 4, term: 1]
98> 2 handling Ready
99 Ready MustSync=false:
100 HardState Term:1 Commit:4
101 Snapshot Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
102 Messages:
103 2->1 MsgAppResp Term:1 Log:0/4
104> 1 receiving messages
105 2->1 MsgAppResp Term:1 Log:0/4
106 DEBUG 1 recovered from needing snapshot, resumed sending replication messages to 2 [StateSnapshot match=4 next=5 paused pendingSnap=4]
107> 1 handling Ready
108 Ready MustSync=false:
109 Messages:
110 1->2 MsgApp Term:1 Log:1/4 Commit:4 Entries:[1/5 EntryConfChangeV2]
111> 2 receiving messages
112 1->2 MsgApp Term:1 Log:1/4 Commit:4 Entries:[1/5 EntryConfChangeV2]
113> 2 handling Ready
114 Ready MustSync=true:
115 Entries:
116 1/5 EntryConfChangeV2
117 Messages:
118 2->1 MsgAppResp Term:1 Log:0/5
119> 1 receiving messages
120 2->1 MsgAppResp Term:1 Log:0/5
121> 1 handling Ready
122 Ready MustSync=false:
123 HardState Term:1 Vote:1 Commit:5
124 CommittedEntries:
125 1/5 EntryConfChangeV2
126 Messages:
127 1->2 MsgApp Term:1 Log:1/5 Commit:5
128 INFO 1 switched to configuration voters=(1 2 3)
129> 2 receiving messages
130 1->2 MsgApp Term:1 Log:1/5 Commit:5
131> 2 handling Ready
132 Ready MustSync=false:
133 HardState Term:1 Commit:5
134 CommittedEntries:
135 1/5 EntryConfChangeV2
136 Messages:
137 2->1 MsgAppResp Term:1 Log:0/5
138 INFO 2 switched to configuration voters=(1 2 3)
139> 1 receiving messages
140 2->1 MsgAppResp Term:1 Log:0/5
141
142# n3 immediately receives a snapshot in the final configuration.
143stabilize 1 3
144----
145> 3 receiving messages
146 1->3 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
147 INFO 3 [term: 0] received a MsgApp message with higher term from 1 [term: 1]
148 INFO 3 became follower at term 1
149 DEBUG 3 [logterm: 0, index: 3] rejected MsgApp [logterm: 1, index: 3] from 1
150> 3 handling Ready
151 Ready MustSync=true:
152 Lead:1 State:StateFollower
153 HardState Term:1 Commit:0
154 Messages:
155 3->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
156> 1 receiving messages
157 3->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
158 DEBUG 1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 3 for index 3
159 DEBUG 1 decreased progress of 3 to [StateProbe match=0 next=1]
160 DEBUG 1 [firstindex: 3, commit: 5] sent snapshot[index: 5, term: 1] to 3 [StateProbe match=0 next=1]
161 DEBUG 1 paused sending replication messages to 3 [StateSnapshot match=0 next=1 paused pendingSnap=5]
162> 1 handling Ready
163 Ready MustSync=false:
164 Messages:
165 1->3 MsgSnap Term:1 Log:0/0 Snapshot: Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
166> 3 receiving messages
167 1->3 MsgSnap Term:1 Log:0/0 Snapshot: Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
168 INFO log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 5, term: 1]
169 INFO 3 switched to configuration voters=(1 2 3)
170 INFO 3 [commit: 5, lastindex: 5, lastterm: 1] restored snapshot [index: 5, term: 1]
171 INFO 3 [commit: 5] restored snapshot [index: 5, term: 1]
172> 3 handling Ready
173 Ready MustSync=false:
174 HardState Term:1 Commit:5
175 Snapshot Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
176 Messages:
177 3->1 MsgAppResp Term:1 Log:0/5
178> 1 receiving messages
179 3->1 MsgAppResp Term:1 Log:0/5
180 DEBUG 1 recovered from needing snapshot, resumed sending replication messages to 3 [StateSnapshot match=5 next=6 paused pendingSnap=5]
181> 1 handling Ready
182 Ready MustSync=false:
183 Messages:
184 1->3 MsgApp Term:1 Log:1/5 Commit:5
185> 3 receiving messages
186 1->3 MsgApp Term:1 Log:1/5 Commit:5
187> 3 handling Ready
188 Ready MustSync=false:
189 Messages:
190 3->1 MsgAppResp Term:1 Log:0/5
191> 1 receiving messages
192 3->1 MsgAppResp Term:1 Log:0/5
193
194# Nothing else happens.
195stabilize
196----
197ok
198
199# Now remove two nodes. What's new here is that the leader will actually have
200# to go to a quorum to commit the transition into the joint config.
201
202propose-conf-change 1
203r2 r3
204----
205ok
206
207# n1 sends out MsgApps.
208stabilize 1
209----
210> 1 handling Ready
211 Ready MustSync=true:
212 Entries:
213 1/6 EntryConfChangeV2 r2 r3
214 Messages:
215 1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
216 1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
217
218# n2, n3 ack them.
219stabilize 2 3
220----
221> 2 receiving messages
222 1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
223> 3 receiving messages
224 1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
225> 2 handling Ready
226 Ready MustSync=true:
227 Entries:
228 1/6 EntryConfChangeV2 r2 r3
229 Messages:
230 2->1 MsgAppResp Term:1 Log:0/6
231> 3 handling Ready
232 Ready MustSync=true:
233 Entries:
234 1/6 EntryConfChangeV2 r2 r3
235 Messages:
236 3->1 MsgAppResp Term:1 Log:0/6
237
238# n1 gets some more proposals. This is part of a regression test: There used to
239# be a bug in which these proposals would prompt the leader to transition out of
240# the same joint state multiple times, which would cause a panic.
241propose 1 foo
242----
243ok
244
245propose 1 bar
246----
247ok
248
249# n1 switches to the joint config, then initiates a transition into the final
250# config.
251stabilize 1
252----
253> 1 handling Ready
254 Ready MustSync=true:
255 Entries:
256 1/7 EntryNormal "foo"
257 1/8 EntryNormal "bar"
258 Messages:
259 1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
260 1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
261 1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
262 1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
263> 1 receiving messages
264 2->1 MsgAppResp Term:1 Log:0/6
265 3->1 MsgAppResp Term:1 Log:0/6
266> 1 handling Ready
267 Ready MustSync=false:
268 HardState Term:1 Vote:1 Commit:6
269 CommittedEntries:
270 1/6 EntryConfChangeV2 r2 r3
271 Messages:
272 1->2 MsgApp Term:1 Log:1/8 Commit:6
273 1->3 MsgApp Term:1 Log:1/8 Commit:6
274 INFO 1 switched to configuration voters=(1)&&(1 2 3) autoleave
275 INFO initiating automatic transition out of joint configuration voters=(1)&&(1 2 3) autoleave
276> 1 handling Ready
277 Ready MustSync=true:
278 Entries:
279 1/9 EntryConfChangeV2
280
281# n2 and n3 also switch to the joint config, and ack the transition out of it.
282stabilize 2 3
283----
284> 2 receiving messages
285 1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
286 1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
287 1->2 MsgApp Term:1 Log:1/8 Commit:6
288> 3 receiving messages
289 1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
290 1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
291 1->3 MsgApp Term:1 Log:1/8 Commit:6
292> 2 handling Ready
293 Ready MustSync=true:
294 HardState Term:1 Commit:6
295 Entries:
296 1/7 EntryNormal "foo"
297 1/8 EntryNormal "bar"
298 CommittedEntries:
299 1/6 EntryConfChangeV2 r2 r3
300 Messages:
301 2->1 MsgAppResp Term:1 Log:0/7
302 2->1 MsgAppResp Term:1 Log:0/8
303 2->1 MsgAppResp Term:1 Log:0/8
304 INFO 2 switched to configuration voters=(1)&&(1 2 3) autoleave
305> 3 handling Ready
306 Ready MustSync=true:
307 HardState Term:1 Commit:6
308 Entries:
309 1/7 EntryNormal "foo"
310 1/8 EntryNormal "bar"
311 CommittedEntries:
312 1/6 EntryConfChangeV2 r2 r3
313 Messages:
314 3->1 MsgAppResp Term:1 Log:0/7
315 3->1 MsgAppResp Term:1 Log:0/8
316 3->1 MsgAppResp Term:1 Log:0/8
317 INFO 3 switched to configuration voters=(1)&&(1 2 3) autoleave
318
319# n2 and n3 also leave the joint config and the dust settles. We see at the very
320# end that n1 receives some messages from them that it refuses because it does
321# not have them in its config any more.
322stabilize
323----
324> 1 receiving messages
325 2->1 MsgAppResp Term:1 Log:0/7
326 2->1 MsgAppResp Term:1 Log:0/8
327 2->1 MsgAppResp Term:1 Log:0/8
328 3->1 MsgAppResp Term:1 Log:0/7
329 3->1 MsgAppResp Term:1 Log:0/8
330 3->1 MsgAppResp Term:1 Log:0/8
331> 1 handling Ready
332 Ready MustSync=false:
333 HardState Term:1 Vote:1 Commit:8
334 CommittedEntries:
335 1/7 EntryNormal "foo"
336 1/8 EntryNormal "bar"
337 Messages:
338 1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
339 1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
340 1->2 MsgApp Term:1 Log:1/9 Commit:8
341 1->3 MsgApp Term:1 Log:1/9 Commit:8
342> 2 receiving messages
343 1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
344 1->2 MsgApp Term:1 Log:1/9 Commit:8
345> 3 receiving messages
346 1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
347 1->3 MsgApp Term:1 Log:1/9 Commit:8
348> 2 handling Ready
349 Ready MustSync=true:
350 HardState Term:1 Commit:8
351 Entries:
352 1/9 EntryConfChangeV2
353 CommittedEntries:
354 1/7 EntryNormal "foo"
355 1/8 EntryNormal "bar"
356 Messages:
357 2->1 MsgAppResp Term:1 Log:0/9
358 2->1 MsgAppResp Term:1 Log:0/9
359> 3 handling Ready
360 Ready MustSync=true:
361 HardState Term:1 Commit:8
362 Entries:
363 1/9 EntryConfChangeV2
364 CommittedEntries:
365 1/7 EntryNormal "foo"
366 1/8 EntryNormal "bar"
367 Messages:
368 3->1 MsgAppResp Term:1 Log:0/9
369 3->1 MsgAppResp Term:1 Log:0/9
370> 1 receiving messages
371 2->1 MsgAppResp Term:1 Log:0/9
372 2->1 MsgAppResp Term:1 Log:0/9
373 3->1 MsgAppResp Term:1 Log:0/9
374 3->1 MsgAppResp Term:1 Log:0/9
375> 1 handling Ready
376 Ready MustSync=false:
377 HardState Term:1 Vote:1 Commit:9
378 CommittedEntries:
379 1/9 EntryConfChangeV2
380 Messages:
381 1->2 MsgApp Term:1 Log:1/9 Commit:9
382 1->3 MsgApp Term:1 Log:1/9 Commit:9
383 INFO 1 switched to configuration voters=(1)
384> 2 receiving messages
385 1->2 MsgApp Term:1 Log:1/9 Commit:9
386> 3 receiving messages
387 1->3 MsgApp Term:1 Log:1/9 Commit:9
388> 2 handling Ready
389 Ready MustSync=false:
390 HardState Term:1 Commit:9
391 CommittedEntries:
392 1/9 EntryConfChangeV2
393 Messages:
394 2->1 MsgAppResp Term:1 Log:0/9
395 INFO 2 switched to configuration voters=(1)
396> 3 handling Ready
397 Ready MustSync=false:
398 HardState Term:1 Commit:9
399 CommittedEntries:
400 1/9 EntryConfChangeV2
401 Messages:
402 3->1 MsgAppResp Term:1 Log:0/9
403 INFO 3 switched to configuration voters=(1)
404> 1 receiving messages
405 2->1 MsgAppResp Term:1 Log:0/9
406 raft: cannot step as peer not found
407 3->1 MsgAppResp Term:1 Log:0/9
408 raft: cannot step as peer not found
View as plain text