# Run a V2 membership change that adds two voters at once and auto-leaves the
# joint configuration. (This is the same as specifying an explicit transition
# since more than one change is being made atomically).

# Bootstrap n1.
add-nodes 1 voters=(1) index=2
----
INFO 1 switched to configuration voters=(1)
INFO 1 became follower at term 0
INFO newRaft 1 [peers: [1], term: 0, commit: 2, applied: 2, lastindex: 2, lastterm: 1]

campaign 1
----
INFO 1 is starting a new election at term 0
INFO 1 became candidate at term 1
INFO 1 received MsgVoteResp from 1 at term 1
INFO 1 became leader at term 1

propose-conf-change 1 transition=auto
v2 v3
----
ok

# Add two "empty" nodes to the cluster, n2 and n3.
add-nodes 2
----
INFO 2 switched to configuration voters=()
INFO 2 became follower at term 0
INFO newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
INFO 3 switched to configuration voters=()
INFO 3 became follower at term 0
INFO newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]

# n1 immediately gets to commit & apply the conf change using only itself. We see that
# it starts transitioning out of that joint configuration (though we will only see that
# proposal in the next ready handling loop, when it is emitted). We also see that this
# is using joint consensus, which it has to since we're carrying out two additions at
# once.
process-ready 1
----
Ready MustSync=true:
Lead:1 State:StateLeader
HardState Term:1 Vote:1 Commit:4
Entries:
1/3 EntryNormal ""
1/4 EntryConfChangeV2 v2 v3
CommittedEntries:
1/3 EntryNormal ""
1/4 EntryConfChangeV2 v2 v3
INFO 1 switched to configuration voters=(1 2 3)&&(1) autoleave
INFO initiating automatic transition out of joint configuration voters=(1 2 3)&&(1) autoleave

# n1 immediately probes n2 and n3.
stabilize 1
----
> 1 handling Ready
  Ready MustSync=true:
  Entries:
  1/5 EntryConfChangeV2
  Messages:
  1->2 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
  1->3 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]

# First, play out the whole interaction between n1 and n2. We see n1's probe to
# n2 get rejected (since n2 needs a snapshot); the snapshot is delivered at which
# point n2 switches to the correct config, and n1 catches it up. This notably
# includes the empty conf change which gets committed and applied by both and
# which transitions them out of their joint configuration into the final one (1 2 3).
stabilize 1 2
----
> 2 receiving messages
  1->2 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
  INFO 2 [term: 0] received a MsgApp message with higher term from 1 [term: 1]
  INFO 2 became follower at term 1
  DEBUG 2 [logterm: 0, index: 3] rejected MsgApp [logterm: 1, index: 3] from 1
> 2 handling Ready
  Ready MustSync=true:
  Lead:1 State:StateFollower
  HardState Term:1 Commit:0
  Messages:
  2->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
> 1 receiving messages
  2->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
  DEBUG 1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 2 for index 3
  DEBUG 1 decreased progress of 2 to [StateProbe match=0 next=1]
  DEBUG 1 [firstindex: 3, commit: 4] sent snapshot[index: 4, term: 1] to 2 [StateProbe match=0 next=1]
  DEBUG 1 paused sending replication messages to 2 [StateSnapshot match=0 next=1 paused pendingSnap=4]
> 1 handling Ready
  Ready MustSync=false:
  Messages:
  1->2 MsgSnap Term:1 Log:0/0 Snapshot: Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
> 2 receiving messages
  1->2 MsgSnap Term:1 Log:0/0 Snapshot: Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
  INFO log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 4, term: 1]
  INFO 2 switched to configuration voters=(1 2 3)&&(1) autoleave
  INFO 2 [commit: 4, lastindex: 4, lastterm: 1] restored snapshot [index: 4, term: 1]
  INFO 2 [commit: 4] restored snapshot [index: 4, term: 1]
> 2 handling Ready
  Ready MustSync=false:
  HardState Term:1 Commit:4
  Snapshot Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
  Messages:
  2->1 MsgAppResp Term:1 Log:0/4
> 1 receiving messages
  2->1 MsgAppResp Term:1 Log:0/4
  DEBUG 1 recovered from needing snapshot, resumed sending replication messages to 2 [StateSnapshot match=4 next=5 paused pendingSnap=4]
> 1 handling Ready
  Ready MustSync=false:
  Messages:
  1->2 MsgApp Term:1 Log:1/4 Commit:4 Entries:[1/5 EntryConfChangeV2]
> 2 receiving messages
  1->2 MsgApp Term:1 Log:1/4 Commit:4 Entries:[1/5 EntryConfChangeV2]
> 2 handling Ready
  Ready MustSync=true:
  Entries:
  1/5 EntryConfChangeV2
  Messages:
  2->1 MsgAppResp Term:1 Log:0/5
> 1 receiving messages
  2->1 MsgAppResp Term:1 Log:0/5
> 1 handling Ready
  Ready MustSync=false:
  HardState Term:1 Vote:1 Commit:5
  CommittedEntries:
  1/5 EntryConfChangeV2
  Messages:
  1->2 MsgApp Term:1 Log:1/5 Commit:5
  INFO 1 switched to configuration voters=(1 2 3)
> 2 receiving messages
  1->2 MsgApp Term:1 Log:1/5 Commit:5
> 2 handling Ready
  Ready MustSync=false:
  HardState Term:1 Commit:5
  CommittedEntries:
  1/5 EntryConfChangeV2
  Messages:
  2->1 MsgAppResp Term:1 Log:0/5
  INFO 2 switched to configuration voters=(1 2 3)
> 1 receiving messages
  2->1 MsgAppResp Term:1 Log:0/5

# n3 immediately receives a snapshot in the final configuration.
stabilize 1 3
----
> 3 receiving messages
  1->3 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
  INFO 3 [term: 0] received a MsgApp message with higher term from 1 [term: 1]
  INFO 3 became follower at term 1
  DEBUG 3 [logterm: 0, index: 3] rejected MsgApp [logterm: 1, index: 3] from 1
> 3 handling Ready
  Ready MustSync=true:
  Lead:1 State:StateFollower
  HardState Term:1 Commit:0
  Messages:
  3->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
> 1 receiving messages
  3->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
  DEBUG 1 received MsgAppResp(rejected, hint: (index 0, term 0)) from 3 for index 3
  DEBUG 1 decreased progress of 3 to [StateProbe match=0 next=1]
  DEBUG 1 [firstindex: 3, commit: 5] sent snapshot[index: 5, term: 1] to 3 [StateProbe match=0 next=1]
  DEBUG 1 paused sending replication messages to 3 [StateSnapshot match=0 next=1 paused pendingSnap=5]
> 1 handling Ready
  Ready MustSync=false:
  Messages:
  1->3 MsgSnap Term:1 Log:0/0 Snapshot: Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
> 3 receiving messages
  1->3 MsgSnap Term:1 Log:0/0 Snapshot: Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
  INFO log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 5, term: 1]
  INFO 3 switched to configuration voters=(1 2 3)
  INFO 3 [commit: 5, lastindex: 5, lastterm: 1] restored snapshot [index: 5, term: 1]
  INFO 3 [commit: 5] restored snapshot [index: 5, term: 1]
> 3 handling Ready
  Ready MustSync=false:
  HardState Term:1 Commit:5
  Snapshot Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
  Messages:
  3->1 MsgAppResp Term:1 Log:0/5
> 1 receiving messages
  3->1 MsgAppResp Term:1 Log:0/5
  DEBUG 1 recovered from needing snapshot, resumed sending replication messages to 3 [StateSnapshot match=5 next=6 paused pendingSnap=5]
> 1 handling Ready
  Ready MustSync=false:
  Messages:
  1->3 MsgApp Term:1 Log:1/5 Commit:5
> 3 receiving messages
  1->3 MsgApp Term:1 Log:1/5 Commit:5
> 3 handling Ready
  Ready MustSync=false:
  Messages:
  3->1 MsgAppResp Term:1 Log:0/5
> 1 receiving messages
  3->1 MsgAppResp Term:1 Log:0/5

# Nothing else happens.
stabilize
----
ok

# Now remove two nodes. What's new here is that the leader will actually have
# to go to a quorum to commit the transition into the joint config.

propose-conf-change 1
r2 r3
----
ok

# n1 sends out MsgApps.
stabilize 1
----
> 1 handling Ready
  Ready MustSync=true:
  Entries:
  1/6 EntryConfChangeV2 r2 r3
  Messages:
  1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
  1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]

# n2, n3 ack them.
stabilize 2 3
----
> 2 receiving messages
  1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
> 3 receiving messages
  1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
> 2 handling Ready
  Ready MustSync=true:
  Entries:
  1/6 EntryConfChangeV2 r2 r3
  Messages:
  2->1 MsgAppResp Term:1 Log:0/6
> 3 handling Ready
  Ready MustSync=true:
  Entries:
  1/6 EntryConfChangeV2 r2 r3
  Messages:
  3->1 MsgAppResp Term:1 Log:0/6

# n1 gets some more proposals. This is part of a regression test: There used to
# be a bug in which these proposals would prompt the leader to transition out of
# the same joint state multiple times, which would cause a panic.
propose 1 foo
----
ok

propose 1 bar
----
ok

# n1 switches to the joint config, then initiates a transition into the final
# config.
stabilize 1
----
> 1 handling Ready
  Ready MustSync=true:
  Entries:
  1/7 EntryNormal "foo"
  1/8 EntryNormal "bar"
  Messages:
  1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
  1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
  1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
  1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
> 1 receiving messages
  2->1 MsgAppResp Term:1 Log:0/6
  3->1 MsgAppResp Term:1 Log:0/6
> 1 handling Ready
  Ready MustSync=false:
  HardState Term:1 Vote:1 Commit:6
  CommittedEntries:
  1/6 EntryConfChangeV2 r2 r3
  Messages:
  1->2 MsgApp Term:1 Log:1/8 Commit:6
  1->3 MsgApp Term:1 Log:1/8 Commit:6
  INFO 1 switched to configuration voters=(1)&&(1 2 3) autoleave
  INFO initiating automatic transition out of joint configuration voters=(1)&&(1 2 3) autoleave
> 1 handling Ready
  Ready MustSync=true:
  Entries:
  1/9 EntryConfChangeV2

# n2 and n3 also switch to the joint config, and ack the transition out of it.
stabilize 2 3
----
> 2 receiving messages
  1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
  1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
  1->2 MsgApp Term:1 Log:1/8 Commit:6
> 3 receiving messages
  1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
  1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
  1->3 MsgApp Term:1 Log:1/8 Commit:6
> 2 handling Ready
  Ready MustSync=true:
  HardState Term:1 Commit:6
  Entries:
  1/7 EntryNormal "foo"
  1/8 EntryNormal "bar"
  CommittedEntries:
  1/6 EntryConfChangeV2 r2 r3
  Messages:
  2->1 MsgAppResp Term:1 Log:0/7
  2->1 MsgAppResp Term:1 Log:0/8
  2->1 MsgAppResp Term:1 Log:0/8
  INFO 2 switched to configuration voters=(1)&&(1 2 3) autoleave
> 3 handling Ready
  Ready MustSync=true:
  HardState Term:1 Commit:6
  Entries:
  1/7 EntryNormal "foo"
  1/8 EntryNormal "bar"
  CommittedEntries:
  1/6 EntryConfChangeV2 r2 r3
  Messages:
  3->1 MsgAppResp Term:1 Log:0/7
  3->1 MsgAppResp Term:1 Log:0/8
  3->1 MsgAppResp Term:1 Log:0/8
  INFO 3 switched to configuration voters=(1)&&(1 2 3) autoleave

# n2 and n3 also leave the joint config and the dust settles. We see at the very
# end that n1 receives some messages from them that it refuses because it does
# not have them in its config any more.
stabilize
----
> 1 receiving messages
  2->1 MsgAppResp Term:1 Log:0/7
  2->1 MsgAppResp Term:1 Log:0/8
  2->1 MsgAppResp Term:1 Log:0/8
  3->1 MsgAppResp Term:1 Log:0/7
  3->1 MsgAppResp Term:1 Log:0/8
  3->1 MsgAppResp Term:1 Log:0/8
> 1 handling Ready
  Ready MustSync=false:
  HardState Term:1 Vote:1 Commit:8
  CommittedEntries:
  1/7 EntryNormal "foo"
  1/8 EntryNormal "bar"
  Messages:
  1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
  1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
  1->2 MsgApp Term:1 Log:1/9 Commit:8
  1->3 MsgApp Term:1 Log:1/9 Commit:8
> 2 receiving messages
  1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
  1->2 MsgApp Term:1 Log:1/9 Commit:8
> 3 receiving messages
  1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
  1->3 MsgApp Term:1 Log:1/9 Commit:8
> 2 handling Ready
  Ready MustSync=true:
  HardState Term:1 Commit:8
  Entries:
  1/9 EntryConfChangeV2
  CommittedEntries:
  1/7 EntryNormal "foo"
  1/8 EntryNormal "bar"
  Messages:
  2->1 MsgAppResp Term:1 Log:0/9
  2->1 MsgAppResp Term:1 Log:0/9
> 3 handling Ready
  Ready MustSync=true:
  HardState Term:1 Commit:8
  Entries:
  1/9 EntryConfChangeV2
  CommittedEntries:
  1/7 EntryNormal "foo"
  1/8 EntryNormal "bar"
  Messages:
  3->1 MsgAppResp Term:1 Log:0/9
  3->1 MsgAppResp Term:1 Log:0/9
> 1 receiving messages
  2->1 MsgAppResp Term:1 Log:0/9
  2->1 MsgAppResp Term:1 Log:0/9
  3->1 MsgAppResp Term:1 Log:0/9
  3->1 MsgAppResp Term:1 Log:0/9
> 1 handling Ready
  Ready MustSync=false:
  HardState Term:1 Vote:1 Commit:9
  CommittedEntries:
  1/9 EntryConfChangeV2
  Messages:
  1->2 MsgApp Term:1 Log:1/9 Commit:9
  1->3 MsgApp Term:1 Log:1/9 Commit:9
  INFO 1 switched to configuration voters=(1)
> 2 receiving messages
  1->2 MsgApp Term:1 Log:1/9 Commit:9
> 3 receiving messages
  1->3 MsgApp Term:1 Log:1/9 Commit:9
> 2 handling Ready
  Ready MustSync=false:
  HardState Term:1 Commit:9
  CommittedEntries:
  1/9 EntryConfChangeV2
  Messages:
  2->1 MsgAppResp Term:1 Log:0/9
  INFO 2 switched to configuration voters=(1)
> 3 handling Ready
  Ready MustSync=false:
  HardState Term:1 Commit:9
  CommittedEntries:
  1/9 EntryConfChangeV2
  Messages:
  3->1 MsgAppResp Term:1 Log:0/9
  INFO 3 switched to configuration voters=(1)
> 1 receiving messages
  2->1 MsgAppResp Term:1 Log:0/9
  raft: cannot step as peer not found
  3->1 MsgAppResp Term:1 Log:0/9
  raft: cannot step as peer not found