Mongodb – n automatic way for a secondary member to promote to primary member in a replica set

MongoDB requires a primary to be able to see a majority of the set in order to remain a primary. The majority is defined as being greater than n/2 machines where n is the number of machines in the replica set. So, if n is 1, then one machine constitutes a majority. When n is 2 or 3, 2 machines constitute a majority.

When the secondary is added, the primary now requires to be able to access two machines, itself and the secondary. When the newly added secondary goes down, the primary no longer sees a majority, and steps down.

This scenario is why an arbiter is recommended in addition to the primary and secondary. If you have two machines and one arbiter in the replica set, assuming the arbiter always stays up (which should be true because the arbiter does barely anything), then as long as one other machine is up, the arbiter will ensure that a primary will exist, because the other machine and the arbiter constitute a majority (2 out of 3).

There is a guide for how to do this (reconfigure a replica set with unavailable members) here:

http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

You essentially have to do a forced reconfig of the set because there is no primary available. If you want to keep a set operational while doing this type of move, you can:

1. Use your hosts file to temporarily point the old names to the new (or similar) and "fool" the primary into thinking they are up (then make the permanent changes one at a time)
2. Temporarily add arbiters to prevent the primary from stepping down (2 more for example, so the primary is kept with 3/5 votes)
3. Reconfigure the set to a single node before doing the moves
4. Do the moves one server at a time, ensuring there are 2/3 nodes up to form a majority at all times (and hence keep a primary up)