Mongodb – replica set unable to add a node

Replica Set Elections

The MongoDB manual describes Replica Set Elections and the associated election mechanics such as Vetoes in detail.

Broadly speaking, the criteria for electing a node as primary includes:

• Is that node currently in PRIMARY or SECONDARY state?

• Is that node eligible to become primary (priority > 0)?

• Do a majority of the nodes of the replica set see that memory as available (by way of replica set heartbeat)?

• Does that node have the least replication lag in relation to the current primary?

• Is that node the highest priority eligible node?

• Would the node veto itself if elected primary? For example, if you step down a primary it will veto itself for election for 60 seconds by default.

The first node to receive a majority of votes subject to the election mechanics will become primary.

There are some further subtleties to the process depending what triggered the election (i.e. a unplanned failover versus stepping down the current primary).

Suppose a replication set R1 contains one primary, 2 secondary and one arbiter node.

In order to ensure consensus can be reached by a strict majority of voting nodes, the general recommendation is to have an odd number of voting nodes. With your example configuration an arbiter node is not required, and you should instead only have the primary and two secondaries. An arbiter node does not have any special role in the election process aside from being available to cast a vote.

With all things being equal (no replication lag, default priorities, no network issues, etc) your two secondaries in this scenario are equally likely to be elected primary in the event the current primary unexpectedly fails.

The context you are missing is that failure of a replica member does not affect the number of configured members the replica set has (or the required voting majority to maintain a primary). Any changes in fault tolerance or replica set election requirements will involve a re-configuration of the replica set (eg. adding/removing members or changing voting members) rather than a replica set member state change (eg. DOWN, RECOVERING, ..).

A three node replica set with one down member still has three members (with a strict majority of two). If you think of it as analogous to a RAID configuration, a three node replica set with one member down is running in degraded mode and cannot tolerate the failure of any further voting members. This allows for continued availability, but you will want to recover or replace the unavailable member to return the replica set to a healthy state.

You can check the current replica set configuration with rs.conf() and the current state of members with rs.status().