MongoDB replica set node failure without arbiter

mongodbreplication

I'm baffled by the fact that this isn't addressed in the official docs (or at my inability to track it down). From the 3.4 docs:

Only add an arbiter to sets with even numbers of voting members. If you add an arbiter to a set with an odd number of voting members, the set may suffer from tied elections.

So we have two possible scenarios:

  1. A replica set with an odd number of data nodes, which requires no arbiter (and in fact, adding an arbiter can be detrimental per above)
  2. A replica set with an even number of data nodes, and an arbiter

This is great so long as things are running smoothly. But what if a data node goes down in each of these scenarios?

  1. We now have an even number of nodes with no arbiter. As I understand it this means you cannot have a successful election.
  2. We now have an odd number of nodes, plus an arbiter. Above docs state that this can lead to tied elections.

This is so painfully obvious, why can't I find any discussion or documentation on it? But what I really want to know is, how do I correctly handle these scenarios?

Best Answer

The context you are missing is that failure of a replica member does not affect the number of configured members the replica set has (or the required voting majority to maintain a primary). Any changes in fault tolerance or replica set election requirements will involve a re-configuration of the replica set (eg. adding/removing members or changing voting members) rather than a replica set member state change (eg. DOWN, RECOVERING, ..).

A three node replica set with one down member still has three members (with a strict majority of two). If you think of it as analogous to a RAID configuration, a three node replica set with one member down is running in degraded mode and cannot tolerate the failure of any further voting members. This allows for continued availability, but you will want to recover or replace the unavailable member to return the replica set to a healthy state.

You can check the current replica set configuration with rs.conf() and the current state of members with rs.status().

Related Question