MongoDB durability and consistency with j= true and w:majority

mongodbreplication

Assume there is a 7 member replica set, where each node is a voting member and no arbiter node is present. For a write w1, Using j=true and w:majority would ensure that a write is written to the journal and let's assume to the disk also after the default time interval.

If 3 nodes fail (all having the write w1 mentioned above) at the same time including the primary, only one of the remaining 4 nodes has the write w1. If this node does not get enough votes (say it's overloaded and will not be suitable for role of a primary), and some other node gets all enough votes to become primary, we have basically lost the write w1.

I know this is a rare case, but it still can happen. So isn't it true that using j= true and w:majority also will not assure that the writes are durable?

Best Answer

This is not how it works. With j:true and w: majority, w1 would only be acknowledged when the majority of nodes actually had reported to the primary that they have put the operation to the journal. Since the primary went down, this acknowledgement would never happen. So in this edge case, you would only have to do what you'd need to do what you always need to do if a failover happens before a write operation is persisted: redo it.

Let us expand the example a bit: let us assume that the write was successful, and then the nodes which already received the writes minus one go down. Now, an election takes place. The new primary is elected basically by each remaining node announcing the latest timestamp they have in the oplog. So, the node which received w1 would win (since the oplog is processed consecutively for each entry) and the remaining nodes would start to process the oplog from their latest position to the new primaries latest position - w1 will be replicated and hence it will be eventually consistent on all nodes.

Replica Set Elections

The MongoDB manual describes Replica Set Elections and the associated election mechanics such as Vetoes in detail.

Broadly speaking, the criteria for electing a node as primary includes:

Is that node currently in PRIMARY or SECONDARY state?
Is that node eligible to become primary (priority > 0)?
Do a majority of the nodes of the replica set see that memory as available (by way of replica set heartbeat)?
Does that node have the least replication lag in relation to the current primary?
Is that node the highest priority eligible node?
Would the node veto itself if elected primary? For example, if you step down a primary it will veto itself for election for 60 seconds by default.

The first node to receive a majority of votes subject to the election mechanics will become primary.

There are some further subtleties to the process depending what triggered the election (i.e. a unplanned failover versus stepping down the current primary).

Suppose a replication set R1 contains one primary, 2 secondary and one arbiter node.

In order to ensure consensus can be reached by a strict majority of voting nodes, the general recommendation is to have an odd number of voting nodes. With your example configuration an arbiter node is not required, and you should instead only have the primary and two secondaries. An arbiter node does not have any special role in the election process aside from being available to cast a vote.

With all things being equal (no replication lag, default priorities, no network issues, etc) your two secondaries in this scenario are equally likely to be elected primary in the event the current primary unexpectedly fails.

Mongodb – Can arbiterOnly replica in MongoDB become SECONDARY and what it means

Some time ago my one of my data replicas which was secondary at a time crushed due to hard drive failure. After I fixed that problem and restarted secondary it went into “Recovering” state. But my arbiter is now “Secondary”

A MongoDB arbiter cannot automatically become a secondary or a primary node, as it does not have a copy of the data set.

If you try to manually reconfigure the arbiter as a regular node via rs.reconfig() you should get an exception similar to:

{
    "errmsg" : "exception: arbiterOnly may not change for members",
    "code" : 13510,
    "ok" : 0
}

Furthermore, the data directory for this Arbiter shows bunch of files of total size >10GB. They indeed look to me like data files. Are they? What is going to happen to these files when Recovery completes?

Assuming this node is definitely an arbiter, I would expect those files are unused (check the timestamps?) and are either:

a local database created if this node was incorrectly initialized with an oplog
unused copy of data directory if this node was copied from another secondary or used as a standalone before

You can always log into the arbiter mongod directly to see what data it appears to have.

Best Answer

Related Solutions

Mongodb – Based on what criteria voting take place in MongoDB

Replica Set Elections

Mongodb – Can arbiterOnly replica in MongoDB become SECONDARY and what it means

Related Question