MongoDB: Can I have a delayed secondary that doesn’t vote

mongodb

I have a replica set setup with 1 Primary, 1 Secondary and 1 Arbiter; each on a different data center.

I want to add a delayed secondary to one of the existing data centers, but I don't want it to participate in any election decisions.

I still want to have a total of 3 votes, so if the data center that hosts the delayed secondary (plus another member) goes down, my replica set would still elect a primary. Is this possible?

Best Answer

A configuration with three voting nodes in different data centres plus an additional non-voting delayed secondary is definitely supported. This is actually a reasonable configuration in terms of satisying core availability requirements using the distributed voting nodes with additional hidden/non-voting nodes for special use cases. A delayed secondary should be always configured as hidden and priority 0, and may have 0 or 1 vote.

However, there is one important caveat with your deployment: you have three data-bearing nodes but one of those is delayed. This will be problematic if you want to use any write concern higher than the usual default (w:1 aka acknowledgement by the primary). In the event one of your non-delayed secondaries is unavailable, a higher write concern (eg w:2 or w:majority) could still be satisfied but the writes would have to wait for acknowledgement from a second data-bearing node which would have to be the delayed secondary.

A more robust configuration would be to replace the arbiter with a data-bearing secondary so that unavailability of any of the three voting nodes would still allow for acknowledgement of majority writes.

Related Solutions

Mongodb – Can arbiterOnly replica in MongoDB become SECONDARY and what it means

Some time ago my one of my data replicas which was secondary at a time crushed due to hard drive failure. After I fixed that problem and restarted secondary it went into “Recovering” state. But my arbiter is now “Secondary”

A MongoDB arbiter cannot automatically become a secondary or a primary node, as it does not have a copy of the data set.

If you try to manually reconfigure the arbiter as a regular node via rs.reconfig() you should get an exception similar to:

{
    "errmsg" : "exception: arbiterOnly may not change for members",
    "code" : 13510,
    "ok" : 0
}

Furthermore, the data directory for this Arbiter shows bunch of files of total size >10GB. They indeed look to me like data files. Are they? What is going to happen to these files when Recovery completes?

Assuming this node is definitely an arbiter, I would expect those files are unused (check the timestamps?) and are either:

a local database created if this node was incorrectly initialized with an oplog
unused copy of data directory if this node was copied from another secondary or used as a standalone before

You can always log into the arbiter mongod directly to see what data it appears to have.

MongoDB ReplicaSet reconfig when primary and majority don’t exist

Yes, you can do this, and in fact with just 2 data centers you really have no other choice. Essentially, whenever you do not have a primary you can still do a reconfig but you must pass in {force:true} to make it happen on a non-primary node.

This is listed in the docs for the rs.reconfig() command and in detail (with examples) as part of a tutorial which describes exactly what needs to be done in the scenario you described:

http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

If you are interested, I cover this and other availability issues in the (free) M202: Advanced Deployment and Operations course on MongoDB University

Best Answer

Related Solutions

Mongodb – Can arbiterOnly replica in MongoDB become SECONDARY and what it means

MongoDB ReplicaSet reconfig when primary and majority don’t exist

Related Question