MongoDB automatic failover between 2 data centers

Replica Set Elections

The MongoDB manual describes Replica Set Elections and the associated election mechanics such as Vetoes in detail.

Broadly speaking, the criteria for electing a node as primary includes:

• Is that node currently in PRIMARY or SECONDARY state?

• Is that node eligible to become primary (priority > 0)?

• Do a majority of the nodes of the replica set see that memory as available (by way of replica set heartbeat)?

• Does that node have the least replication lag in relation to the current primary?

• Is that node the highest priority eligible node?

• Would the node veto itself if elected primary? For example, if you step down a primary it will veto itself for election for 60 seconds by default.

The first node to receive a majority of votes subject to the election mechanics will become primary.

There are some further subtleties to the process depending what triggered the election (i.e. a unplanned failover versus stepping down the current primary).

Suppose a replication set R1 contains one primary, 2 secondary and one arbiter node.

In order to ensure consensus can be reached by a strict majority of voting nodes, the general recommendation is to have an odd number of voting nodes. With your example configuration an arbiter node is not required, and you should instead only have the primary and two secondaries. An arbiter node does not have any special role in the election process aside from being available to cast a vote.

With all things being equal (no replication lag, default priorities, no network issues, etc) your two secondaries in this scenario are equally likely to be elected primary in the event the current primary unexpectedly fails.

I do not understand why the failover to DC2 has to be done manually (even if other parts have to be done manually: one thing less on your to do list in case of a major failure is always a good thing!).

In general, my feeling is that there are conceptual flaws in your setup.

Here is how I would do it and why.

1. I would not have manual failover. It is better to have slow access than none. What will happen in the current configuration is that if the primary fails, there will be a tie and therefor the whole set would enter secondary state, effectively turning the cluster into read-only mode. So even when everything else is fine in DC1 and there is no need for failing over to DC2, a failing primary will be a show stopper. With setup, you are artificially creating a single point of failure, effectively gainst the whole idea of a cluster, let alone a multi DC setup. Sounds like a Very Bad Idea™ to me. Automatic failover, even to DC2 sounds like a better idea. Slower reads and (depending on your write concern) slower writes still are better than read only mode.
2. I would have a third datacenter with only one instance: an arbiter. An arbiter can easily be run on a micro-machine as it will only be called in case of an election and an election is a cheap task in terms of RAM and computation power. The arbiter will help the set to always have a majority: If one DC gets disconnected for whatever reason, the other DC and the arbiter will form a majority. So if one DC goes down, you have only to worry about your other parts of your application. You don't have to wait
3. I am pretty sure that automatic failover for the other parts of your application can be achieved with some time and effort. Especially if you store all data in mongoDB and you have some sort of session replication available, it should be quite easy. Whether implementing automatic failover is worth the effort is pretty easy to calculate: Get your average downtime, find out how big the losses are created by this downtime in terms of money and customer satisfaction (if applicable). If the costs of implementing automatic failover is below or equal, go for automatic failover. I can help you with that if needed.