MongoDB Replica Set – Configuring Two Datacenters with One Primary and One Secondary

mongodb

The tutorial I found on MongoDB's website suggest installing an odd number of servers. For what it's worth, we're not sharding so it should be a simple setup. For even numbers I would need an arbiter. In our environment, DC1 is always the primary with DC2 being the secondary where we would fail over services in the event DC1 fails. With that said …

If I have six nodes (3 node in each DC), would I need an arbiter?
If I have five nodes (3 in DC1 and 2 in DC2), if DC1 fails, wouldn't I need an arbiter in DC2?

I guess my confusion is how to handle failover in the event DC1 fails.

Best Answer

If I have six nodes (3 node in each DC), would I need an arbiter?

Yes - as Mongo docs mentioned an odd number is ideal.

If you have 6 Nodes and 1 goes down, the remaining 5 will elect a new primary. When 2 go down, 4 will elect a primary. When 3 do go down, you only have 3 left which are no longer a majority and all 3 will become read-only until a 4th node comes back online.

If you have 6 + an arbiter, then the scenario plays out slightly more beneficially. 1 node goes down, 5+1 elect a new primary. 2 go down, 4+1 elect a new primary. 3 go down, 3+1 still has a majority (3+1 / 7) so they can elect a new primary. Finally, when 4 nodes are down you are left with 2 nodes up and 1 arbiter, which is less than the majority (2+1 / 7) so you'll enter into read-only until a new node comes back online.

An odd number gives you one more level of outage before you lose the majority of your voting nodes and an arbiter is a cheap way to gain that benefit.

If I have five nodes (3 in DC1 and 2 in DC2), if DC1 fails, wouldn't I need an arbiter in DC2?

In this config, adding an arbiter would give you an even number, so that's not ideal (see above). However, if DC1 goes down you're also losing your majority automatically and thus down to read-only by default. Although it might seem strange, this is the recommended setup from Mongo.

But why?

You could have 3 in DC1 and 3 in DC2 with an arbiter in DC2, which would allow you to maintain a voting majority in DC2 if DC1 were to completely fail. This works if you want automatic failover into DC2. But if DC2 goes down and a node in DC1 goes down, you also lose DC1 majority and end up with read-only nodes - probably not ideal.

That's why Mongo recommends that you set the priority for nodes in DC2 to 0. In this case you should have 3 nodes + 1 arbiter in DC1 and 3 nodes in DC2. You will need to manually fail over to DC2 if DC1 completely fails, but it gives you the best case scenario as far as keeping DC1 active and primary.

Related Solutions

MONGODB primary became secondary after servers restart

You need to change the config such that there is only one member remaining, and you have to do it on that host, i.e. Ubuntu12041vanilla:27017

The others are unreachable from its perspective, so even if you go with 2 nodes, 1 out of 2 down means you cannot form a majority and you will still only have secondary state.

Given your current rs.status() output, try this when connected via the mongo shell to Ubuntu12041vanilla:27017:

cfg = rs.conf()
cfg.members = [cfg.members[0]]
rs.reconfig(cfg, {force : true})

That will give you a config only one node and should mean that Ubuntu12041vanilla:27017 will be the only remaining member and primary.

MongoDB: Manual failover and failback between two datacenters

I do not understand why the failover to DC2 has to be done manually (even if other parts have to be done manually: one thing less on your to do list in case of a major failure is always a good thing!).

In general, my feeling is that there are conceptual flaws in your setup.

Here is how I would do it and why.

I would not have manual failover. It is better to have slow access than none. What will happen in the current configuration is that if the primary fails, there will be a tie and therefor the whole set would enter secondary state, effectively turning the cluster into read-only mode. So even when everything else is fine in DC1 and there is no need for failing over to DC2, a failing primary will be a show stopper. With setup, you are artificially creating a single point of failure, effectively gainst the whole idea of a cluster, let alone a multi DC setup. Sounds like a Very Bad Idea™ to me. Automatic failover, even to DC2 sounds like a better idea. Slower reads and (depending on your write concern) slower writes still are better than read only mode.
I would have a third datacenter with only one instance: an arbiter. An arbiter can easily be run on a micro-machine as it will only be called in case of an election and an election is a cheap task in terms of RAM and computation power. The arbiter will help the set to always have a majority: If one DC gets disconnected for whatever reason, the other DC and the arbiter will form a majority. So if one DC goes down, you have only to worry about your other parts of your application. You don't have to wait
I am pretty sure that automatic failover for the other parts of your application can be achieved with some time and effort. Especially if you store all data in mongoDB and you have some sort of session replication available, it should be quite easy. Whether implementing automatic failover is worth the effort is pretty easy to calculate: Get your average downtime, find out how big the losses are created by this downtime in terms of money and customer satisfaction (if applicable). If the costs of implementing automatic failover is below or equal, go for automatic failover. I can help you with that if needed.

Best Answer

Related Solutions

MONGODB primary became secondary after servers restart

MongoDB: Manual failover and failback between two datacenters

Related Question