MongoDB ReplicaSet reconfig when primary and majority don’t exist

cmongodbreplication

http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

Is there anyway to reconfigure a replica set in the event the primary as well as a majority of the servers are down ? For example if data center 1 goes does, a procedure can be run to reconfigure and set a new primary with the remaining servers?

I keep getting this "Command '$eval' failed: not master (response: { "note" : "from execCommand", "ok" : 0.0, "errmsg" : "not master" })"

We only have 2 data centers. (I we had 3 i realize this would solve my problem, but we don't.
)
I was thinking about running a reconfigure on a 5 member replica set every time the fail over happens and changing the host of the arbiter from one data center to another so that the active data center will always have the arbiter, but I can't do that if 3 of the 5 members of the replica set aren't available…because not new primary will ever be set.

Any suggestions?

Best Answer

Yes, you can do this, and in fact with just 2 data centers you really have no other choice. Essentially, whenever you do not have a primary you can still do a reconfig but you must pass in {force:true} to make it happen on a non-primary node.

This is listed in the docs for the rs.reconfig() command and in detail (with examples) as part of a tutorial which describes exactly what needs to be done in the scenario you described:

http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

If you are interested, I cover this and other availability issues in the (free) M202: Advanced Deployment and Operations course on MongoDB University