MongoDB – Adding a new server to an existing replica set with auth enabled

mongodbreplication

Short version: Is there a way to add a new SECONDARY server to an existing production MongoDB replica set without downtime, and without temporarily disabling the "auth" and "keyfile" config settings on the existing servers?

Long version: My current production environment consists of three Windows 2012 R2 DataCenter servers running MongoDB version 2.4.9 as a service:

PRIMARY   - ServerA.example.org:27017
SECONDARY - ServerB.example.org:27017
ARBITER   - ServerC.example.org:27017

I have two MongoDB user accounts in the environment:

{ "_id" : ObjectId("XXXXXXXXXX"), "pwd" : "XXXXXXXXXX", "user" : "mongo-admin", "roles" : [  "userAdminAnyDatabase" ] }
{ "_id" : ObjectId("XXXXXXXXXX"), "pwd" : "XXXXXXXXXX", "user" : "mongo-cluster-admin", "roles" : [  "clusterAdmin" ] }

And the config file on the existing servers looks like this:

logpath=C:\mongo\logs\mongo.log
dbpath=C:\mongo\data
replSet=rs
oplogSize=700
port=27017
auth=true
keyFile=C:\mongo\config\keyfile.txt

I want to add ServerD and ServerE as new SECONDARY servers.

I've tried the following from my workstation and from the local console on ServerA, with and without the "auth" and "keyfile" in the config on ServerD, but I get the same result in all cases:

D:\MongoDB\bin>mongo ServerA.example.org:27017
MongoDB shell version: 2.4.7
connecting to: ServerA.example.org:27017/test
> use admin
switched to db admin
> db.auth("mongo-cluster-admin", "XXXXXXXXXX")
1
rs:PRIMARY> rs.add("ServerD.example.org:27017")
Mon May 19 11:41:42.050 count failed: { "ok" : 0, "errmsg" : "unauthorized" } at src/mongo/shell/query.js:180

The only way I've been able to get this working in a test environment is if I disable the "auth" and "keyfile" settings on all five servers, then add the new servers to the replica set using "rs.add()" on the PRIMARY, and finally re-enable "auth" and "keyfile" on all servers. This requires a couple of restarts of the MongoDB service on each box to update the config settings, which I really want to avoid.

Is there a way to add the new SECONDARY servers to the replica set without changing the "auth" and "keyfile" settings on the existing servers first?

Best Answer

Just add the new secondary with auth enabled and using identical keyfiles from the start. If you do that, they will connect, use the keyfile to authenticate, and then sync from scratch - there is no reason to change the original members of the set, just make sure that the new ones you add have the correct (identical) keyfile as the original members.

The other thing to make sure of is that the user you are authenticated as has the correct permissions to do the additions (since turning off auth would solve that issue also). For reference, the built in role with the relevant permissions is clusterManager for 2.6+ - clusterAdmin should also work, and has the advantage of existing in 2.4.

Per the comments discussion, the caveat with clusterAdmin in 2.4 is that although you have permission to run the relevant commands, you will need to have read privileges on the local database (per the Combined Access section of the 2.4 docs) to use the rs.conf() command. It's not strictly required but without it you will have to construct the entire config document from scratch for a reconfiguration command which is certainly inconvenient.

Related Question