MongoDB – Is it Safe to Transfer Database with rsync While Working?

The solution to this issue was as follows: Starting with the secondary:

Inside of Mongo I ran the following:

db.shutdownServer()
exit

On the command line:

rm -rf /var/lib/mongodb/*
sudo service mongodb start

It then takes a couple of hours to resync. Once resynced I did the following on the primary:

rs.status() //To make sure that the everything was synced
rs.stepDown(9200)
db.shutdownServer()
exit

On the command line I ran:

rm -rf /var/lib/mongodb/*
sudo service mongodb start

Then in Mongo I ran:

rs.status() //To make sure that everything was synced
rs.freeze(0) //To remove the re.stepDown(9200) lock

On the secondary I ran:

rs.stepDown()

Note: If possible, back up all of your data first.

The usual reason for sharding is that your workload has exceeded the resources of a single server, so the expectation is that you would not be running mongodump via mongos for a full sharded cluster backup. Backups are instead done by stopping the balancer and then backing up a config server as well as a mongod from each shard.

As at MongoDB 3.0, mongodump does not provide an option to specify read preferences or tags. If you have specific secondaries in your DR data centre for backup purposes, you can use these mongod nodes in your backup procedure.

Recommended backup strategies are included in the MongoDB manual:

• Stop the balancer and backup all components of the sharded cluster with mongodump

• Stop the balancer and backup all components of the sharded cluster with file system snapshots

Backups with filesystem snapshots are faster to complete (and to restore) because they include all data and indexes. Backups using mongodump export the data and index definitions, but indexes will have to be rebuilt as part of the mongorestore procedure.

FYI, there is an open feature request you can watch/upvote in the MongoDB issue tracker: TOOLS-630: Allow specifying readPreference (including tags).