Mongodb – Reliability of mongodump

mongodbmongodump

I understand that the recommended way to backup MongoDB data is to use file system level tools. However, mongodump utility is promoted as a valid alternative for smaller instances, when file system snapshots are not available for some reason. Many resources recommend running it with –oplog option to enable point-in-time restores.

I tried to understand how –oplog could enable point-in-time snapshots without proper multi-version concurrency control from the database engine. I think it can only work in the following way:

  1. We assume that _id values in all collections are monotonously increasing
  2. Mongodump blocks all writes and records last operation id in the oplog
  3. Mongodump starts dumping documents from all collections by navigating _id index in decreasing order
  4. After dump has started, mongodump unlocks writes to the collections
  5. We assume that all inserts from now on will have higher _id values and will appear only in oplog dump, not collections dump.

This system quickly breaks if _id values are not monotonously increasing or if write locks are not implemented and there is a race condition. Also, there is no support for secondary unique indexes.

From what I see, the following statements are true, meaning that mongodump cannot possibly enable point-in-time restores, so it's specification is misleading as it comes to its snapshot-like capabilities:

  1. Even build-in _id values generators do not guarantee total order
  2. mongodump does not issue any write locks
  3. mongodump does not traverse _id index in descending order, so inserted records will appear both in collection dump and oplog dump.

Is my understanding correct? Is there any value in –oplog option of mongodump utility?

Following are the tickets which seem to enforce my suggestions:

  1. https://jira.mongodb.org/browse/SERVER-24231
  2. https://jira.mongodb.org/browse/TOOLS-176

Best Answer

Is my understanding correct? Is there any value in --oplog option of mongodump utility?

As MongoDB Blog documentation here Ensures that mongodump creates a dump of the database that includes a partial oplog containing operations from the duration of the mongodump operation. This oplog produces an effective point-in-time snapshot of the state of a mongod instance. To restore to a specific point-in-time backup, use the output created with this option in conjunction with mongorestore --oplogReplay. Without --oplog, if there are write operations during the dump operation, the dump will not reflect a single moment in time. Changes made to the database during the update process can affect the output of the backup.

--oplog has no effect when running mongodump against a mongos instance to dump the entire contents of a sharded cluster. However, you can use --oplog to dump individual shards.

--oplog only works against nodes that maintain an oplog. This includes all members of a replica set, as well as master nodes in master/slave replication deployments.

--oplog does not dump the oplog collection.

I understand that the recommended way to backup MongoDB data is to use file system level tools. However, mongodump utility is promoted as a valid alternative for smaller instances, when file system snapshots are not available for some reason. Many resources recommend running it with --oplog option to enable point-in-time restores.

You could use oplog option if you are taking backups of all the DBs.

mongodump  --oplog --out /dump_location

Now while restoring do something like this :

mongorestore --oplogReplay

This way mongodump won't require a write lock. Any writes that are made during the backup process will be written to oplog.bson. While restoring, mongorestore will use this oplog to include the write operations.

For your further ref here, here and here

Related Question