Mongodb – threshold replica lag for secondaryPrefered mongod

It is not a great practice to mix the app and database on the same server. You're competing for resources. You also mentioned remote execution versus local execution on the same disk. I'd contend that you could suffer from disk contention if it was all on the same disk and the overhead of the remote replication is questionable by comparison. A lot of factors play into that including the type of disk(s) and if you are separating the OS, logs, and data.

Regarding your topology, yes a best practice in any DB technology is to place replica set members in a different datacenter. Pending your disaster recovery requirements, having at least one of those close to the primary node would be beneficial. There are other considerations such as an emergency on one coast of the US for example that makes all 3 unavailable. That is a case by case scenario to consider though. You also should consider your read and write concerns when addressing performance and availability. The default write concern will be 1. If your greater concern is availability; you may want to use a WC of 2 or majority in the example you gave. Testing your workload will only prove this for your use case and the WC can be set per operation so if you have some operations that are less important you could just use a less reliable WC. You should explore the bulk insert method as well. Personally for time-series data I have found the default and maximum batch of 1000 to be very good. I've also had good results increasing it to 5000 and allowing the DB engine break the batches out into 1k batches. Anything after that has demonstrated poorer inserts per second.

It is quite a lengthy topic but these are some ideas.

The usual reason for sharding is that your workload has exceeded the resources of a single server, so the expectation is that you would not be running mongodump via mongos for a full sharded cluster backup. Backups are instead done by stopping the balancer and then backing up a config server as well as a mongod from each shard.

As at MongoDB 3.0, mongodump does not provide an option to specify read preferences or tags. If you have specific secondaries in your DR data centre for backup purposes, you can use these mongod nodes in your backup procedure.

Recommended backup strategies are included in the MongoDB manual:

• Stop the balancer and backup all components of the sharded cluster with mongodump

• Stop the balancer and backup all components of the sharded cluster with file system snapshots

Backups with filesystem snapshots are faster to complete (and to restore) because they include all data and indexes. Backups using mongodump export the data and index definitions, but indexes will have to be rebuilt as part of the mongorestore procedure.

FYI, there is an open feature request you can watch/upvote in the MongoDB issue tracker: TOOLS-630: Allow specifying readPreference (including tags).