Once you add a user to the individual shards, which you indicate you have done for MMS
, you must then have valid credentials to connect for any purpose, including mongodump
. Up until you added that user for MMS
, the shards were running with authentication enabled but with no users populated (this only happens if all your users are in the admin database ans using delegated auth for other databases, otherwise with 2.4 and below you would have at least one shard with users for each database - 2.6+ changes this behavior) and so you were able to connect without credentials.
Essentially this is a loophole left open so that you don't accidentally lock yourself out of your instances when you turn on auth with no users (and one that would probably have stopped working at some point as default security is tightened anyway).
The bottom line is that you will need to add a user for use with mongodump
, and it's a good idea to do so anyway rather than allowing non-authenticated users free access to your instances. If you are running 2.6 or later, then the built in backup role exists precisely for this purpose, if you are on 2.4 or earlier, then the description for that role gives you a great outline of what is needed to backup successfully (and in particular if you want to backup the users themselves).
The problem (from what I can see)
From what I can see from your output, you have a monotonically increasing shard key like ObjectId
or a Date
or something similar.
MongoDB sharding is done with key ranges over the selected shard key. Put simply, it works like "shard0000
is supposed to hold the rang from -infinity to $i
, shard0001
the keys from $h to $p
and shard0002
from $q to infinity
. With a monotonically increasing shard key, every new value will be bigger than the last one inserted, heading to infinity, so the shard which will be used is always shard0002
, instead of the writes being balanced to all shards.
So all of the write operations seem to hit shard0002
with individual chunks being migrated to the other shards. The necessary chunk migrations together with added network latency and a lot of metadata updates going on, it may well be that you have a poor write performance.
The solution
Really bad news: you can't change the shard key once it is set. You basically have to create new sharded collection with a better shard key. For choosing a proper shard key, please read Considerations for Selecting Shard Keys in the MongoDB docs.
P.S. I wasn't sure wether to answer this question, since it is slightly off topic on Stackoverflow, which is aimed at programming questions. But since usually the developers are supposed to choose the shard key, my intention was to clarify things for them. Additionally, the data modeling might be influenced by sharding considerations, so – by a small margin – I thought it was ok to answer here. However, dear reviewer, feel free to flag this answer for deletion if you think differently. I am really not sure.
Best Answer
You cannot replay oplogs through a
mongos
process: oplog entries can only be applied via amongod
that is the primary of a replica set.Please see the MongoDB documentation for tutorials on Supported Backup Methods including Backing up a Sharded Cluster with Database Dumps.