The problem explained
As per your comment, your shard key is the _id field of the document. This field is monotonically increasing, basically like an incremented integer.
Put simply, sharding works this way: documents are stored in chunks. Those chunks are spread over the cluster based on ranges of the shard key. Let's look at a simple example:
- s1: chunks holding id 0-50
- s2: chunks holding id 51-100
- s3: chunks above 101-?
And here is where the problem occurs: there is one shard were all new documents go to. What happens then is that new chunks are created all the time (as chunks have fixed size). So not only are all writes focussed on one server, this server also has to do some overhead work. After a certain threshold is met, the cluster will start to move chunks to the other shards, which only will balance out disk space, though. In out example, it might look like this:
- s1: chunks holding id 0-100
- s2: chunks holding id 101-200
- s3: chunks holding id 201-?
Obviously, all write operations will still go to s3.
What went wrong?
You chose the wrong shard key. Monotonically increasing shard keys lead to the problem explained above. Altough ObjectIds look like hash sums, but they aren't. They are monotonically increasing.
What can be done?
You need a better shard key. Since a shard key can not be changed after a collection is sharded, migrating to a new shard key is not a simple thing to do. However, there is a quite detailed explanation in mongoDB's ticket system. Basically, it works like this:
- Remove all but one shard from the cluster
- Wait for the chunk migrations to be finished
- Shut down the remaining shard
- Change the shard key through a
mongos instance
- Restart the last shard
- Add the other shards to the cluster
You can use below commands to remove the shards, though I havn't tried ever but looks straight forward.
use admin
db.runCommand( { removeShard: "mongodb0" } )
--Response
{
"msg" : "draining started successfully",
"state" : "started",
"shard" : "mongodb0",
"ok" : 1
}
link: http://docs.mongodb.org/manual/tutorial/remove-shards-from-cluster
Best Answer
From the database perspective you can try using, MongoDB 3.0 with the Wired Tiger storage engine to eliminate these waits. MongoDB 3.0 with the Wired Tiger uses Document level blocking and thus brings down the waits considerably.
In case this is not possible, then you cannot do much on the database layer and will either need to live with these waits or will need to handle it from the Application layer.
A tentative approach, which can be used to provide control of the waits from the Application layer can be to queue the respective transactions in the application so that the user does not need to make the changes multiple times.