Mongodb – Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database

The problem explained

As per your comment, your shard key is the _id field of the document. This field is monotonically increasing, basically like an incremented integer.

Put simply, sharding works this way: documents are stored in chunks. Those chunks are spread over the cluster based on ranges of the shard key. Let's look at a simple example:

• s1: chunks holding id 0-50
• s2: chunks holding id 51-100
• s3: chunks above 101-?

And here is where the problem occurs: there is one shard were all new documents go to. What happens then is that new chunks are created all the time (as chunks have fixed size). So not only are all writes focussed on one server, this server also has to do some overhead work. After a certain threshold is met, the cluster will start to move chunks to the other shards, which only will balance out disk space, though. In out example, it might look like this:

• s1: chunks holding id 0-100
• s2: chunks holding id 101-200
• s3: chunks holding id 201-?

Obviously, all write operations will still go to s3.

What went wrong?

You chose the wrong shard key. Monotonically increasing shard keys lead to the problem explained above. Altough ObjectIds look like hash sums, but they aren't. They are monotonically increasing.

What can be done?

You need a better shard key. Since a shard key can not be changed after a collection is sharded, migrating to a new shard key is not a simple thing to do. However, there is a quite detailed explanation in mongoDB's ticket system. Basically, it works like this:

1. Remove all but one shard from the cluster
2. Wait for the chunk migrations to be finished
3. Shut down the remaining shard
4. Change the shard key through a mongos instance
5. Restart the last shard
6. Add the other shards to the cluster

In the future, ask to have the question migrated here instead of double posting.

From the MongoDB Documentation,

To calculate how much RAM you need, you must calculate your working set size, or the portion of your data that clients use most often. This depends on your access patterns, what indexes you have, and the size of your documents. Because MongoDB uses a thread per connection model, each database connection also will need up to 1MB of RAM, whether active or idle.

If the query is not completely covered by an index then the entire document is loaded into RAM by MongoDB. This is the same regardless of the type of query.

As for whether or not you should split your collection, that's really a design decision that requires a lot more background to understand and weigh in on. You'll need to perform tests to see if the gains from it based on the query patterns are worth it compared to the loss of atomicity (a business driven decision potentially).

I probably wouldn't recommend this approach though and would assess what your true SLA's are and whether or not you're using the right amount of hardware (RAM) for the desired results. Your real question here is "Do I have enough RAM to satisfy my query performance requirements when it comes to cached data?" If the answer is no, figure out what design works best with your constraints.