MongoDB: enable sharding automatically

OK, flights.flight is sharded but basically only has one chunk so far, and hence all current data is on one shard and so all writes can only go to one shard - you will need more before the collection can be balanced across multiple shards. Unless you pre-split (something like this) or use a hashed shard key (which you are not) then the mongos process is what is responsible for automatically splitting the chunks.

It will attempt the first split after it has seen writes of approximately 20% of the max chunk size (default is 64MiB, so 20% is ~13MiB). It looks like this has not yet happened, or has failed for some reason in your case.

It should be noted that although the mongos will track all the writes, it will lose its "history" if restarted. It should also be noted that the config servers must be up and working for any splits to happen. Check the mongos logs for any evidence of split failures or similar.

If you want to kick start things manually, just pick two (or more) reasonable split points and use the splitAt() command to break up your current single chunk. You can then wait for the balancer to move things, or you can move them manually with the moveChunk() command. You can find examples of this in the answer to the question I linked above.

• For your 7 collections on the primary shard:

  Enter as administrator on the primary replica (if you shard your replica) and create your collection there by inserting 1 document or creating an index there. When you create a collection via the mongos - shard client - then the collection is started on a random shard; if you create it on the shard itself first, then you know it's on the one you want. shard1$ mongo --port 27018 localhost/mydb --eval 'db.mycol.insert({firstdocument:'hello'})'

• For the 25 collections:

  Standard sharding indeed as you mention. Be patient with the distribution. The first shards will start distributing after a certain amount of data, can be 100k documents if the documents are small.

• For the big collection:

  Have a look at shard-tags. So, you have to tag your collection on shard1 with let's say : {status:'in_use'}; shard2 is tagged {status:'freeBeer'}. Then depending on the status in the document, the shard is chosen.

• For the disk issue, I think you gonna have to write a script. Get a warning when 60% of a disk is used, and another warning at 80% to take action. Then you'll need to upgrade your disk or redistribute the chunks. Certainly the 2TB collection could give you space via the tag, to send more to shard2. Or, add a shard3 ... what is actually the purpose of sharding, to grow horizontally.

• Choose shardKeys well, practice configuration and setup on a small server, practice moving chunks, adding tags, ....

• Have a look at the AllChunkInfo script of Adam Comerford for you scripts and testing : https://github.com/comerford/mongodb-scripts