MongoDB Stuck with Jumbo chunks while exhausting a shard

migrationmongodbsharding

I'm running a shared MongoDB 2.4 cluster.
I'm currently in a process of exhausting a shard and I'm down to the last 15 chunks needs to be migrated out of 9660.

My issue is that these chunks are too big as the shard key wasn't the best one, and dumping/restoring the 2 collections that has these issues is not an option at this moment.

I've tried to manually split the chunks, but it keeps on failing with this error:
{ "cause" : { }, "ok" : 0, "errmsg" : "split failed" }

I also tried to raise the chunk size, but that didn't work out as well.

Is there another solution I might be missing?

EDIT – Added chunk information as requested – Based on AdamC function.

mongos> AllChunkInfo("db.coll1", true)
ChunkID,Shard,ChunkSize,ObjectsInChunk
db.coll1-company_name_-6727988435900812827,shard0003,148256192,248752
db.coll1-company_name_-6615369920888926287,shard0003,88895188,149153
db.coll1-company_name_-6595735839523245520,shard0003,87448696,146726
db.coll1-company_name_-6490024596613592713,shard0003,32099368,53858
db.coll1-company_name_-3125507653387230887,shard0003,89171732,149617
db.coll1-company_name_-3084465672933415339,shard0003,105551600,177100
db.coll1-company_name_-2997585070019050064,shard0003,120763904,202624
db.coll1-company_name_-2993901072357262199,shard0003,86105908,144473
db.coll1-company_name_-2543788723715016046,shard0003,123507888,207228
db.coll1-company_name_4754688535895874026,shard0003,91061648,152788
db.coll1-company_name_5062850413490905708,shard0003,118145676,198231
db.coll1-company_name_5483223634690519314,shard0003,111475244,187039
Summary Chunk Information
Total Chunks: 12
Average Chunk Size (bytes): 100206920.33333333
Empty Chunks: 0
Average Chunk Size (non-empty): 100206920.33333333

mongos> AllChunkInfo("db.coll2", true)
ChunkID,Shard,ChunkSize,ObjectsInChunk
db.coll2-company_name_-5389355967913336416,shard0003,95282040,94808
Summary Chunk Information
Total Chunks: 1
Average Chunk Size (bytes): 95282040
Empty Chunks: 0
Average Chunk Size (non-empty): 95282040

mongos> AllChunkInfo("db.coll3", true)
ChunkID,Shard,ChunkSize,ObjectsInChunk
db.coll3-company_name_-3231146661862124155,shard0003,101878644,173263
db.coll3-company_name_658930972165413978,shard0003,142545900,242425
Summary Chunk Information
Total Chunks: 2
Average Chunk Size (bytes): 122212272
Empty Chunks: 0
Average Chunk Size (non-empty): 122212272

Thanks,
Meny

Best Answer

I was able to complete the exhaust by increasing the chunk size and removing the jumbo tag from the problematic chunks.
After getting the results from AdamC function, I found out the biggest chunk of that shard was ~148 MB of size.
After figuring this one out, the first step is connect to the config database and change the max chunk size:

use config
db.settings.save( { _id:"chunksize", value: 150 } )

The second step was to clear the jumbo flag from the relevant chunks. This can be found in the MongoDB Documentation: Clear jumbo flag

Once the chunks were no longer marked as 'jumbo' the balancer had no problem migrating them to another shard.
When all of this was done, I restored the chunkSize back to 64

db.settings.save( { _id:"chunksize", value: 64 } )

As I do not want that all of my chunks will be as big. when possible, I'll correct the hashed shard key on the problematic collections to create a more divisible shard keys.

Thank you AdamC, your initial guidance set me on the correct path to solve this problem.

Related Solutions

MongoDB presplitting chunks for compound shard key

Just a slight issue with how you are passing the $minKey values in, try this instead:

db.adminCommand( { split : "mydb.mycollection" , middle : { "region" : "region1", "foo" : MinKey , "bar" : MinKey } } );
db.adminCommand( { split : "mydb.mycollection" , middle : { "region" : "region2", "foo" : MinKey , "bar" : MinKey } } );

This got me the following layout:

sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "version" : 4,
    "minCompatibleVersion" : 4,
    "currentVersion" : 5,
    "clusterId" : ObjectId("53a2cd9d98b4ace818666544")
}
  shards:
    {  "_id" : "shard0000",  "host" : "localhost:30000" }
    {  "_id" : "shard0001",  "host" : "localhost:30001" }
    {  "_id" : "shard0002",  "host" : "localhost:30002" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "mydb",  "partitioned" : true,  "primary" : "shard0001" }
        mydb.mycollection
            shard key: { "region" : 1, "foo" : 1, "bar" : 1 }
            chunks:
                shard0000   1
                shard0001   2
            {
    "region" : { "$minKey" : 1 },
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} -->> {
    "region" : "region1",
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} on : shard0000 Timestamp(2, 0) 
            {
    "region" : "region1",
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} -->> {
    "region" : "region2",
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} on : shard0001 Timestamp(2, 2) 
            {
    "region" : "region2",
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} -->> {
    "region" : { "$maxKey" : 1 },
    "foo" : { "$maxKey" : 1 },
    "bar" : { "$maxKey" : 1 }
} on : shard0001 Timestamp(2, 3)

The use of the $minKey (MinKey) and $maxKey (MaxKey) values is a bit tough to tease out (they are rarely used except internally), but there is a decent, and illustrative example here in the docs.

Mongodb – Mongo large chunks will not split

Lot to go through here, so I'll take it piece by piece, first off splitting:

I thought this meant that when a chunk hits 64mb, it splits into two equal chunks both of size 32mb. That's what is demonstrated here. Is that not correct?

That's not quite how it works. If you have a 64MB chunk and you manually run a splitFind command, you will get (by default) 2 chunks split at the mid-point. Auto splitting is done differently though - the details are actually quite involved, but use what I explain as a rule of thumb and you will be close enough.

Each mongos tracks how much data it has seen inserted/updated for each chunk (approximately). When it sees that ~20% of the maximum chunk size (so 12-13MiB by default) has been written to a particular chunk it will attempt an automatic split of that chunk. It sends a splitVector command to the primary that owns the chunk asking for it to evaluate the chunk range and return any potential split points. If the primary replies with valid points, then the mongos will attempt to split on those points. If there are no valid split points, then the mongos will retry this process when it the updates/writes get to 40%, 60% of the max chunk size.

As you can see, this does not wait for a chunk to reach the max size before splitting, in fact it should happen long before that and with a normally operating cluster you should not see such large chunks in general.

What's up with this? How can the first 3 shards/replica sets have an average size greater than 64mb when that is set to be chunkSize? Rs_2 is 119mb!

The only thing preventing large chunks from occurring is the auto-split functionality described above. Your average chunk sizes suggest that something is preventing the chunks from being split. There are a couple of possible reasons for this, but the most common is that the shard key is not granular enough.

If your chunk ranges get down to a single key value then no further splits are possible and you get "jumbo" chunks. I would need to see the ranges to be sure, but you can probably manually inspect them easily enough from sh.status(true) but for a more easily digestible version take a look at this Q&A I posted about determining the chunk distribution.

If that is the issue you only really have 2 choices - either live with the jumbo chunks (and possibly increase the max chunk size to allow them to move around - anything over the max will be aborted and tagged as "jumbo" by the mongos), or re-shard the data with a more granular shard key that prevents the creation of single key chunks.

Rs_2 has 27.53% of the data when it should have 16.6%.

This is a fairly common misconception about the balancer - it does not balance based on the data size, it just balances the number of chunks (which you can see are nicely distributed) - from that perspective a chunk with 0 documents in it counts just the same as one with 250k documents. Hence, the reason for the imbalance in terms of the data is because of the imbalance in the chunks themselves (some contain a lot more data than others).

What should I do here? I can manually find chunks that are large and split them, but that is a pain. Do I lower my chunkSize?

Lowering the chunk size would cause the mongos to check for split points more frequently, but it won't help if the splits are failing (which your chunk size averages suggest is the case), it will just fail more often. As a first step I would find the largest chunks (see the Q&A link above) and split those as a priority first.

If you are going to do any manual splitting or moving, I recommend turning off the balancer so that it is not holding the meta data lock and does not kick in as soon as you start splitting. It's also generally a good idea to do it at a low traffic time because otherwise the auto-splitting I described above could interfere also.

After a quick search I don't have anything generic immediately to hand, but I have in the past seen scripts used to automate this process. It tends to need to be customized to fit the particular issue (imagine an imbalance due to a monotonic shard key versus an issue with chunk data density for example).

Best Answer

Related Solutions

MongoDB presplitting chunks for compound shard key

Mongodb – Mongo large chunks will not split

Related Question