Mongodb – Read concern level of majority error while upgrading Sharded Cluster from 3.2 to 3.4

How did you change the chunk size? Did you follow this guide?

To recap, one should:

1. connect to mongos
2. switch to config db
3. issue change chunk size command:
   db.settings.save( { _id:"chunksize", value: <sizeInMB> } )

AFAIK,

warning: chunk is larger than 1024 bytes because of key ...

won't bring down a shard, it will only result in a unbalanced cluster.

The current wording in the documented steps may be somewhat confusing given the relative references to first/last/second config servers.

I've added some context for the documented steps below, but you should consider the manual the definitive source. It's also worth noting that you do not have to upgrade the config servers to use WiredTiger (even if your shards are using WiredTiger). Config servers typically have a small data set and are not under high write load.

Annotated version of the steps from the MongoDB 3.0 Upgrade Guide

1. Disable the balancer.

   Disabling the balancer ensures any active migrations have completed.

2. Stop the last config server listed in your mongos' configDB setting (will call that config3 for the purpose of these steps):

   At this stage you should have:

   • config1 (running mmap)
   • config2 (running mmap)
   • config3 (stopped)

   Stopping one of the config servers ensures there are no changes to the metadata in the cluster (chunk splits or migrations cannot be committed without all three config servers available).

3. Use mongodump to export the config database from config2

   After running the mongodump you should have a dump directory with bson files.

4. Create a new data directory on config2.

   The storage format for WiredTiger data is different from the existing mmap data, and cannot use the same dbpath as mmap.

5. Restart the config2 server with the WiredTiger and appropriate storage options:

   mongod --storageEngine wiredTiger --dbpath <newWiredTigerDBPath> ...

   At this stage you should have:

   • config1 (running mmap)
   • config2 (running WiredTiger with no data)
   • config3 (stopped)

   Note that config2 doesn't have any data yet, because it has just been started up with the new WiredTiger dbpath.

6. Use mongorestore to load the config database backup you created in step 3.

   At this stage you should have:

   • config1 (running mmap)
   • config2 (running WiredTiger)
   • config3 (stopped)

7. Shut down config2:

   At this stage you should have:

   • config1 (running mmap)
   • config2 (stopped)
   • config3 (stopped)

   config2 is stopped at this point to ensure no metadata changes can happen when we start config3 up in the next step.

8. Restart config3.

   At this stage you should have:

   • config1 (running mmap)
   • config2 (stopped)
   • config3 (running mmap)

(steps 9-15) These steps are just repeating the same mongodump and mongorestore for each config server. There's a bit of shuffling to ensure you always have at least one config server available, and do not have all three up while you are still migrating data.

16. Upload the data into config1

    At this stage you should have:

    • config1 (running WiredTiger)
    • config2 (stopped)
    • config3 (running WiredTiger)

17. Start config2

    With all three config servers are available & upgraded, changes to the sharded cluster metadata can now resume.

18. Re-enable the balancer so normal balancing activity & chunk migration can resume.