Mongodb – how to upgrade the config servers to use WiredTiger in mongodb 3.0.0

Restoring config servers, particularly if you have had some sort of catastrophic event is tricky, but not impossible. But, before we go any further, a big bold caveat:

BACK UP EVERYTHING

That means taking a back up of all three config servers. I am going to give you some advice, and it is generally correct, but please, please take a back up of every current config server instance before you overwrite/replace anything

As a quick explanation, config servers are not configured as a replica set - each config server instance is supposed to be identical (at least for all the collections that matter) to the others. Hence, any healthy config server can be used to replace a non-healthy config server and you can then follow the tutorial you mentioned to get back to a good config.

The key to recovery is to identify the healthy config server and then use that to replace the others - you then end up with 3 identical config servers.

There is more than one way to do this, they basically fall into three categories:

1) Use the error message

The error message that is printed out actually lets you know which config server it believes is health, though that is not obvious from the messaging. Here's how to read it generically:

ERROR: config servers not in sync! config servers <healthy-server> and <out-of-sync-server> differ

Basically the first one in the list is the healthy one, in your case that would be mongocfg1.testing.com:27000. That is our first candidate for a healthy config database.

2) Use dbhash to compare all three and pick the ones that agree

On each config server switch to the config database using use config, run db.runCommand("dbhash") and compare the hashes for the collections below:

• chunks
• databases
• settings
• shards
• version

You are looking for two servers that agree, and using that as the basis to determine that the version of the config database on those hosts is basically trustworthy and should be used to seed the rest.

3. Manually inspect the collections in the config database

Finally, take a look at the config database, and pay attention to the collections listed in the second option above. This is a straight judgement call based on your familiarity with your data.

Hopefully all three methods point you at the same host (or hosts). That config server should be used to seed the other two (after you have taken backups so you can go back). That is basically your best bet. Should that fail, then you may want to try one of the other versions (from the backups) - always making sure that when you start them, all three are identical.

Finally, always ensure that all mongos processes are using the same config server string, and that all 3 servers are always listed in the same order on every process - not doing so across all mongos processes can lead to (very) odd results.

It looks like the current MongoDB 3.0 upgrade instructions are missing mention of two important parameters for backing up and restoring users and roles:

• mongodump --dumpDbUsersAndRoles (see also: Required Access to Backup User Data).
• mongorestore --restoreDbUsersAndRoles (see also: Required Access to Restore User Data)

I can think of several approaches to fix:

1. If you don't have many user accounts on the config servers, recreate the administrator & user accounts. This isn't ideal, but is probably the fastest approach.

2. Export the users from your mmap database. This is more involved, but saves you recreating the users & roles. I've described steps for this below.

3. Redo the config server migration with the user & role information included. I expect this is the least desirable option.

Exporting the users

Assuming you have already upgraded all of your config servers to WiredTiger, here are some steps to add the user information:

1. Disable the balancer

2. Stop the last config server listed in your mongos' configDB setting (will call that config3 for the purpose of these steps). This will ensure your sharded cluster metadata remains read-only for the following steps.

3. Re-start config2 using the mmap data directory

   At this stage you should have:

   • config1 (running WiredTiger)
   • config2 (running mmap with user/role data)
   • config3 (stopped)

4. Export the data from config2:

   mongodump --db config --dumpDbUsersAndRoles --username .. --password ..

   Add any other parameters needed, eg --authenticationDatabase .. if you need to auth against another database.

5. If you have users in the admin database on your config server, you will also want to dump that as well.

6. (optional) Remove files from your dump except for the user/role information. If you are certain nothing has changed since you did the original migration from mmap to WiredTiger you could skip this step, however it would be safer to not overwrite any existing data.

   Preview the files to remove:

   find ./dump -type f -not -name "\$admin.system*"

   WARNING: removing files, make sure you have previewed to confirm:

   find ./dump -type f -not -name "\$admin.system*" | xargs rm

7. Re-start config2 using the wiredTiger storage engine

8. Run: mongorestore --db config --restoreDbUsersAndRoles dump/config/

   You should see messages about restoring users & roles, for example:

   2015-03-18T02:41:34.887+1100 restoring users from dump/config/$admin.system.users.bson

   2015-03-18T02:41:34.887+1100 restoring roles from dump/config/$admin.system.roles.bson

9. Login to config2 and confirm the users are correctly setup (i.e. auth with admin account, use db.getUsers() to check).

   At this stage you should have:

   • config1 (running WiredTiger)
   • config2 (running WiredTiger with user/role data)
   • config3 (stopped)

10. Copy the dump directory to config1 and repeat the mongorestore step.

11. Shutdown config2 (to keep the sharded cluster metadata readonly for the next step).

    At this stage you should have:

    • config1 (running WiredTiger with user/role data)
    • config2 (stopped)
    • config3 (stopped)

12. Start config3. Copy the dump directory to config3, and repeat the mongorestore step.

    At this stage you should have:

    • config1 (running WiredTiger with user/role data)
    • config2 (stopped)
    • config3 (running WiredTiger with user/role data)

13. Start config2. At this point all config servers should be online with the user information.

14. Re-enable the balancer so normal balancing activity & chunk migration can resume.