Restoring config servers, particularly if you have had some sort of catastrophic event is tricky, but not impossible. But, before we go any further, a big bold caveat:
BACK UP EVERYTHING
That means taking a back up of all three config servers. I am going to give you some advice, and it is generally correct, but please, please take a back up of every current config server instance before you overwrite/replace anything
As a quick explanation, config servers are not configured as a replica set - each config server instance is supposed to be identical (at least for all the collections that matter) to the others. Hence, any healthy config server can be used to replace a non-healthy config server and you can then follow the tutorial you mentioned to get back to a good config.
The key to recovery is to identify the healthy config server and then use that to replace the others - you then end up with 3 identical config servers.
There is more than one way to do this, they basically fall into three categories:
1) Use the error message
The error message that is printed out actually lets you know which config server it believes is health, though that is not obvious from the messaging. Here's how to read it generically:
ERROR: config servers not in sync! config servers <healthy-server> and <out-of-sync-server> differ
Basically the first one in the list is the healthy one, in your case that would be mongocfg1.testing.com:27000
. That is our first candidate for a healthy config database.
2) Use dbhash
to compare all three and pick the ones that agree
On each config server switch to the config database using use config
, run db.runCommand("dbhash")
and compare the hashes for the collections below:
- chunks
- databases
- settings
- shards
- version
You are looking for two servers that agree, and using that as the basis to determine that the version of the config database on those hosts is basically trustworthy and should be used to seed the rest.
3. Manually inspect the collections in the config database
Finally, take a look at the config database, and pay attention to the collections listed in the second option above. This is a straight judgement call based on your familiarity with your data.
Hopefully all three methods point you at the same host (or hosts). That config server should be used to seed the other two (after you have taken backups so you can go back). That is basically your best bet. Should that fail, then you may want to try one of the other versions (from the backups) - always making sure that when you start them, all three are identical.
Finally, always ensure that all mongos
processes are using the same config server string, and that all 3 servers are always listed in the same order on every process - not doing so across all mongos
processes can lead to (very) odd results.
Have you tried using the sh.startBalancer()
helper instead?
Rather than a straight update, it does takes an timeout argument as how long to wait for balancing to start as well as a sleep interval in terms of how long to sleep between waiting. Here's the code from the shell by way of explanation:
mongos> sh.startBalancer
function ( timeout, interval ) {
sh.setBalancerState( true )
sh.waitForBalancer( true, timeout, interval )
}
So, you could even break it up and use the waitForBalancer
helper if you wished. For reference, here is the equivalent stopBalancer
command erroring out when I tried to stop it with a config server down:
mongos> sh.stopBalancer(2000, 100)
Waiting for active hosts...
Waiting for active host adamc-mbp.local:30999 to recognize new settings... (ping : Tue Dec 31 2013 19:51:32 GMT+0000 (GMT))
Waiting for the balancer lock...
Waiting again for active hosts after balancer is off...
Tue Dec 31 19:51:39.243 error: {
"$err" : "error creating initial database config information :: caused by :: SyncClusterConnection::udpate prepare failed: localhost:29000:9001 socket exception [FAILED_STATE] server [localhost:29000] ",
"code" : 8005
} at src/mongo/shell/query.js:128
Best Answer
As Per MongoDB BOL Starting in
3.4
, the use of the deprecated mirrored mongod instances as config servers (SCCC
) is no longer supported. Before you can upgrade your sharded clusters to 3.4, you must convert your config servers fromSCCC
toCSRS
.To convert your config servers from
SCCC
toCSRS
, see Upgrade Config Servers to Replica Set.Where Config servers store the metadata for a sharded cluster. The metadata reflects state and organization for all data and components within the sharded cluster. The metadata includes the list of chunks on every shard and the ranges that define the chunks.
The mongos instances cache this data and use it to route read and write operations to the correct shards. mongosupdates the cache when there are metadata changes for the cluster, such as Chunk Splits or adding a shard. Shards also read chunk metadata from the config servers.
MongoDB also uses the config servers to manage distributed locks.
Each sharded cluster must have its own config servers. Do not use the same config servers for different sharded clusters.
WARNING: Administrative operations conducted on config servers may have significant impact on sharded cluster performance and availability. Depending on the number of config servers impacted, the cluster may be read-only or offline for a period of time.
For your further Ref Here and Here