Mongodb – what if mongos server goes down in shard cluster setup

Have you tried using the sh.startBalancer() helper instead?

Rather than a straight update, it does takes an timeout argument as how long to wait for balancing to start as well as a sleep interval in terms of how long to sleep between waiting. Here's the code from the shell by way of explanation:

mongos> sh.startBalancer
function ( timeout, interval ) {
    sh.setBalancerState( true )
    sh.waitForBalancer( true, timeout, interval )
}

So, you could even break it up and use the waitForBalancer helper if you wished. For reference, here is the equivalent stopBalancer command erroring out when I tried to stop it with a config server down:

mongos> sh.stopBalancer(2000, 100)
Waiting for active hosts...
Waiting for active host adamc-mbp.local:30999 to recognize new settings... (ping : Tue Dec 31 2013 19:51:32 GMT+0000 (GMT))
Waiting for the balancer lock...
Waiting again for active hosts after balancer is off...
Tue Dec 31 19:51:39.243 error: {
    "$err" : "error creating initial database config information :: caused by :: SyncClusterConnection::udpate prepare failed:  localhost:29000:9001 socket exception [FAILED_STATE] server [localhost:29000] ",
    "code" : 8005
} at src/mongo/shell/query.js:128

When you say "shutdown node-01 or node-02" what are you actually shutting down? (mongod service, config server service, the host?) Please also state what node you are shutting down in the example you gave, as well as the which node you took the mongos log from.

You can run your config server on the same host as your replica set. However, you may wish to run your config servers on separate hosts. That way, if a replica set host goes down, you don't loose a config server also. If you do loose a config server, your cluster won't be able to migrate chunk data on sharded collections. (this is what is happening in your logs, the balancer will remain disabled until your config server comes back on line.)

Note: I probably would not have apps connecting to the mongos running on the replica sets. It is fine that they are running there, admins have easy access to a mongos to connect to. But, I would run a mongos that sits outside these hosts. Otherwise a host going down in your cluster means your app, or part of it at least, is also down.

Replication is unaffected by a config server going down (just chunk migration). Do you have a sharded collection in this cluster?

Follow up:

Not sure what happened with my comment. I'll try it this way:

You can absolutely run your config servers on the same hosts as your replicas. Here is a quote from Chodorow, Kristina. MongoDB: The Definitive Guide:. Beijing: O'Reilly, 2013. page 243.

"In terms of provisioning, config servers do not need much space or many resources. A generous estimate is 1 KB of config server space per 200 MB of actual data: they really are just tables of contents. As they don't use many resources, you can deploy config servers on machines running other things, like app servers, shard mongods, or mongos processes."

Something else is going on. When you connect your mongos to the config servers, are you adding them using IP addresses or host names? What is the output of rs.conf() and rs.status() before and after shutting down the node? I notice mongos is attempting to connect on port 27017. Typically, in a sharded setup, your mongod port will be 27018 and config port will be 27019. Did you change the port numbers?

I suspect you have something misconfigured. Mongo's fail-over is pretty rock solid.