Mongodb – Help – Error : Could not connect to master of shard replica set

If i have 10 shards, each being a 3 node replica set. If i loose 2 servers in a replica set causing the primary to change to a read-only secondary. Where does the mongos write new chunks to that the hash key would have sent to this shards primary?

Each shard contains a subset of data for the sharded cluster. If the replica set backing a shard becomes read-only you will get exceptions trying to write new data to that shard. The mechanics of failover and high availability for an individual shard are governed by your Replica Set configuration and deployment.

Does the config server detect the shard being read only and inform the mongos that writes should be redirected to another shard?

No. In order to achieve this, the existing data would have to be migrated to another shard. The config servers and mongos do not monitor whether a shard is read-only or attempt to work around this case. When write operations are directed to that shard, they will generate exceptions.

If you do have an outage for a shard, one of the steps you should take is to disable the balancer. This will prevent attempts to migrate or rebalance data while you work on restoring that shard.

when the shard gets fixed and has a primary again. Will the chunks be re-balanced to it?

Once the shard has a primary, normal operation can resume. You should also remember to re-enable the balancer if you disabled it during your outage.

How did you change the chunk size? Did you follow this guide?

To recap, one should:

1. connect to mongos
2. switch to config db
3. issue change chunk size command:
   db.settings.save( { _id:"chunksize", value: <sizeInMB> } )

AFAIK,

warning: chunk is larger than 1024 bytes because of key ...

won't bring down a shard, it will only result in a unbalanced cluster.