MongoDb: sharded replica set: data distribution

If i have 10 shards, each being a 3 node replica set. If i loose 2 servers in a replica set causing the primary to change to a read-only secondary. Where does the mongos write new chunks to that the hash key would have sent to this shards primary?

Each shard contains a subset of data for the sharded cluster. If the replica set backing a shard becomes read-only you will get exceptions trying to write new data to that shard. The mechanics of failover and high availability for an individual shard are governed by your Replica Set configuration and deployment.

Does the config server detect the shard being read only and inform the mongos that writes should be redirected to another shard?

No. In order to achieve this, the existing data would have to be migrated to another shard. The config servers and mongos do not monitor whether a shard is read-only or attempt to work around this case. When write operations are directed to that shard, they will generate exceptions.

If you do have an outage for a shard, one of the steps you should take is to disable the balancer. This will prevent attempts to migrate or rebalance data while you work on restoring that shard.

when the shard gets fixed and has a primary again. Will the chunks be re-balanced to it?

Once the shard has a primary, normal operation can resume. You should also remember to re-enable the balancer if you disabled it during your outage.

Once you add a user to the individual shards, which you indicate you have done for MMS, you must then have valid credentials to connect for any purpose, including mongodump. Up until you added that user for MMS, the shards were running with authentication enabled but with no users populated (this only happens if all your users are in the admin database ans using delegated auth for other databases, otherwise with 2.4 and below you would have at least one shard with users for each database - 2.6+ changes this behavior) and so you were able to connect without credentials.

Essentially this is a loophole left open so that you don't accidentally lock yourself out of your instances when you turn on auth with no users (and one that would probably have stopped working at some point as default security is tightened anyway).

The bottom line is that you will need to add a user for use with mongodump, and it's a good idea to do so anyway rather than allowing non-authenticated users free access to your instances. If you are running 2.6 or later, then the built in backup role exists precisely for this purpose, if you are on 2.4 or earlier, then the description for that role gives you a great outline of what is needed to backup successfully (and in particular if you want to backup the users themselves).