Mongodb – In shard cluster, if 2 replica set nodes are offline leaving only 1 secondary..where do writes go

Sometimes, the 3 config server are out of sync. What does it mean? How can I quickly get rid of these issues?

Most convenient way would be to copy (cp or rsync) the configdb data directory content between hosts in cases of inconsistency

When should I choose MongoDb Shard with Replica Set vs MongoDb Replica set only?

Replica ONLY: When you desire to make your mongo data highly available and add redundancy in your solution. The benefit can be to offload reporting requests to secondary nodes and/or make mongodb cluster sustain network partitions or disaster recovery

Sharding ONLY: When you desire to scale your mongodb cluster horizontally to achieve load balancing and distribute traffic to multiple hosts you must choose sharding. For example, you can achieve partitioning benefits with sharding enabled on databases.

Sharding with replica set: When you desire to enable all benefits stated above i.e. scale you cluster horizontally to handle large volumes of data for load balancing and also achieve HA within each shard you must choose this approach

First, restart the mongos process on mongosHost3, then recheck the locks as you did in the question.

If the same lock (make sure it is the same and not a new lock by the same process - check the ts field etc.) is still present after that restart it is safe to remove it manually, it is basically stale and should be cleaned up eventually in any case. You can remove it just like any other document (assuming you have permissions to edit the config DB of course). MongoDB 3.2 introduced the deleteOne command which I would recommend using here.

It is possible that this is not an issue with a stale lock and is instead an issue with balancing (i.e. it is seeing errors or is stuck/slow for some reason), take a look in the mongos logs of whatever process holds the lock once balancing kicks off again and see what it reports as progress for balancing.

You can also look in the the changelog for more information related to balancing and migrations. If balancing is seeing issues I would recommend getting the logs, changelog information and posting a separate question to troubleshoot that issue.