I will try to be brief. Most of the answers are documented.
Q1: There are many ways. Shard a collection using _id:hashed and set initial number of chunks 2 and then start inserting documents (100 is a good number of docs). You will see documents on both shards.
Q2: It uses range partitioning. Each shard holds ranges of the shard key.
Q3: Sharding is an automated process, all you need to do is to determine a shard key.
Q4: Stop the balancer, backup shard rs1, rs2 and the config server, start the balancer.
Q5: On a sharded cluster the access point should be the mongos
My understanding is that each shard is a physical machine. holding below two sets of data(primary and secondary) and host two separate mongod server/instance.
No, each replica-set member is a physical machine. In the above graphic, each shard consists of two physical machines. While you can technically run multiple members of a replica-set on the same machine, there is nothing to gain from that.
. One will hold primary data and second will hold secondary data which is back up of primary data on another shard
No, the secondary will back up the data from the other member(s) of the same shard. But if you want to build the server-level equivalent of a mirroring RAID, you could of course put two members of different shards on the same physical machine.
i believe i need to start only 1 mongos instance
You only need one, but to avoid a single point of failure, you might want to have more than one.
Similarly I need to have single mongo arbiter for all 3 shards instead of 3 on each shard?
An arbiter can only be member of a single replica-set. So you need 3 arbiters. However, the arbiters are very lightweight. They don't hold any data and don't do anything unless there is a primary election. So you do not need a dedicated server for the arbiter processes.
Best Answer
If eavesdropping of traffic intra-cloud is a potential issue for you, you really should implement the connection over SSL. You should also consider running all of your resources in a VPC (virtual private cloud). This allows you to isolate all of your EC2 resources in a private network that is isolated from the rest of the traffic. It also enables you to control where the points of connection to the external world are and how they behave.