Mongodb – sharding and replication on mongodb

mongodb

My question is on the answer given at Difference between Sharding And Replication on MongoDB

I need to split thedata of 75GB into 3 shards of 25GB each with replication factor of 3. Answer depicts below picture

                            Sharded Cluster             
             /                    |                    \
      Shard A                  Shard B                  Shard C
        / \                      / \                      / \
+-------+ +---------+    +-------+ +---------+    +-------+ +---------+
|Primary| |Secondary|    |Primary| |Secondary|    |Primary| |Secondary|
|  25GB |=| 25GB    |    | 25 GB |=| 25 GB   |    | 25GB  |=| 25GB    |   
+-------+ +---------+    +-------+ +---------+    +-------+ +---------+

and says we need at least 6 database servers organized in three replica-sets. Each replica-set consists of two servers who have the same 25GB of data.

My understanding is that each shard is a physical machine holding below two sets of data(primary and secondary) and host two separate mongod server/instance . One will hold primary data and second will hold secondary data which is back up of primary data on another shard

Primary data :- This is 1/3 of the 75GB data distributed over some partition key
Secondary :- This will hold secondary data which is back up of primary data on another shard

Application query will go only to primary data on a given shard based on partition key. It is possible that on a shard a secondary gets converted to primary if another shard primary goes down . Is my understanding correct here ?

No of instances of Mongos and Arbiter

Also i believe i need to start only 1 mongos instance(which will be separate from mongod instance) ? Similarly I need to have single mongo arbiter for all 3 shards instead of 3 on each shard ? Both mongos and aribiter can be on separate machine or share the hardware of one of the shard depending on requirement ?

Best Answer

My understanding is that each shard is a physical machine. holding below two sets of data(primary and secondary) and host two separate mongod server/instance.

No, each replica-set member is a physical machine. In the above graphic, each shard consists of two physical machines. While you can technically run multiple members of a replica-set on the same machine, there is nothing to gain from that.

. One will hold primary data and second will hold secondary data which is back up of primary data on another shard

No, the secondary will back up the data from the other member(s) of the same shard. But if you want to build the server-level equivalent of a mirroring RAID, you could of course put two members of different shards on the same physical machine.

i believe i need to start only 1 mongos instance

You only need one, but to avoid a single point of failure, you might want to have more than one.

Similarly I need to have single mongo arbiter for all 3 shards instead of 3 on each shard?

An arbiter can only be member of a single replica-set. So you need 3 arbiters. However, the arbiters are very lightweight. They don't hold any data and don't do anything unless there is a primary election. So you do not need a dedicated server for the arbiter processes.

Related Solutions

Mongodb – Difference between Sharding And Replication on MongoDB

A Replica-Set means that you have multiple instances of MongoDB which each mirror all the data of each other. A replica-set consists of one Master (also called "Primary") and one or more Slaves (aka Secondary). Read-operations can be served by any slave, so you can increase read-performance by adding more slaves to the replica-set (provided that your client application is capable to actually use different set-members). But write-operations always take place on the master of the replica-set and are then propagated to the slaves, so writes won't get faster when you add more slaves.

Replica-sets also offer fault-tolerance. When one of the members of the replica-set goes down, the others take over. When the master goes down, the slaves will elect a new master. For that reason it is suggested for productive deployment to always use MongoDB as a replica-set of at least three servers, two of them holding data (the third one is a data-less "arbiter" which is required for determining a new master when one of the slaves goes down).

A Sharded Cluster means that each shard of the cluster (which can also be a replica-set) takes care of a part of the data. Each request, both reads and writes, is served by the cluster where the data resides. This means that both read- and write performance can be increased by adding more shards to a cluster. Which document resides on which shard is determined by the shard key of each collection. It should be chosen in a way that the data can be evenly distributed on all clusters and so that it is clear for the most common queries where the shard-key resides (example: when you frequently query by user_name, your shard-key should include the field user_name so each query can be delegated to only the one shard which has that document).

The drawback is that the fault-tolerance suffers. When one shard of the cluster goes down, any data on it is inaccessible. For that reason each member of the cluster should also be a replica-set. This is not required. When you don't care about high-availability, a shard can also be a single mongod instance without replication. But for production-use you should always use replication.

So what does that mean for your example?

                            Sharded Cluster             
             /                    |                    \
      Shard A                  Shard B                  Shard C
        / \                      / \                      / \
+-------+ +---------+    +-------+ +---------+    +-------+ +---------+
|Primary| |Secondary|    |Primary| |Secondary|    |Primary| |Secondary|
|  25GB |=| 25GB    |    | 25 GB |=| 25 GB   |    | 25GB  |=| 25GB    |   
+-------+ +---------+    +-------+ +---------+    +-------+ +---------+

When you want to split your data of 75GB into 3 shards of 25GB each, you need at least 6 database servers organized in three replica-sets. Each replica-set consists of two servers who have the same 25GB of data.

You also need servers for the arbiters of the three replica-sets as well as the mongos router and the config server for the cluster. The arbiters are very lightweight and are only needed when a replica-set member goes down, so they can usually share the same hardware with something else. But Mongos router and config-server should be redundant and on their own servers.

Mongodb – Mongo db and sharding / replica sets concept questions

1 - You can have more than 1 mongos instance and connect to whichever you want (the client driver should have that option). Nevertheless, a mongos is just a router, meaning it will only route the requests to the correct shard(s).

2 - Yes, a config server can be in the same machine as a primary/secondary, just don't put config instances together (you are required to have 3 because of redundancy).

3 - A replica set is a group consisting of 1 primary and N secondaries. If you are sharding, each group will be a shard. That means that the single primary and all secondaries of the same group will have the same data (replicated). Also, having more secondaries won't help you increase the write performance as only the primary is able to do perform inserts/updates/deletes. Considering this, the only way to increase the write performance is by having more shards (horizontal scaling) and, of course, choosing a good sharding key that will balance data across all shards (you don't want all your data to be in 1 shard and all others remain empty). For further info check this official doc explaining the concept of replica set and its members http://docs.mongodb.org/manual/core/replica-set-members/

Also, instances of a replica set shard (the primary and N secondaries) are not required to be in the same machine. You might want to put them in separated machines to increase redundancy and perhaps have a better distribution of the load.

Best Answer

Related Solutions

Mongodb – Difference between Sharding And Replication on MongoDB

So what does that mean for your example?

Mongodb – Mongo db and sharding / replica sets concept questions

Related Question