Mongodb – what is so special about MongoDB indexing

Bottom line, is this a necessary point to consider when mongodb are used in production?

Is it worth figuring out your bandwidth requirements? Yes

Is it worth including the heartbeat in that calculation? No

The heartbeats between the nodes in a set are basically TCP based and unicast - it is done by background threads that monitor the other members of the set from each of the constituent node and fire every ~2 seconds with minimal amounts of data. If you are not doing enough traffic in terms of data to make that an insignificant amount of traffic, then you probably don't need to bother modeling for traffic at all.

The real thing you need to determine is the data traffic - each secondary will be replicating data from a source. Your primary will need to both ingest that data from whatever application or other client is feeding it as well as replicate the data to at least one secondary (other secondaries can be "chained" off the first, but at least one must replicate from the primary).

Then, of course, for most non-archival databases, something is going to be consuming data as well as writing it. Again, this can all be from the primary, or you might be OK with eventual consistency and go for secondary reads also. In read heavy applications, this can dwarf write traffic and replication traffic.

If you are talking about a sharded environment, then you get some extra overhead - each shard is usually a replica set and you will get some communication between them as chunks are balanced. But again, this is usually dwarfed by the application reads and writes.

To summarize, unless you are going to be doing something out of the ordinary, the amount of data you read and write will be your main source of bandwidth usage and hence what you have to model.

1 - Yes, the balancing operation can affect the performance, that's why you can schedule the balancing window. By default, the balancer is enabled and may run at any time.

2 - The balancer moves data chunks between shards.

Just to clarify, each shard is a replica set. All members of the replica set (the primary and N secondaries) have the same data (replicated). Therefore, each shard has a different subset of the total data.