Mongodb – Deploy Across Cloud and On-Premises

In order :

1. There is one reason to have multiple mongod instances in one machine (other than machine efficient repset setups) and that one reason is the process wide write lock. If you run into a situation where you're bottlenecked by the write lock rather than any hardware resource bottleneck you can go for multiple mongod instances. In all other situations it's usually more practical to have one mongod process, especially given it's memory mapped based storage engine.
2. There's no ideal number. Adding shards (almost) linearly scales up write and read performance for sharded collections. It's a basic horizontal scaling solution and follows the rules for that sort of scaling.
3. Replica sets always consist of at least 3 members although 1 of them can be an arbiter (votes but does not replicate). If you add more you gain read performance if your application uses slaveOk flagged queries. Again, ideal number depends on your use case.
4. Depends. Most people use a hot backup scheme of some sort. There's no difference between MySQL backup conventions and MongoDB here.
5. Depends on whether or not you want to clone a single database or the entire cluster? Both are well documented.

What happened

I have asked in the comments of the question that OP provides the output of rs.status()

The reason for that was that the primary reverted to secondary status once a single member was shut down. It was obvious that the cluster lost the quorum necessary to elect a new primary. This could only be the case when one additional of the voting members of the original members of the replica set is or became unavailable.

As it turned out, the arbiter of the replica set in question was not reachable by the primary, which after the shutdown of the PC was the only remaining member (from it's point of view) of the replica set. So it wasn't possible to hold an election with a quorum of the configured replica set members and it consequently reverted to secondary state.

How to prevent

• Always run rs.status() after setting up a replica set.
• Always run rs.status() when encountering problems with a replica set
• Always do fail tests (down to loosing write capabilities) and ensure your application handles those situations gracefully (as OP did)

Using these rules, you will eliminate the vast majority of problems one can face when using a replica set.

Personally, I think MongoDB Inc.'s Cloud Manager is a must for production environments, since it shows such problems os OP had right away and has alerting built in.

Side note

Never, ever (and yes, that means no exception for no reason, however sound the reasons may seem to be) put an arbiter on a data bearing node of the same replica set.

Imagine the node with the arbiter and the data bering node goes down.

If you have a 3 member replica set, you wouldn't have a quorum of the original members, your remaining member would automatically revert to secondary, loosing the failover capability.

In a 5 member replica set, two voting members would be eliminated. Fine as long as all others are up and running, right? Except, it isn't fine. If another node fails, you'd loose your quorum again. So with only two nodes failed, the other two nodes become more or less useless. Given the price of a virtual server today (and even the smallest ones are well sufficient to run an arbiter), this simply does not make sense. You'd be paying 4 data bearing nodes anyway and loose failover capabilities because you tried to save a tiny fraction of the overall costs.

With a 7 member replica set, those costs become an even tinier fraction of the overall costs.

Conclusion: It's simply a bad decision business wise to have a arbiter running on the same machine as a data bearing node, even when setting aside the technical aspects.