MongoDB Replica Set: Can’t Add Member/Connect

You currently have a member running in secondary, because it cannot form a majority. This is why you should always have an odd number of nodes in a replica set (one can be an arbiter) and I would recommend adding a third node as soon as possible once you get things back to normal.

In terms of how to get the second node up and running, do the following:

1. Shut down the remaining Secondary server (not strictly necessary, since it is read only as a Secondary, but safer)
2. Now, copy the whole data directory (everything in dbpath) over to the other host
3. Restart both nodes

One of the advantages to replica sets (over classic master/slave) is that they are intended to be functionally identical to each other, so you can simply use the data from one "good" node to seed any other "bad" node.

There is a guide for how to do this (reconfigure a replica set with unavailable members) here:

http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

You essentially have to do a forced reconfig of the set because there is no primary available. If you want to keep a set operational while doing this type of move, you can:

1. Use your hosts file to temporarily point the old names to the new (or similar) and "fool" the primary into thinking they are up (then make the permanent changes one at a time)
2. Temporarily add arbiters to prevent the primary from stepping down (2 more for example, so the primary is kept with 3/5 votes)
3. Reconfigure the set to a single node before doing the moves
4. Do the moves one server at a time, ensuring there are 2/3 nodes up to form a majority at all times (and hence keep a primary up)