MongoDB Reverse SSH Replica Set Config Fails when leader changes

mongodbreplication

We have a MongoDB replica set with 3 machines outside our network and one inside network. The inside one is connected to leader by reverse SSH tunnel such as :

ssh -R 37017:inside_machine:37017 user@leadermachine

Everything works well if the leader doesn't change. When it does, we need to renew the tunnel and hosts entry for the new leader (host entry should resolve the same ip as the leader) but replica set configs don't match anymore,

InvalidReplicaSetConfig: Our replica set configuration is invalid or does not include us

And we need to remove the inside machine, drop local db and readd to replica set. This means the inside machine drops everything and starts to replicate from the very beginning.

While this is not much of a problem for litte db's, ours are large and replication may as well take 2 days.

Are we doing something wrong or is there anything we can do to prevent this?

Thanks a lot.

Best Answer

You should use TLS/SSL for connections between those machines. Then you don't need to play around with ssh. With SSL connections between nodes are encrypted and IP addresses don't change.

Related Solutions

MONGODB primary became secondary after servers restart

You need to change the config such that there is only one member remaining, and you have to do it on that host, i.e. Ubuntu12041vanilla:27017

The others are unreachable from its perspective, so even if you go with 2 nodes, 1 out of 2 down means you cannot form a majority and you will still only have secondary state.

Given your current rs.status() output, try this when connected via the mongo shell to Ubuntu12041vanilla:27017:

cfg = rs.conf()
cfg.members = [cfg.members[0]]
rs.reconfig(cfg, {force : true})

That will give you a config only one node and should mean that Ubuntu12041vanilla:27017 will be the only remaining member and primary.

Mongodb – Steps to setup mongo replica set

tl;dr

Your hosts were unable to resolve the hostnames you gave to actual IPs, since those were missing in the /etc/hosts file.

How to solve

The hosts files on the all the machines must resolve the IPs of the respective machines to their respective hostnames.

For example

192.168.0.1 myhost1
192.168.0.2 myhost2
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Make sure all hostnames are resolved correctly by pinging each other host from each host, eg.

myhost1 $ ping myhost2

Please read the docs thoroughly and make sure you understood them.
Proceed to step 2 until you really got it.
You only have to call rs.initiate() once and only on one machine, then use rs.add() on the same machine to add replica set members.

With your setup, you need an additional node or an arbiter

Personal note

Please do not take this personal, but you lack basic system administration skills, and that's put politely.

For your own sake: Do not run MongoDB for production until you acquired basic system and MongoDB administration skills!

I'd look for RHCSA first (works for CentOS as well) and M102, M202 and maybe even MongoDB DBA at MongoU later (without basic sysadmin skills, the latter won't help you much).