Mongodb – Adding secondary via rs.add on MongoDB

mongodb

I have a master and slave on Linux (centOS 7). The rs.initiate(cfg) was done, but its shows the below output when trying to add secondary via rs.add:

genienational:PRIMARY> rs.add("localhost.localdomain:27017")  
{  
     "ok" : 0,  
     "errmsg" : "The hosts test.appraisergenie.com:27017 a  
nd localhost.localdomain:27017 all map to this node new co  
nfiguration version 2 for replica set genienational",  
     "code": 103  
}  
genienational:PRIMARY> use local  
switched to db local

MongoDB

How do I resolve?

Version: The master is on Enterprise. I think 3.2 and the slave is on 2.6.

Best Answer

The error message in your screenshot indicates you are trying to add identical members to your replica set under different names:

"The hosts x:27017 and y:27017 all map to this node in new configuration".

If you want to configure multiple replica set members on a single machine for testing purposes, all members need to use different ports and/or IP addresses.

Version: The master is on Enterprise. I think 3.2 and the slave is on 2.6.

Mixed-version replica sets are only supported for consecutive major versions (i.e. 3.2 and 3.0) and expected to be used transiently during a rolling upgrade to a new major version of MongoDB. A mixed replica set with 3.2 and 2.6 members is definitely not supported.

Related Solutions

Mongodb – Adding wiredtiger nodes to a non-wiredtiger replica set

To migrate data between different storage engines in MongoDB 3.0 you will need to perform an initial sync to the new replica set node or use mongorestore to seed a new replica set with your data.

If you want to add a node with a new storage engine to a live production replica set, rs.add() is the correct approach. To ensure a smooth upgrade I would also upgrade all existing nodes to the latest version of MongoDB 3.0.x with MMAP storage (the default) before adding your new WiredTiger node(s).

The key information in your question (and comments) is:

socket exceptions related to networking timeouts:

I NETWORK  [rsSync] Socket recv() timeout  x.x.x.x:x
I NETWORK  [rsSync] SocketException: remote: x.x.x.x:x error: 9001 socket exception [RECV_TIMEOUT] server [x.x.x.x:x]

deploying on Azure, which has some known issues with timeouts

As per the MongoDB production notes on Azure, you should reduce the TCP keepalive setting to avoid network timeouts:

The TCP keepalive on the Azure load balancer is 240 seconds by default, which can cause it to silently drop connections if the TCP keepalive on your Azure systems is greater than this value. You should set tcp_keepalive_time to 120 to ameliorate this problem.

As further preparation for successful initial sync with WiredTiger you should review the MongoDB production notes for your operating system.

Since you mention using Linux, I would recommend checking:

ulimits are appropriately set. WiredTiger creates more files than MMAP (separate data & index files per collection rather than per database) so will use more file handles than MMAP nodes in the same replica set. If you don't increase the ulimits from the default you may find the initial data transfer completes but initial sync fails at the index build stage with a fatal error like Too many open files.
the WiredTiger data directory is on a volume using the XFS file system. As at the time of writing, there are known performance issues (such as throughput stalls) with EXT4.

The production notes in the MongoDB manual are updated regularly based on issues and experiences reported by MongoDB users, so should be part of your preflight checklist for production O/S deployments or upgrades.

Mongodb – Steps to setup mongo replica set

tl;dr

Your hosts were unable to resolve the hostnames you gave to actual IPs, since those were missing in the /etc/hosts file.

How to solve

The hosts files on the all the machines must resolve the IPs of the respective machines to their respective hostnames.

For example

192.168.0.1 myhost1
192.168.0.2 myhost2
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Make sure all hostnames are resolved correctly by pinging each other host from each host, eg.

myhost1 $ ping myhost2

Please read the docs thoroughly and make sure you understood them.
Proceed to step 2 until you really got it.
You only have to call rs.initiate() once and only on one machine, then use rs.add() on the same machine to add replica set members.

With your setup, you need an additional node or an arbiter

Personal note

Please do not take this personal, but you lack basic system administration skills, and that's put politely.

For your own sake: Do not run MongoDB for production until you acquired basic system and MongoDB administration skills!

I'd look for RHCSA first (works for CentOS as well) and M102, M202 and maybe even MongoDB DBA at MongoU later (without basic sysadmin skills, the latter won't help you much).