Mongodb – Up to sync new members in replica set

Its not a good idea to run the backup against the Primary. The extra load you adding on the Primary might cause heartbeat failures and trigger an election. Now in your case, overloading the primary might caused an election, you can see that info on your logs, but the other secondary didn't become new primary which expected to. I see some cases why didn't. 1) Replication lag was big 2) you run the replica set on the same server or storage so all nodes get affected but here logs are your best friend on what really happened.

To migrate data between different storage engines in MongoDB 3.0 you will need to perform an initial sync to the new replica set node or use mongorestore to seed a new replica set with your data.

If you want to add a node with a new storage engine to a live production replica set, rs.add() is the correct approach. To ensure a smooth upgrade I would also upgrade all existing nodes to the latest version of MongoDB 3.0.x with MMAP storage (the default) before adding your new WiredTiger node(s).

The key information in your question (and comments) is:

• socket exceptions related to networking timeouts:

  I NETWORK  [rsSync] Socket recv() timeout  x.x.x.x:x
  I NETWORK  [rsSync] SocketException: remote: x.x.x.x:x error: 9001 socket exception [RECV_TIMEOUT] server [x.x.x.x:x]
  

• deploying on Azure, which has some known issues with timeouts

As per the MongoDB production notes on Azure, you should reduce the TCP keepalive setting to avoid network timeouts:

The TCP keepalive on the Azure load balancer is 240 seconds by default, which can cause it to silently drop connections if the TCP keepalive on your Azure systems is greater than this value. You should set tcp_keepalive_time to 120 to ameliorate this problem.

As further preparation for successful initial sync with WiredTiger you should review the MongoDB production notes for your operating system.

Since you mention using Linux, I would recommend checking:

• ulimits are appropriately set. WiredTiger creates more files than MMAP (separate data & index files per collection rather than per database) so will use more file handles than MMAP nodes in the same replica set. If you don't increase the ulimits from the default you may find the initial data transfer completes but initial sync fails at the index build stage with a fatal error like Too many open files.

• the WiredTiger data directory is on a volume using the XFS file system. As at the time of writing, there are known performance issues (such as throughput stalls) with EXT4.

The production notes in the MongoDB manual are updated regularly based on issues and experiences reported by MongoDB users, so should be part of your preflight checklist for production O/S deployments or upgrades.