MongoDB Backup – How to Copy Files for mongorestore

backupmongodb

I've got 3 nodes in a replica set.

Each time when performing a mongorestore from the Master, the secondaires go into a RECOVERING state. The oplog cannot keep up. unfortunately I cannot increase the oplog size because lack of disk space.

Now, I want to perform a manual restore by copying the files.
https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/

Here is a snippet of the files which I got:

meee@mongo-p01:~$ sudo ls -la /var/lib/mongodb
[sudo] password for meee:
total 44052516
drwxr-x---   4 mongodb root          4096 Feb 22 17:54 .
drwxr-xr-x. 32 root    root          4096 Feb 23 01:59 ..
-rw-------   1 mongodb mongodb   67108864 Feb 22 17:54 mymongobase.0
-rw-------   1 mongodb mongodb  134217728 Feb 22 17:08 mymongobase.1
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:14 mymongobase.10
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:12 mymongobase.11
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:14 mymongobase.12
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:14 mymongobase.13
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:14 mymongobase.14
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:14 mymongobase.15
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:14 mymongobase.16
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:15 mymongobase.17
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:21 mymongobase.18
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:51 mymongobase.19
-rw-------   1 mongodb mongodb  268435456 Feb 22 17:08 mymongobase.2
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:51 mymongobase.20
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:54 mymongobase.21
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:54 mymongobase.22
-rw-------   1 mongodb mongodb  536870912 Feb 22 17:08 mymongobase.3
-rw-------   1 mongodb mongodb 1073741824 Feb 22 17:08 mymongobase.4
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:11 mymongobase.5
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:53 mymongobase.6
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:54 mymongobase.7
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:54 mymongobase.8
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:53 mymongobase.9
-rw-------   1 mongodb mongodb   16777216 Feb 22 17:54 mymongobase.ns
drwxr-xr-x   2 mongodb mongodb         40 Feb 22 17:54 journal
-rw-------   1 mongodb mongodb   67108864 Feb 22 17:07 local.0
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:54 local.1
-rw-------   1 mongodb mongodb 2146435072 Feb 22 17:54 local.2
-rw-------   1 mongodb mongodb   16777216 Feb 22 17:54 local.ns
-rwxr-xr-x   1 mongodb mongodb          6 Feb 22 17:06 mongod.lock
drwxr-xr-x   2 mongodb mongodb          6 Feb 22 17:54 _tmp

What are the correct steps I should take?

1. Reset all mongos by reïnstalling
1. Perform a mongorestore on the master
1. Right after the restore, stop all mongod
1. Copy all local.x and mymongobase.x files to other nodes
1. Start mongod on all nodes?

I'm a bit lost here.

Best Answer

Ensure you have enough disk space!
Shutdown mongos on all 3 nodes
From Master node; tar all local* and mymongobase* files
SCP files to 2 secondaries
Unpack files in mongo data location (/var/lib/mongodb)
Start mongod on all nodes

Works for me

Related Solutions

MongoDB Root Access – Why Run mongod as Root?

First of all when you run mongod as a normal user you need to make sure that the particular user has the following permissions:

Access to network interface ( open socket ) in order to be able to listen to connections
Access to the database folder. The default is accessible only by root or mongod user (/data/db)
Access to logging folder /var/log/mongod

All of the above are automatically set and granted for the user mongodb (mongod in earlier versions). If you take a look at /etc/init.d/mongodb you will see that the service will be stared under the user mongodb DAEMONUSER=${DAEMONUSER:-mongodb} and not as root.

MongoDB Replication – Adding WiredTiger Nodes to Non-WiredTiger Replica Set

To migrate data between different storage engines in MongoDB 3.0 you will need to perform an initial sync to the new replica set node or use mongorestore to seed a new replica set with your data.

If you want to add a node with a new storage engine to a live production replica set, rs.add() is the correct approach. To ensure a smooth upgrade I would also upgrade all existing nodes to the latest version of MongoDB 3.0.x with MMAP storage (the default) before adding your new WiredTiger node(s).

The key information in your question (and comments) is:

socket exceptions related to networking timeouts:

I NETWORK  [rsSync] Socket recv() timeout  x.x.x.x:x
I NETWORK  [rsSync] SocketException: remote: x.x.x.x:x error: 9001 socket exception [RECV_TIMEOUT] server [x.x.x.x:x]

deploying on Azure, which has some known issues with timeouts

As per the MongoDB production notes on Azure, you should reduce the TCP keepalive setting to avoid network timeouts:

The TCP keepalive on the Azure load balancer is 240 seconds by default, which can cause it to silently drop connections if the TCP keepalive on your Azure systems is greater than this value. You should set tcp_keepalive_time to 120 to ameliorate this problem.

As further preparation for successful initial sync with WiredTiger you should review the MongoDB production notes for your operating system.

Since you mention using Linux, I would recommend checking:

ulimits are appropriately set. WiredTiger creates more files than MMAP (separate data & index files per collection rather than per database) so will use more file handles than MMAP nodes in the same replica set. If you don't increase the ulimits from the default you may find the initial data transfer completes but initial sync fails at the index build stage with a fatal error like Too many open files.
the WiredTiger data directory is on a volume using the XFS file system. As at the time of writing, there are known performance issues (such as throughput stalls) with EXT4.

The production notes in the MongoDB manual are updated regularly based on issues and experiences reported by MongoDB users, so should be part of your preflight checklist for production O/S deployments or upgrades.

Best Answer

Related Solutions

MongoDB Root Access – Why Run mongod as Root?

MongoDB Replication – Adding WiredTiger Nodes to Non-WiredTiger Replica Set

Related Question