we have a mongodb cluster which takes about 3.7TB on the filesystem. One of our secondary nodes died, so we replaced it and started the resync.
The resync was started on 12 Jul 2017 and took until yesterday (6 Aug 2017) I can see the index rebuilding and immediately after that, I got the magical message (there are dbs before the test one):
[rsSync] initial sync cloning db: test
[rsSync] build index on: test.t1 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "test.t1" }
[rsSync] building index using bulk method
[rsSync] build index done. scanned 0 total records. 0 secs
[rsSync] initial sync data copy, starting syncup
[rsSync] oplog sync 1 of 3
[ReplicationExecutor] syncing from: fr-mongo-c3-p3:27017
[rsBackgroundSync] we are too stale to use fr-mongo-c3-p3:27017 as a sync source
[ReplicationExecutor] syncing from: de-mongo-c3-b1:27017
[ReplicationExecutor] syncing from: de-mongo-c3-p1:27017
[ReplicationExecutor] syncing from: de-mongo-c3-p2:27017
[ReplicationExecutor] could not find member to sync from
[rsBackgroundSync] too stale to catch up -- entering maintenance mode
[rsBackgroundSync] our last optime : (term: 10, timestamp: Jul 12 05:29:26:14)
[rsBackgroundSync] oldest available is (term: 10, timestamp: Aug 3 06:24:29:9f0)
Now what can I do with this? First thing that comes to mind is to increase the oplog size and repeat the process. Second thing that comes to mind is to take the backup data, copy them to this new node and if this process takes less then 4 days, I might be able to make it.
Any other ideas? Or is my assumption completely wrong? Thanks for answers!
(Mongo version is 3.2.10, OS is Ubuntu Trusty)
Best Answer
Yes, you are correct in both cases! Actually, it would be best to do both!
First, increase the opLog size (at least 4x, 10x if you have space) and after that use backup method to copy all data to that node (lvm snapshot is best).
During that time what copy takes, your new opLog (what is this time much bigger) will collect all changes and still keep that starting point.