MongoDB stuck in startup2 state

mongodbmongodb-4.0recovery

I have a Mongo replica set (MongoDB server version: 4.0.3) with primiry, secondary and arbiter. Size of database is about 7 Tb. I add a member to replica set. It stucks in sturtup2 mode for a month! In first day it synhronize 3 Tb and then go slower and slower synchronyzing about 200 Gb each day. Previous member I add to this replica set was added for about a two or three days. I remove all load from replica set, there is nothing else take place – only synchronisation. Maybe there is a way to tell it to synchronyse faster?

Best Answer

Maybe there is a way to tell it to synchronize faster?

Since the amount of data copied to the new secondary is very large (7TB) so the issue happens. MongoDB suggests the following procedure for a large amount of data set.

Sync by Copying Data Files from Another Member

This approach “seeds” a new or stale member using the data files from an existing member of the replica set. The data files must be sufficiently recent to allow the new member to catch up with the oplog. Otherwise the member would need to perform an initial sync.

Copy the Data Files

You can capture the data files as either a snapshot or a direct copy. However, in most cases you cannot copy data files from a running mongod instance to another because the data files will change during the file copy operation.

Sync the Member

After you have copied the data files from the “seed” source, start the mongod instance and allow it to apply all operations from the oplog until it reflects the current state of the replica set.

STARTUP2

Each member of a replica set enters the STARTUP2 state as soon as mongod finishes loading that member’s configuration, at which time it becomes an active member of the replica set and is eligible to vote. The member then decides whether or not to undertake an initial sync. If a member begins an initial sync, the member remains in STARTUP2 until all data is copied and all indexes are built. Afterwards, the member transitions to RECOVERING.