MongoDB – Replication Headroom Below One Day

mongodb

getting alert that "Replication Headroom has gone below 1 days(avg/sec)".
when checked OPS manager for replication lag, max lag I can see that is of 16 seconds. Oplog size has been set to 50gigs. also when checked rs.status(), there is no difference in time for "optimeDate" and "lastHeartbeat".
I'm not understanding how to resolve this alert. Also is there any way I can sync secondary and reduce replication lag to zero. Please suggest.
Concept explaining shrinking replication headroom/replication lags with its workaround welcomed.
Thanks!

Best Answer

"Replication headroom" means "how long time of changes your opLog can store". To get larger "replication headroom" you need to expand your opLog size at least one node, of course it is preferred to have same size opLog at every node.

Reason why this headroom has gone under one day, is that because you have more changes (opLog documents) nowadays, than before.

Related Solutions

High Background Flush Average in MongoDB – Troubleshooting Guide

Your mongo stat shows higher number of updates vs inserts. One thing that could cause high write lock issues is if your updates typically are increasing the document size and causing the document to move in the data file. We ran into this ourselves, but we were working with mongo support at the time to figure out so I don't remember what metric or stat would tell you this is the case. This would likely only be an issue if your document sizes were very large. We ended up splitting out a sub array that was always being added to into its own collection so that we were just adding new documents instead of modifying an existing one.

The usePowerOf2Sizes flag on the collection can also help alleviate this by giving the documents room for growth. This is apparently the default now on 2.6, but you would need to turn it on if you're not on 2.6 yet. Setting that is described here: http://docs.mongodb.org/manual/reference/command/collMod/

Mongodb master slave replication

One reason could be that your oplog is too small.

On the master:

db.getReplicationInfo()

If timeDiffHours is shorter than the time that your initial sync takes, then the initial sync will fail and start again...

Resizing the oplog is quite easy, if this is indeed your problem. The documentation is here: http://docs.mongodb.org/manual/tutorial/change-oplog-size/

Best Answer

Related Solutions

High Background Flush Average in MongoDB – Troubleshooting Guide

Mongodb master slave replication

Related Question