MongoDB Replica Set member cannot start due “84 key/value already in index”

mongodbmongodb-3.0replication

I have a Replica Set with 3 members (1 primary and 2 secondaries), and one of the secondaries was working just fine and thrown the following error

> 2018-03-12T11:03:54.868-0400 E REPL [repl writer worker 13] writer
> worker caught exception: :: caused by :: 84 key/value already in index
> on: { ts: Timestamp 1520867034000|8, h: 287037468373256260, v: 2, op:
> "u", ns: "Sitecore_analytics_PROD.Interactions", o2:
> 
> { _id: BinData(3, EFC3DD1A15B75442986344FDE9CC71EB) } , o: { $set:
> 
> { ContactVisitIndex: 2 } } } 2018-03-12T11:03:54.868-0400 I - [repl
> writer worker 13] Fatal Assertion 16360 2018-03-12T11:03:54.868-0400 I
> - [repl writer worker 13] 
> ***aborting after fassert() failure

After that the MongoDB cannot start anymore, and keep throwing the same error!

How can I determine the root cause of it? Is there a way to prevent? How to solve it?

Best Answer

Do initial sync for that problematic secondary node. Just delete files from dbpath and start mongod. Node will do initial sync automatically. Other and faster way (if you can) is copy snapshot of dbpath of existing node (can be primary or secondary) and start mongod.

The problem here is that there is already entry at unique index and system tries to apply opLog line and crashes to that error.