Mongodb 2.6 Server crash and not restarting

crashmongodb

We are using mongodb 2.6 with one Primary and 2 secondaries. One of the secondary has gone bad and not restarting.

2014-04-18T21:13:34.765+0000 [rsSync] replSet still syncing, not yet to minValid optime 535185c4:fe
2014-04-18T21:13:34.766+0000 [rsBackgroundSync] replset setting syncSourceFeedback to server134:27017
2014-04-18T21:13:34.880+0000 [rsSync] replSet SECONDARY
2014-04-18T21:13:34.972+0000 [rsSync] SEVERE: Invalid access at address: 0x7f9694fdc13c
2014-04-18T21:13:34.982+0000 [rsSync] SEVERE: Got signal: 7 (Bus error).
Backtrace:0x11bd301 0x11bc6de 0x11bc7cf 0x7fb08d53fcb0 0xf1d43a 0x8aef63 0xf0e4a7 0xf14efe 0xf14f6f 0xf1e02f 0xf1d9c0 0x8afd8f 0x8b1c5a 0xe42251 0xe9b0eb 0xea52e4 0xea5c6a 0xea5da0 0xea6098 0x1201c99
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11bd301]
 /usr/bin/mongod() [0x11bc6de]
 /usr/bin/mongod() [0x11bc7cf]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fb08d53fcb0]
 /usr/bin/mongod(_ZN5mongo17RecordStoreV1Base12deleteRecordERKNS_7DiskLocE+0xea) [0xf1d43a]
 /usr/bin/mongod(_ZN5mongo10Collection14deleteDocumentERKNS_7DiskLocEbbPNS_7BSONObjE+0x633) [0x8aef63]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPNS_10CollectionERKNS_10StringDataEi+0x267) [0xf0e4a7]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPNS_10CollectionERKNS_10StringDataEi+0x2e) [0xf14efe]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPNS_10CollectionERKNS_10StringDataEi+0x5f) [0xf14f6f]
 /usr/bin/mongod(_ZN5mongo19CappedRecordStoreV111allocRecordEii+0x3f) [0xf1e02f]
 /usr/bin/mongod(_ZN5mongo17RecordStoreV1Base12insertRecordEPKcii+0x60) [0xf1d9c0]
 /usr/bin/mongod(_ZN5mongo10Collection15_insertDocumentERKNS_7BSONObjEbPKNS_16PregeneratedKeysE+0x7f) [0x8afd8f]
 /usr/bin/mongod(_ZN5mongo10Collection14insertDocumentERKNS_7BSONObjEbPKNS_16PregeneratedKeysE+0x22a) [0x8b1c5a]
 /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x341) [0xe42251]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0xe9b0eb]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x4f4) [0xea52e4]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x14a) [0xea5c6a]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x30) [0xea5da0]
 /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xa8) [0xea6098]
 /usr/bin/mongod() [0x1201c99]

Is there a way by which we can start the secondry to work or workaround this bug?

Best Answer

I just know a workaround: manually resync the member of the replica set.

Basically, it is done like this:

  1. Shutdown the mongod instance in question
  2. Delete the contents of the dbpath directory
  3. Restart the instance. It will sync then