Mongodb – mongod 2.6 corruption (Assertion failure isOk() src/mongo/db/storage/extent.h 80)

corruptionlinuxmongodb

I have been using mongo 2.6.0 for some time on 64-bit centos (with journaling enabled). One fine day when we were testing to insert a document to one of the collections, we started seeing data corruption issues.
When I ran db.collection.validate() from mongos, the response had pointed to one of the clusters that had an index corruption. I restarted all the nodes in the cluster but continue to see the issue.
Here is the logs on the start up of one of the nodes

2015-09-02T15:35:43.940-0700 ***** SERVER RESTARTED *****
2015-09-02T15:35:44.004-0700 [initandlisten] MongoDB starting : pid=2017        port=27017 dbpath=/opt/ebrary/mongodb/data 64-bit host=mongod01-f.ebrary.com
2015-09-02T15:35:44.005-0700 [initandlisten] db version v2.6.0
2015-09-02T15:35:44.005-0700 [initandlisten] git version:  1c1c76aeca21c5983dc178920f5052c298db616c
2015-09-02T15:35:44.005-0700 [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-09-02T15:35:44.005-0700 [initandlisten] allocator: tcmalloc
2015-09-02T15:35:44.005-0700 [initandlisten] options: { config: "/etc/mongod.conf", processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSet: "mongod-f-shard01" }, security: { keyFile: "****" }, storage: { dbPath: "****" }, systemLog: { destination: "file", logAppend: true, path: "****" } }
2015-09-02T15:35:44.024-0700 [initandlisten] journal dir=/****/journal
2015-09-02T15:35:44.024-0700 [initandlisten] recover : no journal files present, no recovery needed
2015-09-02T15:35:46.641-0700 [initandlisten] waiting for connections on port 27017
2015-09-02T15:35:46.728-0700 [rsStart] replSet I am mongod01-f.xxx.com:27017
2015-09-02T15:35:46.731-0700 [rsStart] replSet STARTUP2
2015-09-02T15:35:46.732-0700 [rsSync] replSet SECONDARY
2015-09-02T15:35:46.732-0700 [rsMgr] replSet info electSelf 0
2015-09-02T15:35:46.807-0700 [initandlisten] connection accepted from 10.10.25.47:51612 #1 (1 connection now open)
2015-09-02T15:35:47.733-0700 [rsMgr] replSet PRIMARY
2015-09-02T15:36:19.931-0700 [conn1]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "xxx", key: "xxx" }
2015-09-02T15:36:46.623-0700 [clientcursormon] mem (MB) res:66 virt:794147
2015-09-02T15:36:46.623-0700 [clientcursormon]  mapped (incl journal view):793464
2015-09-02T15:36:46.623-0700 [clientcursormon]  connections:1
2015-09-02T15:36:46.623-0700 [clientcursormon]  replication threads:32
2015-09-02T15:44:34.983-0700 [conn1]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "mmsmon", key: "xxx" }
2015-09-02T15:45:29.948-0700 [conn1]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "mmsmon", key: "xxx" }
2015-09-02T15:46:11.005-0700 [conn1]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "mmsmon", key: "xxx" }
**2015-09-02T15:46:11.103-0700 [conn1] core-mongo Assertion failure isOk() src/mongo/db/storage/extent.h 80**
2015-09-02T15:46:11.164-0700 [conn1] core-mongo 0x11bd301 0x115f819 0x114395e 0xee2ec0 0x8ae1e9 0xa28f3f 0xa1cfda 0xa1df4e 0xa1f806 0xd4c6a7 0xb96382 0xb98962 0x76b76f 0x117367b 0x7f20c68e8a51 0x7f20c5c8d9ad          /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11bd301]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x115f819]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x114395e]
 /usr/bin/mongod(_ZNK5mongo13ExtentManager9getExtentERKNS_7DiskLocEb+0x60) [0xee2ec0]
/usr/bin/mongod(_ZNK5mongo10Collection11storageSizeEPiPNS_16BSONArrayBuilderE+0x779) [0x8ae1e9]
 /usr/bin/mongod(_ZN5mongo7DBStats3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x2df) [0xa28f3f]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa1cfda]

/usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xc5e) [0xa1df4e]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa1f806]
/usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x2307) [0xd4c6a7]
/usr/bin/mongod() [0xb96382]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x442) [0xb98962]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x76b76f]
/usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4fb) [0x117367b]
/lib64/libpthread.so.0(+0x7a51) [0x7f20c68e8a51]
/lib64/libc.so.6(clone+0x6d) [0x7f20c5c8d9ad]

Best Answer

This error means the data files MongoDB are corrupted.

You should run mongod with --repair option to fix them.

mongod --repair

After finishing repairing, stop it and try start mongod again.