Mongodb crashed with the error or permission denied

mongodb

Our mongodb server was running for the last two months without any error. Suddenly it crashed and below is the log.

2017-08-11T13:41:41.320+0100 E STORAGE  [thread1] WiredTiger (13) [1502455301:320924][1281:0x7f11bce8e700], log-server: /var/lib/mongo/journal: opendir: Permission denied
2017-08-11T13:41:41.320+0100 E STORAGE  [thread1] WiredTiger (13) [1502455301:320991][1281:0x7f11bce8e700], log-server: dirlist journal prefix WiredTigerPreplog: Permission denied
2017-08-11T13:41:41.321+0100 E STORAGE  [thread1] WiredTiger (13) [1502455301:321013][1281:0x7f11bce8e700], log-server: log pre-alloc server error: Permission denied
2017-08-11T13:41:41.321+0100 E STORAGE  [thread1] WiredTiger (13) [1502455301:321021][1281:0x7f11bce8e700], log-server: log server error: Permission denied
2017-08-11T13:41:42.000+0100 I -        [ftdc] Assertion: 13538:couldn't open [/proc/1281/stat] errno:13 Permission denied
2017-08-11T13:41:42.000+0100 W FTDC     [ftdc] Uncaught exception in 'Location13538 couldn't open [/proc/1281/stat] errno:13 Permission denied' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
2017-08-11T13:41:50.006+0100 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:45196 #205521 (23 connections now open)
2017-08-11T13:41:50.006+0100 I NETWORK  [conn205521] end connection 127.0.0.1:45196 (22 connections now open)
2017-08-11T13:42:36.004+0100 E STORAGE  [thread2] WiredTiger (13) [1502455356:4361][1281:0x7f11bb68b700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: fstat: Permission denied
2017-08-11T13:42:36.004+0100 E STORAGE  [thread2] WiredTiger (13) [1502455356:4466][1281:0x7f11bb68b700], checkpoint-server: checkpoint server error: Permission denied
2017-08-11T13:42:36.004+0100 E STORAGE  [thread2] WiredTiger (-31804) [1502455356:4480][1281:0x7f11bb68b700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-08-11T13:42:36.004+0100 I -        [thread2] Fatal Assertion 28558
2017-08-11T13:42:36.004+0100 I -        [thread2] 

***aborting after fassert() failure

Please tell me why it happened and what should I do to avoid this crash in future. I am using mongodb version 3.2. How come the file permission changed while the server is running.

Best Answer

It's not mongodb fault. Your system went to "read only" mode. You can see errors on that log file. Positive is that system managed to write that information to log file because it makes easier to debug this problem.

/var/lib/mongo/journal: opendir: Permission denied

couldn't open [/proc/1281/stat] errno:13 Permission denied

Check /var/log/syslog (or /messages) for more info, something fatal happened on that node and it's operating system!

Related Question