Mongodb crash after WiredTiger Cannot allocate memory error

This will be hard to diagnose without the full logs, but the easiest way to get that secondary back to normal is to resync it from scratch from the primary. Basically stop that mongod, delete all data in the dbpath (including subdirectories), then restart. It will kick off the sync process as if it were a new member.

For what it's worth, running two full instances (primary/secondary) on what sound like m1.large instances is probably going to see a lot of resource contention, especially on the disk side of things.

In terms of the crash, based on the limited evidence, it could be the OOM Killer at work:

http://www.mongodb.org/display/DOCS/The+Linux+Out+of+Memory+OOM+Killer

To avoid it, make sure you have some swap space configured.

Okay, so after following the clues given by loicmathieu and jstell, and digging it up a little, these are the things I found out about MongoDB using WiredTiger storage engine. I'm putting it here if anyone encountered the same questions.

The memory usage threads that I mentioned, all belonged to 2012-2014, all pre-date WiredTiger and are describing behavior of the original MMAPV1 storage engine which doesn't have a separate cache or support for compression.

The WiredTiger cache settings only controls the size of memory directly used by the WiredTiger storage engine (not the total memory used by mongod). Many other things are potentially taking memory in a MongoDB/WiredTiger configuration, such as the following:

• WiredTiger compresses disk storage, but the data in memory are uncompressed.

• WiredTiger by default does not fsync the data on each commit, so the log files are also in RAM which takes its toll on memory. It's also mentioned that in order to use I/O efficiently, WiredTiger chunks I/O requests (cache misses) together, that also seems to take some RAM (In fact dirty pages (pages that has changed/updated) have a list of updates on them stored in a Concurrent SkipList).

• WiredTiger keeps multiple versions of records in its cache (Multi Version Concurrency Control, read operations access the last committed version before their operation).

• WiredTiger Keeps checksums of the data in cache.

• MongoDB itself consumes memory to handle open connections, aggregations, serverside code and etc.

Considering these facts, relying on show dbs; was not technically correct, since it only shows the compressed size of the datasets.

The following commands can be used in order to get the full dataset size.

db.getSiblingDB('data_server').stats()
# OR
db.stats()

This results is the following:

{
    "db" : "data_server",
    "collections" : 11,
    "objects" : 266565289,
    "avgObjSize" : 224.8413545621088,
    "dataSize" : 59934900658, # 60GBs
    "storageSize" : 22959984640,
    "numExtents" : 0,
    "indexes" : 41,
    "indexSize" : 7757348864, # 7.7GBs
    "ok" : 1
}

So it seems that the actual dataset size + its indexes are taking about 68GBs of that memory.

Considering all these, I guess the memory usage is now pretty expected, good part being it's completely okay to limit the WiredTiger cache size, since it handles I/O operations pretty efficiently (as described above).

There also remains the problem of OOM, to overcome this issue, since we didn't have enough resources to take out mongodb, we lowered the oom_score_adj to prevent OOM from killing important processes for the time being (Meaning we told OOM not to kill our desired processes).