Currently running two VMs with fresh installs of Ubuntu 14.04 x64 and Mongo 3 Stable. Each VM has the same specs with 8 cores and 56GBs of RAM. Our processing VM is doing great under heavy loads, however our production VM seems to have Mongo crashing once every few hours and can't pinpoint why or how.
My config for the production Mongo instance looks like this
Here is the last three logs outputted. The first two links are cut near the bottom as they were too big, and the last is a full log top to bottom (as it was a short lived run).
https://gist.github.com/dustinblackman/e32778344ec5f21af0d8
https://gist.github.com/dustinblackman/acb0a3aef77114275de4
https://gist.github.com/dustinblackman/908d63e7bd386aae6bc7
The applications that are communicating with this instance are using the latest PHP, Python, and NodeJS drivers.
Does anything in the logs stand out to anyone? All that I see is that the only proper operation that happens before a crash is auth (and I've tried disabling auth, but no deal). Any help is appreciated. Thanks!
Best Answer
Open up
/var/log/messagesand search formongo.If you see "
oom-killer" or "Out of memory" then this points to your problem.We had that problem when running a
mongosand multiplemongods on the same machine, for testing purposes.Since there is no way to limit mongo's memory usage, we just added a 4GB swapfile, so there was some leeway when memory was at its limit. (Using swap is the recommendation here.)
More on the OOM killer here.