Mongodb – Mongo 3.6 performance issue

mongodbmongodb-3.6performanceperformance-tuning

We have a 3-node Mongodb replica set that resides entirely in AWS. The primary is an i3.2xlarge (8 cores 61G RAM), while the secondary members are i3.xlarge (4 cores 30.5G RAM). These instances are each backed by two SSD EBS volumes, with one of the pair formatted XFS solely for the Mongodb installation. The Mongodb installations use the WiredTiger storage engine. The mongod.conf is here.

We have a roughly weekly ETL process that uses Pentaho to write to our replica set via a Tomcat8-based REST API. This process has completed smoothly for the last two years until our upgrade to Mongodb 3.6 at the end of February. During the first run of this process on Mongodb 3.6, the mongod process ran the instance out of memory. The OS invoked oom-killer to stop the process. After the subsequent election, the process then also killed the new primary.

We’d previously never configured swap space for the instances, but after this incident, swap space was configured on each, on a [RAM x 2] sized AWS NVMe instance store volume. Since implementing the swap space, any intensive process will continue to run the instance down to <1G of memory, then roll over into the swap space. We’re no longer experiencing a mongod service crash, but the system will regularly use up all available memory, and up to half of the swap space. As an example, the system was in such a state yesterday, and while restarting the service freed up the memory and swap space, the nightly dump has left the system using 26/60G of memory. It seems that the service isn’t releasing memory/killing threads once they’re no longer actively in use (see below image).24hr % avail. Memory. Service was restarted at 10:15am, and dump started at 2am..

Other system details:

  • Ubuntu 14.04.5
  • 4.4.0-112-generic #135~14.04.1-Ubuntu SMP
  • Mongodb 3.6.2
  • transparent_hugepage & defrag are set to “never”
  • readahead is set to 16K
  • numactl is installed, allowing NUMA to be disabled via the init script

Best Answer

As per your system details the readahead is set to 16K. But as per MongoDB BOL for the WiredTiger storage engine. Set the readahead setting to 0 regardless of storage media type (spinning, SSD, etc.).

Setting a higher readahead benefits sequential I/O operations. However, since MongoDB disk access patterns are generally random, setting a higher readahead provides limited benefit or performance degradation. As such, for most workloads, a readahead of 0 provides optimal MongoDB performance. In general, set the readahead setting to 0 unless testing shows a measurable, repeatable, and reliable benefit in a higher readahead value. MongoDB Professional Support can provide advice and guidance on non-zero readahead configurations.

As MongoDB documentation for the WiredTiger, consider the following recommendations on Linux environment:

  1. Turn off atime for the storage volume containing the database files.
  2. Set the file descriptor limit, -n, and the user process limit (ulimit), -u, above 20,000, according to the suggestions in the ulimit reference. A low ulimit will affect MongoDB when under heavy use and can produce errors and lead to failed connections to MongoDB processes and loss of service.
  3. Disable Transparent Huge Pages. MongoDB performs better with normal (4096 bytes) virtual memory pages. see Transparent Huge Pages Settings.
  4. Disable NUMA in your BIOS. If that is not possible, see MongoDB on NUMA Hardware.
Related Question