Mongodb – First query is really slow with WiredTiger, not with mmapv1

mongodbwiredtiger

We recently upgraded to MongoDB 4.0. Loving the performance improvements with WiredTiger, but all of a sudden our test suite is timing out on certain machines.

Before running our tests, we run the following command:

mongod --dbpath ./.test-data --bind_ip 127.0.0.1 --fork --logpath ../testlog.txt

Everything seems great. The database starts. Node connects to it. We create a bunch of collections and indexes.

Then once we run the first query, it takes 10+ seconds to run.

To be clear, at this point the database has never had any data inserted into it. It's brand new.

Using MMAPv1 (with the following command) resolves the problem, but I think it'd be great to run our tests with WiredTiger, if possible.

mongod --dbpath ./.test-data --storageEngine mmapv1 --bind_ip 127.0.0.1 --fork --logpath ../testlog.txt

Running mongod v4.0.3 and OS X 10.13.6. Connecting with node v6.9.1 and mongoose v5.4.19.

After each test run, we nuke the data directory:

mongo admin --eval 'db.shutdownServer()' > /dev/null
rm -rf ./.test-data/*

I also tried setting --wiredTigerJournalCompressor none --wiredTigerCacheSizeGB 1.0 --wiredTigerCollectionBlockCompressor none --wiredTigerIndexPrefixCompression false and it didn't make a difference.

Best Answer

First query is really slow with WiredTiger, not with mmapv1

As per MongoDB blog documentation How to Tell if Your MongoDB Server Is Correctly Sized For Your Working Set For MongoDB users, knowledge of working sets is critical. Understanding the interactions between your working set and physical memory can make a major difference in how your system performs.

In order to optimize MongoDB performance you should consider and know the following: How working sets fit into the scheme of MongoDB in the first place; Why it's important to fit your working set within your available RAM; Why you should avoid pulling from disk; and When and how it's possible to tell that there's a mismatch between your working set and your RAM.

What is a working set?

MongoDB's documentation defines a working set as follows:

A working set represents the total body of data that the application uses in the course of normal operations… For best performance, the majority of your active set should fit in RAM.

Why's this important? Fitting your working set to RAM is vital for performance: Going to disk is costly. As this breakdown from a UC Berkeley study on latency numbers shows, reading from memory is significantly faster than reading from disk. As this visualization of 2017 speeds shows, even if you're using SSD, accessing from memory is an order of magnitude more efficient than accessing from disk.

There are two primary storage engines available for MongoDB:

MMAP

  • As Mongo's documentation explains, "MMAPv1 is MongoDB’s original storage engine based on memory mapped files. It excels at workloads with high volume inserts, reads, and in-place updates." As of MongoDB version 3.2, however, MMAP is no longer MongoDB's default storage engine—WiredTiger now holds that spot.
  • With MMAP, MongoDB will use all available memory, as necessary. There are no configuration options to limit the data kept in-memory.

WiredTiger

  • WiredTiger has an internal data cache, but is configured to also leave memory for the Operating System’s file cache. It's generally recommended to leave your cache size at its default setting.
  • These defaults settings will use the larger of either of the following: 50% of RAM minus 1 GB.
  • 256 MB.
  • Outside WiredTiger's cache, the Operating System will cache frequently accessed files for the database in its own file cache. WiredTiger compresses data files by default, so leveraging the OS cache actually lets more data stay in the server's physical memory.

How can I tell if my working set fits my RAM?

For MMAP, a high number of page faults is a leading indicator that instead of finding data in RAM, it's going to disk to fetch it. That, of course, means the working set isn't fitting within your available RAM. Inversely, you can look directly at your RAM usage: As explained above, MMAP uses all available memory automatically. Therefore, if you're not using all of the RAM, then you can safely conclude that the working set fits into RAM, even with room to spare.

The total number of page faults. The extra_info.page_faults counter may increase dramatically during moments of poor performance and may correlate with limited memory environments and larger data sets. Limited and sporadic page faults do not necessarily indicate an issue.

Windows draws a distinction between hard page faults involving disk I/O, and soft page faults that only require moving pages in memory. MongoDB counts both hard and soft page faults in this statistic.

"freeMonitoring" : {
   "state" : <string>,
   "retryIntervalSecs" : <NumberLong>,
   "lastRunTime" : <string>,
   "registerErrors" : <NumberLong>,
   "metricsErrors" : <NumberLong>
},

For WiredTiger, you can also begin by looking directly at memory usage. If your server has free RAM, that inherently means the working set fits into your memory, as WiredTiger automatically uses its set cache plus your operating system's cache. If your server looks to be using most of its memory, take a look at the pages-read-into-cache and unmodified pages evicted metrics reported for WiredTiger by MongoDB’s serverStatus command. High activity for those two metrics indicates that the server is cycling data through the cache, which means your working set does not fit in memory.

The last metric to consider, of course, is volume of disk reads — if you see a lot of read activity on your drive, it's generally a good sign that your working set is overflowing your memory. Because WiredTiger interacts with your OS cache, this may be a more reliable metric to evaluate working set ratio — e.g. you may be able to see high cache evictions but low IO because of how your cache behaves.

Even though as documented here WiredTiger allows for tuning some of the storage engine parameters. The 3 most important to consider are the cache size, checkpoint interval and logging.

for further your ref here , here and here