MongoDB failure scenario

The lock spike coincides with a spike in page faults in MMS, which means that for that period of time (which looks pretty brief), there is data being paged in off disk (that is: not in memory). Since you mention that you are doing updates I would guess that something about the updates during that time frame is hitting a rarely used section of the data that is no longer in memory (the kernel uses LRU to decide what stays in memory). The other possibility is that you are evicting db2 pages right before this because of some other activity and then attempting to use that data again (i.e. classic memory contention).

To get a better picture of this you could do before and after comparisons using something like mongomem but that may not tell you what the usage pattern is that is causing this, unless the data impacted is indicative.

On the usage front, with lock percentage hitting higher values like that you should see the occasional slow operation logged in the log files during that time. If not then perhaps drop your slowms value down a little (to 50ms for example) from the 100ms default and then see if anything gets logged. You can do this without a restart and without using profiling itself by using db.setProfilingLevel() as follows:

db.setProfilingLevel(0, 50)

This has less impact than full profiling and you can leave it on for a longer period to catch the slow operations, though it will cause your log files to be larger, so be careful about disk space for logs if things are tight.

For help filtering and visualizing log files, I highly recommend using mtools, though the usual suspects like grep, awk, sed etc. all work too. As always, beware the false positive - high lock can cause other regularly run operations (like db.serverStatus()) to show up as slow. They are symptoms rather than causes most of the time and they are showing up because they are being run continuously and can be blocked by a busy database or high lock.

You have a major mistake in your code. MongoClient creates a connection pool. Even in large applications, it is hence usually a singleton. So you should have it as a global variable, initialize it in main and reuse it in each runnable. Which is perfectly fine, since MongoClient is thread safe.

Another thing to keep in mind is that although the single document sure is in the working set and hence should be in RAM, you application still needs to communicate with mongod. So your query will be translated to MongoDB's wire protocol, sent to the server where it will be executed, the matching documents identified (in this case only one, though this is not necessarily transparent before execution), finally sent back to the client and the answer is translated from MongoDB's wire protocol to Java terms. This is obviously going to be slower than simple Java native accesses in the same JVM without match conditions.

Finally, let's do some maths:

which even not taking the overhead of unnecessary MongoClients into account is pretty fast in my book.