Mongodb – Why does restarting MongoDB 3.0.x boost performance

mongodbmongodb-3.0

I have a MongoDB in my Development environment running on Windows, the server has 64GB of RAM, but MongoDB has only consumed 26.9GB (here we are building the database), so it should be able to hold all the indexes and data in ram:

DB Stats:
Data Size: 27.8071 GB
Index Size: 3.96 GB

After 3 or 4 hours running, insert performance drops to only 2-3k inserts per second, despite all indexes being in RAM (or they should be).

However, I've noticed if I restart the server, insert speeds rates increase to x3 or x4.

I'm wondering why this is, and could it be caused by running on Windows?

The collections being inserted (with any frequency) have (at this time) 20 mil, 44 mil, and 51 mil.

The first collection of 20mil has a very random index based on a 256 bit hash, the other two collections are indexed on ObjectId. This is the bottle neck I cannot shift without sharding, but it is concerning that the insert rate changes so much after a restart.

I do not want to periodically restart my Primary node, causing clients to have to fail over to a secondary.

Edit: I should also say, I am running with write concern unacknowledged.

Please find example image of logs, apologies I did not capture a text output of this log.
enter image description here

Some typical slow queries when they are popped up by the profiler, set at 100ms:

2015-08-18T11:56:34.768+0100 I COMMAND  [conn7] command Slice_BTC.$cmd command: insert { insert: "Transactions", writeConcern: { w: 0 }, ordered: false, documents: 765 } keyUpdates:0 writeConflicts:0 numYields:
 acquireCount: { w: 7 } }, Database: { acquireCount: { w: 7 } }, Collection: { acquireCount: { w: 7 } } } 790ms
2015-08-18T11:56:35.184+0100 I QUERY    [conn10] query Slice_BTC.Outputs query: { $query: { t: ObjectId('55d2cf76137e6e233c231ecf'), i: 96 } } planSummary: IXSCAN { t: 1, i: 1 } ntoreturn:1 ntoskip:0 nscanned:1
:0 writeConflicts:0 numYields:0 nreturned:1 reslen:224 locks:{ Global: { acquireCount: { r: 1 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } 378ms

Typical size of Transactions object: 178 bytes.
Typical size of Outputs object: 204 bytes.

Best Answer

I believe the slowdown is caused by running in Windows. I am going to attempt this on a Linux box once our new server arrives.

However, the 26GB limit is because of the WiredTiger default configuration which only caches 50% up to max ram.

You can change this using the http://docs.mongodb.org/v3.0/reference/configuration-options/#storage.wiredTiger.engineConfig.cacheSizeGB option.