Mongodb – “[ftdc] serverStatus was very slow” – MongoDB 4.0.4

mongodbmongodb-4.0

I'm seeing very frequent messages like this in mongod.log and doesn't give an exact reason for the server status was very slow.
looks like ftdc logs more information but interpreting ftdc log is very tedious method and is not human readable format.

Due to this issue mongo daemon stops working abruptly and becomes stale which results in replication issues and loosing quorum and hence DB inaccessible.

Error log:

2021-04-21T04:56:39.729+0000 I COMMAND [ftdc] serverStatus was very slow: { after basic: 2165, after asserts: 3087, after backgroundFlushing: 8785, after connections: 11612, after dur: 12922, after extra_info: 19620, after globalLock: 27972, after locks: 41890, after logicalSessionRecordCache: 51379, after network: 55947, after opLatencies: 62013, after opcounters: 67473, after opcountersRepl: 70586, after repl: 74168, after security: 78481, after storageEngine: 78629, after tcmalloc: 78629, after transactions: 78629, after transportSecurity: 78629, after wiredTiger: 78813, at end: 79004 }

Question :

how to troubleshoot further ?
Enabling verbose 5 logs does gives more information ?
Is it truly a memory issue ?
what's the next step to avoid this issue ?

Best Answer

That means, your server is overload. It can't even reply to the serverStatus command without delay.

Some people solved that by adding more hardware to their server. But, I think first of all you should check your queries and indexes, maybe you can optimize them.

Check out your queries:

db.currentOp()

Related Solutions

Mongodb – see when a database was dropped

If the drop command ran slowly, then it will be recorded in the logs (by default >100ms), otherwise the only record of it will be in the oplog (assuming you are running a replica set, even a single node replica set) and that is assuming that it did not occur so far in the past that it has "fallen out" of the oplog (which is a capped collection).

NOTE: Before running queries against your oplog, be aware that any such queries will be table scans, and that running such a query will potentially be slow, especially when run on an active replica set with a large oplog. If you have a secondary available, you may wish to use that for this type of query rather than add load to your primary.

With that said, let's show an example. We will call the database "foo", drop it and show how you would search the oplog for evidence of the drop. From the mongo shell your query would look like this:

use local;
db.oplog.rs.find({ns : "foo.$cmd",  "o" : { "dropDatabase" : 1 }})

And the result, if found, would look like this:

{
        "ts" : Timestamp(1412246712, 1),
        "h" : NumberLong("-6606042550253448275"),
        "v" : 2,
        "op" : "c",
        "ns" : "foo.$cmd",
        "o" : {
                "dropDatabase" : 1
        }
}

After that, the only other place to gather such evidence would be the filesystem, since the files are unlinked/deleted when the DB is dropped. There are plenty of answers on how to do that (and the potential problems) - I've used this one successfully in the past.

MongoDB – Fixing SymInitialize Error with Large Page File on Windows

Under Windows, in a worst case scenario, your pagefile size might have to be set to the size of your data files + physical memory size. So if your data files take up 50GB on disk, the rough guidance, in your case, is to set pagefile size to 53.5GB. This will improve with MongoDB 2.8 release since the new storage engine does not rely on virtual memory services provided by the OS. On a related subject, your memory size of 3.5GB sounds very low. Take a look at the Hard Page Faults per second under the Resource Monitor -- if the number is in hundreds, you need to dramatically increase your memory size

Related Question