Mongodb – All queries waiting for lock in mongodb

mongodb

I have 1 primary, 1 secondary and 1 arbiter instance in my mongo cluster and no sharding. The priority of secondary is lower than priority of primary to keep the same machine as primary.

During a very high load today, I saw 1176 queries in db.currentOp(). All of these queries except 1 are waiting for some lock. And these queries are on various collections (not just on 1 collection). One such query waiting for the lock is

        {
        "desc" : "conn802243",
        "threadId" : "0x2055e1a0",
        "connectionId" : 802243,
        "opid" : 946990251,
        "active" : false,
        "op" : "query",
        "ns" : "db.collection",
        "query" : {
            "foo" : "bar",
        },
        "client" : "10.240.0.14:49355",
        "numYields" : 0,
        "locks" : {
            "Global" : "r",
            "MMAPV1Journal" : "r"
        },
        "waitingForLock" : true,
        "lockStats" : {
            "Global" : {
                "acquireCount" : {
                    "r" : NumberLong(2)
                }
            },
            "MMAPV1Journal" : {
                "acquireCount" : {
                    "r" : NumberLong(1)
                },
                "acquireWaitCount" : {
                    "r" : NumberLong(1)
                }
            }
        }
    }

The only query which is not waiting for the lock is a "dbstat" query which was fired by the arbiter("client" is the arbiter's ip in the below sample) and is running for more than 5 seconds.

{
        "desc" : "conn909",
        "threadId" : "0x7f53dc0",
        "connectionId" : 909,
        "opid" : 946985942,
        "active" : true,
        "secs_running" : 5,
        "microsecs_running" : NumberLong(5239773),
        "op" : "query",
        "ns" : "tb_dev",
        "query" : {
            "dbstats" : 1
        },
        "client" : "10.240.0.3:44403",
        "numYields" : 0,
        "locks" : {
            "Global" : "r",
            "MMAPV1Journal" : "r",
            "Database" : "R"
        },
        "waitingForLock" : false,
        "lockStats" : {
            "Global" : {
                "acquireCount" : {
                    "r" : NumberLong(2)
                }
            },
            "MMAPV1Journal" : {
                "acquireCount" : {
                    "r" : NumberLong(1)
                }
            },
            "Database" : {
                "acquireCount" : {
                    "R" : NumberLong(1)
                }
            }
        }
    }

Also, this looks to be a db level lock even though I am using 3.0(with mmapv1 as the storage engine).What is causing this db level lock to be held for so long?

Best Answer

In MMAPv1 write locks are collection level (version 3.0 and forward; Before 3.0, locks where database level). So, if you have 1174 clients reading "db.collection" and one client who is writing that same collection, readers will wait! Especially if you have write concern w:2 (or w:"majority") and you (only) secondary is lagging.

With wiredTiger locking mechanism is on document level.

You check here how concurrency works.

Related Question