Mongodb background indexing – no progress

mongodbmongodb-3.0

I have added a couple of background indexes on few collections on mongodb replicaset (3.0.2, MMAPv1 storage), and since then I'm seeing following entry in db.currentOp(true) response in all nodes (this one from Primary):

{
    "desc" : "rsSync",
    "threadId" : "0x2923ee0",
    "opid" : 5653,
    "active" : false,
    "op" : "none",
    "ns" : "local.replset.minvalid",
    "query" : {

    },
    "msg" : "Index Bulk Build: (3/3) btree-middle",
    "numYields" : 0,
    "locks" : {

    },
    "waitingForLock" : false,
    "lockStats" : {
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(2),
                "R" : NumberLong(1),
                "W" : NumberLong(1)
            }
        },
        "MMAPV1Journal" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(1)
            }
        },
        "Database" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "R" : NumberLong(1)
            }
        },
        "Collection" : {
            "acquireCount" : {
                "R" : NumberLong(1)
            }
        }
    }
},

It has been running for nearly 5 days now. From "Index Bulk Build: (3/3) btree-middle" I would guess it's still rebuilding indexes, but it seems to be taking unusually long time. Whole database is ~140Gb on disk, and I wasn't adding indexes on biggest collections either.

Database performance is 2-3 times slower than it was before adding background indexes.

One of collections that I added index on has many slow (30ms) "count" queries (had them before new indexes).

Servers are idling, resource usage (CPU load, Disk I/O, Network traffic, RAM usage) looks unchanged in graphs. 3 bare metal machines in use, one Primary and 2 Secondary nodes. Rotational disks are used, but 128G RAM, 122G cached.

Database is doing ~25K RPM, collections that indexes were added upon are in use. Collection sizes that I added background indexes on vary from 100-1K to 100K-1M records.

How to tell what exactly mongodb is doing right now, and if it's possible / safe to kill this indexing operation if it's stuck?

Or perhaps indexing is over, but that message does not have to disappear, and my indexes somehow degraded the performance?

How to troubleshoot this situation?

Best Answer

Solved this one. While indexing was running, new code was released in production, which contained a bad query. I was checking Primary performance only, and killer query was happening on Secondary node. After adding one more necessary index, everything went back to normal, performance is stellar again.

However message "msg" : "Index Bulk Build: (3/3) btree-middle" is still there.

Related Question