MongoDB: movechunk failed after adding a new share to cluster

mongodbsharding

I have a MongoDB cluster version 2.6.8, which already has two shares: share1 and share2.

When I add a new share3 to the cluster, data chunk can't migrate to share3.

This is from the mongos log:

2015-08-25T11:50:48.145+0800
[Balancer] balancer move failed:
{ cause: { ok: 0.0, errmsg: "migrate already in progress" }, ok: 0.0,
errmsg: "moveChunk failed to engage TO-shard in the data transfer: 
migrate already in progress" } from: share1 to: share3 
chunk:  min: { taskid: "51f7b17f56240isjduf2716b", uid: "111234322333" }
max: { taskid: "51f7b17f56240isjduf2716b", uid: "111234322333" }

I also tried to restart the balancer process, but it doesn't work.

Is the only thing I can do is to restart the mongos process?

    {
        "_id" : "mongodb9.hostname.xx3-2015-08-25T05:47:37-55dc0179c599c005ae1b4fb1",
        "server" : "mongodb9.hostname.xx3",
        "clientAddr" : ":27017",
        "time" : ISODate("2015-08-25T05:47:37.735Z"),
        "what" : "moveChunk.to",
        "ns" : "tbplus.col",
        "details" : {
                "min" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "s8j32djs80jsia4ae8d99es219aaab"
                },
                "max" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "es8djidual923ns5153f84a1228be6"
                },
                "step 1 of 5" : 3,
                "step 2 of 5" : 89063,
                "note" : "aborted",
                "errmsg" : "_migrateClone failed: { ok: 0.0, errmsg: \"not active\" }"
        }
}
{
        "_id" : "mongodb9.hostname.xx3-2015-08-25T03:18:01-55dbde6927d0376a83244f3f",
        "server" : "mongodb9.hostname.xx3",
        "clientAddr" : ":27017",
        "time" : ISODate("2015-08-25T03:18:01.651Z"),
        "what" : "moveChunk.to",
        "ns" : "tbplus.col",
        "details" : {
                "min" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "s8j32djs80jsia4ae8d99es219aaab"
                },
                "max" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "es8djidual923ns5153f84a1228be6"
                },
                "step 1 of 5" : 1,
                "step 2 of 5" : 0,
                "step 3 of 5" : 198662,
                "step 4 of 5" : 20,
                "note" : "aborted",
                "errmsg" : "{ ok: 0.0, errmsg: \"no active migration!\" }"
        }
}
{
        "_id" : "mongodb9.hostname.xx3-2015-08-25T03:13:16-55dbdd4c27d0376a83244f3e",
        "server" : "mongodb9.hostname.xx3",
        "clientAddr" : ":27017",
        "time" : ISODate("2015-08-25T03:13:16.153Z"),
        "what" : "moveChunk.to",
        "ns" : "tbplus.col",
        "details" : {
                "min" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "s8j32djs80jsia4ae8d99es219aaab"
                },
                "max" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "es8djidual923ns5153f84a1228be6"
                },
                "step 1 of 5" : 0,
                "step 2 of 5" : 151066,
                "note" : "aborted",
                "errmsg" : "_migrateClone failed: { ok: 0.0, errmsg: \"not active\" }"
        }
}
{
        "_id" : "mongodb9.hostname.xx3-2015-08-25T03:10:44-55dbdcb427d0376a83244f3d",
        "server" : "mongodb9.hostname.xx3",
        "clientAddr" : ":27017",
        "time" : ISODate("2015-08-25T03:10:44.331Z"),
        "what" : "moveChunk.to",
        "ns" : "tbplus.col",
        "details" : {
                "min" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "s8j32djs80jsia4ae8d99es219aaab"
                },
                "max" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "es8djidual923ns5153f84a1228be6"
                },
                "step 1 of 5" : 0,
                "step 2 of 5" : 0,
                "note" : "aborted",
                "errmsg" : "_migrateClone failed: { ok: 0.0, errmsg: \"not active\" }"
        }
}
{
        "_id" : "mongodb9.hostname.xx3-2015-08-25T03:08:25-55dbdc2927d0376a83244f3c",
        "server" : "mongodb9.hostname.xx3",
        "clientAddr" : ":27017",
        "time" : ISODate("2015-08-25T03:08:25.380Z"),
        "what" : "moveChunk.to",
        "ns" : "tbplus.col",
        "details" : {
                "min" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "s8j32djs80jsia4ae8d99es219aaab"
                },
                "max" : {
                        "colkey" : "51f7b174nm23i84ngals9d87",
                        "coluid" : "es8djidual923ns5153f84a1228be6"
                },
                "step 1 of 5" : 0,
                "step 2 of 5" : 199817,
                "note" : "aborted",
                "errmsg" : "_migrateClone failed: { ok: 0.0, errmsg: \"not active\" }"
        }
}

I have restart the mongos, but still have this problem. There is no packet loss problem on the network.

Best Answer

Wiki answer generated from an edit to the question, and a comment by Markus W Mahlberg.
Feel free to edit this to improve it.


Tom: I stopped the balancer process, and moved the chunk manually. There is no problem now.

Markus: May have been related to: movechunk failed after adding a new share to cluster