MongoDB Clustering – How to Ensure MongoDB is Balancing

clusteringmongodbUbuntu

I have the following structure:

Two Ubuntu 14.04.2, both with MongoDB v3.0.4 instaled. Both in the same LAN. I want to make them work as a cluster.

In the first one, that I call "MC1", I run the three config server, mongos and mongod. In the second one, that I call "MC2", I run mongos and mongod.

MC1 is the shard1.
MC2 is the shard0.

I have a DB with 2.031GB size.

I have the following problem:

When I connect via Robomongo to MC1 mongos I can see all the data of every collection. If I connect to MC2 mongos I see the DB, the collections but the collections are empty.

Testing the balancer status I get the following results:

sh.getBalancerState()
true

db.getSiblingDB("config").collections.findOne({_id : "cluster.username"}).noBalance;
false

db.databases.find()
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "test", "partitioned" : false, "primary" : "shard0001" }
{ "_id" : "db", "partitioned" : false, "primary" : "shard0001" }
{ "_id" : "cluster", "partitioned" : true, "primary" : "shard0000" }

With this data I know that the balancer is active and the collections and DB have the balancer activated.

I do a sh.status() too. Here you can see that all collections have data only in the shard0, any chunk haven't been migrate to the shard1. And I only can view the data connecting to MC1 mongos.

sh.status()
--- Sharding Status --- 
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("55a778641066cdf8d093fe97")
}
shards:
{  "_id" : "shard0000",  "host" : "172.31.37.215:27018" } #This one is MC2#
{  "_id" : "shard0001",  "host" : "172.31.35.191:27018" }#This one is MC1#
balancer:
Currently enabled:  yes
Currently running:  no
Failed balancer rounds in last 5 attempts:  0
Migration Results for the last 24 hours: 
    No recent migrations
databases:
{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
{  "_id" : "test",  "partitioned" : false,  "primary" : "shard0001" }
{  "_id" : "db",  "partitioned" : false,  "primary" : "shard0001" }
{  "_id" : "cluster",  "partitioned" : true,  "primary" : "shard0000"     }
    cluster.fs.chunks
        shard key: { "files_id" : 1, "n" : 1 }
        chunks:
            shard0000   1
        { "files_id" : { "$minKey" : 1 }, "n" : { "$minKey" : 1 } } -->> { "files_id" : { "$maxKey" : 1 }, "n" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0) 
    cluster.fs.files
        shard key: { "filename" : 1, "_id" : 1 }
        chunks:
            shard0000   1
        { "filename" : { "$minKey" : 1 }, "_id" : { "$minKey" : 1 } } -->> { "filename" : { "$maxKey" : 1 }, "_id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0) 
    cluster.licenses
        shard key: { "license" : 1 }
        chunks:
            shard0000   1
        { "license" : { "$minKey" : 1 } } -->> { "license" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0) 
    cluster.movusers
        shard key: { "username" : 1 }
        chunks:
            shard0000   1
        { "username" : { "$minKey" : 1 } } -->> { "username" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0) 
    cluster.roles
        shard key: { "roleId" : 1 }
        chunks:
            shard0000   1
        { "roleId" : { "$minKey" : 1 } } -->> { "roleId" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0) 
    cluster.sessions
        shard key: { "sessionId" : 1, "_id" : 1 }
        chunks:
            shard0000   1
        { "sessionId" : { "$minKey" : 1 }, "_id" : { "$minKey" : 1 } } -->> { "sessionId" : { "$maxKey" : 1 }, "_id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0) 
    cluster.webusers
        shard key: { "username" : 1, "_id" : 1 }
        chunks:
            shard0000   1
        { "username" : { "$minKey" : 1 }, "_id" : { "$minKey" : 1 } } -->> { "username" : { "$maxKey" : 1 }, "_id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)

Also I check the shardDistribution:

db.fs.chunks.getShardDistribution()

Shard shard0000 at 172.31.37.215:27018
data : 0B docs : 0 chunks : 1
estimated data per chunk : 0B
estimated docs per chunk : 0

Totals
data : 0B docs : 0 chunks : 1
Shard shard0000 contains NaN% data, NaN% docs in cluster, avg obj size on shard : NaNGiB

How can I fix this?

Best Answer

Reproducing the situation :

mc1 collection already created on shard0000 before sharding
mc2 collection already created on shard0001 before sharding

Create a test-shard:

$ mongo --nodb
mongo> config={d0:{smallfiles:"",noprealloc:"",nopreallocj : ""}, d1 : {smallfiles : "", noprealloc : "", nopreallocj : "" }};
mongo> cluster = new SharingTest({shards:config});

In another terminal (shard0000 on port 30000, shard0001 on port 30001, mongos on port 30999) :

$ ps aux | grep mongo
root      9668  0.0  0.7 655804 23484 pts/1    Sl+  15:57   0:00 mongo --nodb
root      9693  0.4  2.4 344400 74108 pts/1    Sl+  15:59   0:00 mongod --port 30000 --dbpath /data/db/test0 --smallfiles --noprealloc --nopreallocj --setParameter enableTestCommands=1
root      9712  0.3  2.1 271672 62828 pts/1    Sl+  15:59   0:00 mongod --port 30001 --dbpath /data/db/test1 --smallfiles --noprealloc --nopreallocj --setParameter enableTestCommands=1
root      9732  0.2  0.5 117672 16200 pts/1    Sl+  15:59   0:01 mongos --port 30999 --configdb localhost:30000 --chunkSize 50 --setParameter enableTestCommands=1

Create mc1 on shard0000 and mc2 on shard0001 with some data:

$ mongo --port 30000
> use cluster
> for(var i=0; i<=1000; i++) {db.mc1.insert({'a':i})}
> db.mc1.createIndex({a:1})
$ mongo --port 30001
> use cluster
> for(var i=0; i<=1000; i++) {db.mc2.insert({'a':i})}
> db.mc2.createIndex({a:1})

Connect to mongos, and shard cluster db, mc1 collection and mc2 collection:

$ mongo --port 30999
mongos> show dbs
admin    (empty)
cluster  0.063GB
config   0.016GB
mongos> use cluster
switched to db cluster
mongos> show collections
mc2     #only mc2 is shown because see in sh.status => primary : shard0001
system.indexes
mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("55dacecf2c2855020c52454e")
}
  shards:
    {  "_id" : "shard0000",  "host" : "localhost:30000" }
    {  "_id" : "shard0001",  "host" : "localhost:30001" }
  balancer:
    Currently enabled:  no
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        No recent migrations
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "cluster",  "partitioned" : false,  "primary" : "shard0001" }
mongos> sh.enableSharding("cluster")
{ "ok" : 1 }
mongos> sh.shardCollection("cluster.mc1", {'a':1})
{ "collectionsharded" : "cluster.mc1", "ok" : 1 }
mongos> sh.shardCollection("cluster.mc2", {'a':1})
{ "collectionsharded" : "cluster.mc2", "ok" : 1 }

Check amount of documents in the shard-distribution:

mongos> db.mc1.getShardDistribution()

Shard shard0001 at localhost:30001
 data : 0B docs : 0 chunks : 1
 estimated data per chunk : 0B
 estimated docs per chunk : 0

Totals
 data : 0B docs : 0 chunks : 1
 Shard shard0001 contains NaN% data, NaN% docs in cluster, avg obj size on shard : NaNGiB


mongos> db.mc2.getShardDistribution()

Shard shard0001 at localhost:30001
 data : 46KiB docs : 1001 chunks : 1
 estimated data per chunk : 46KiB
 estimated docs per chunk : 1001

Totals
 data : 46KiB docs : 1001 chunks : 1
 Shard shard0001 contains 100% data, 100% docs in cluster, avg obj size on shard : 48B

Problem : No documents in mc1. See shard.status() => primary is pointing to Shard001. So only the collections in cluster-database on Shard0001 got sharded.

Solution 1:

Dump mc1 collection from shard0000
Drop mc1 collection from shard0000
Restore mc1 collection into mongos

(!!!! make sure nobody can add more records !!!!)

$ mkdir mc1
$ mongodump --port 30000 --db=cluster --collection=mc1 --out mc1/
$ mongo --port 30000 --eval "db.mc1.drop()" cluster
$ mongorestore --port 30999 --db=cluster --collection=mc1 mc1/cluster/mc1.bson
$ mongo --port 30999
MongoDB shell version: 3.0.5
connecting to: 127.0.0.1:30999/test
Server has startup warnings: 
2015-08-24T15:59:11.178+0800 I CONTROL  ** WARNING: You are running this process as the root user, which is not recommended.
2015-08-24T15:59:11.178+0800 I CONTROL  
mongos> use cluster
switched to db cluster
mongos> show collections
mc1
mc2
system.indexes
mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("55dacecf2c2855020c52454e")
}
  shards:
    {  "_id" : "shard0000",  "host" : "localhost:30000" }
    {  "_id" : "shard0001",  "host" : "localhost:30001" }
  balancer:
    Currently enabled:  no
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        No recent migrations
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "cluster",  "partitioned" : true,  "primary" : "shard0001" }
        cluster.mc1
            shard key: { "a" : 1 }
            chunks:
                shard0001   4
            { "a" : { "$minKey" : 1 } } -->> { "a" : 1 } on : shard0001 Timestamp(1, 1) 
            { "a" : 1 } -->> { "a" : 651 } on : shard0001 Timestamp(1, 2) 
            { "a" : 651 } -->> { "a" : 977 } on : shard0001 Timestamp(1, 3) 
            { "a" : 977 } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 4) 
        cluster.mc2
            shard key: { "a" : 1 }
            chunks:
                shard0001   1
            { "a" : { "$minKey" : 1 } } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 0)

Solution 2 (without having to dump):

(Dropping a collection or a database will remove it from the cluster configuration automatically)

Drop 'empty'-mc1 collection from cluster (test first if empty):

mongos> use cluster
switched to db cluster
mongos> db.mc1.find().count()
0
mongos> db.mc1.drop();
true

Move primary to shard0000:

mongos> use admin
switched to db admin
mongos> db.runCommand({movePrimary:"cluster", to:"shard0000"});
{ "primary " : "shard0000:localhost:30000", "ok" : 1 }

Checking the status:

mongos> use cluster
switched to db cluster
mongos> sh.status();
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("55dacecf2c2855020c52454e")
}
  shards:
    {  "_id" : "shard0000",  "host" : "localhost:30000" }
    {  "_id" : "shard0001",  "host" : "localhost:30001" }
  balancer:
    Currently enabled:  no
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours: 
        No recent migrations
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "cluster",  "partitioned" : true,  "primary" : "shard0000" }
        cluster.mc2
            shard key: { "a" : 1 }
            chunks:
                shard0001   1
            { "a" : { "$minKey" : 1 } } -->> { "a" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 0) 

mongos> show collections
mc1
system.indexes
mongos> db.mc1.find().count()
1001
mongos> db.mc2.find().count()
1001

Shard mc1 now into the collection:

mongos> sh.shardCollection("cluster.mc1", {'a':1});
{ "collectionsharded" : "cluster.mc1", "ok" : 1 }
mongos> db.mc1.getShardDistribution()

Shard shard0000 at localhost:30000
 data : 46KiB docs : 1001 chunks : 1
 estimated data per chunk : 46KiB
 estimated docs per chunk : 1001

Totals
 data : 46KiB docs : 1001 chunks : 1
 Shard shard0000 contains 100% data, 100% docs in cluster, avg obj size on shard : 48B


mongos> db.mc2.getShardDistribution()

Shard shard0001 at localhost:30001
 data : 46KiB docs : 1001 chunks : 1
 estimated data per chunk : 46KiB
 estimated docs per chunk : 1001

Totals
 data : 46KiB docs : 1001 chunks : 1
 Shard shard0001 contains 100% data, 100% docs in cluster, avg obj size on shard : 48B

Joy joy joy, that added it!!!

So, in your case (connect to mongos):

> use cluster
> db.fs.chunks.find().count()    #check if 0
> db.fs.chunks.drop()
> use admin
> db.runCommand({movePrimary:"cluster", to:"shard0001"});
> use cluster
> sh.shardCollection("cluster.fs.chunks", {'files_id':1,'n':1});

Best Answer

Related Solutions

MongoDB – Write Behavior When Only One Secondary Node is Online in Shard Cluster

Mongodb – Mongo sharding issue with chunk split and Data transfer

Related Question