Mongodb – Not able to access a database on shard mongodb cluster

mongodbmongodb-3.2sharding

We have a mongoDB cluster with 2 shards, 3 config servers and 5 mongos instances.
We create new database for each day and drop database(s) that are older than 10 days.

On any one mongos, when I list all databases, there is one database shown that is occupying 40 GB space. This database should have been removed by timely scripts, but that did not happen.

When I try to drop this database, error Occurs as – "database does not exist".
Find below console output of step carried out –

    mongos> show dbs
    dailyDB_A23    37.890GB
    mongos> use dailyDB_A23
    switched to db dailyDB_A23
    mongos> show collections
    mongos> db.dropDatabase()
    { "info" : "database does not exist", "ok" : 1 }
    mongos>

On using this database and listing all collections, there is nothing shown on console.
Plus when I tried taking a dump of this database, nothing is backed up and command exits withing 2 to 3 seconds.

On connecting to PRIMARY of each replica set in shard, this database is visible (ran 'show dbs' after connecting to mongo shell) and files are also present in to data directory.

Kindly help me out here. If this database does not exist (as per error on dropping) why it is showing it in list with size of ~ 40 GB?

Thanks and Regards,

Vibhav

Best Answer

Try db.repairDatabase() which fixes inconsistency across database. Check mongod and mongos logs for results.

Related Solutions

Mongodb – Removing Database from sharded cluster in Mongo DB

Unfortunately there is some stuff that will not be cleaned up through dropDatabase. On the other hand if you delete it by hand may not will be harmful. Some data like statistics will reside on your nodes anyway till the restart of mongods. (Try it out first! Not on the production), as the configuration database is relatively small and does not have huge requirements in terms of performance, that maybe even not a problem, just to be on the safe side and leave them as they are.

Also something to check out how to make the manual actions: https://stackoverflow.com/questions/9407838/mongodb-dropdatabase-not-working

and as it is mentioned on this question the dropDatabase in such case a bit bugy:https://jira.mongodb.org/browse/SERVER-4804

Mongodb – Update tag range on MongoDB range shard

New to MongoDB, but this is how I learned how to do the same thing following the MongoDB 202 class. There are other ways to balance traffic, such as moving, splitting and merging chunks. I haven't read anywhere that directly updating the tags collection in the config database was dirty or incorrect.

The code below is untested, and you will have to replace certain values like nameofShard and the minimum, maximum values. You will notice that I stop the balancer before modifying the tag ranges

// stop the balancer
sh.stopBalancer()

// remove the tags from the shards
sh.removeShardTag("nameofshardN", "rangeTime1")
sh.removeShardTag("nameofshardN+1", "rangeTime2")

// remove the existing tags
db.tags.remove({ "_id" : { "ns" : "testlab.range", "min" : { "_id" : 1 } }, "ns" : "testlab.range", "min" : { "_id" : 1 }, "max" : { "_id" : 100 }, "tag" : "rangeTime1" })
db.tags.remove({ "_id" : { "ns" : "testlab.range", "min" : { "_id" : 100 } }, "ns" : "testlab.range", "min" : { "_id" : 100 }, "max" : { "_id" : 200 }, "tag" : "rangeTime2" })

// Add new tags
sh.addTagRange("testlab.range", {_id : new_minimum_value}, {_id : new_maximum_value}, "rangeTime1");

sh.addTagRange("testlab.range", {_id : new_minimum_value}, {_id : new_maximum_value}, "rangeTime2");


// Add shardTag
sh.addShardTag("nameofshardN", "rangeTime1")
sh.addShardTag("nameofshardN+1", "RangeTime2")


// start balancer
sh.startBalancer()

sh.status() or mongostat --port 27017 --discover to verify balancing

test.lab is not a valid database name, so I changed my database name to testlab.

One additional remark about the _id field not being updated. When I query the tags collection like so

mongos> db.tags.find({}, {_id:1})
{ "_id" : { "ns" : "testlab.range", "min" : { "_id" : 1 } } }
{ "_id" : { "ns" : "testlab.range", "min" : { "_id" : 100 } } }

It shows that the _id field contains the min value or the tag range. If you are updating the tags collection and modifying the tags min field value, then wouldn't it also be reasonable to update the _id field's min value as well?

Related Question