Mongodb – How much free space do I need for mongoDB? and how to reclaim lost space

mongo-repairmongodb

I ran out of disk space on my mongo database. I saw it and dropped some collections (120 out of 1700 i had). And ran db.compact(...) thinking it would release free space. well, that was a stupid idea.

Anyway, the database is around 800GB, and I dont have any space (~5GB) in the current disk. I symlinked a folder from another disk that has around 35GB and tried db.repairDatabase() (and mongod --repair --repairpath ...), but that failed too saying not enough space. What else can I do? short of getting another 1TB HDD?

Why is even 35GB not enough? Since I dropped the collections, I know I should get another ~50GB free after the repair. Is that going to be enough?

The partition I use for data would have around ~800GB for mongo. How much free space should I ideally have?

Mongo says I need working set+2GB free space. But no queries are running and mongod stopped. So I'm not sure why 35GB is not enough.

Best Answer

There are two operations here
1. Compact - This is an in place compact of data inside the data files. This will however not release any of the data files. Even thought it does not return the free space to the file system mongo should be able to use the space for growth.
2. Repair - This will release any unused space back to the filesystem. This is what you need. However repair requires the same amount of space as your database. So if you want to repair you will 800GB + few extra GB for repair to complete.

Related Solutions

MongoDB Not Using All Available RAM

The resident memory size represents the number of pages in memory actually touched by the mongod process. If that is significantly lower than the available memory and data exceeds the available memory (yours does), then it could be a case of simply not having actively touched enough pages yet.

To determine if this is the case, you should run free -m, the output should look something like this:

free -m
             total       used       free     shared    buffers     cached
Mem:          3709       3484        224          0         84       2412
-/+ buffers/cache:        987       2721
Swap:         3836        156       3680

In my example, cached is not close to the total, which means that not only has mongod not touched enough pages, the filesystem cache has not yet even been filled by pages being read from the disk in general.

A quick remedy for this would be the touch command (added in 2.2) - it should be used with caution on large data sets as it will attempt to load everything into RAM even if the data is far too large to fit (causing a lot of disk IO and page faults). It will certainly fill up the memory effectively though :)

If your cached value is close to the total available, then your issue is that a large number of pages being read into memory from disk are not relevant to (and hence not touched by) the mongod process. The usual candidate for this kind of discrepency is readahead. I've already covered that particular topic elsewhere in detail, so I'll just link those two answers for future reading if necessary.

Mongodb – Mongo repairDatabase procedure

The solution to this issue was as follows: Starting with the secondary:

Inside of Mongo I ran the following:

db.shutdownServer()
exit

On the command line:

rm -rf /var/lib/mongodb/*
sudo service mongodb start

It then takes a couple of hours to resync. Once resynced I did the following on the primary:

rs.status() //To make sure that the everything was synced
rs.stepDown(9200)
db.shutdownServer()
exit

On the command line I ran:

rm -rf /var/lib/mongodb/*
sudo service mongodb start

Then in Mongo I ran:

rs.status() //To make sure that everything was synced
rs.freeze(0) //To remove the re.stepDown(9200) lock

On the secondary I ran:

rs.stepDown()

Note: If possible, back up all of your data first.

Best Answer

Related Solutions

MongoDB Not Using All Available RAM

Mongodb – Mongo repairDatabase procedure

Related Question