Try sudo lsof| grep deleted
and check if any PostgreSQL process appears. This command looks for files which have been deleted but its file descriptors are still open by any process. Another side effect is that df -h
and du -sh /
differs. This is because du
looks at the file system and sum up the size of all files, and df
looks at the physical device.
I have just had an issue with a database which didn't free any space after a DROP table
and that was the cause.
The only solution I know is to restart the database. Maybe you can try to send a reload (SIGHUP).
First, if you haven't done the first two that vyegorov suggests do it now. If you have already vacuumed, it is not clear what sort of damage you may have already done in this case. This error should not be possible on new versions of PostgreSQL due to a number of checks, and had you upgraded some time ago, this wouldn't have happened. I highly recommend trying to stay on supported branches in the future.
I want to take a moment and describe what likely caused the problem and what it means. The prognosis IMO is not good and recovering the data, if it is even possible, is likely to be expensive and time consuming. I sincerely hope you have a good backup from before the transaction wraparound. If not, ouch....
What Went Wrong
PostgreSQL uses something called MVCC, which means that old versions of rows are kept around until they are clearly no longer used. MVCC as practiced by PostgreSQL stamps each row with a minimum transaction and a maximum transaction, and uses these for visibility management. When a transaction rolls back those rows entered are no longer visible and those rows deleted remain visible. Transaction ID's are 32-bit integers.
Periodically you are supposed to vacuum PostgreSQL instances. This, among other things, manages MVCC so that fewer transaction id's need to be checked, manages free space in tables, and can reset the transaction id sequence.
When the transaction ID wraps around, bad things happen but they boil down to the fact that PostgreSQL can no longer be sure what rows are visible and what rows are not. Note thais applies not only to rows in your own tables but in the system catalogs as well. This is a very, very bad thing. The best approach, if you can do it, is to restore from a backup before the wraparound occurred. This is why databases and tables are not showing up and why vacuuming full duplicates them.
Why Upgrading Would have Fixed This
Modern versions of PostgreSQL come with autovacuum automatically enabled which runs vacuum processes in the background to help prevent this sort of problem. More recent versions also will refuse to start new transactions for non-superusers once wraparound is approaching. This gives you a chance to detect and correct the problem before you suffer possibly catastrophic data loss.
PostgreSQL 7.4 hasn't been supported in nearly three years. I don't know when the last time anything was vacuumed but it must have been billions of transactions ago. This is not good.
Best Answer
At worst
pg_total_relation_size(table) * 2
. Usually lots less because the new table and indexes are significantly more compact.Clean up unrelated files on the volume. Drop some tables or indexes you don't need. Move tables or indexes to other tablespaces on other storage. Expand the underlying volume.
There isn't really a good way to compact in-place and free space to the OS without first needing to allocate more space. I think most people agree this is a flaw, but nobody seems to care enough to develop any improvement for it.