VMware: Why is zero-filling ext4 free space needed to shrink *.vmdk files

ext4vmware

On my Linux Mint 17.2 VMware guest, df -h was reporting my total disk usage steady at about 10GB total. I'm using this machine for Ruby on Rails development inside a Windows host running Workstation 12.1.1 Pro.

The *.vmdk files kept on steadily growing to approx 100GB. Trying to shrink with vmware-vdiskmanager -k or vmware-toolbox-cmd disk shrinkonly made no difference.

I have Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize. A more complete dumpe2fs output is available.

Running e4defrag / and e2fsck -E discard didn't allow any more space to be reclaimed (I killed the shrink after the first few vmdks (which started at around 6GB) showed no sign of shrinking.)

What finally did the trick was filling all free space with NULbytes:

dd if=/dev/zero of=wipefile bs=1M; sync; rm wipefile

I could now shrink to a total *.vmdk size of 15.6GB with vmware-toolbox-cmd disk shrinkonly.

That's about 85GB space saving on a VM with 10GB actual data.

It seems that ext4 is not re-using previously used blocks when asked for an unused block, often preferring to give out never-before-used blocks.

Questions

  1. Having old data hanging around longer seems less secure. Why would ext4 not re-use recently-used blocks as soon as possible?

  2. Is there a way to force ext4 to reuse just-used blocks?

  3. Is there a way to prevent a VMware guest's *.vmdk files from continually growing without needing to 0-fill free space on a regular basis?

    • How do you safely (eg not entirely filling the filesystem) automate this?

Best Answer

  1. Yes, from security standpoint it would be better to immedeately erase any unused blocks. The reason that is not done (ignoring chattr secure-delete flag and patches) is performance. The same is a reson for not using any recenty freed block - it would lead to heavy fragmentations which hurts performance.

  2. No, not really. You could make your whole image much smaller (formatting it to say 15G and then growing only if/when that becomes necessary.) - then it will never grow bigger than 15Gb.

  3. You can try to mount filesystem with discard option - see fstab(5), but I'm not sure if your vmware will heed this

Related Question