I am using rsync for backing up several systems, either to external hard drives or over the network to another machine. The idea is to maintain a replica of the original filesystem, with a few things like /dev, /proc, /mnt, and what not excluded, which is then snapshotted at the destination (i.e. with btrfs or zfs). To that end, I use a command along the lines of
rsync -avP --delete --exclude /dev/* --exclude /proc/* --exclude /mnt/* etc. etc. / backup@fileserver:/mnt/array/backup/machine/
Which works reasonably well. However, I just noticed that rsync is not reliably deleting things on the destination. As a result, there are many more files on the destination than exist at any time on the source, in one case the back up was 100 GB larger than the source. The leftover files seem to be mostly old versions of various software and libraries. For example, on the original machine, /usr/include/qt/QtWidgets/ contains
5.9.2
QAbstractButton
qabstractbutton.h
....
But on the backup target, the same folder contains:
5.6.0
5.6.1
5.7.0
5.7.1
5.8.0
5.9.0
5.9.1
5.9.2
QAbstractButton
qabstractbutton.h
rsync seems to be copying over the new files just fine, but forgetting to remove all the old ones. The odd part is that some files are in fact removed (rsync always reports deleting a lot of files when synchronizing) but it seems that many files are not actually deleted. As a result, my backups are taking up a lot more space than they should, and they can't be directly restored because they can take up more space than the original partition to be due to all the extra files, and things like flat-file package databases are all screwed up, containing many duplicate entries for old versions packages.
Any ideas on what might be going on here? Is there some other flag I need to pass to rsync so that it actually deletes everything it's supposed to?
Best Answer
Some suggestions for things to try come from the contents of the man page (the section about
--delete
). There are a lot of caveats and warnings. Have you considered all of these?Works only "for the directories that are being synchronized". I guess this means they must have a parent/ancestor directory being copied.
Similarly, it doesn't apply to anything being explicitly excluded:
If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. Override with
--ignore-errors
.Try with
--dry-run
first? Combine with--verbose
and closely examine everything that's happening with problem directories.So consider trying
--delete-excluded
,--ignore-errors
, and/or--dry-run
/--verbose
...not necessarily as a "fix" but at the very least there may be helpful data to be gleaned.Also, take a look at the
--delete-before
flag. This is supposed to do the receive-side deletes before the actual transfer. Again, this might not fix everything but could be revealing. It'd be interesting to see if it helps reduce disk space usage, too.