Alternative of rsync –delete to move files to another directory instead of deleting

backuprsync

I use rsync to back up large hard drives on a remote machine. By default, rsync does not delete files on the remote machine when they no longer exist on the local machine. This is useful, because it makes it possible to recover, e.g., if I want to restore a file which was deleted by mistake, and it also means that I don't need to worry about rsync'ing an empty directory (if the hard drive dies or is not mounted). The problem is that it takes lots of disk space on the remote host, and also that if I need to restore a backup it will be polluted with these deleted files.

The alternative is rsync --delete, but then it means that files that have been deleted will also get deleted the first time I run the backup, and if the mountpoint ever becomes empty then I risk deleting the backup as well.

Ideally I would like to have a way to tell rsync to do like --delete, but instead of deleting files, it could move them to a different backup hierarchy. That way, I could still restore these files if I wanted, and I could clean up the larger files in that second hierarchy to save space, while the main backup hierarchy would be a clean back-up of my drive.

Is there a way to emulate this behavior? (or some similar behavior)

(I don't have enough disk space to maintain two complete backups on the remote host. Also, I know about the existence of incremental backup tools like Borg Backup, but I'd like a solution that doesn't rely on heavy tools with a custom storage format. I'm also a bit wary of solutions using hard links but I'm OK with them if it's the best option.)

Best Answer

You can use --backup

-b, --backup With this option, preexisting destination files are renamed as each file is transferred or deleted.

in combination with --backup-dir

--backup-dir=DIR In combination with the --backup option, this tells rsync to store all backups in the specified directory on the receiving side. This can be used for incremental backups.

to keep a copy of any modified and removed files in a separate directory, on a receiving side.

If you want to only keep the deleted files in the backup, you can separately purge the updated files after the sync (e.g. like this).

Related Solutions

Rsync, delete files on receiving side that were deleted on sending side. (But don’t delete everything)

What you want to do is reasonable, but using rsync to do it on its own is not. So the answer is no.

The reason is simple: rsync keeps no history of what was in each directory and has no way of knowing what needs to be deleted and what not. Not without additional support.

You should ask yourself why you like to do this with rsync and make that more clear. There are other programs that use librsync1.so that are more intelligent.

With the relaxed constraints that you don't need rsync per se, you can take a look at rdiff-backup:

mkdir a
touch a/xx
touch a/yy
rdiff-backup a b
ls b

This shows xx and yy are in b.

touch b/zz
rm a/xx
rdiff-backup a b

This shows xx and zz are in b. rdiff-backup also keeps a directory rdiff-backup-data in b so you can rollback any changes, you should purge this on a regular basis using the rdiff-backup commands. (The example is with local files to show extra data in the target does not get deleted, but rdiff-backup works over a network as well).

Another alternative is to setup some distributed revision control system (mercurial, bazaar, git). With mercurial e.g. you can have a script (I use a Makefile for that), that pushes all the changes to the server and then does an update of the checked out files there, ignore any additional files that are on the remote server (but have not been put under revision control).

On the server you would do:

hg init
hg add file_list_excluding_that_should_not_should_be_deleted_if_not_on_client
hg commit -m "initial setup"

On the client:

hg clone ssh://username@server/dir_to_repository

Now if you remove a file on the client and do:

hg commit -m "removed file"
ssh username@server "cd dir_to_repository; hg update --clean"

Your removed file is removed on the server, but any other data (not added to the repository) does not get deleted.

Rsync delete only excluded files

The use of filters might meet your requirements:

--filter='P /some/dir'

Best Answer

Related Solutions

Rsync, delete files on receiving side that were deleted on sending side. (But don’t delete everything)

Rsync delete only excluded files

Related Question