Resumable transfer for btrfs send

backupbtrfsencryption

I am planing to backup data via sending btrfs snapshots to an online storage. The storage is mounted as a LUKS encrypted container file on a cifs share.

Later transfers will be reasonably fast, but for the first one is 1,3TB, which, when I use tc to leave me with a reasonable amount of upstream, will take 23 days.
Now, in theory, my connection could handle that, but reconnections are possible. As far as I understand, this would force me to start all over again, if I just use

 btrfs send ... | btrfs receive ...

Is there any save, that is, resumable way to do this? I found buttersink, but it only seems to allow resume for S3.

Any ideas?

Feel free to use the comments to suggest a completely different solution. It is a Hetzner Storagebox (https://www.hetzner.de/de/hosting/storagebox/bx40). I have FTP, FTPS, SFTP, SCP (but not SSH and paramiko also doesn't work), Samba/CIFS, HTTPS, WebDAV access. The storage is not to be trusted with the unencrypted data. Free space on both sites is not excessive. There will be lots of changing smaller files, so duplicity without a regular full backup (that would take a month again) does not seem to be feasible. For the same reason, rsync would most likely be slow when comparing the local version with the one locally mounted from remote.
EncFS seems to be unsave, since the other side will potentially be able to gather multiple versions of a single file over time.

Best Answer

As I understand it, your remote storage is exposed as a filesystem. I don't use btrfs myself but I assume the snapshots are equivalent to one large "full backup" file followed by a number of smaller "incremental" files.

On that basis I'd still go with rsync because it's restartable. You can't use its snazzy delta differences algorithm unless there's an rsync server available on the remote host, but you can tell rsync to assume the source file hasn't changed and to continue after a break from the byte offset it had reached:

test -t 2 && progress=--progress
rsync -av $progress --partial --append --sparse /path/to/source.img /path/to/remote/storage/

If you can usefully gzip your source file before transferring it, do so. (Neither --rsyncable nor rsync -z is relevant for what rsync sees as a local to local file transfer.)

Related Question