I am planing to backup data via sending btrfs snapshots to an online storage. The storage is mounted as a LUKS encrypted container file on a cifs share.
Later transfers will be reasonably fast, but for the first one is 1,3TB, which, when I use tc to leave me with a reasonable amount of upstream, will take 23 days.
Now, in theory, my connection could handle that, but reconnections are possible. As far as I understand, this would force me to start all over again, if I just use
btrfs send ... | btrfs receive ...
Is there any save, that is, resumable way to do this? I found buttersink, but it only seems to allow resume for S3.
Any ideas?
Feel free to use the comments to suggest a completely different solution. It is a Hetzner Storagebox (https://www.hetzner.de/de/hosting/storagebox/bx40). I have FTP, FTPS, SFTP, SCP (but not SSH and paramiko also doesn't work), Samba/CIFS, HTTPS, WebDAV access. The storage is not to be trusted with the unencrypted data. Free space on both sites is not excessive. There will be lots of changing smaller files, so duplicity without a regular full backup (that would take a month again) does not seem to be feasible. For the same reason, rsync would most likely be slow when comparing the local version with the one locally mounted from remote.
EncFS seems to be unsave, since the other side will potentially be able to gather multiple versions of a single file over time.
Best Answer
As I understand it, your remote storage is exposed as a filesystem. I don't use
btrfs
myself but I assume the snapshots are equivalent to one large "full backup" file followed by a number of smaller "incremental" files.On that basis I'd still go with
rsync
because it's restartable. You can't use its snazzy delta differences algorithm unless there's anrsync
server available on the remote host, but you can tellrsync
to assume the source file hasn't changed and to continue after a break from the byte offset it had reached:If you can usefully
gzip
your source file before transferring it, do so. (Neither--rsyncable
norrsync -z
is relevant for whatrsync
sees as a local to local file transfer.)