How to copy multiple snapshots at once without duplicating data

btrfsdeduplicationsnapshot

I have a live btrfs filesystem of 3.7TiB that's >90% full including old snapshots and a fresh 4TB backup harddisk. How to copy all existing snapshots to the backup harddisk?

I tried

# btrfs send home_1 home_2 home_3 share_1 share_2 share_3 ...

but the backup harddisk was full before 2/3 of the snapshots were transmitted. So I did some research:

# ### create test image
# dd bs=1M count=1000 > btrfs_test.dd
# mkfs.btrfs btrfs_test.dd

# ### create snapshots
# btrfs subvol create testvol/
# btrfs subvol snapshot -r testvol/ testvol_0/
# ### (copy some ISO image)
# btrfs subvol snapshot -r testvol/ testvol_1/
# ### (proceed until testvol_3)

My test filesystem then was 91% full with 818MiB used.

# btrfs send testvol_* | wc -c 
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
1466441978     # 1398MiB >> 1000MiB

When simply sending all the snapshots with 1 command, data is duplicated and the size of the stream as well as the size of the snapshots at the receiver's end exceed the original used space and the harddisk's capacity.

So the actual question became: How to copy multiple snapshots without duplicating data that is contained in 2 or more of them?

For this simple test case I successfully tried the incremental approach:

# ( btrfs send testvol_0; btrfs send -p testvol_0 testvol_1; btrfs send -p testvol_1 testvol_2; btrfs send -p testvol_2 testvol_3 ) | wc -c 
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546    # 800 MiB < 1000MiB

But on the real filesystem there are multiple subvolumes, each of them with multiple snapshots. I can't define any order to use with -p. If a data block is shared among the subvolumes home_1, home_2 and share_3 I want to transmit and store it only once, of course. Any ideas how to do this?

Best Answer

TL;DR: Using the -c parameter as described at the top works in general. When the snapshots contain hardlinks there's a bug in the Linux kernel that triggers an error when sending the snapshots—for details see conclusion at the end of the answer.

I am experimenting with the -c parameter and it looks promising:

# for i in {0..3}; do btrfs send testvol_$i $(ls -d testvol_* | head -n$i | sed 's/^/-c /'); done | wc -c
### btrfs send testvol_0
### btrfs send testvol_1 -c testvol_0
### btrfs send testvol_2 -c testvol_0 -c testvol_1
### btrfs send testvol_3 -c testvol_0 -c testvol_1 -c testvol_2
At subvol testvol_0
At subvol testvol_1
At subvol testvol_2
At subvol testvol_3
838778546    # also 800MiB

I am still not sure if this is what I need. Any comments on this solution?

Update: To test this with my real filesystem I wrote a Perl script to comfortably send a set of subvolumes (creating the lists for -c quickly became tedious) and sent some data to /dev/null:

#!/usr/bin/env perl

use strict;
use warnings;

my @subvols = @ARGV
  or die "Usage: $0 SUBVOLUME ...\n";

for(@subvols) {
    -d $_
      or die "Not a directory: $_\n";
}

for my $i (0 .. $#subvols) {
    my $subvol = $subvols[$i];
    my @clones = map { ('-c', $_) } @subvols[ 0 .. $i-1 ];
    print "btrfs send $subvol @clones\n";
}

Results:

  • btrfs send some-subvolme_* | pv > /dev/null: 24GiB 0:04:17 [95.2MiB/s]
  • perl btrfs-send-all.pl some-subvolume_* | bash | pv > /dev/null: 12.7GiB 0:03:58 [54.7MiB/s]

That's not much of a performance gain but nearly a 50% decrease in storage space! I am now trying to run this in real …

Update: I successfully transferred 2 snapshots but for the 3rd one btrfs receive failed with the following error message:

ERROR: unlink path/to/some/file/in/the/snapshot failed. No such file or directory

The named file is present in subvol_2 and subvol_3 but not yet in subvol_1.

I tried to compare the sent and the received snapshots:

# ### sender
# du -s subvol_{1,2,3}
132472304       subvol_1
117069504       subvol_2
126015636       subvol_3

# ### receiver
# du -s subvol_*
132472304       subvol_1
117069504       subvol_2
132472304       subvol_3

The first two snapshots seem to be transferred correctly but subvol_3 to be a clone of subvol_1. It's definitely a clone because the used space on the backup disk is only 39% of the snapshots' combined size—tightly above 1/3 as they share most files.

Why does btrfs send subvol_3 -c subvol_1 -c subvol_2 | btrfs receive not correctly transfer the snapshot but clone subvol_1 and then fail to delete a file that should be kept in subvol_3 and is only present in subvol_2??

Update: Could it be this bug? https://patchwork.kernel.org/patch/10073969/

I am running Debian 9 Stretch with Kernel 4.9 which seems to be older than the patch.

Update: Because I could not find any solution I simply copied the most recent snapshot of each subvolume. Then I had roughly 500GiB free space and tried to add the previous snapshot with the the already copied snapshot as -p parameter. Then I got the same error message for the same snapshot as above.

Conclusion: I conclude that I hit the bug linked above. I had to reboot this machine with a newer Linux kernel or access the filesystem from another computer but this is not feasible because this is a production system.

I had no problems with btrfs so far but running rsnapshot—which creates a lot of hardlinks—and sending btrfs snapshots still might have problems on current Debian stable.

Related Question