Linux – Rsync backups on btrfs very slow

Maybe using Btrfs send / receive could be made to work, but it is not so simple, because to send a filesystem, a read only snapshot must first be made, and then the name of this snapshot is used on the external disk. I don't think there is a way to receive the root filesystem, at /

This is the best solution. I use snapshots to make fast incremental backups of my servers. You can back up the root subvolume just like any other, but I don't think you can receive at the root. More to the point, you shouldn't do so, because that prevents you from enjoying one of the benefits of snapshotting: incremental backups. Done properly, it will only send the data that has changed and consume only the disk space for the changed data.

This script runs as a cron job and takes a daily snapshot of my root partition, then uses btrfs send to send an incremental copy to my backup partition. The script as written uses pv, but if for some reason you don't want to install it, you can simply remove pv from the middle of the pipes.

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
date=$(date +%Y-%m-%d)

# the path to the partition mount point that we are backing up
source_partition=/

# where backup snapshots will be stored on the local partition
# this is needed for incremental backups
source_snapshot_dir=/snapshots

# where backups will be stored on the backup drive
target_snapshot_dir=/mnt/media/backups/root

if [ ! -d $source_snapshot_dir ]; then
    echo 'Creating initial snapshot...'
    mkdir --parents $source_snapshot_dir $target_snapshot_dir

    # create a read-only snapshot on the local disk
    btrfs subvolume snapshot -r $source_partition $source_snapshot_dir/$date

    # clone the snapshot as a new subvolume on the backup drive
    # you could also pipe this through ssh to back up to a remote machine
    btrfs send $source_snapshot_dir/$date | pv | \
        btrfs receive $target_snapshot_dir
elif [ ! -d $source_snapshot_dir/$date ]; then
    echo 'Creating root volume snapshot...'

    # create a read-only snapshot on the local disk
    btrfs subvolume snapshot -r $source_partition $source_snapshot_dir/$date

    # get the most recent snapshot
    previous=$(ls --directory $source_snapshot_dir/* | tail -n 1)

    # send (and store) only the changes since the last snapshot
    btrfs send -p $previous $source_snapshot_dir/$date | pv | \
        btrfs receive $target_snapshot_dir
fi

echo 'Cleaning up...'

# keep the 3 most recent snapshots on the source partition
ls --directory $source_snapshot_dir/* | \
    head --lines=-3 | \
    xargs --no-run-if-empty --verbose \
    btrfs subvolume delete --commit-after

# keep the 28 most recent snapshots on the backup partition
ls --directory $target_snapshot_dir/* | \
    head --lines=-28 | \
    xargs --no-run-if-empty --verbose \
    btrfs subvolume delete --commit-after

(Note: I've adapted the script somewhat to make it into a general solution, and haven't tested it as written. Please feel free to submit revisions if necessary.)

Fragmentation is a largely unavoidable side effect of a file system being designed with copy-on-write. It's also what allows the nearly free file system snapshots in the first place.

The reason for this is fairly simple: Every time a block is changed, the new block must be written to some location other than that of the original block. So even if the file was contiguous originally, it won't be after it has been modified.

I don't know how Btrfs nodatacow interacts with snapshots, but I have a feeling that the instant you have a snapshot on a dataset, you force at least partial copy-on-write behavior no matter what flags you are using; otherwise how would you be able to access the old data via the snapshot?

However, it is not a given that this will necessarily severely affect your MySQL performance, for two reasons:

• Modern disks are really pretty fast for single-user workloads (which I take it that you are most interested in because you mention that your system is a "workstation")
• Modern operating systems have pretty good caching algorithms, thereby reducing the need to actually hit physical storage

Just to give you an idea, I'm running ZFS myself (from which Btrfs borrows many ideas), and there's currently a scrub in progress. The pool in question is a six-disk raidz2, which isn't really known for its stellar performance, physically backed by six 7200 rpm disks (two SATA, four SAS) which also aren't exactly known for stellar IOPS in particular. A ZFS scrub navigates the entire on-disk Merkle tree, reads all data, and verifies checksums on everything to make sure everything reads back as it was previously written; in my case, computing SHA-256 hashes of everything along the way. The current scrub speed (after it got past the initial, metadata-heavy portion, which involves heavy seeking) is hovering right around 200 MB/s and actually climbing slowly. And that's for actual platter I/O, with no caching involved (because caching doesn't make any sense when you want to verify what's on persistent storage).

Sure, it's very likely that you will see some performance degredation from fragmentation if you move to a copy-on-write file system. But you can't eat the cake and keep it, too; if fast, low cost snapshots is something you want, it's likely that you are going to have to give up something else to get them.

What I would do in your case is benchmark. Set up some Btrfs storage, put a copy of the MySQL database there, and see how the two perform under reasonable workloads.