Linux – Backup to external drive with full disk encription and bit rot prevention

backupencryptionlinuxubuntu 14.04

I have a new USB external 2TB HDD and would like to use it as a backup for my home server.

My wishes are:

full disk encryption (the disk will be stored off-site)
bit rot prevention

Currently, I am using an older 320GB external disk with TrueCrypt partition-level encryption. But since this does not provide bit rot protection and since TrueCrypt is deprecated, I would like to try something else.

My server runs Ubuntu Server 14.04.3 LTS and has ECC memory with 2x 3TB disks in RAID 1.

For the encryption part, using LUKS is probably the best choice, right?

But when it come with dealing with bit rot, things get a bit complicated. At least for single disk configurations. One option would be to create 2 identical partitions and make a RAID 1 with regular scrubbing. But this solutions seems a bit weird on the same drive.

Would btrfs be an option? Is it possible to add redundancy for single drive configurations?

I don't mind sacrificing half of the storage for redundancy. Loss in performance is also not an issue for me.

Best Answer

Given that you have all the basic prerequisites, particularly ECC RAM, I think ZFS (through the ZFS On Linux implementation) might be a usable option.

Unlike btrfs, which borrows a lot of design ideas from ZFS, ZFS is a tried and true (volume manager and) file system. Sure, the Linux port has some rough edges that are being smoothed out over time, but the code and design has been tested in quite a number of real-world failure scenarios.

You could use ZFS either with two separate partitions in a mirror configuration, or with one partition and setting copies=2 on the pool's root file system. The disk space and I/O performance overhead of these would be similar. Either will allow you to migrate to larger disks, or multi-disk configurations, as your needs change with time. You can use raidz vdevs (with varying levels of redundancy: one, two, or three disks' worth) but this presents problems should you ever want to change redundancy levels.

I would suggest seriously considering running ZFS on top of LUKS.

The opposite (running LUKS on top of ZFS) is also possible, but significantly more involved. You could also run something like ecryptfs on top of unencrypted ZFS, but this potentially leaks significant amounts of file system metadata.

In other words, create LUKS containers (one for each drive or partition), then use those containers to create a ZFS pool. Running ZFS on top of LUKS should be sufficient to prevent an offline attacker in most scenarios, although it will present little obstacle for an online attacker. Whether this is a problem depends on your threat model, but for off-site backups, offline access is often the more important aspect to consider.

Running two separate partitions as distinct LUKS containers should help against damage to the LUKS header making both copies inaccessible, but other methods can do this as well (for example, a securely stored LUKS header backup). Running a single partition LUKS container for each drive will allow ZFS to make decisions about storing file system metadata in diverse, redundant locations.

If you go with copies=2, make sure you set that immediately when creating the pool. In other words, you want something like:

cryptsetup luksFormat /dev/sdx
cryptsetup luksOpen /dev/sdx sdx-crypt
zpool create -O copies=2 tank /dev/mapper/sdx-crypt

and not

cryptsetup luksFormat /dev/sdx
cryptsetup luksOpen /dev/sdx sdx-crypt
zpool create tank /dev/mapper/sdx-crypt
zfs set copies=2 tank

because the latter does not fully replicate the root file system metadata until that metadata gets rewritten.

Note that as with any copy-on-write file system, ZFS does best when disk utilization is kept below about 75-80%, and you should expect fragmentation over time. For backups, this should not be a major concern.

Related Solutions

Linux – Mirror / backup laptop btrfs drive to sometimes connected external USB disk

Maybe using Btrfs send / receive could be made to work, but it is not so simple, because to send a filesystem, a read only snapshot must first be made, and then the name of this snapshot is used on the external disk. I don't think there is a way to receive the root filesystem, at /

This is the best solution. I use snapshots to make fast incremental backups of my servers. You can back up the root subvolume just like any other, but I don't think you can receive at the root. More to the point, you shouldn't do so, because that prevents you from enjoying one of the benefits of snapshotting: incremental backups. Done properly, it will only send the data that has changed and consume only the disk space for the changed data.

This script runs as a cron job and takes a daily snapshot of my root partition, then uses btrfs send to send an incremental copy to my backup partition. The script as written uses pv, but if for some reason you don't want to install it, you can simply remove pv from the middle of the pipes.

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
date=$(date +%Y-%m-%d)

# the path to the partition mount point that we are backing up
source_partition=/

# where backup snapshots will be stored on the local partition
# this is needed for incremental backups
source_snapshot_dir=/snapshots

# where backups will be stored on the backup drive
target_snapshot_dir=/mnt/media/backups/root

if [ ! -d $source_snapshot_dir ]; then
    echo 'Creating initial snapshot...'
    mkdir --parents $source_snapshot_dir $target_snapshot_dir

    # create a read-only snapshot on the local disk
    btrfs subvolume snapshot -r $source_partition $source_snapshot_dir/$date

    # clone the snapshot as a new subvolume on the backup drive
    # you could also pipe this through ssh to back up to a remote machine
    btrfs send $source_snapshot_dir/$date | pv | \
        btrfs receive $target_snapshot_dir
elif [ ! -d $source_snapshot_dir/$date ]; then
    echo 'Creating root volume snapshot...'

    # create a read-only snapshot on the local disk
    btrfs subvolume snapshot -r $source_partition $source_snapshot_dir/$date

    # get the most recent snapshot
    previous=$(ls --directory $source_snapshot_dir/* | tail -n 1)

    # send (and store) only the changes since the last snapshot
    btrfs send -p $previous $source_snapshot_dir/$date | pv | \
        btrfs receive $target_snapshot_dir
fi

echo 'Cleaning up...'

# keep the 3 most recent snapshots on the source partition
ls --directory $source_snapshot_dir/* | \
    head --lines=-3 | \
    xargs --no-run-if-empty --verbose \
    btrfs subvolume delete --commit-after

# keep the 28 most recent snapshots on the backup partition
ls --directory $target_snapshot_dir/* | \
    head --lines=-28 | \
    xargs --no-run-if-empty --verbose \
    btrfs subvolume delete --commit-after

(Note: I've adapted the script somewhat to make it into a general solution, and haven't tested it as written. Please feel free to submit revisions if necessary.)

Best Answer

Related Solutions

Linux – Mirror / backup laptop btrfs drive to sometimes connected external USB disk

Related Question