Recover deleted file from ZFS snapshot (still viewable vith lsof)

If you store the snapshots in files, as opposed to in the file system (e.g. with zfs receive), I'm afraid, this is not possible.

ZFS on the receiving side

If you use ZFS on the sending and on the receiving side you can avoid having to transfer the whole snapshot and only transfer the differences of the snapshot compared to the previous one:

ssh myserver 'zfs send -i pool/dataset@2014-02-04 pool/dataset@2014-02-05' | \
  zfs receive

ZFS knows about the snapshots and stores mutual blocks only once. Having the file system understand the snapshots enables you to delete the old ones without problems.

Other file system on the receiving side

In your case you store the snapshots in individual files, and your file system is unaware of the snapshots. As you already noticed, this breaks rotation. You either have to transmit entire snapshots, which will waste bandwidth and storage space, but enables you to delete individual snapshots. They don't depend on each other. You can do incremental snapshots like this:

ssh myserver 'zfs send -i pool/dataset@2014-02-04 pool/dataset@2014-02-05' \
  > incremental-2014-02-04:05

To restore an incremental snapshot you need the previous snapshots as well. This means you can't delete the old incrementals.

Possible solutions

You could do incrementals as shown in my last example and do a new non-incremental every month. The new incrementals depend on this non-incremental and you're free to delete the old snapshots.

Or you could look into other backup solutions. There is rsnapshot, which uses rsync and hard links. It does a very good job at rotation and is very bandwidth efficient, since it requires a full backup only once.

Then there is bareos. It does incrementals, which are bandwith- and space-saving. It has a very nice feature; it can calculate a full backup from a set of incrementals. This enables you to delete old incrementals. But it's a rather complex system and intended for larger setups.

The best solution, however, is to use ZFS on the receiving side. It will be bandwidth efficient, storage efficient and much faster than the other solutions. The only really drawback I can think of is that you should have a minimum of 8 GiB ECC memory on that box (you might be fine with 4 GiB if you don't run any services and only use it to zfs receive).

After much tinkering and experimentation I've found a solution, albeit with a fairly large tradeoff.

First off, the options I had to rule out:

• Having a second offsite ZFS server with a mirrored pool wasn't an option due to cost. Had it been an option this would by far have been the best approach, utilizing ZFS send / receive to ship snapshots to the remote pool.

• Having a second onsite ZFS mirrored pool, which I could remove disks from to take home. This is more feasible than the first option, but I would need the second pool to always have two disks onsite (or to use two data-copies on a single onsite disk). At present I have four disks, and no more space for a fifth in the server. This would be a fair approach but still not ideal.

• Using ZFS attach and detach to rotate the backup disk into and out of the mirrored pool. This works well, but has to perform a full resilver every time the disk is added. This takes unacceptably long, and so I couldn't rely on this.

My solution is similar to using attach and detach, however it uses online and offline. This has the advantage of performing a delta resilvering versus a full resilvering, but the drawback that the pool always reports a DEGRADED state (the pool always has two disks; the rotating offsite disks are marked offline when they are in remote storage and resilver and then come online when they are onsite).

So, a quick recap and overview of my setup:

I have one ZFS server and four identical disks. ZFS is setup to use a mirrored pool. Two of the four disks are permanent members of this pool. The other two disks rotate; one is always in offsite storage, the other is part of the pool to act as a ready-to-go backup.

When it comes time to rotate the backups:

• I wait for a zfs scrub to complete to reasonably assure the backup disk is error free

• I zfs offline the disk which will be taken remote. After its offline'd I hdparm -Y /dev/id to spin it down. After a minute I partially remove the disk sled (just enough to ensure its lost power) and then give it another minute before fully pulling the drive to guarantee it has stopped spinning. The disk goes in a static bag and then a protective case and goes offsite.

• I bring in the other offsite disk. It gets installed in the hotswap tray and spins up. I use zfs online to restore the disk to the pool and kick off a partial resilvering to make it concurrent.

This system guarantees that at any given time I have two ONLINE mirror disks and one OFFLINE remote disk (which has been scrubbed). The fourth disk is either being resilvered or online, which has the benefit that in case a running drive fails it's probably the pool will still consistent of two online disks.

It's worked well for the past couple weeks, but I'd still consider this a hackish approach. I'll follow up if I run into any major issues.

Update: After running with this for a couple months I've found that in my real-world use the resilvering is taking the same time for either detach/attach and offline/online. In my testing I don't think I was running a scrub--my hunch is that if a drive is offline for a scrub then it requires a full resilver.