ZFS Zpool – How to Create a Zpool Readable by GRUB

arch linuxgrub2zfs

The Arch Linux ZFS wiki page explains grub-compatible pool creation, as does this page about booting Fedora, but I have not been able to create a pool that is readable by Grub. The Arch Linux wiki page about Installing Arch Linux on ZFS highlights certain bugs but doesn't really explain how to overcome them.

The linked pages explain that Grub supports a subset of zpool features and cannot read a pool that uses features that it doesn't support. They go on to explain how to configure a suitable pool but I have been unable to make it work. The supported feature subset does not appear to be documented anywhere.

I am using a virtual machine to test with Grub 2.02 and Arch Linux kernel 4.16.13-1-ARCH which is the most recent and is compatible with the current zfs-linux package version (zfs-linux-0.7.9.4.16.13.1-1). I am not (yet) trying to make a bootable system, only to prove that Grub can read the zpool. Here is what I have tried:

First, like the arch wiki page suggests, by disabling unwanted features:

# zpool create \
    -o feature@multi_vdev_crash_dump=disabled \
    -o feature@large_dnode=disabled           \
    -o feature@sha512=disabled                \
    -o feature@skein=disabled                 \
    -o feature@edonr=disabled                 \
    testpool mirror \
    /dev/disk/by-id/ata-VBOX_HARDDISK_VB{5f2d4170-647f16b7,f38966d8-57bff7df}

which results in these features:

testpool  feature@async_destroy          enabled                       local
testpool  feature@empty_bpobj            active                        local
testpool  feature@lz4_compress           active                        local
testpool  feature@multi_vdev_crash_dump  disabled                      local
testpool  feature@spacemap_histogram     active                        local
testpool  feature@enabled_txg            active                        local
testpool  feature@hole_birth             active                        local
testpool  feature@extensible_dataset     active                        local
testpool  feature@embedded_data          active                        local
testpool  feature@bookmarks              enabled                       local
testpool  feature@filesystem_limits      enabled                       local
testpool  feature@large_blocks           enabled                       local
testpool  feature@large_dnode            disabled                      local
testpool  feature@sha512                 disabled                      local
testpool  feature@skein                  disabled                      local
testpool  feature@edonr                  disabled                      local
testpool  feature@userobj_accounting     active                        local

Then, like the fedora example, by enabling wanted features:

zpool create -d \
    -o feature@async_destroy=enabled \
    -o feature@empty_bpobj=enabled \
    -o feature@spacemap_histogram=enabled \
    -o feature@enabled_txg=enabled \
    -o feature@hole_birth=enabled \
    -o feature@bookmarks=enabled \
    -o feature@embedded_data=enabled \
    -o feature@large_blocks=enabled \
    testpool mirror \
    /dev/disk/by-id/ata-VBOX_HARDDISK_VB{5f2d4170-647f16b7,f38966d8-57bff7df}

which results in these features:

# zpool get all testpool | grep feature
testpool  feature@async_destroy          enabled                       local
testpool  feature@empty_bpobj            active                        local
testpool  feature@lz4_compress           disabled                      local
testpool  feature@multi_vdev_crash_dump  disabled                      local
testpool  feature@spacemap_histogram     active                        local
testpool  feature@enabled_txg            active                        local
testpool  feature@hole_birth             active                        local
testpool  feature@extensible_dataset     enabled                       local
testpool  feature@embedded_data          active                        local
testpool  feature@bookmarks              enabled                       local
testpool  feature@filesystem_limits      disabled                      local
testpool  feature@large_blocks           enabled                       local
testpool  feature@large_dnode            disabled                      local
testpool  feature@sha512                 disabled                      local
testpool  feature@skein                  disabled                      local
testpool  feature@edonr                  disabled                      local
testpool  feature@userobj_accounting     disabled                      local

In each case, I loaded some content:

# cp -a /boot /testpool

And then, rebooted into Grub:

grub> search --set --label testpool
grub> ls /
@/
grub> ls /@
error: compression algorithm 80 not supported
.
grub> ls /@/
error: compression algorithm inherit not supported
.

I tried enabling/disabling some features, most notably lz4_compress. I also tried creating a dataset on the pool. Nothing I tried worked inside Grub.

I expected to be able to list /boot or /@/boot.

Errors encountered include

compression algorithm inherit not supported
compression algorithm 66 not supported
compression algorithm 80 not supported
incorrect dnode type

How should a ZFS zpool be created in order for it to be readable by Grub?

Best Answer

Grub cannot reliably perform a directory listing of a zpool due to a bug as confirmed via the mailing list:

listing directory contents in Grub is broken, I have a patch lying around that fixes that specific issue. if you get strange error messages (e.g., something like invalid BP type or compression algorithm), it's likely this issue.

This probelm is present in Grub installed from ArchLinux and also Fedora 28. Ubuntu, however, appears to have patched their Grub to fix this (confirmed with Ubuntu 16.10).

Grub can, however, read the files required to boot. You can create a zpool that Grub will boot like this:

zpool create -m none "$ZPOOL" "$RAIDZ" "${DISKS[@]}"

where the variables define the name of the pool, e.g mypool, the raid level, e.g. mirror and the disks, e.g /dev/disk/by-id/... (two disks being required for a mirror).

You need to create a dataset

zfs create -p "$ZPOOL"/ROOT/archlinux

And you need to set the dataset's mount point:

zfs set mountpoint=/ "$ZPOOL"/ROOT/archlinux

You can then boot it with Grub using commands like:

insmod part_gpt
search --set --label mypool
linux /ROOT/archlinux@/boot/vmlinuz-linux zfs=mypool rw
initrd /ROOT/archlinux@/boot/initramfs-linux.img
boot

I scripted this to test it in a VirtualBox machine. I installed Arch from the ISO onto a normal ext4 root and then used that to instal a new ZFS root onto two mirrored virtual disks. The above is a summary - see the script for full details.

Related Solutions

freebsd – How to Create a GRUB Entry for FreeBSD on ZFS Installation

Never mind I found the answer thanks to the FreeBSD forums. This is what I needed in my grub.cfg:

menuentry "FreeBSD" --class freebsd --class bsd --class os {
    insmod zfs
    insmod bsd
    search -s -l zroot
    kfreebsd /@/boot/zfsloader
    kfreebsd_loadenv /@/boot/device.hints
}

To be clear I have tried this successfully, I even tried writing a file and rebooting to check it wasn't just read-only and the file was still there and had not been altered by the reboot in any way (not even its permissions were).

Linux – Recover/import ZFS pool from single old mirror

Well, I'm close and think I've found a path to recovery. Since I haven't received advice from others, I'll post what I've learned so far.

Summary:

There is an unmaintained, not-officially-supported labelfix utility for fixing labels on certain kinds of corrupted (and offline/detached) ZFS volumes, which can be used to make an unimportable pool importable.
Before doing anything, be sure to clone the old spare devices and only work on the clones.
If you have a situation as described in the question with two pools of the same name (due to a mistaken create or other error), make sure only devices are plugged in with the specific pool you want to recover.
Also, remove any devices that may have ever been associated with the pool you want to recover but which are faulted. (That applies even if you think you've completely destroyed any other pools and disassociated these devices. The recovery tools will try to piece together fragments of old pools, and may read old labels/uberblocks to combine devices and data in unpredictable ways.)

More details:

There appears to be a way to recover offline and detached drives from zpools on Linux. A user jjwhitney created a port of the labelfix utility I mentioned in the question, originally written by Jeff Bonwick (the inventor of ZFS) nearly 12 years ago. For reasons I cannot fathom, this utility has not been incorporated into ZFS builds, even though it would allow recovery of data for intact pools when an import fails for a number of reasons due to invalid labels. (Some discussion on the issue here.)

(Sidenote: One thing this process has led me to realize is that ZFS recovery tools are severely lacking, and no one should be using this filesystem for anything without complete backups of data running all the time. And don't depend on that old mirrored drive sitting in your closet to be a last-chance backup unless you're sure it's importable. ZFS is apparently great at maintaining data integrity when ZFS cooperates, but incredibly fragile. And when it breaks -- or you do something minor but stupid -- your data can simply all be inaccessible and unreadable, even if intact.)

In any case, the labelfix utility hasn't been updated in 5 years, so it doesn't compile with modern ZFS library files. Luckily, I had the original OS version still installed and could boot to that, then download an old ZFS on Linux source tarball and use that to get the appropriate ZFS libraries and build environment on a system where it all still works. (I started tweaking the labelfix utility to try to work with modern ZFS libraries, but that seemed a little dangerous given how little I understand about all the internals I'd need to fix to correspond to the current codebase. Easier to just build it on an old version.)

And lo, labelfix immediately and easily rewrote the label on my device to something that zpool import could at least interpret!

I should say that I used ddrescue to copy the whole thing from the original drive before attempting any of this. And I would highly recommend that, as it's possible to make mistakes, as I did. The original pool I accidentally wrote over was named backup, so zdb started seeing multiple versions of the different backup pools and couldn't figure out why all the metadata didn't match. I had to tweak vdev_validate_skip=1 in the ZFS kernel module to get the pool to import, but that then just imported the newer backup pool (not the one I wanted). Note that this happened even though I specified the exact path to the drive I wanted to import from: when forcing imports with this method, it seemed to completely ignore my specification and draw on a completely different configuration from a device that wasn't listed in the command.

Luckily, I had made another clone of the drive, so I could attempt another run. However, labelfix also is smart and seems to read the current drive configuration, so it picked up on the fact that I had two old drives with "corrupted data" from the first backup pool. The corruption unfortunately meant that the "fixed" label listed the pool not only as DEGRADED but also FAULTED and thus un-import-able.

At this point I realized I simply had to unplug all the old drives and work without them in the system at all to avoid corrupting recovery attempts. Unfortunately, labelfix only seems to fix things once, so I am now on to clone #3 of this drive (which is currently copying from my first backup clone). Once that cloning process finishes, I'll run labelfix without any of the other old drives present, and hopefully I'll get a DEGRADED pool that I can then import.

Best Answer

Related Solutions

freebsd – How to Create a GRUB Entry for FreeBSD on ZFS Installation

Linux – Recover/import ZFS pool from single old mirror

Related Question