On Gentoo, ZFS stops working after 0.6.5 update

gentooinitramfszfs

I just updated my Gentoo system, updating my world, including ZFS. Everything but the kernel. I figured I'd do a reboot first, then build a new kernel. Currently, it's running on kernel 3.17.8-gentoo-r1.

My kernel is supported by ZFS. Also, in the past I've been able to do an update from ZFS 0.6.2 to 0.6.3. Today, I went from said 0.6.3 to 0.6.5.4.

Since everything but /boot is on ZFS, I can't boot anymore and get stuck in a half-life. No kernel panic, but empty filesystems. Whenever I try any zfs/zpool operation, I get an error:

# zpool status
internal error: Success
Aborted

Amusing though that success-statement may be, I'm adrift.

Calling it by name gives a different error:

# zpool status system
zpool: libzfs_status.c:201: Assertion `nvlist_lookup_uint64(config, ZPOOL_CONFIG_VERSION, &version) == 0' failed.
Aborted

At first I remembered that ZFS checks for the kernel version to compile itself against that kernel. Also that ZFS is a module.
So I already did a make modules and make modules_install. Do I need to rebuild my kernel and initramfs? The second one is annoying, since genkernel has a bug. But it can be done.

I'm currently chrooted into the system (with a mounted ZFS) using Funtoo's ZFS rescue CD. FYI, same errors happen in the chrooted environment. The 'host'-environment has no issues with the volumes.

I will continue to search for answers. I did, however, wish to pose the question at this time since you may be able to reduce my time searching. The exact procedure/order for a ZFS update hasn't exactly been documented very well.

PS It's running on SystemD, if that matters. I don't believe it would.

Best Answer

Well, it was indeed the part I didn't look forward to. Having updated all modules, there was only one place they hadn't been updated yet. As soon as I updated the initrd-file she booted fine.

So the process would be:

  1. (Optional) Snapshot root-filesystem.
  2. Emerge ZFS updates.
  3. If you use an initramfs, update/rebuild it to get the new modules in.
  4. Reload the kernel module. If your root-fs is ZFS, that means you reboot. If it's not, rebooting is still the simplest.

Now if you also have a kernel update to do, I would suggest doing it after updating ZFS. That will mean you're doing certain things twice, but for me it's better than wondering if it was a new option.

When you've updated the kernel, after generating the kernel image, but before generating the initrd-file, do an emerge -va @module-rebuild, which creates the modules for the new kernel. Modules that will be tucked away into your initrd-file.

So that's it. I just wish there was documentation out there...

During the process I did also find out Genkernel-Next does not like it when you compress the modules, so I had to deactivate that feature. And compressing the kernel with LZ4 works fine, but as soon as I do the same to the InitRD, she panics.

So Genkernel created 12 different InitRD-files that I had to manually break open to change one single line or it wouldn't work. But it works now. Running on 4.1.15. And those ZFS performance increases are noticeable. I'll let it run for a while before removing all the old.

Now if someone can tell me if I can upgrade the pool features yet without becoming unbootable, I'd appreciate it.

Related Question