Ubuntu – Ubuntu 18.04 on kernel 4.15.0-34-generic boots to BusyBox

18.04bootencryptionkernelluks

The machine was a fresh install of Ubuntu 18.04.1 and used LVM and LUKS during install.
The machine was working fine on kernel 4.15.0-33-generic.

The kernel upgraded from 4.15.0-33 to 4.15.0-34-generic.
After the kernel updated, it no longer prompts for LUKS passphrase and boots straight to BusyBox v1.27.2

When booting from the older kernel, it prompts for LUKS like normal and the machine is able to boot properly.
The issue is when booting from the newer kernel.

I tried to run fsck from a liveCD of 18.04.
I unlocked the drive first then ran 

fsck -f /deb/sda3

However I only get

fsck from util-linux 2.31.1

and nothing else.

Is there a fix that I can run?

Best Answer

The problem is that /sbin/cryptsetup is not in initramfs file /boot/initrd.img-4.15.0-36-generic.

You need to add CRYPTSETUP=y in /etc/cryptsetup-initramfs/conf-hook. You also need to add the initramfs option in /etc/crypttab. Then, you have to run update-initramfs, which solves this problem.

Detailed solution:

  1. In an other Ubuntu machine, use "Startup Disk Creator" to create a bootable Ubuntu 18.04 USB drive
  2. Plug in this USB drive in a USB port of the computer that boots to BusyBox

  3. When the live USB Ubuntu is ready, connect to Internet (WiFi or wired)

  4. Open a terminal

    ubuntu@ubuntu:~$ sudo su

  5. Find the encrypted block device.

    root@ubuntu:~# blkid|grep LUKS
/dev/nvme0n1p3: UUID="4b206e76-1531-48ae-95be-ae0ed7a244c1" TYPE="crypto_LUKS" PARTUUID="21db499d-b87b-41c6-864f-04d1531cb083"

  6. Decrypt the device

    root@ubuntu:~# cryptsetup open UUID="4b206e76-1531-48ae-95be-ae0ed7a244c1" nvme0n1p3_crypt
Enter passphrase for /dev/disk/by-uuid/4b206e76-1531-48ae-95be-ae0ed7a244c1:

  7. List mapped devices. control is a file used to send command to the device mapper. nvme0n1p3_crypt is the decrypted device. ubuntu--vg-root is a LVM logical volume in the LVM ubuntu-vg volume group.

    root@ubuntu:~# ls /dev/mapper/*
/dev/mapper/control  /dev/mapper/ubuntu--vg-root  /dev/mapper/nvme0n1p3_crypt

  8. Create mount point. This is where we will mount the / of our system that won't boot.

    root@ubuntu:~# mkdir -p /mnt/ubuntu-root

  9. Mount root logical volume

    root@ubuntu:~# mount /dev/mapper/ubuntu--vg-root /mnt/ubuntu-root/

  10. Mount pseudo file systems

    root@ubuntu:~# mount -o bind /sys /mnt/ubuntu-root/sys
root@ubuntu:~# mount -o bind /proc /mnt/ubuntu-root/proc
root@ubuntu:~# mount -o bind /dev /mnt/ubuntu-root/dev

  11. Copy DNS information

    root@ubuntu:~# cp /etc/resolv.conf /mnt/ubuntu-root/etc/

  12. Change root

    root@ubuntu:~# chroot /mnt/ubuntu-root/

  13. Mount /boot, which contains the initramfs file. This partition is unencrypted.

    root@ubuntu:/# mount /boot/

  14. Install binwalk (to see the content of the init ram file system)

    root@ubuntu:~# apt update
root@ubuntu:~# apt install binwalk

  15. Find offset of gzipped initramfs content

    root@ubuntu:~# binwalk /boot/initrd.img-4.15.0-36-generic | grep gzip
1605632       0x188000        gzip compressed data, from Unix, last modified: 2018-10-18 13:00:32

  16. The problem is that the initramfs file system does not contain cryptsetup. So, that is why there is no LUKS password prompt.

    root@ubuntu:/# cd /root/
root@ubuntu:~# mkdir initramfs-4.15.0-36-generic
root@ubuntu:~# cd initramfs-4.15.0-36-generic
root@ubuntu:~/initramfs-4.15.0-36-generic# dd if=/boot/initrd.img-4.15.0-36-generic bs=1605632 skip=1 | gunzip | cpio -i
root@ubuntu:~/initramfs-4.15.0-36-generic# ls sbin/cryptsetup
ls: cannot access 'sbin/cryptsetup': No such file or directory

  17. To decrypt root at boot, the initramfs needs to contain:

    sbin/cryptsetup
lib/modules/4.15.0-36-generic/kernel/drivers/md/dm-crypt.ko

  18. Add the initramfs option in /etc/crypttab

    root@ubuntu:/# cat /etc/crypttab 
nvme0n1p3_crypt UUID=4b206e76-1531-48ae-95be-ae0ed7a244c1 none luks,discard,initramfs

  19. Add CRYPTSETUP=y in /etc/cryptsetup-initramfs/conf-hook

  20. Run update-initramfs:

    root@ubuntu:~# update-initramfs -k 4.15.0-36-generic -c -v &> update-initramfs-4.15.0-36-generic.cryptsetup.log

  21. Now, we have cryptsetup in initramfs and the Linux kernel module dm-crypt.ko too:

    root@ubuntu:~# grep /sbin/cryptsetup update-initramfs-4.15.0-36-generic.cryptsetup.log
Adding binary /sbin/cryptsetup

root@ubuntu:~# grep dm-crypt.ko update-initramfs-4.15.0-36-generic.cryptsetup.log
Adding module /lib/modules/4.15.0-36-generic/kernel/drivers/md/dm-crypt.ko

  22. Now, run update-initramfs, without -c (new) and without -v (verbose):

    root@ubuntu:~# update-initramfs -k 4.15.0-36-generic -u

  23. Verify that the initramfs is actually correct

    root@ubuntu:~# binwalk /boot/initrd.img-4.15.0-36-generic | grep gzip
1605632       0x188000        gzip compressed data, from Unix, last modified: 2018-10-18 14:26:29

root@ubuntu:~# dd if=/boot/initrd.img-4.15.0-36-generic bs=1605632 skip=1 2> /dev/null | gunzip | cpio -t 2> /dev/null |grep sbin/crypt 
sbin/cryptsetup

root@ubuntu:~# dd if=/boot/initrd.img-4.15.0-36-generic bs=1605632 skip=1 2> /dev/null | gunzip | cpio -t 2> /dev/null |grep dm-crypt.ko
lib/modules/4.15.0-36-generic/kernel/drivers/md/dm-crypt.ko

  24. Now, reboot.

Related Question