Linux – GRUB Bootloader with root LUKS encryption: Only grub shell

bootloaderdisk-encryptiongrublinux

I would appreciate help and hints to solve a problem with the GRUB bootloader.
I have a system with root LUKS encryption. Means that except the partition with the bootloader, the whole system is in an encrypted LUKS container. I am unsure where the possible problems are, so I tell you some more details.

After a system update, GRUB now hangs in the GRUB shell (not rescue mode).

I can however start manually with:

insmod cryptodisk
insmod luks
insmod lvm
insmod ext2
cryptomount -a
set root=(lvm/system-root)
linux (lvm/system-boot)/vmlinuz-[...]-generic root=/dev/system/root
initrd (lvm/system-boot)/initrd.img-[...]-generic

Running lsblk on my system looks like this:

nvme0n1           259:0    0 953,9G  0 disk
├─nvme0n1p1       259:1    0   450M  0 part
├─nvme0n1p2       259:2    0   100M  0 part  /boot/efi
├─nvme0n1p3       259:3    0    16M  0 part
├─nvme0n1p4       259:4    0  97,9G  0 part
└─nvme0n1p5       259:5    0 839,4G  0 part
  └─system        253:0    0 839,4G  0 crypt
    ├─system-boot 253:1    0   512M  0 lvm   /boot
    ├─system-swap 253:2    0  16,1G  0 lvm   [SWAP]
    └─system-root 253:3    0 822,8G  0 lvm   /

p1, p3, p4 belong to an installed windows 10.
p2 is the fat32 EFI system partition
p5 is the luks encrypted system with boot, swap and root partition.

Maybe Grub cannot load the grub.cfg?

I thought Grub does not find the config file in the p2 partition in /boot/efi/grub/grub.cfg.

But if this would be the case, the manual execution of the config file should succeed. So I executed (used auto-completion, so the path should be right) in the grub shell at startup:

configfile (hd0,gpt2)/grub/grub.cfg

But i just get a cleared grub shell (without any errors).

Why I thought grub does not find the grub.cfg is because of the content of the grub config file in the EFI folder (which gets loaded first as I understood corecctly):
/boot/efi/EFI/ubuntu/grub.cfg:

search.fs_uuid db041d94-37fa-42a6-b2f3-87572a38f23c root lvmid/uWUnvj-b70J-Gfam-Tfr3-87I3-GQAP-bL2Lym/gEpImK-yuWK-W6Ip-v5KF-ZGKm-8GZS-ALhXxF 
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

Here the UUID(list at the end) it looks for is the one of theencrypted boot partition. This seems weired: Shouldn't it be visible for grub till I entered the password?.

But as my command to load the config file manually did not work I guess this is not the issue for my problem.

Checking /etc/default/grub

So it could be my configuration file:
I cannot judge how right this file is, so here is the summarized content of the file:

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=menu
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_ENABLE_CRYPTODISK=y

GRUB_CMDLINE_LINUX="cryptdevice=UUID=8d82f654-43fd-40ba-a185-1ead48838f54:system root=/dev/system/system-root resume=/dev/system/system-swap ro"

GRUB_PRELOAD_MODULES="luks cryptodisk lvm ext2"

To me this does not look too bad. The UUID of the cryptdevice looks fine.
I tried changed the "/dev/system/system-root" to
"/dev/system/system" as this is the path while running the system, but without any effect.
i also did run update-grup and grub-install.

In addition I installed grub-emu (emulator for the grub bootloader).
This did show me at least the gurb menu. When selecting the entry to start ubuntu I get at least some errors:

error: no such cryptodisk found.
error: no such device: db041d94-37fa-42a6-b2f3-87572a38f23c.
error: can't find command `linux'.
error: can't find command `initrd'.

What would help me

Of course point me to the problem and telling me the fix.
As I now have to deal with it, i am curious about the boot process with grub:
I do not understand the content of the /boot/efi/EFI/ubuntu/grub.cfg.
What is the content of ($root) here? And why is it looking for an encrypted partition?
Where is it determined what content is in this file?
Why does it not show me at least the grub menu even if there are errors that would prevent the system to boot?

Thank you very much,
Ecco

Additional content

The content of /boot/efi/grub/grub.cfg can be found at https://pastebin.com/D1X7jPsS

UUIDs (i omitted the windows specific partitions)

/dev/nvme0n1p2: UUID="2885-45AC" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="1043df12-d11d-4023-bf6a-6acc2919c06f"
/dev/nvme0n1p5: UUID="8d82f654-43fd-40ba-a185-1ead48838f54" TYPE="crypto_LUKS" PARTLABEL="system" PARTUUID="2457b24a-7695-4737-ae46-5687aca3ab01"

UUIDs of the LUKS container

/dev/mapper/system: UUID="X7XHJG-pQ21-edgj-d73H-kK8Z-DxM8-eXBCh6" TYPE="LVM2_member"
/dev/mapper/system-root: LABEL="root" UUID="07e1d1e4-cd00-42dd-96cf-4f99864d0b7f" TYPE="ext4"
/dev/mapper/system-swap: UUID="02664e3e-3ba7-4a03-8b03-eb0335ca40f9" TYPE="swap"
/dev/mapper/system-boot: LABEL="boot" UUID="db041d94-37fa-42a6-b2f3-87572a38f23c" TYPE="ext4"

Best Answer

Finally it works (again): The cmd refreshgrub did the trick This command changed the content of the file /boot/efi/EFI/ubuntu/grub.cfg to:

search.fs_uuid 2885-45AC root 
set prefix=($root)'/EFI/ubuntu/grub'
configfile $prefix/grub.cfg

This makes more sense than before: Searching for the unencrypted partition. But now the bigger grub.cfg file is stored in another folder, which is ok.

It was now showing the menu, but the boot failed, I had to change in the /etc/default/grub.cfg the following line:

GRUB_CMDLINE_LINUX="cryptdevice=UUID=8d82f654-43fd-40ba-a185-1ead48838f54:system root=/dev/system/root resume=/dev/system/swap ro"

Related Solutions

Uefi and full disk encryption with lvm on luks

Yeah, i know it's a pretty late answer but, better late than never...

I don't know if Debian has the tools to do it, but using Arch Linux you can create a disk layout like this:

EFI partition(mounted /boot/efi) with grub EFI bootloader, formated fat32 EFI type partition EF00. Could be your /dev/sda1. This partition will only hold grub stub, to ask password to mount your boot partition.
boot partition(mounted /boot) that is a luks crypto device. After crypto unlocking this partition, you can format it using any filesystem that grub supports to but(ext4 for example). This will be your /dev/sda2
Crypto device that will store all remaining partitions as logical volumes. Crypto device, with LVM and its logical volumes(3 layers). This will be your /dev/sda3.
- Here, you can create as many logical volumes as you want/need. The key that unlocks this partition will be used to access data on all it's logical volumes.

Borrowing from Arch Wiki, this is how your disk layout will looks like:

+---------------+----------------+----------------+----------------+----------------+
|ESP partition: |Boot partition: |Volume 1:       |Volume 2:       |Volume 3:       |
|               |                |                |                |                |
|/boot/efi      |/boot           |root            |swap            |home            |
|               |                |                |                |                |
|               |                |/dev/store/root |/dev/store/swap |/dev/store/home |
|/dev/sdaX      |/dev/sdaY       +----------------+----------------+----------------+
|unencrypted    |LUKS encrypted  |/dev/sdaZ encrypted using LVM on LUKS             |
+---------------+----------------+--------------------------------------------------+

Caveats:

Grub will ask for a password to unlock /boot, initial ram disk will ask for a password AGAIN(cause for him, /boot is locked), and probably while mounting your root partition this will happen once more. The trick here is to use a master key inside your /boot(and maybe inside your initrd with the FILES= option of mkinitcpio and add it with luksAddKey. Your boot partition is encrypted so, there is no need to be worried cause the key is inside an encrypted partition. chmod 000 keyfile.bin is your friend.
Add encrypt lvm2 to mkinitcpio hooks.
If for some reason your system is not able to use the key, a password will be asked again.
You will still be vulnerable to Evil Maid attacks that explore Cold Boot failures. The best you can do here is:
- Enable Secure Boot.
- Sign your Grub EFI.
- Revoke Microsoft CA on your Motherbord(you know, can't trust anyone).
- Remember to Sign you grub efi whenever you have a grub-efi package update.

Further Reading:

After some research about the need of keeping boot partition outside lvm(as far as i know, grub-pc/bios had lvm and luks modules) i found this guy on Arch Forums confirming that there is no need to keep /boot on a spare crypto partition.

You can install grub on your ESP, and also tell it to read the configuration files from esp like this:

# grub-install --target=x86_64-efi --efi-directory=esp --bootloader-id=grub --boot-directory=esp

after that, regenerate conf:

# grub-mkconfig -o esp/grub/grub.cfg

And, it seems that grub is the only bootloader that have support to boot lvm+crypto without a separated /boot partition.