Alternatives to Debian Live for Persistent USB Systems

debianlive-usbpersistence

For the last six years, my main workstation has consisted of a pen drive running the Debian Live images with a persistent partition. The images were simple, brilliant and reliable, and the online web builder for images was perfect for my use.

Recently I was looking to update my core system and discovered that Debian Live has undergone an "abrupt end." Both that article and other mails mention alternatives; some imply that live.debian.net is still active, but it just redirects to the main Debian wiki, which in turn only refers to the official CD images. Another article mentions that vmdebootstrap is being updated to be the replacement for live-build and other Debian Live tools, but I can't find any useful documentation on that either. And no one seems to be running a web image builder any more.

Can someone point me to alternatives? In an ideal world, there would be some straightforward workflow to produce custom images similar to those that Debian Live used to make possible, and with the kernel options that it supported (some of which are very useful in a persistent USB situation). Is that possible in Debian any more? Can someone point me to a sequence of steps for that?

Best Answer

Alternatives to Debian Live for persistent Debian system on USB

You can create a debian Live USB with persistence using the mkusb tool :

How to install mkusb in Debian

These methods are tested in Debian Wheezy. It contains an instruction to install the ppa 'manually'. There is also an alternative to download the file(s) and check the download manually.

The mkusb is tested and work fine on debian jessie and debian Stretch. There is no dependencies problem.

To install mkusb , add the following line to your /etc/apt/sources.list

deb http://ppa.launchpad.net/mkusb/ppa/ubuntu xenial main

Import the gpg key :

apt-key adv --keyserver keyserver.ubuntu.com --recv 54B8C8AC

Update and install mkusb:

apt-get update
apt-get install mkusb

The command line tool is mkusb-nox (txt mode) , the GUI is mkusb.

The are a few steps to create a Debian Live Persistent USB using the mkusb GUI :

Download the Debian Live ISO from the official website.

Run mkusb from the terminal. Choose d option then validate:

d:  dus , guidus, mkusb-dus    - New, easy to use

Choose : Install (make a boot device)
Choose p : 'Persistent live' only Debian and Ubuntu
Select your debian.iso file and validate
Select your USB device
Select upefi : usb-pack-efi (default grub from ISO file) then validate
Choose the percentage reserved to your persistent partition then validate
Select GO and validate (Yes , i want to go ahead), your USB will be formatted and partitioned

The 5 partitions :

partition 1 - ntfs 'usbdata'
partition 2 - bios_grub
partition 3 - fat32 boot,efi
partition 4 - iso9660 - cloned system
partition 5 - ext4 - 'casper-rw'

You can reboot into your Persistent USB when you receive the following message :

 Done :-) 
 The target device is ready to use.

Feeling lucky

If you feel particularly lucky today, you can try a bash script automating the process for you. Give it your ISO image path as first parameter and the USB drive block device name as the second. Note that this script is insanely dangerous and that you should not execute it without reading and understanding it first.

TL;DR

Get Debian live ISO image, then do the following:

umount /dev/sdX*
parted /dev/sdX --script mktable gpt
parted /dev/sdX --script mkpart EFI fat16 1MiB 10MiB
parted /dev/sdX --script mkpart live fat16 10MiB 3GiB
parted /dev/sdX --script mkpart persistence ext4 3GiB 100%
parted /dev/sdX --script set 1 msftdata on
parted /dev/sdX --script set 2 legacy_boot on
parted /dev/sdX --script set 2 msftdata on

mkfs.vfat -n EFI /dev/sdX1
mkfs.vfat -n LIVE /dev/sdX2
mkfs.ext4 -F -L persistence /dev/sdX3

mkdir /tmp/usb-efi /tmp/usb-live /tmp/usb-persistence /tmp/live-iso
mount /dev/sdX1 /tmp/usb-efi
mount /dev/sdX2 /tmp/usb-live
mount /dev/sdX3 /tmp/usb-persistence
mount -oro live.iso /tmp/live-iso

cp -ar /tmp/live-iso/* /tmp/usb-live

echo "/ union" > /tmp/usb-persistence/persistence.conf

grub-install --no-uefi-secure-boot --removable --target=x86_64-efi --boot-directory=/tmp/usb-live/boot/ --efi-directory=/tmp/usb-efi /dev/sdX

dd bs=440 count=1 conv=notrunc if=/usr/lib/syslinux/mbr/gptmbr.bin of=/dev/sdX
syslinux --install /dev/sdX2

mv /tmp/usb-live/isolinux /tmp/usb-live/syslinux
mv /tmp/usb-live/syslinux/isolinux.bin /tmp/usb-live/syslinux/syslinux.bin
mv /tmp/usb-live/syslinux/isolinux.cfg /tmp/usb-live/syslinux/syslinux.cfg

sed --in-place 's#isolinux/splash#syslinux/splash#' /tmp/usb-live/boot/grub/grub.cfg

sed --in-place '0,/boot=live/{s/\(boot=live .*\)$/\1 persistence/}' /tmp/usb-live/boot/grub/grub.cfg /tmp/usb-live/syslinux/menu.cfg

sed --in-place '0,/boot=live/{s/\(boot=live .*\)$/\1 keyboard-layouts=de locales=en_US.UTF-8,de_DE.UTF-8/}' /tmp/usb-live/boot/grub/grub.cfg /tmp/usb-live/syslinux/menu.cfg

umount /tmp/usb-efi /tmp/usb-live /tmp/usb-persistence /tmp/live-iso
rmdir /tmp/usb-efi /tmp/usb-live /tmp/usb-persistence /tmp/live-iso

In Detail and with some explanation

You will need to execute most of the following commands with elevated privileges, i.e., using sudo on most GNU/Linux systems.

Download

Download a Debian live ISO image with the window manager of your choice:

https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/

We'll refer to the downloaded ISO image simply as "live.iso".

Determine target drive

Find the device that is your USB drive using lsblk. We'll call that /dev/sdX.

Unmount

Unmount existing partitions on your drive using umount /dev/sdX*

Create partitions

We need an EFI boot partition for UEFI PCs to boot from the USB drive. Then we need a sufficiently large partition to hold the original live ISO filesystem image contents. That partition must have the legacy_boot flag set. Then we add the persistence partition, using up all the remaining space of the USB drive. You can do that with any GPT capable partitioning tool (mind the legacy_boot flag). Here is an example using parted:

parted /dev/sdX --script mktable gpt
parted /dev/sdX --script mkpart EFI fat16 1MiB 10MiB
parted /dev/sdX --script mkpart live fat16 10MiB 3GiB
parted /dev/sdX --script mkpart persistence ext4 3GiB 100%
parted /dev/sdX --script set 1 msftdata on
parted /dev/sdX --script set 2 legacy_boot on
parted /dev/sdX --script set 2 msftdata on

This creates a GPT partition table and a protective MBR partition table.

Create Filesystems

We want FAT on the EFI and live partition and we want ext4 on the persistence parition and we require the label persistence for the persistence feature to work.

mkfs.vfat -n EFI /dev/sdX1
mkfs.vfat -n LIVE /dev/sdX2
mkfs.ext4 -F -L persistence /dev/sdX3

Mounting the resources

We'll need to mount the source ISO and target partitions at temporary mount points.

mkdir /tmp/usb-efi /tmp/usb-live /tmp/usb-persistence /tmp/live-iso
mount /dev/sdX1 /tmp/usb-efi
mount /dev/sdX2 /tmp/usb-live
mount /dev/sdX3 /tmp/usb-persistence
mount -oro live.iso /tmp/live-iso

Install live system

Copy the live ISO filesystem content to the LIVE partition.

cp -ar /tmp/live-iso/* /tmp/usb-live

persistence.conf

Prepare the persistence filesystem with the required configuration file. The persistence feature will not work without this file.

echo "/ union" > /tmp/usb-persistence/persistence.conf

Grub for UEFI support

Install grub2 for UEFI booting support (this requires the grub-efi-amd64-bin package on Debian). We force grub-install to not use UEFI secure boot, which apparently does not work with the --removable option.

grub-install --no-uefi-secure-boot --removable --target=x86_64-efi --boot-directory=/tmp/usb-live/boot/ --efi-directory=/tmp/usb-efi /dev/sdX

Syslinux for legacy BIOS support

Install syslinux gptmbr.bin bootloader to the drive (download syslinux or install package syslinux-common). Then install syslinux to the live partition.

dd bs=440 count=1 conv=notrunc if=/usr/lib/syslinux/mbr/gptmbr.bin of=/dev/sdX
syslinux --install /dev/sdX2

Isolinux fixup

Reuse the isolinux config of the original live ISO to work with syslinux.

mv /tmp/usb-live/isolinux /tmp/usb-live/syslinux
mv /tmp/usb-live/syslinux/isolinux.bin /tmp/usb-live/syslinux/syslinux.bin
mv /tmp/usb-live/syslinux/isolinux.cfg /tmp/usb-live/syslinux/syslinux.cfg

Kernel parameters

Now that we copied the live system files to an actual read-write filesystem, we can manipulate the grub and syslinux config.

Add the persistence kernel parameter to menu.cfg and grub.cfg. In both files, add the keyword persistence at the end of the respective first line with boot=live in it.

sed --in-place '0,/boot=live/{s/\(boot=live .*\)$/\1 persistence/}' /tmp/usb-live/boot/grub/grub.cfg /tmp/usb-live/syslinux/menu.cfg

Set the keyboard-layout kernel parameter. In both files, add the keywords at the end of the respective first line with boot=live in it.

sed --in-place '0,/boot=live/{s/\(boot=live .*\)$/\1 keyboard-layouts=de locales=en_US.UTF-8,de_DE.UTF-8/}' /tmp/usb-live/boot/grub/grub.cfg /tmp/usb-live/syslinux/menu.cfg

Grub splash

Fix the grub splash image (optional; we moved it into another directory).

sed --in-place 's#isolinux/splash#syslinux/splash#' /tmp/usb-live/boot/grub/grub.cfg

Unmounting and Cleanup

umount /tmp/usb-efi /tmp/usb-live /tmp/usb-persistence /tmp/live-iso
rmdir /tmp/usb-efi /tmp/usb-live /tmp/usb-persistence /tmp/live-iso

Why this should work for both UEFI and BIOS

When starting in UEFI mode, the PC will scan the FAT partitions we defined in the GPT partition table. The first FAT partition carries the UEFI grub bootloader, which is found because it is located in the path specified by UEFI for removable drives (the --removable switch to grub-install did this). No UEFI boot entry is necessary for that to work, we only need to make the PC try to boot from the USB drive. That grub is configured to take it from there (load the grub.cfg, show the menu, etc.).

When starting in BIOS mode and selecting to boot from the USB drive, the PC will execute the gptmbr.bin bootloader code we have written to the protective MBR of the USB drive. That bootloader looks for the GPT partition marked with the legacy_boot flag and chainload syslinux from that partition. Syslinux then takes over (load menu.cfg, show the menu, etc.).

Encrypted Persistence

Instead of using plain ext4 on the persistence partition, one could first encrypt the persistence partition with LUKS (using cryptsetup), then format that with ext4 (using the proper label). However, as the documentation says, the live system must include the cryptsetup package. Otherwise, the encrypted partition cannot be decrypted by the live system. This means one has to build a custom live ISO first. That is, however, out of the scope of this answer.

History

The --no-uefi-secure-boot option was previously not part of the call to grub-install. The stick worked fine for me, but that stopped with Debian buster, even though secure boot is still disabled on my machine.