Debian Jessie – Fixing Boot Hanging Issues

bootdebiangrublvmudev

After installing kernel 4.6, we booted a development server with Debian 8.5 Jessie. It does not go past the error message "Setting up LVM Volume Groups".

The VM has Debian 8, with backports repository, booting with sysV instead of systemd, and using LVM.

I am able to boot it with a knoppix live image, and make the 6 LVM partitions in the volume group appear with:

vgchange -ay

I am also able to mount the LVM partitions, and edit them, so it does not seem any kind of LVM problem.

I also took advantage of this to mount all the partitions in their natural order, including mount binding proc, sys and dev, and thus running a chroot to run more naturally other debugging/fix commands down the line.

Already tried changing the booting algorithm to CONCURRENCY=none / legacy without much success.

Also regenerated the initrd file, has the server had disk space filled up while upgrading the kernel, with the command:

sudo dpkg-reconfigure linux-image-4.6.0-0.bpo.1-amd64

It also did not make any difference.

I also reinstalled grub, using:

sudo update-grub
sudo grub-install /dev/sda

It also did not work.

I also added debug options to the kernel options in grub, however, the system, upon boot neither prints relevant errors, nor prints any additional message after the error.

There are also not neither any logs nor dmesg logs to inspect, as by this time at boot, syslog is still not working.

Best Answer

I ended up locating the offending script as /etc/rcS.d/S05lvm2 and the command that halted the VM in that script as /sbin/lvm vgchange -aay --sysinit >/dev/null

As it can be seen in the source file, the dependencies of S05lvm2 are:

# Should-Start:      udev mdadm-raid cryptdisks-early multipath-tools-boot

Ultimately it dawned on me someone or something had disabled the udev daemon at startup (a short while ago after the last successful boot). I also determined it was not the kernel post install routines disabling udev.

The init script(s) not having the udev service dependency satisfied stalled either waiting for the a dependency that was never satisfied, or for a device that never appeared (or both).

The problem was fixed running in the aforementioned chroot:

sudo chkconfig udev on

(has we have installed chkconfig). In the Debian way, it would be:

sudo update-rc.d udev defaults

Next for remaking the initrd file (Debian trick for easily regenerating the initrd file):

sudo dpkg-reconfigure linux-image-4.6.0-0.bpo.1-amd64
Related Question