Debian – Startup services on Debian 9

The problem

It turns out that my problem is a known issue between the latest Intel microcode on (some?) Skylake CPUs and recent Linux kernels, which is mainly triggered by sssd. See Ubuntu bug #1759920 “intel-microcode 3.20180312.0 causes lockup at login screen(w/ linux-image-4.13.0-37-generic)”, and also a number of other bugs which turn out to be about the same issue, such as Ubuntu bug #1746806 “sssd appears to crash AWS c5 and m5 instances, cause 100% CPU” and Ubuntu bug #1746418 “System freezes when starting Xorg after installing linux-image-4.13.0-32-generic”. You are likely to encounter this bug if:

• You have a very recent Intel CPU. As far as I can tell, this bug only arises on Skylake CPUs.
• You have the intel-microcode package installed. Reverting to an earlier, tested kernel didn't work for me because I'd only run that kernel with an earlier microcode.
• Your computer is connected to a corporate network (typically LDAP or Active Directory) for user authentication. Although there are other ways to trigger the bug, running sssd seems to be the most common culprit. There are also reports of Xorg crashing.

The bug is due to mitigations for the Spectre security issue that was published in January 2018. There's an incompatibility between some kernel code and some processor microcode that causes a lock-up in certain circumstances.

How to repair

1. If you can't boot normally, you'll need to edit the kernel command line at the Grub prompt. See the question for explanations and possible ways to get a root shell.
2. A workaround for this specific bug is to add the noibpb parameter to the kernel command line (1746418/14, 1759920/56). This should let you boot normally and perform some repairs.
   This disables the vulnerability mitigation that causes the problem, which means that your computer is now vulnerable to some attacks. They're local attacks, i.e. the attacker needs to run code on your machine, but these attacks may potentially be carried out e.g. through JavaScript in a web browser.
   If you don't have any other way, you can make this permanent by adding noibpb to the kernel command line until you can get a fixed kernel.
3. In Ubuntu, the fix is expected on the week of 23 April 2018, in what will presumably be kernel 4.4.0-117 and 4.13.0-39. In the meantime, Tyler Hicks has published test kernels for 4.4 and 4.13.

How I diagnosed the issue

I tried several things (see the question) and determined that the bug was triggered somewhere between reaching basic.target and reaching multi-user.target. So I set the default systemd target to basic.target (systemctl set-default basic.target) and enabled the debug-shell service (systemctl enable debug-shell) to get a root shell.

I ran systemctl list-dependencies multi-user.target and manually started the listed dependencies one by one. This did not trigger the crash.

Not all services are managed directly by systemd. Some are managed as Upstart services and some are managed as SysVinit scripts. The shell script below runs all of them. Note: I only tested it once, and it crashed by design.

#!/bin/sh
wants=$(systemctl show -p Wants multi-user.target | sed 's/^Wants=//' | tr ' ' '\n' | sort)
log=/var/tmp/multi-user-steps-$(date +%Y%m%d-%H%M%S)

log () {
  echo "$* ..." | tee -a "$log"
  sync
  "$@"
  ret=$?
  echo "$* -> $ret" | tee -a "$log"
  sync
  return $ret
}

# systemd services
for service in $wants; do
  log systemctl start $service
  sleep 2
done

# upstart services
for conf in /etc/init/*.conf; do
  service=${conf##*/}; service=${service%.conf}
  log service ${service} start
  sleep 2
done

# sysvinit services
for service in /etc/rc3.d/S*; do
  log ${service} start
  sleep 2
done

My computer crashed after starting sssd. From there, a web search on “sssd linux kernel hang” led me to https://bugs.launchpad.net/cloud-images/+bug/1746806 and to the diagnosis and solution.