How to debug systemd “init” freeze during boot

bootinit-scriptinitramfssystemd

I'm trying to get my own init script running on some ARM hardware and Debian Jessie with systemd. The overall boot concept and init script is almost identical to a x86 variant, which works fine. For both hardwares the whole SD card image is pre-built on a x86 host.

When running on the ARM hardware, having serial console access I see that my init script runs fine up to the point where switch_root is called:

exec switch_root -c /dev/console /newroot /sbin/init

Nothing happens after that. No error message is printed – which makes it difficult to find the problem.

The kernel command line is…

ro root=LABEL=IM_BOOT1 panic=10 mem=256M console=ttyS0,115200 systemd.log_level=debug

…and AFAIK this should make systemd print maximum debug messages. Unfortunately nothing is printed.

Any idea/hint what I could do to understand what could be causing the freeze, or with other words – understand what is going on?

The only strange detail are some warnings in the kernel log before calling switch_root:


<snip>
ext4: Unknown symbol jbd2_journal_errno (err 0)
ext4: Unknown symbol jbd2_journal_begin_ordered_truncate (err 0)
ext4: Unknown symbol jbd2_journal_flush (err 0)
ext4: Unknown symbol mb_cache_entry_find_next (err 0)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
aufs 3.16-20150928
usbhid: Unknown symbol hid_output_report (err 0)
usbhid: Unknown symbol hidinput_count_leds (err 0)
usbhid: Unknown symbol hid_allocate_device (err 0)
usbhid: Unknown symbol hid_destroy_device (err 0)
usbhid: Unknown symbol hid_alloc_report_buf (err 0)
usbhid: Unknown symbol hid_set_field (err 0)
usbhid: Unknown symbol hid_check_keys_pressed (err 0)
usbhid: Unknown symbol hid_input_report (err 0)
usbhid: Unknown symbol hid_debug (err 0)
usbhid: Unknown symbol __hid_request (err 0)
usbhid: Unknown symbol hid_parse_report (err 0)
usbhid: Unknown symbol hid_add_device (err 0)
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
usb_storage: Unknown symbol scsi_report_device_reset (err 0)
usb_storage: Unknown symbol scsi_remove_host (err 0)
usb_storage: Unknown symbol scsi_report_bus_reset (err 0)
usb_storage: Unknown symbol scsi_sense_desc_find (err 0)
usb_storage: Unknown symbol scsi_eh_prep_cmnd (err 0)
usb_storage: Unknown symbol scsi_host_put (err 0)
usb_storage: Unknown symbol scsi_scan_host (err 0)
usb_storage: Unknown symbol scsi_is_host_device (err 0)
<snip>

…however these modules appear to work fine anyway. I don't know if that is related.

Some additional details:

  • Debian installation is prepared using debootstrap (same method works perfectly with x86)
  • Custom built Kernel: Linux (none) 3.16.7 #1 Mon Nov 16 08:32:55 UTC 2015 armv5tejl GNU/Linux, patched with AUFS and hardware-specific device tree
  • both Kernel and root fs are built with help of QEMU on a x86 host
  • initramfs boots with BusyBox, sets up the AUFS final root fs and finally invokes switch_root. The same method (nearly 100% identical init script) works fine on x86
  • I can run an interactive shell just before running switch_root and I can't find anything that isn't working as expected at that point

Best Answer

It turned out that the Kernel was missing some important settings, that are required by systemd.

http://cgit.freedesktop.org/systemd/systemd/tree/README shows what Kernel config settings were necessary.

Once configured, systemd started working immediately.

Related Question