I recently upgraded systemd from version 208 to 212 (also tried any other version between these two numbers)
And the system can't boot anymore. Looks like it cannot mount /tmp, but in emergency shell mount /tmp just works fine, so does every other file system. (including a LUKS volume)
Pretty sure this is not about initramfs or kernel modules, if I simply downgrade systemd again, everything is just fine. Also, mkinitcpio -p linux-custom doesn't help either
UPDATE
Looks like dbus connection is broken, when I try to run systemd-analyze blame it says failed to get dbus connection or something
EDIT
Other method I tried:
-
Disable plymouth target (not even present in systemctl list-units)
-
Disable the
tmp.mounttarget, does not help either
Logs
I'm running Arch Linux, here's the fstab structure about tmpfs:
/dev/zram0 /tmp tmpfs nodev,size=512M,nosuid,noatime,nodiratime 0 0
/dev/zram0 /var/tmp tmpfs nodev,size=256M,nosuid,noexec,noatime,nodiratime 0 0
Here's the attached journal log (only systemd part)
systemd-journal[198]: Runtime journal is using 8.0M (max allowed 388.0M, trying to leave 582.1M free of 3.7G available → current limit 388.0M).
systemd-journal[198]: Runtime journal is using 8.0M (max allowed 388.0M, trying to leave 582.1M free of 3.7G available → current limit 388.0M).
systemd-udevd[148]: starting version 212
systemd[1]: RTC configured in localtime, applying delta of 480 minutes to system time.
systemd 212 running in system mode. (+PAM -AUDIT -SELINUX -IMA -SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ +SECCOMP -APPARMOR)
systemd[1]: Detected architecture 'x86-64'.
systemd[1]: Failed to insert module 'ipv6'
systemd[1]: No hostname configured.
systemd[1]: Set hostname to <localhost>.
systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
systemd[1]: Starting Forward Password Requests to Wall Directory Watch.
systemd[1]: Started Forward Password Requests to Wall Directory Watch.
systemd[1]: Starting Remote File Systems.
systemd[1]: Reached target Remote File Systems.
systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
systemd[1]: Starting Dispatch Password Requests to Console Directory Watch.
systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
systemd[1]: Starting Paths.
systemd[1]: Reached target Paths.
systemd[1]: Starting Swap.
systemd[1]: Reached target Swap.
systemd[1]: Expecting device dev-zram0.device...
systemd[1]: Expecting device dev-disk-by\x2duuid-fc660063\x2d302d\x2d4c49\x2d9a69\x2d3ffcf9217590.device...
systemd[1]: Expecting device dev-sdb3.device...
systemd[1]: Expecting device dev-sdb5.device...
systemd[1]: Expecting device dev-sdb7.device...
systemd[1]: Expecting device dev-mapper-secure.device...
systemd[1]: Expecting device dev-sda2.device...
systemd[1]: Expecting device dev-sda1.device...
systemd[1]: Starting Root Slice.
systemd[1]: Created slice Root Slice.
systemd[1]: Starting User and Session Slice.
systemd[1]: Created slice User and Session Slice.
systemd[1]: Starting Device-mapper event daemon FIFOs.
systemd[1]: Listening on Device-mapper event daemon FIFOs.
systemd[1]: Starting Delayed Shutdown Socket.
systemd[1]: Listening on Delayed Shutdown Socket.
systemd[1]: Starting /dev/initctl Compatibility Named Pipe.
systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
systemd[1]: Starting Syslog Socket.
systemd[1]: Listening on Syslog Socket.
systemd[1]: Starting udev Control Socket.
systemd[1]: Listening on udev Control Socket.
systemd[1]: Starting udev Kernel Socket.
systemd[1]: Listening on udev Kernel Socket.
systemd[1]: Starting LVM2 metadata daemon socket.
systemd[1]: Listening on LVM2 metadata daemon socket.
systemd[1]: Starting Journal Socket.
systemd[1]: Listening on Journal Socket.
systemd[1]: Starting System Slice.
systemd[1]: Created slice System Slice.
systemd[1]: Starting File System Check on Root Device...
systemd[1]: Starting system-getty.slice.
systemd[1]: Created slice system-getty.slice.
systemd[1]: Starting Apply Kernel Variables...
systemd[1]: Starting Load Kernel Modules...
systemd[1]: Mounting Huge Pages File System...
systemd[1]: Mounting POSIX Message Queue File System...
systemd[1]: Starting Setup Virtual Console...
systemd[1]: Starting udev Coldplug all Devices...
systemd\x2dcryptsetup.slice.
systemd\x2dcryptsetup.slice.
systemd[1]: Starting Set Up Additional Binary Formats...
systemd[1]: Mounted Debug File System.
systemd[1]: Starting Create list of required static device nodes for the current kernel...
systemd[1]: Starting Journal Service...
systemd[1]: Started Journal Service.
systemd-journal[198]: Journal started
systemd-modules-load[191]: Module 'acpi_cpufreq' is builtin
systemd-modules-load[191]: Module 'acpi_call' is builtin
systemd-modules-load[191]: Module 'aesni_intel' is builtin
systemd-modules-load[191]: Module 'coretemp' is builtin
systemd-modules-load[191]: Module 'microcode' is builtin
systemd-modules-load[191]: Module 'tp_smapi' is builtin
systemd-modules-load[191]: Module 'uinput' is builtin
systemd-sysctl[189]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sysctl.conf'.
systemd[1]: Starting Slices.
systemd[1]: Reached target Slices.
systemd-binfmt)
systemd-modules-load[191]: Inserted module 'vboxdrv'
systemd[1]: Mounted POSIX Message Queue File System.
systemd[1]: Mounted Arbitrary Executable File Formats File System.
systemd[1]: Started udev Coldplug all Devices.
systemd-modules-load[191]: Inserted module 'vboxnetadp'
systemd-modules-load[191]: Inserted module 'vboxnetflt'
systemd[1]: Started Set Up Additional Binary Formats.
systemd-modules-load[191]: Inserted module 'vboxpci'
systemd-modules-load[191]: Module 'zram' is builtin
systemd-modules-load[191]: Module 'fuse' is builtin
systemd[1]: Started Load Kernel Modules.
systemd[1]: Mounting Configuration File System...
systemd[1]: Mounting FUSE Control File System...
systemd[1]: Starting Create Static Device Nodes in /dev...
systemd[1]: Mounted FUSE Control File System.
systemd[1]: Mounted Configuration File System.
systemd[1]: Started Apply Kernel Variables.
systemd-fsck[188]: ArchRoot: clean, 281073/812800 files, 1963679/3249138 blocks
systemd[1]: Started File System Check on Root Device.
systemd[1]: Starting Remount Root and Kernel File Systems...
systemd[1]: Started Remount Root and Kernel File Systems.
systemd[1]: Starting Load/Save Random Seed...
systemd[1]: Started Load/Save Random Seed.
systemd[1]: Started Setup Virtual Console.
systemd[1]: Started Create Static Device Nodes in /dev.
systemd[1]: Starting udev Kernel Device Manager...
systemd[1]: Starting Local File Systems (Pre).
systemd[1]: Reached target Local File Systems (Pre).
systemd[1]: Mounting /run/shm...
systemd[1]: Mounted /run/shm.
systemd[1]: Started udev Kernel Device Manager.
systemd-udevd[234]: starting version 212
systemd-sysctl[253]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sysctl.conf'.
systemd-sysctl[295]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sysctl.conf'.
systemd-udevd[239]: error opening ATTR{/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.3/1-1.3:1.0/power/control} for writing: No such file or directory
systemd-journal[198]: Forwarding to syslog missed 40 messages.
systemd[1]: Job dev-sda1.device/start timed out.
systemd[1]: Timed out waiting for device dev-sda1.device.
systemd[1]: Dependency failed for /data.
systemd[1]: Dependency failed for Local File Systems.
systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
systemd[1]: Job dev-sda2.device/start timed out.
systemd[1]: Timed out waiting for device dev-sda2.device.
systemd[1]: Dependency failed for /vm.
systemd[1]: Job dev-sdb7.device/start timed out.
systemd[1]: Timed out waiting for device dev-sdb7.device.
systemd[1]: Dependency failed for Cryptography Setup for secure.
systemd[1]: Dependency failed for Encrypted Volumes.
systemd[1]: Dependency failed for dev-mapper-secure.device.
systemd[1]: Dependency failed for /secure.
systemd[1]: Job dev-sdb5.device/start timed out.
systemd[1]: Timed out waiting for device dev-sdb5.device.
systemd[1]: Dependency failed for /ubuntu_ssd_vm.
systemd[1]: Job dev-sdb3.device/start timed out.
systemd[1]: Timed out waiting for device dev-sdb3.device.
systemd[1]: Dependency failed for /win7_ssd_vm.
systemd[1]: Job dev-disk-by\x2duuid-fc660063\x2d302d\x2d4c49\x2d9a69\x2d3ffcf9217590.device/start timed out.
systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-fc660063\x2d302d\x2d4c49\x2d9a69\x2d3ffcf9217590.device.
systemd[1]: Dependency failed for /Application.
systemd[1]: Job dev-zram0.device/start timed out.
systemd[1]: Timed out waiting for device dev-zram0.device.
systemd[1]: Dependency failed for /tmp.
systemd[1]: Dependency failed for /var/tmp.
systemd[1]: Dependency failed for /home/USER/.cache.
systemd[1]: Dependency failed for /var/log.
systemd[1]: Dependency failed for Update UTMP about System Boot/Shutdown.
systemd[1]: Requested transaction contradicts existing jobs: Transaction is destructive.
systemd[1]: syslog.socket failed to queue service startup job (Maybe the service file is missing or not a non-template unit?): Transaction is destructive.
systemd[1]: Unit syslog.socket entered failed state.
systemd[1]: Stopped Daily Cleanup of Temporary Directories.
systemd[1]: Stopped Getty on tty1.
systemd[1]: Stopped Getty on tty3.
systemd[1]: Stopped target Graphical Interface.
systemd[1]: Stopped target Multi-User System.
systemd[1]: Stopped Initialize hardware monitoring sensors.
systemd[1]: Stopped Set brightness.
systemd[1]: Stopped Restore pax attributes.
systemd[1]: Stopped sets battery thresholds.
systemd[1]: Stopped Name Service Cache Daemon.
systemd[1]: Stopped Periodic Command Scheduler.
systemd[1]: Stopped Network Manager.
systemd[1]: Stopped My GoAgent Service.
systemd[1]: Starting Network.
systemd[1]: Reached target Network.
systemd[1]: Stopped Run my.local script.
systemd[1]: Stopped Load apparmor profiles.
systemd[1]: Stopped Login Service.
systemd[1]: Starting Login Prompts.
systemd[1]: Reached target Login Prompts.
systemd[1]: Stopped ACPI event daemon.
systemd[1]: Closed ACPID Listen Socket.
systemd[1]: Stopped D-Bus System Message Bus.
systemd[1]: Closed D-Bus System Message Bus Socket.
systemd[1]: Stopped System Logger Daemon.
systemd[1]: Stopped Permit User Sessions.
systemd[1]: Starting Trigger Flushing of Journal to Persistent Storage...
systemd[1]: Stopped target Basic System.
systemd[1]: Starting Timers.
systemd-journal[198]: Forwarding to syslog missed 26 messages.
systemd[1]: Reached target Timers.
systemd[1]: Starting Sockets.
systemd[1]: Reached target Sockets.
systemd[1]: Stopped target System Initialization.
systemd[1]: Started Manage Sound Card State (restore and store).
systemd[1]: Starting Restore Sound Card State...
systemd[1]: Starting Create Volatile Files and Directories...
systemd[1]: Starting Emergency Shell...
systemd[1]: Started Emergency Shell.
systemd[1]: Starting Emergency Mode.
systemd[1]: Reached target Emergency Mode.
systemd[1]: Started Restore Sound Card State.
systemd-journal[198]: Runtime journal is using 8.0M (max allowed 388.0M, trying to leave 582.1M free of 3.7G available → current limit 388.0M).
systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.
systemd[1]: Started Create Volatile Files and Directories.
systemd[1]: Startup finished in 2.835s (kernel) + 1min 30.235s (userspace) = 1min 33.070s.
systemd[884]: Failed at step EXEC spawning /bin/plymouth: No such file or directory
Best Answer
Just connected with the developers, and my problem is resolved.
My customized kernel does not support the open by file handle syscall, so I enabled the
CONFIG_FHANDLEoption, and it boots again.