Systemd Can’t Mount Partition – Troubleshooting Guide

hard drivelinuxmountsystemd

I use a ssh tunnel at work to browse the internet.
Yesterday I realize that my terminal is very slow responding to commands, I inspect htop, iotop, netop and see nothing in particular, only the console is slow, the tunnel works fine.

Coming back home I try to log in my X session, and the computer hangs for 5min after I press the Enter key.

I decide to reboot, and end up in systemd emergency mode.

Looking at the journalctl (here pastebined) I try a few things, among them commenting some partition / devices in my /etc/fstab.

It happens that when I comment my /home I can log in normally. Here is my /etc/fstab:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda5 during installation
UUID=f0e6eea4-264d-42f2-97c1-0ad6d60e9e76 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda6 during installation
#UUID=d464c414-6e10-4fb1-adaf-43096cf52ab5 none            swap    sw              0       0
# changed home on new partition
#UUID=83192ee9-fd8a-4d8c-9988-bca2d0fdf630   /home   ext4   nodev,nosuid   0   2
#/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
#/dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0
#/dev/md0   /media/raid5   ext4   defaults   0   0

#UUID=ea59484a-3652-4d1d-a398-f417cbd0ae7c   /media/backup   ext4   defaults   0   0

# for virtualbox, changed on june 5th
#none   /proc/bus/usb   usbfs   devgid=126,devmode=664   0   0

# for sftp in chroot
#/media/raid5/TV\040SHOWS/   /home/quest/tv/      ext4   bind,ro      0   0
#/media/raid5/MOVIES/      /home/quest/movies/   ext4   bind,ro      0   0
#/media/raid5/ANIME/      /home/quest/anime/   ext4   bind,ro      0   0

#garmin fenix 2
#UUID=489A-9E97         /media/raid5/fenix2   vfat   defaults,user,noauto,errors=remount-ro   0   1

So I try to understand a little better what can happen and decide to see if there is a mismatch in uuid (I saw a few posts talking about that), but unfortunately the uuid do match. See blkid output:

/dev/sdc1: UUID="e7c87d29-37d0-d6b4-9a33-6d4f0e239a13" UUID_SUB="4cc4d747-5c88-9bc6-59c1-5f29603fba57" LABEL="bigbenn:0" TYPE="linux_raid_member" PARTLABEL="Linux filesystem" PARTUUID="af78f725-7354-448b-a3f7-9149937d440c"
/dev/sdb1: UUID="83192ee9-fd8a-4d8c-9988-bca2d0fdf630" TYPE="ext4" PARTUUID="8aaed40e-01"
/dev/sdb5: UUID="f0e6eea4-264d-42f2-97c1-0ad6d60e9e76" TYPE="ext4" PARTUUID="8aaed40e-05"
/dev/sdb6: UUID="d464c414-6e10-4fb1-adaf-43096cf52ab5" TYPE="swap" PARTUUID="8aaed40e-06"
/dev/sde1: UUID="ea59484a-3652-4d1d-a398-f417cbd0ae7c" TYPE="ext4" PARTUUID="1433120b-01"
/dev/sdf1: UUID="e7c87d29-37d0-d6b4-9a33-6d4f0e239a13" UUID_SUB="69383fe0-2113-a305-87fe-51057bfdd681" LABEL="bigbenn:0" TYPE="linux_raid_member" PARTUUID="873db73f-01"
/dev/sda1: UUID="e7c87d29-37d0-d6b4-9a33-6d4f0e239a13" UUID_SUB="cfdfaca9-f350-3c3a-a6a7-75a9844e9660" LABEL="bigbenn:0" TYPE="linux_raid_member" PARTUUID="4483559b-01"
/dev/sdd1: UUID="e7c87d29-37d0-d6b4-9a33-6d4f0e239a13" UUID_SUB="efb1db76-0169-ee95-ba36-d64adf36de65" LABEL="bigbenn:0" TYPE="linux_raid_member" PARTUUID="092c2257-01"
/dev/sdg1: UUID="e7c87d29-37d0-d6b4-9a33-6d4f0e239a13" UUID_SUB="c6dc6576-3d94-e0e9-77b8-abb62563277d" LABEL="bigbenn:0" TYPE="linux_raid_member" PARTUUID="000d77f7-01"
/dev/md0: UUID="0a0dc09a-0714-4fed-99e2-31a08b753373" TYPE="ext4"

Now I'm at a loss, I tried also to smartctl the disk where my /home is, and it seems normal (test passed) but I'm not sure as there are errors at the end… here is the smartctl

Best Answer

You have a strange situation, whereby udev does not seem to start mounting partitions until systemd has completed, but this runs into problems when such partitions are mentioned in /etc/fstab: in that case udev waits for systemd to complete (and this leads to a time out), and only then it mounts the partition.

I have no idea whatsoever about how this may have come about. I know though that you can solve your problem by creating a new service: create a file /etc/systemd/system/mounthome.service into which you place this code:

 [Unit]
 Description=Mount Home
 DefaultDependencies=no

 [Service]
 Type=oneshot
 RemainAfterExit=yes
 ExecStart=/bin/mount /dev/sdb1

and now add to the file /usr/lib/systemd/system/systemd-udev-trigger.service the following line, 

 Wants=systemd-udevd.service mounthome.service

so that it looks as follows:

  [Unit]
  Description=udev Coldplug all Devices
  Documentation=man:udev(7) man:systemd-udevd.service(8)
  DefaultDependencies=no
  Wants=systemd-udevd.service mounthome.service
  After=systemd-udevd-kernel.socket systemd-udevd-control.socket
  Before=sysinit.target
  ConditionCapability=CAP_MKNOD

  [Service]
  Type=oneshot
  RemainAfterExit=yes
  ExecStart=/usr/bin/udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm trigger

I believe this will solve your problem.

Related Question