Debian – bind mounts by systemd don’t magically work with systemd-tmpfiles

bind-mountdebiansystemd

I have an weird challenge from on my Debian 8 box.

The background is I want to mount some directories as tmpfs, to avoid physical IO (disk wakeup / flash wear).

Probably I should just mount a separate tmpfs for each directory. However what I tried first was bind-mounts under /tmp/mnts. (My previous task was to move IO from disk to a small flash storage, to avoid spinups, so I just tried using the same pattern).

So I want to create directories on the tmpfs at boot time. I.e. systemd-tmpfiles. And then bind-mount them in various places under /var.

# /etc/tmpfiles.d/tmpfs-mnts.conf snippet
# Type Path    Mode UID  GID  Age Argument
d /tmp/mnts/var-lib-icinga-spool-checkresults 0750 nagios nagios -

# /etc/fstab snippet
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/tmp/mnts/var-lib-icinga-spool-checkresults /var/lib/icinga/spool/checkresults none bind

systemd-tmpfiles --create + mount -a works fine. But it doesn't work at boot time, so there's a race condition or something. But the failure is a bit interesting – findmnt shows the source directory has been deleted.

# findmnt|grep /var/lib/icinga/spool/checkresults
└─/var/lib/icinga/spool/checkresults                     tmpfs[/mnts/var-lib-icinga-spool-checkresults//deleted] tmpfs       rw
# cd /var/lib/icinga/spool/checkresults/
# mkdir ./test
mkdir: cannot create directory ‘./test’: No such file or directory
# ls --inode /tmp/mnts
7414 var-lib-icinga-spool-checkresults
# ls --inode /var/lib/icinga/spool/
6254 checkresults

So it looks like

The mount happened correctly after systemd-tmpfiles created the source directory
systemd-tmpfiles then deleted the source directory
You're allowed to delete the source directory of a bind mount (?!)
systemd-tmpfiles then created the source directory a second time

I guess there's a number of questions. Can we rely on 1) working? Could 1) still work if something other than systemd-tmpfiles created the source directory? What's the reason for 2) and 4) happening? And what's up with 3), has it always been like that?

Best Answer

bind is unreliable when defined in fstab on a system with systemd. Systemd parses the fstab and tries to work out what order to mount and bind things in. From my own experience it gets this wrong 100% of the time. Best option is to move all you binds out of fstab and make you own xxx.mount system files for systemd. That was you gain controo over the order etc.

Related Solutions

Linux – How does systemd-tmpfiles work

I don't believe tmpfiles.d is the proper way to go here. You really should do the udev rules. Look:

udevadm info -a -p /sys/class/scsi_host/host*

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:11.0/ata1/host0/scsi_host/host0':
    KERNEL=="host0"
    SUBSYSTEM=="scsi_host"
    DRIVER==""
    ATTR{unchecked_isa_dma}=="0"
    ATTR{state}=="running"
    ATTR{cmd_per_lun}=="1"
...
    ATTR{ahci_host_version}=="10200"
    ATTR{prot_guard_type}=="0"
    ATTR{eh_deadline}=="off"
    ATTR{link_power_management_policy}=="max_performance"
    ATTR{host_busy}=="0"

  looking at parent device '/devices/pci0000:00/0000:00:11.0/ata1/host0':
    KERNELS=="host0"
    SUBSYSTEMS=="scsi"
    DRIVERS==""
...

And it goes on, walking up the parent device tree. But consider that, using just the information above, you can do:

KERNEL=="host[0-5]", SUBSYSTEM=="scsi_host", ATTR{link_power_management_policy}="min_power"

And I believe that does it for the majority of your script. You'll want to put the above after rule 60, I think. And really, you should do this for the rest - just the sleep bit in your script is reason enough - it implies a race condition. udev is the one adding and setting these parameters - it's the one that populates sysfs. Just ask it to do the work it is already doing.

And for your keyboard you should definitely do the same - and the backlight. Just get the information you need about these devices from udevadm, write some rules and udevadm test them.

Debian – systemd slow boot – systemd-tmpfiles-setup

When systemd start a system, one of the first service units launched is systemd-tmpfiles-setup. This service runs the command:

# systemd-tmpfiles --create --remove

This command reads configuration files from (less relevant first):

/usr/lib/tmpfiles.d/*.conf - these files are provided by the relevant RPM package and shouldn't be edited by system admin.
/run/tmpfiles.d/*.conf - these files are normally used by daemons to manage their own runtime temporary files
/etc/tmpfiles.d/*.conf - these files are meant for sysadmis to configure custom temporary locations, and to override vendor-provided default

Also there are three places where temporary files are stored:

/var - Variable data specific to this system that should persist between boots
/run - Runtime data for processes started since the last boot. This includes process ID files and lock files, among other things. The contents of this directory are recreated on reboot.
/tmp - A world-writale space for temporary files. Files which have not been accessed, changed, or modified for 10 days are deleted automatically. Another temporary directory exists in /var/tmp in which files that have not been accessed, changed, or modified in more than 30 days are deleted automatically.

Summing up:
check tmp configuration files to see why tmp setup take so much time, especially note entries in /run directory because it's recreated at boot time.

Best Answer

Related Solutions

Linux – How does systemd-tmpfiles work

Debian – systemd slow boot – systemd-tmpfiles-setup

Related Question