Systemd linked unit files on mounted disk fail to load

systemd

We have an internal application with systemd services that we want to deploy outside of the normal systemd directories (/etc/systemd/system and /usr/lib/systemd/system). That location is on another disk (/mnt/data in the example).

The systemd service is enabled by:

systemctl enable /mnt/data/sprinterd.service

which creates a symbolic link in /etc/systemd/system

lrwxrwxrwx. 1 root root   27 Jun 20 22:47 sprinterd.service -> /mnt/data/sprinterd.service

After rebooting, the service is not loaded because the unit file can't be found. From journalctl, first an error that the service failed to load, then after that the mount of the disk where the unit is located.

Cannot add dependency job for unit sprinterd.service, ignoring: Unit sprinterd.service failed to load: No such file or directory.
systemd[1]: Mounted /mnt/data.

From /etc/fstab:

/dev/disk/by-uuid/c55e944f-5c63-48ad-8cd2-bd32d7b35c82 /mnt/data auto nosuid,nodev,nofail,x-gvfs-show 0 0

For completeness the service unit file:

[Unit]
Description=sprinterd

[Service]
Type=simple
Environment=TERM=linux
ExecStart=/srv/s1.erp/bin/sprinterd
Restart=always
RestartSec=5
KillSignal=SIGKILL

[Install]
WantedBy=multi-user.target

I have tested this on RHEL 7 and on openSuSE 13.2.

Is it supported to have a system service unit file on another disk than /etc or /usr?
How could the order of execution between mounting the disk and loading the systemd unit files be changed?

Best Answer

As explained by @StephenHarris the problem is at the moment systemd tries to read the unit, the file that's symlinked isn't available yet

To just have systemd reload the units after it has mounted :

[Unit]
Description=reloads units stored in /mnt/data
DefaultDependencies=no
After=mnt-data.mount
Requires=mnt-data.mount

[Service]
Type=oneshot
ExecStart=/bin/systemctl daemon-reload

[Install]
WantedBy=local-fs.target

This will cause the units to become available, because this time this time the target of the symlinks is mounted.

But by that time, the list of jobs needing to be run to reach the default.target is already built, and the service won't be started.

To have it also restart your service:

[Unit]
Description=restart unit stored in /mnt/data
Requires=mnt-data.mount

[Service]
Type=oneshot
ExecStart=/bin/systemctl daemon-reload
ExecStartPost=/bin/systemctl start sprinterd.service

[Install]
WantedBy=multi-user.target

Alternatives:

  • I've tested with ExecStart= & ExecStartPost=, but it should obviously work with ExecStartPre= & ExecStart=
  • if it's all about 1 single unit, you might as well : ExecStart=/bin/systemctl enable /mnt/data/sprinterd.service instead of daemon-reload
  • if there are multiple services, do the daemon-reload, then start a single unit that uses ConsistsOf= or PartOf= to load all the multiple services.
  • If its NFS (or other networked system), of course local-fs.target isn't your best Installation option, obviously.

For a more old-school SysVinit-style approach, put the systemctl commands inside /etc/rc.local and chmod +x that file.

And then go smuggly post on Devuan's mailing list how you needed SysVInit to fix b0rked SystemD ;-)

Related Question