Debian – Removing Debian package automatically masks systemd service – causes a systemd warning

debiansystemd

I booted a Debian unstable container. Early on, systemd inside the container shows rsync.service: Cannot add dependency job, ignoring: Unit rsync.service is masked.

I determined that rsync.service was masked automatically because the rsync package was removed. Re-installing the package unmasked it again.

Is there documentation for this behaviour?
What is the conflict that causes systemd to warn, when faced with this behaviour of Debian?
I'm pleasantly surprised to see that this behaviour somehow detects if I masked rsync while it was installed, and avoids unmasking it automatically if I remove and reinstall rsync. How is this implemented?? Are there any more subtle limitations to it?

Discovery of automatic masking on package removal

I knew that rsync had originally been installed, but was now removed. I removed the mask, and was left with this:

$ sudo systemctl status rsync
● rsync.service - LSB: fast remote file copy program daemon
   Loaded: loaded (/etc/init.d/rsync; generated; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

$ # reset status of all systemd services
$ # DO NOT TRY THIS COMMAND INSIDE A REAL, NON-CONTAINER SYSTEM...
$ # IT DOES NOT GO WELL.
$ sudo systemctl isolate default.target

$ sudo systemctl status rsync
● rsync.service - LSB: fast remote file copy program daemon
   Loaded: loaded (/etc/init.d/rsync; generated; vendor preset: enabled)
   Active: active (exited) since Wed 2017-06-07 11:35:27 BST; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 432 ExecStart=/etc/init.d/rsync start (code=exited, status=0/SUCCESS)
   CGroup: /machine.slice/machine-unstable.scope/system.slice/rsync.service

Jun 07 11:35:27 unstable systemd[1]: Starting LSB: fast remote file copy program daemon...
Jun 07 11:35:27 unstable systemd[1]: Started LSB: fast remote file copy program daemon.

This output is misleading. Debian's /etc/init.d/rsync was not starting rsync --daemon, because I had not changed the default RSYNC_ENABLE=false in /etc/default/rsync. (The init script itself would exit silently in this case; however I believe a systemd boot would show a starting message for it, similar to the log messages shown above). So the masking was serving a useful purpose here.

(The reason /etc/init.d/rsync remains when the package is removed, is that initscripts are considered to be user-editable configuration files)

It turns out that if I install rsync again, rsync.service is unmasked. If I remove it, rsync.service becomes masked again.

I'm pleased to say that if I install rsync, mask it, then remove and reinstall rsync, rsync remains masked.

If I use apt-get remove --purge rsync to completely remove it, including residual configuration files, then the mask is removed.

Since I have etckeeper installed, I noticed that complete removal also removes
/etc/systemd/system/multi-user.target.wants/rsync.service, as well as removing the mask (/etc/systemd/system/rsync.service -> /dev/null). Neither of these files were owned by the package (dpkg-query -L rsync), so it looks like these removals are caused by a package script.

Software versions

Up to date Debian unstable container. This question was asked soon before the release of stretch.

Host used systemd-container version 231-15.fc25.

More context for systemd message "ignoring: Unit rsync.service is masked"

$ sudo systemd-nspawn -b -D unstable
Spawning container unstable on /home/nspawn/unstable.
Press ^] three times within 1s to kill container.
systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization systemd-nspawn.
Detected architecture x86-64.

Welcome to Debian GNU/Linux 9 (stretch)!

Set hostname to <unstable>.
Failed to install release agent, ignoring: File exists
rsync.service: Cannot add dependency job, ignoring: Unit rsync.service is masked
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.

Best Answer

Is there documentation for this behaviour?

It is written down. The pointer as to why this is implemented is found in a commit message to a file which was since moved elsewhere...

I'm pleasantly surprised to see that this behaviour somehow detects if I masked rsync while it was installed, and avoids unmasking it automatically if I remove and reinstall rsync. How is this implemented?? Are there any more subtle limitations to it?

Look hard, and you can still find the source history. It links to an issue, which confirms that masking is used to handle the system V initscripts as best as possible under systemd.

Tangent: there's an unimplemented proposal which would remove the need for this, #749400 - dh_installinit: disable init scripts on removal of package. Not that it's an unambiguously good idea. IIUC, it loses track of whether the script was enabled by the user. (Note this is a separate setting for each runlevel, in system V init).

The clue to this was in the package script, which I found at /var/lib/dpkg/info/rsync.postrm.

## from /usr/share/debhelper/autoscripts/postrm-systemd :

if [ "$1" = "remove" ]; then
    if [ -x "/usr/bin/deb-systemd-helper" ]; then
        deb-systemd-helper mask rsync.service >/dev/null
    fi
fi

What this does is documented in man deb-systemd-helper. 'The "mask" action will keep state on whether the service was enabled/disabled before and will properly return to that state on "unmask".' It is also commented in rsync.postinst:

## from /usr/share/debhelper/autoscripts/postinst-systemd-enable :

# This will only remove masks created by d-s-h on package removal.
deb-systemd-helper unmask rsync.service >/dev/null || true

Fedora Linux (version 25) doesn't implement this behaviour. Arguably because they don't support system V init and had a policy to completely remove the old-style init scripts. I don't know how they handled this issue during the transition... but they could have ignored it without causing any functional problem.

What is the conflict that causes systemd to warn, when faced with this behaviour of Debian?

In the general case, masking an enabled service seems a bit suspicious, perhaps?

It looks like the rpm based distributions don't / didn't try to preserve the enabled status of initscripts. Because they run checkconf --del on removal. https://www.cyberciti.biz/faq/centos-rhel-suse-rpm-see-installation-uninstallation-scripts/

The modern Fedora rpms have equivalent-looking code

$ rpm -q --scripts rsync
...
    # Package removal, not upgrade 
    systemctl --no-reload disable --now avahi-daemon.socket avahi-daemon.service > /dev/null 2>&1 || :
...

I looked at this as I noticed removing rsync-daemon doesn't remove /etc/systemd/system/multi-user.target.wants/rsyncd.service. ~~Because systemctl disable does not currently remove symlinks if they point to a file which has already been removed.~~ This is a bug specific to the rsync package: the service file is in package rsync, but the rpm scripts referring to the service are in package rsync-daemon.

Related Solutions

Debian – Minecraft server startup/shutdown with systemd

After browsing through the manpages few more times (yeah, the answer is never there the first time...), I've come up with a solution... which did not work. After browsing even more, I've finally come up with the most elegant solution possible.

[Unit]
Description=Minecraft server
After=local-fs.target network.target

[Service]
WorkingDirectory=/home/minecraft/minecraft_server
User=minecraft
Group=minecraft
Type=forking
# Run it as a non-root user in a specific directory

ExecStart=/usr/bin/screen -h 1024 -dmS minecraft ./minecraft_server.sh
# I like to keep my commandline to launch it in a separate file
# because sometimes I want to change it or launch it manually
# If it's in the WorkingDirectory, then we can use a relative path

# Send "stop" to the Minecraft server console
ExecStop=/usr/bin/screen -p 0 -S minecraft -X eval 'stuff \"stop\"\015'
# Wait for the PID to die - otherwise it's killed after this command finishes!
ExecStop=/bin/bash -c "while ps -p $MAINPID > /dev/null; do /bin/sleep 1; done"
# Note that absolute paths for all executables are required!

[Install]
WantedBy=multi-user.target

This really does look better than my original script! However, there are a few regressions.

If I want to pass commands to the server console, then I'll have to make a separate script to do it.
After running systemctl start minecraft or systemctl stop minecraft, be sure to check systemctl status minecraft, because those commands give no output at all, even if they actually fail. This is the only major regression compared to scripts - "always examine your output" is #1 rule in IT, but systemd doesn't seem to care about it...
Also, I expected systemd to be able to manage the service shutdown without the "wait for the PID to die" workaround. In the old init script, I had to do this manually, because it was a script, and systemd is trying to eliminate the need for complex scripts that to the same things; it eliminates the need to manually script in all the timeouts and killing whoever did not die, but "waiting for the pid to die" is the next most common thing, and we still need to script it.

Starting systemd service inside systemd service causes deadlock

What appears to be a deadlock is actually the Type=oneshot directive working as advertised. The "hang" can be triggered by running your another.service directly:

systemctl start another

Once cancelled, one can review journalctl to find it wasn't "hung" at all, but running an infinite loop as expected. In man systemd.service, we find the documentation for Type=oneshot:

Behavior of oneshot is similar to simple; however, it is expected that the process has to exit before systemd starts follow-up units.

In other words, you have written an infinite loop and then instructed systemd to wait till it finishes before moving on.

You either need to use a different Type=, like Type=simple, or quit using an infinite loop for your main process.