How do you keep systemd user services alive over mosh

moshsystemd

I have a few services (such as syncthing) set up as user-level systemd services on my Arch machine. They work great when I ssh in, but when I connect using mosh, they seem to start and then immediately stop again. For instance, I can connect with mosh, and do systemctl --user status syncthing and get back that it's running or shutting down; then, repeating the command, I get

Failed to connect to bus: No such file or directory

Based on other similar questions, I've checked that $XDG_RUNTIME_DIR is set within the mosh session:

$ echo $XDG_RUNTIME_DIR
/run/user/1000

Indeed, it seems the user manager shuts down cleanly, even though I'm still connected to the session:

$ systemctl status user@1000.service
● user@1000.service - User Manager for UID 1000
   Loaded: loaded (/usr/lib/systemd/system/user@.service; static; vendor preset: disabled)
   Active: inactive (dead)

[...]
Aug 16 18:36:56 ip-172-70-3-138 systemd[7804]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Aug 16 18:36:56 ip-172-70-3-138 systemd[7804]: Reached target Shutdown.
Aug 16 18:36:56 ip-172-70-3-138 systemd[7804]: Starting Exit the Session...
Aug 16 18:36:56 ip-172-70-3-138 systemd[7804]: Received SIGRTMIN+24 from PID 7877 (kill).
Aug 16 18:36:56 ip-172-70-3-138 systemd[1]: Stopped User Manager for UID 1000.

How do I keep the services alive?

Update: Tmux sessions themselves don't start or keep systemd services alive, at least on my system. I haven't been able to find out whether that's correct behavior or not, but it seems to me a tmux session should keep systemd from closing things down. Consider the case where I'm editing a file using emacsclient: I would expect that if my connection dropped for a minute, whether using mosh or tmux, the emacs daemon would stay alive.

Best Answer

There is, once again, a mismatch between systemd's concept of user-space login sessions and how a program such as mosh works. (This is far from the only such conflict. There have been problems with tmux, screen, emacs in its new daemon mode, deluged, and others. They are not within the scope of this answer, though.)

The systemd notion is that a bodge PAM plug-in communicates the set-up and tear-down of login sessions over to logind, which in turn handles starting and stopping the per-user service management, at first log-on and at last log-off.

That comes into force with the SSH session that you are using to initiate mosh-server. But that session is short-lived, and finishes after mosh-server is up and running. mosh-server, furthermore, is not a login program and does not have any dealings with PAM, so the bodge PAM plug-in does not work. The consequence is that logind sees only one very shortlived SSH session, and as a result starts and then quickly stops again your per-user service management subsystem.

The only way that systemd has for dealing with this is to tell logind that your per-user service management "lingers around" after final log-off. You do this with the enable-linger subcommand of the loginctl command.

This does not just apply to mosh, moreover. Any system that involves shortlived SSH sessions, especially if it involves lots of them, has this problem of logind starting and stopping per-user service management over and over.

Further reading

Related Question