Ssh – Tmux under Kitty terminal

Theory

Some init systems including systemd provide a feature to kill all processes belonging to the service. The service typically starts a single process which that creates more processes by forking and those processes can do that as well. All such processes are typically considered part of the service. In systemd this is done using cgroups.

In systemd, all processes belonging to a service are killed when the service is stopped by default. The SSH server is obviously part of the service. When you connect to the server, SSH server typically forks and the new process handles your SSH session. By forking from the SSH session process or its children, other server side processes are started, including your screen or tmux.

Killmode and socket activation

The default behavior can be changed using the KillMode directive. The upstream project doesn't AFAIK include any .service files and so those vary by distribution. There are typically two ways to enable SSH on your system. One is the classic ssh.service that maintains a long running SSH daemon listening on the network. The other is via socket activation handled by the ssh.socket that in turn starts sshd@.service which only runs for a single SSH session.

Solutions

If your processes get killed at the end of the session, it is possible that you are using socket activation and it gets killed by systemd when it notices that the SSH session process exited. In that case there are two solutions. One is to avoid using socket activation by using ssh.service instead of ssh.socket. The other is to set KillMode=process in the Service section of ssh@.service.

The KillMode=process setting may also be useful with the classic ssh.service, as it avoids killing the SSH session process or the screen or tmux processes when the server gets stopped or restarted.

Future notes

This answer apparently gained a level of popularity. While it worked for the OP it might happen that it doesn't work for someone in the future due to systemd-logind development or configuration. Please check documentation on logind sessions if you experience behavior different from the description in this answer.

Thanks to Mark Plotnick I was able to identify and fix the issue.

The problem is the interaction between the AFS file system used by the server and Kerberos handling the authentication. The same issue was brought up in this question on SO.

Basically what is happening is that when I ssh into the server, Kerberos gives the authentication token to the session. This token is used also to access the AFS file system. When closing the SSH session this token gets destroyed and the processes running start to get permission denied errors when trying to access files on the AFS.

The way to fix this is to start a new window inside screen/tmux and launch the command:

kinit && aklog

After that you can detach from screen/tmux and close the ssh session safely.

The commands above create new Kerberos tokens and associate those with the screen/tmux session, in this way when the ssh connection is closed the initial tokens get revoked but since the subprocesses now use those you created they don't suffer permission denied errors.

To summarize:

• ssh username@server
• tmux
• Launch the process you need to keep running
• Create a new window with Ctrl+B, C
• kinit && aklog
• Detach from the session with Ctrl+B, D
• Close ssh session