Ssh – How to prevent a program from dying when its ssh session dies

gnu-screennohupsshterminal

I have a really old console application that I want to make a bit more resilient. The program is used this way:

the user uses a custom terminal emulator to connect to a remote machine through ssh
the user starts a shell script
the shell script might start a long-running progress database process.

Obviously, sometimes users simply lose the ssh connection to the machine, and in this case, the ssh session is terminated, the shell script running inside is terminated, and finally the progress database process is terminated as well. In like one case out of a thousand this causes corruption in the database, so I'd like to prevent it from happening.

What I tried so far:

starting a screen or tmux session before starting the shell script – this doesn't work because the application needs the TERM variable to be set to at386 (and it bypasses termcap/terminfo completely… ugh…)
nohup/disown the progress process – this doesn't work because the shell script and the progress process are continuously communication with each other in obscure ways it seems

Any other idea of how to make sure that the progress process doesn't terminate when the ssh session is killed?

Best Answer

Add term at386 to your .screenrc in order to override TERM. If this doesn't help, try dtach instead of Screen (which doesn't do any terminal emulation itself).

Related Solutions

Bash – How to completely detach a process

It's a little hard to diagnose, as you don't give source for myproc, but I suspect that your problem has something to do with "controlling TTY". I wrote a small shell script that just calls sleep 100. I ran it under nohup:

$ nohup ./sleeper > sleeper.out &

[1] 25305
-bash-3.2$ jobs
[1]+  Running                 nohup ./sleeper > sleeper.out &
-bash-3.2$ ps -l -p 25305
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S 429624 25305 18252  0 77   0 - 15961 wait   pts/0    00:00:00 sleeper

If you look at the ps output, you see a column named "TTY". At least some versions of nohup (the one I used above is from GNU coreutils, version 5.97, so relatively old). When I exited the bash shell from which I started sleeper, the TTY column changed to '?', meaning that sleeper didn't have one.

If myproc doesn't deliberately detach itself from it's controlling TTY, it's still possible to get things like a SIGPIPE signal if it tries to write to stdout. It seems to me other things are possible, but I can't recall or google anything up.

If you can find or compile "daemonize", you might want to try it. If myproc is compiled, you could modify the source to call the daemon(3) library function.

Why nohup background process is getting killed

Your Python program undoes `nohup`.

nohup ignores the hangup signal with SIG_IGN and then chain loads your program in the same process.

Your Python program promptly resets the signal handling for the hangup signal, installing its own signal handler. That handler checks an internal function (that is not designed very well, being based upon some flawed assumptions, if it is the one that I have seen) and decides that the appropriate course of action on receipt of a hangup signal is to print that message and exit.

Your Python program by design is not nohup-able. On a system with a job control shell and POSIX session/job semantics, you need to be disowning the job so that the shell never knows about it to send a hangup signal to it in the first place.

(Even that is not enough on systemd operating systems. Because the systemd people have made a bit of a pig's ear of their user-space login session mechanism, you also need to ensure that systemd's mechanism that signals system shutdown, rather than hangup, to login sessions at every logout is also not kicking in.)

Best Answer

Related Solutions

Bash – How to completely detach a process

Why nohup background process is getting killed

Your Python program undoes nohup.

Further reading

Related Question

Your Python program undoes `nohup`.