Why can’t I run programs after closing SSH connection using setsid, nohup or disown?

bashssh

I am attempting to run a program 'a.o' on a remote server. To keep it running after closing SSH connection, I tried the following:

setsid ./a.o

nohup ./a.o

./a.o &
disown -h %1

None of these options seem to work, and I cannot figure out why. Any ideas on how to fix this problem?

P.S. I am using a tunnel connection to connect to the server via a jump machine. Also, this is a program that involves CUDA code running on GPU. Do these have anything to do with the issue?

Best Answer

The nohup Wikipedia Page has a reference to this issue specifically:

Note that nohupping backgrounded jobs is typically used to avoid terminating them when logging off from a remote SSH session. A different issue that often arises in this situation is that ssh is refusing to log off ("hangs"), since it refuses to lose any data from/to the background job(s).[6][7] This problem can also be overcome by redirecting all three I/O streams:

$ nohup ./myprogram > foo.out 2> foo.err < /dev/null &

Also note that a closing SSH session does not always send a HUP signal to dependent processes, such as when a pseudo-terminal has not been allocated.

As mentioned in comments, screen and tmux are perfect for this too. Personally, I prefer tmux.

Related Question