Bash – Why does the Python background process end when SSH session is terminated

background-processbashpythonssh

I have a bash script that starts up a python3 script (let's call it startup.sh), with the key line:

nohup python3 -u <script> &

When I ssh in directly and call this script, the python script continues to run in the background after I exit. However, when I run this:

ssh -i <keyfile> -o StrictHostKeyChecking=no <user>@<hostname> "./startup.sh"

The process ends as soon as ssh has finished running it and closes the session.

What is the difference between the two?

EDIT: The python script is running a web service via Bottle.

EDIT2: I also tried creating an init script that calls startup.sh and ran ssh -i <keyfile> -o StrictHostKeyChecking=no <user>@<hostname> "sudo service start <servicename>", but got the same behavior.

EDIT3: Maybe it's something else in the script. Here's the bulk of the script:

chmod 700 ${key_loc}

echo "INFO: Syncing files."
rsync -azP -e "ssh -i ${key_loc} -o StrictHostKeyChecking=no" ${source_client_loc} ${remote_user}@${remote_hostname}:${destination_client_loc}

echo "INFO: Running startup script."
ssh -i ${key_loc} -o StrictHostKeyChecking=no ${remote_user}@${remote_hostname} "cd ${destination_client_loc}; chmod u+x ${ctl_script}; ./${ctl_script} restart"

EDIT4: When I run the last line with a sleep at the end:

ssh -i ${key_loc} -o StrictHostKeyChecking=no ${remote_user}@${remote_hostname} "cd ${destination_client_loc}; chmod u+x ${ctl_script}; ./${ctl_script} restart; sleep 1"

echo "Finished"

It never reaches echo "Finished", and I see the Bottle server message, which I never saw before:

Bottle vx.x.x server starting up (using WSGIRefServer())...
Listening on <URL>
Hit Ctrl-C to quit.

I see "Finished" if I manually SSH in and kill the process myself.

EDIT5: Using EDIT4, if I make a request to any endpoint, I get a page back, but the Bottle errors out:

Bottle vx.x.x server starting up (using WSGIRefServer())...
Listening on <URL>
Hit Ctrl-C to quit.


----------------------------------------
Exception happened during processing of request from ('<IP>', 55104)

Best Answer

I would disconnect the command from its standard input/output and error flows:

nohup python3 -u <script> </dev/null >/dev/null 2>&1 &

ssh needs an indicator that doesn't have any more output and that it does not require any more input. Having something else be the input and redirecting the output means ssh can safely exit, as input/output is not coming from or going to the terminal. This means the input has to come from somewhere else, and the output (both STDOUT and STDERR) should go somewhere else.

The </dev/null part specifies /dev/null as the input for <script>. Why that is useful here:

Redirecting /dev/null to stdin will give an immediate EOF to any read call from that process. This is typically useful to detach a process from a tty (such a process is called a daemon). For example, when starting a background process remotely over ssh, you must redirect stdin to prevent the process waiting for local input. https://stackoverflow.com/questions/19955260/what-is-dev-null-in-bash/19955475#19955475

Alternatively, redirecting from another input source should be relatively safe as long as the current ssh session doesn't need to be kept open.

With the >/dev/null part the shell redirects the standard output into /dev/null essentially discarding it. >/path/to/file will also work.

The last part 2>&1 is redirecting STDERR to STDOUT.

There are three standard sources of input and output for a program. Standard input usually comes from the keyboard if it’s an interactive program, or from another program if it’s processing the other program’s output. The program usually prints to standard output, and sometimes prints to standard error. These three file descriptors (you can think of them as “data pipes”) are often called STDIN, STDOUT, and STDERR.

Sometimes they’re not named, they’re numbered! The built-in numberings for them are 0, 1, and 2, in that order. By default, if you don’t name or number one explicitly, you’re talking about STDOUT.

Given that context, you can see the command above is redirecting standard output into /dev/null, which is a place you can dump anything you don’t want (often called the bit-bucket), then redirecting standard error into standard output (you have to put an & in front of the destination when you do this).

The short explanation, therefore, is “all output from this command should be shoved into a black hole.” That’s one good way to make a program be really quiet!
What does > /dev/null 2>&1 mean? | Xaprb

Short answer

In bash (and dash) the various "job status" messages are not displayed from signal handlers, but require an explicit check. This check is performed only before a new prompt is provided, probably not to disturb the user while he/she is typing a new command.

The message is not shown just before the prompt after the kill is displayed probably because the process is not dead yet - this is particularly probable condition since kill is an internal command of the shell, so it's very fast to execute and doesn't need forking.

Doing the same experiment with killall, instead, usually yields the "killed" message immediately, sign that the time/context switches/whatever required to execute an external command cause a delay long enough for the process to be killed before the control returns to the shell.

matteo@teokubuntu:~$ dash
$ sleep 60 &
$ ps
  PID TTY          TIME CMD
 4540 pts/3    00:00:00 bash
 4811 pts/3    00:00:00 sh
 4812 pts/3    00:00:00 sleep
 4813 pts/3    00:00:00 ps
$ kill -9 4812
$ 
[1] + Killed                     sleep 60
$ sleep 60 &
$ killall sleep
[1] + Terminated                 sleep 60
$

Long answer

`dash`

First of all, I had a look at the dash sources, since dash exhibits the same behavior and the code is surely simpler than bash.

As said above, the point seems to be that job status messages are not emitted from a signal handler (which can interrupt the "normal" shell control flow), but they are the consequence of an explicit check (a showjobs(out2, SHOW_CHANGED) call in dash) that is performed only before requesting new input from the user, in the REPL loop.

Thus, if the shell is blocked waiting for user input no such message is emitted.

Now, why doesn't the check performed just after the kill show that the process was actually terminated? As explained above, probably because it's too fast. kill is an internal command of the shell, so it's very fast to execute and doesn't need forking, thus, when immediately after the kill the check is performed, the process is still alive (or, at least, is still being killed).

`bash`

As expected, bash, being a much more complex shell, was trickier and required some gdb-fu.

The backtrace for when that message is emitted is something like

(gdb) bt
#0  pretty_print_job (job_index=job_index@entry=0, format=format@entry=0, stream=0x7ffff7bd01a0 <_IO_2_1_stderr_>) at jobs.c:1630
#1  0x000000000044030a in notify_of_job_status () at jobs.c:3561
#2  notify_of_job_status () at jobs.c:3461
#3  0x0000000000441e97 in notify_and_cleanup () at jobs.c:2664
#4  0x00000000004205e1 in shell_getc (remove_quoted_newline=1) at /Users/chet/src/bash/src/parse.y:2213
#5  shell_getc (remove_quoted_newline=1) at /Users/chet/src/bash/src/parse.y:2159
#6  0x0000000000423316 in read_token (command=<optimized out>) at /Users/chet/src/bash/src/parse.y:2908
#7  read_token (command=0) at /Users/chet/src/bash/src/parse.y:2859
#8  0x00000000004268e4 in yylex () at /Users/chet/src/bash/src/parse.y:2517
#9  yyparse () at y.tab.c:2014
#10 0x000000000041df6a in parse_command () at eval.c:228
#11 0x000000000041e036 in read_command () at eval.c:272
#12 0x000000000041e27f in reader_loop () at eval.c:137
#13 0x000000000041c6fd in main (argc=1, argv=0x7fffffffdf48, env=0x7fffffffdf58) at shell.c:749

The call that checks for dead jobs & co. is notify_of_job_status (it's more or less the equivalent of showjobs(..., SHOW_CHANGED) in dash); #0-#1 are related to its inner working; 6-8 is the yacc-generated parser code; 10-12 is the REPL loop.

The interesting place here is #4, i.e. from where the notify_and_cleanup call comes. It seems that bash, unlike dash, may check for terminated jobs at each character read from the command line, but here's what I found:

      /* If the shell is interatctive, but not currently printing a prompt
         (interactive_shell && interactive == 0), we don't want to print
         notifies or cleanup the jobs -- we want to defer it until we do
         print the next prompt. */
      if (interactive_shell == 0 || SHOULD_PROMPT())
    {
#if defined (JOB_CONTROL)
      /* This can cause a problem when reading a command as the result
     of a trap, when the trap is called from flush_child.  This call
     had better not cause jobs to disappear from the job table in
     that case, or we will have big trouble. */
      notify_and_cleanup ();
#else /* !JOB_CONTROL */
      cleanup_dead_jobs ();
#endif /* !JOB_CONTROL */
    }

So, in interactive mode it's intentional to delay the check until a new prompt is provided, probably not to disturb the user entering commands. As for why the check doesn't spot the dead process when displaying the new prompt immediately after the kill, the previous explanation holds (the process is not dead yet).

Bash – How to check whether a particular port is open on a machine from a shell script and perform action based on that

The program lsof allows you to check which processes are using which ressources, like files or ports.

To show which processes are listening on port 8080:

lsof -Pi :8080 -sTCP:LISTEN

In your case, you want to test whether a process is listening on 8080 - the return value of this command tells you just that. It also prints the pid of the process.

lsof -Pi :8080 -sTCP:LISTEN -t

If you need just the test, with no output, redirect it to /dev/null:

if lsof -Pi :8080 -sTCP:LISTEN -t >/dev/null ; then
    echo "running"
else
    echo "not running"
fi

If you use multiple host names with multiple IP addresses locally, specify the hostname too like
lsof -Pi @someLocalName:8080 -sTCP:LISTEN

Best Answer

Related Solutions

Bash – Why does bash show ‘Terminated’ after killing a process

Short answer

Long answer

dash

bash

Bash – How to check whether a particular port is open on a machine from a shell script and perform action based on that

Related Question

`dash`

`bash`