Why does “su -c &” seemingly allow a command to run in the background without hanging up

background-processdaemonsu

I was helping a colleague who was having problems with a background process intermittently dying.

I found out that they were starting the background process by logging in to the server and executing:

su - <user> -c '<command>' &

"Aha", I exclaimed. "If you start a command with "&" it will hang-up when you exit the controlling terminal. You need to use something like nohup to achieve this. Really this process needs to support running as a daemon, tut tut."

We tested the above command to demonstrate my point and… it seemed to work: the process started by command didn't exit when we exited the terminal that ran the above command.

command is a custom Python script whose output goes to a file. As far as I can tell, there is no intelligent "daemonize" like capability in the script. It doesn't do any of the things needed to run as a daemon listed in the Wikipedia: Daemon (computing): Creation page.

Running the command like so behaves as expected:

<command> &
exit

In the above case the background process started by command exits when we exit the terminal.

My question is this:

What is happening when we add "su – -c &" that prevents the process from exiting when our terminal exits. I would like to understand in detail with respect to the controlling terminal, standard input and output etc.
Is this a reasonable way to achieve the goal of running this command as a background process. If not why, not?

I want to propagate best practices within my company, but I need to be able to demonstrate and back-up any recommendations I make.

I also want to understand what is going on exactly.

Best Answer

There are several ways a process might be killed because of a dying terminal.

The first way is that the terminal driver in the kernel sends a SIGHUP signal to the controlling process for which the terminal is the controlling terminal. In most cases, the controlling process is the shell that is initially started in the terminal, and its controlling terminal is the one that its stdin, stdout and stderr are connected to. A process becomes disconnected from the controlling terminal if it calls setsid.
The second way is that the shell, when it receives SIGHUP, re-sends that signal to its subprocesses — more precisely, to its background jobs. Some shells (ksh, bash, zsh) have a disown builtin to tell the shell not to send a SIGHUP to a particular job.
If the terminal goes away and a program tries to read from it, it is notified of an end-of-file condition (or possibly EIO for a background job). If the terminal goes away and a program tries to write to it, the write returns with an EIO error. These might cause the program to exit.

So what su changes here is that (2) would normally cause the initial shell to kill the background job, but since that background process is running as a different user, the signal cannot be delivered, and the background process lives on.

Related Solutions

Background Process – Why Process Isn’t Running in Background Using &

When you close a terminal window, the terminal emulator sends a SIGHUP to the process it is running, your shell. Your shell then forwards that SIGHUP to everything it's running. On your local system, this is the ssh. The ssh then forwards the SIGHUP to what it's running, the remote shell. So your remote shell then sends a SIGHUP to all its processes, your backgrounded program.

There are 2 ways around this.

Disassociate the backgrounded program from your shell.
1. Use the disown command after backgrounding your process. This will make the shell forget about it.
2. Prefix your command with nohup (nohup $python program.py &). This accomplishes the same thing, but by using an intermediate process. Basically it ignores the SIGHUP signal, and then forks & executes your program which inherits the setting, and then exits. Because it forked, the program being launched is not a child of the shell, and the shell doesn't know about it. And unless it installs a signal handler for SIGHUP, it keeps the ignore action anyway.
Use logout (or Ctrl+d) instead of closing the terminal window. When you use logout, this isn't a SIGHUP, and so the shell won't send a SIGHUP to any of its children.

Additionally you must make sure that your program doesn't write to the terminal through STDOUT or STDERR, as both of those will no longer exist once the terminal exits. If you don't redirect them to something like /dev/null, the program will still run, but if it tries to write to them, it'll get a SIGPIPE, and the default action of SIGPIPE is to kill the process).

Linux Daemon – How to Daemonize a Process in Shell

From the Wikipedia article on daemon:

In a Unix environment, the parent process of a daemon is often, but not always, the init process. A daemon is usually either created by a process forking a child process and then immediately exiting, thus causing init to adopt the child process, or by the init process directly launching the daemon. In addition, a daemon launched by forking and exiting typically must perform other operations, such as dissociating the process from any controlling terminal (tty). Such procedures are often implemented in various convenience routines such as daemon(3) in Unix.

Read the manpage of the daemon function.

Running a background command from a shell that immediately exits results in the process's PPID becoming 1. Easy to test:

# bash -c 'nohup sleep 10000 &>/dev/null & jobs -p %1'
1936
# ps -p 1936
      PID    PPID    PGID     WINPID   TTY         UID    STIME COMMAND
     1936       1    9104       9552  cons0       1009 17:28:12 /usr/bin/sleep

As you can see, the process is owned by PID 1, but still associated with a TTY. If I log out from this login shell, then log in again, and do ps again, the TTY becomes ?.

Read here why it's important to detach from TTY.

Using setsid (part of util-linux):

# bash -c 'cd /; setsid sleep 10000 </dev/null &>/dev/null & jobs -p %1'
9864
# ps -p 9864
      PID    PPID    PGID     WINPID   TTY         UID    STIME COMMAND
     9864       1    9864       6632  ?           1009 17:40:35 /usr/bin/sleep

I think you don't even have to redirect stdin, stdout and stderr.

Best Answer

Related Solutions

Background Process – Why Process Isn’t Running in Background Using &

Linux Daemon – How to Daemonize a Process in Shell

Related Question