Linux – Prevent Background Process from Stopping After SSH Close

Use the RunasJob parameter via a PSSession:

(from Help about_remote_jobs)

START A REMOTE JOB THAT RETURNS THE RESULTS TO THE LOCAL COMPUTER (ASJOB)

To start a background job on a remote computer that returns the command results to the local computer, use the AsJob parameter of a cmdlet such as the Invoke-Command cmdlet.

When you use the AsJob parameter, the job object is actually created on the local computer even though the job runs on the remote computer. When the job is completed, the results are returned to the local computer.

You can use the cmdlets that contain the Job noun (the Job cmdlets) to manage any job created by any cmdlet. Many of the cmdlets that have AsJob parameters do not use Windows PowerShell remoting, so
you can use them even on computers that are not configured for
remoting and that do not meet the requirements for remoting.

STEP 1: INVOKE-COMMAND -ASJOB

The following command uses the AsJob parameter of Invoke-Command to start a background job on the Server01 computer. The job runs a Get-Eventlog command that gets the events in the System log. You can use the JobName parameter to assign a display name to the job.

  invoke-command -computername Server01 -scriptblock {get-eventlog system} -asjob

The results of the command resemble the following sample output.

  SessionId   Name    State      HasMoreData     Location   Command
  ---------   ----    -----      -----------     --------   -------
  1           Job1    Running    True            Server01   get-eventlog system

When the AsJob parameter is used, Invoke-Command returns the same type of job object that Start-Job returns. You can save the job object in a variable, or you can use a Get-Job command to get the job.

Note that the value of the Location property shows that the job ran on the Server01 computer.

STEP 2: GET-JOB

To manage a job started by using the AsJob parameter of the Invoke-Command cmdlet, use the Job cmdlets. Because the job object that represents the remote job is on the local computer, you do not need to run remote commands to manage the job.

To determine whether the job is complete, use a Get-Job command. The following command gets all of the jobs that were started in the current session.

   get-job

Because the remote job was started in the current session, a local Get-Job command gets the job. The State property of the job object shows that the command was completed successfully.

  SessionId   Name   State      HasMoreData     Location   Command
  ---------   ----   -----      -----------     --------   -------
  1           Job1   Completed  True            Server01   get-eventlog system

STEP 3: RECEIVE-JOB

To get the results of the job, use the Receive-Job cmdlet. Because the job results are automatically returned to the computer where the job object resides, you can get the results with a local Receive-Job command.

The following command uses the Receive-Job cmdlet to get the results of the job. It uses the session ID to identify the job. This command saves the job results in the $results variable. You can also redirect the results to a file.

  $results = receive-job -id 1

A process isn't "killed with SIGHUP" -- at least, not in the strict sense of the word. Rather, when the connection is dropped, the terminal's controlling process (in this case, Bash) is sent a hang-up signal*, which is commonly abbreviated the "HUP signal", or just SIGHUP.

Now, when a process receives a signal, it can handle it any way it wants**. The default for most signals (including HUP) is to exit immediately. However, the program is free to ignore the signal instead, or even to run some kind of signal handler function.

Bash chooses the last option. Its HUP signal handler checks to see if the "huponexit" option is true, and if so, sends SIGHUP to each of its child processes. Only once its finished with that does Bash exit.

Likewise, each child process is free to do whatever it wants when it receives the signal: leave it set to the default (i.e. die immediately), ignore it, or run a signal handler.

Nohup only changes the default action for the child process to "ignore". Once the child process is running, however, it's free change its own response to the signal.

This, I think, is why some programs die even though you ran them with nohup:

1. Nohup sets the default action to "ignore".
2. The program needs to do some kind of cleanup when it exits, so it installs a SIGHUP handler, incidentally overwriting the "ignore" flag.
3. When the SIGHUP arrives, the handler runs, cleaning up the program's data files (or whatever needed to be done) and exits the program.
4. The user doesn't know or care about the handler or cleanup, and just sees that the program exited despite nohup.

This is where "disown" comes in. A process that's been disowned by Bash is never sent the HUP signal, regardless of the huponexit option. So even if the program sets up its own signal handler, the signal is never actually sent, so the handler never runs. Note, however, that if the program tries to display some text to a user that's logged out, it will cause an I/O error, which could cause the program to exit anyway.

* And, yes, before you ask, the "hang-up" terminology is left over from UNIX's dialup mainframe days.

** Most signals, anyway. SIGKILL, for instance, always causes the program to terminate immediately, period.