Systemd – Child Processes Killed When Main Process Exits

process-managementsystemdsysvinit

Don't normally post here but I am ripping my hair out over this one.
I have a Python script that forks when it launches, and is responsible for starting a bunch of other processes. This script used to be launched at startup via sysvinit, but recently I upgraded to Debian Jessie so have adapted it to launch via systemd.

Unfortunately, I'm running into an issue I can't work out. When you launch the script directly in a user shell, it launches it's child processes correctly, and when the script exits the child processes are orphaned and continue to run.

When launched Via systemd, if the parent process exits, the children all exit too (Well, the screens that they launch in die and appear as Dead).

Ideally I need to be able to restart the parent script without killing all the child processes, is there something that I am missing?

Thanks!

[Unit]
Description=Server commander
After=network.target

[Service]
User=serveruser
Type=forking
PIDFile=/var/Server/Server.pid

ExecStart=/var/Server/Server.py
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

Edit:

It's probably relevant for me to point out that the Python script is essentially a 'controller' for its child processes. It starts and stops servers in GNU screens as requested from a central server. It is normally always running, it doesn't spawn services and exit.

There are cases however where I would like to be able to reload the script without killing child processes, even if that means the processes are orphaned off to pid 1. In fact, it wouldn't even matter if the Python script started off processes as a parent process, if that is even possible.

A better explanation of how it works:

systemd spawns Server.py
Server.py forks and writes the pid file for systemd
Server.py then spawns server processes in gnu screen based on its instructions
Server.py continues to run to perform any restarts requested from the server

When launching without systemd, Server.py can be restarted and the GNU screens it launches are unaffected. When launching with systemd, when Server.py shuts down, instead of those screen processes being orphaned off to pid 1, they are killed.

Best Answer

I managed to fix this simply by setting KillMode to process instead of control-group (default). Thanks all!

Related Solutions

Systemd service can’t access user home

In theory it could be private mount namespace. It's a security feature. I guess you could try ruling it out with

ProtectHome=off

Creating a systemd script for Sybase ASE

Try adding to the [Service] a PIDFile= entry with the name of a file in which you should put the process id of your dataserver. Edit your ExecStart line to get this pid somehow, eg from the showserver command. Such a file may already be created by Sybase. You probably need to use Type=forking again.

Ensure the exit code from your Exec scripts is 0 if they work but systemd says they dont, eg by adding exit 0 at the end.

Perhaps your process is leaving the control group somehow. I used the following as a testing service: in /etc/systemd/system/myscript.service

[Unit]
Description=Try Forking using PIDFile
After=syslog.target
[Service]
Type=Forking
PIDFile=/tmp/mypid
ExecStart=/bin/bash -c '( (nohup sleep 99999 & echo $! >/tmp/mypid )& sleep 5 )'
ExecStop=/bin/bash -c 'kill -1 $(</tmp/mypid); >/tmp/mypid; exit 0'
[Install]
WantedBy=multi-user.target

I started this with

$ sudo systemctl daemon-reload
$ sudo systemctl enable myscript
$ sudo systemctl start myscript
$ sudo systemctl status myscript;echo $?
   Loaded: loaded (/etc/systemd/system/myscript.service; enabled)
   Active: active (running) since  09:01:33 CEST; 2s ago
 Main PID: 25485 (bash)
   CGroup: /system.slice/myscript.service
       |-25485 /bin/bash -c ( (nohup sleep 99999 & echo $! >/tmp/mypid )&...
       |-25486 /bin/bash -c ( (nohup sleep 99999 & echo $! >/tmp/mypid )&...
       |-25488 sleep 5
       `-25489 sleep 99999

After 5 seconds the status shows just the sleep running. Its pid is in /tmp/mypid.

$ sudo systemctl status myscript;echo $?
   Active: active (running) since  09:01:33 CEST; 11s ago
 Main PID: 25489 (sleep)
   CGroup: /system.slice/myscript.service
       `-25489 sleep 99999

A stop works, as the status shows:

$ sudo systemctl stop myscript;echo $?
0
$ sudo systemctl status myscript;echo $?
   Active: inactive (dead) since  09:01:55 CEST; 4s ago
  Process: 25504 ExecStop=/bin/bash -c kill ...
  Process: 25489 ExecStart=/bin/bash -c ( (nohup sleep ...
 Main PID: 25489 (code=killed, signal=TERM)
3

Starting again and killing the sleep by hand makes systemd notice the process death immediately and it runs the ExecStop and removes the /tmp/mypid file without needing to do a stop:

$ sudo systemctl start myscript
$ sudo kill -15 $(</tmp/mypid) # pid 25529
$ ps -p $(</tmp/mypid)
   bash: /tmp/mypid: No such file or directory
$ sudo systemctl status myscript;echo $?
   Active: inactive (dead) since  09:03:53 CEST; 9s ago
  Process: 25568 ExecStop=/bin/bash -c kill
  Process: 25529 ExecStart=/bin/bash -c ( (nohup sleep
 Main PID: 25529 (code=killed, signal=TERM)
   09:02:38 systemd[1]: Started Try Forking using PIDFile.
   09:03:53 bash[25568]: /bin/bash: line 0: kill: (25529) - No suc...ss
3
$ sudo systemctl disable myscript;echo $?

Best Answer

Related Solutions

Systemd service can’t access user home

Creating a systemd script for Sybase ASE

Related Question