Restart systemd service when output is no longer generated

outputsystemdtimeout

Is it possible to restart a systemd service when there is no output generated for a predefined amount of time?

I have a script which apparently can hang, but that is not detected by systemd (or Python for that case), and thus it does not enter the failed state.

However, it does stop logging output, so I should be able to restart the service after no output is given for a minute. Is this possible with systemd?

My current systemd file:

[Unit]
Description=SOmething
After=network.target

[Service]
WorkingDirectory=/home/user/system/something
User=nobody
ExecStart=/usr/bin/python2 something.py
Restart=on-watchdog
RestartSec=10s

[Install]
WantedBy=multi-user.target

Best Answer

I don't think systemd allows you to do that, at least the systemd.service(5) manual page doesn't seem to mention anything like that.

However, what you could do is use systemd's builtin watchdog.

You would do that by settings WatchdogSec= and then having your service send WATCHDOG=1 with sd_notify regularly. If you set WatchdogSec=30 then your service must notify systemd at most every 30 seconds.

When your service hangs, it won't notify systemd anymore, and systemd will kill your service as a result.

With restart=on-watchdog (which you already have, but it doesn't do anything without WatchdogSec=), then systemd will restart your service after it's been killed by the watchdog.

Related Question