Systemd service isn’t restarted, no explanation of failure

systemd

When my service fails I get the the following output:

Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Unit entered failed state.
Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Failed with result 'exit-code'.
Nov 06 10:23:58 irgprod2 systemd[1]: emsxsub.service: Service hold-off time over, scheduling restart.
Nov 06 10:23:58 irgprod2 systemd[1]: Stopped EMSX API Subscription.

There is no explanation of what has gone wrong or why the service wasn't restarted, no mention of "start-limit" or anything. This is the process:

[Unit]
Description=EMSX API Subscription
After=multi-user.target

[Service]
Type=simple
Restart=always
RestartSec=3
User=root
Environment=PYTHONPATH=/k/svn_prod
Environment=LD_LIBRARY_PATH=/opt/lib
ExecStart=/usr/bin/python /k/svn_prod/com_wrappers/bbg_emsxsub.py
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

I have tried without the User arg, without the RemainAfterExit arg, etc. The systemctl process is enabled, and the process runs and restarts successfully when I execute "service emsxsub restart". It is only when it fails that systemd schedules the restart and then stops the service without successfully restarting it. I cannot find anybody else with this problem – other issues seem to be a "start-limit" error which is not the case here. Has anybody seen this issue before? I cannot figure out why it all works when it is manually restarted but the service is unable to restart itself on failure.

Here is the journalctl output:

Nov 06 10:23:55 irgprod2 python[18451]: Traceback (most recent call last):
Nov 06 10:23:55 irgprod2 python[18451]:   File "/k/svn_prod/com_wrappers/bbg_emsxsub.py", line 533, in <module>
Nov 06 10:23:55 irgprod2 python[18451]:     main()
Nov 06 10:23:55 irgprod2 python[18451]:   File "/k/svn_prod/com_wrappers/bbg_emsxsub.py", line 505, in main
Nov 06 10:23:55 irgprod2 python[18451]:     raise RuntimeError()  # raise error so process restarts
Nov 06 10:23:55 irgprod2 python[18451]: RuntimeError
Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Main process exited, code=exited, status=1/FAILURE
Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Unit entered failed state.
Nov 06 10:23:55 irgprod2 systemd[1]: emsxsub.service: Failed with result 'exit-code'.
Nov 06 10:23:58 irgprod2 systemd[1]: emsxsub.service: Service hold-off time over, scheduling restart.
Nov 06 10:23:58 irgprod2 systemd[1]: Stopped EMSX API Subscription.

Output of systemctl status emsxsub.service:

* emsxsub.service - EMSX API Subscription
   Loaded: loaded (/lib/systemd/system/emsxsub.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Wed 2018-11-07 08:18:49 EST; 40s ago
  Process: 3936 ExecStart=/usr/bin/python /k/svn_prod/com_wrappers/bbg_emsxsub.py (code=exited, status=1/FAILURE)
 Main PID: 3936 (code=exited, status=1/FAILURE)

Nov 07 08:18:45 irgprod2 systemd[1]: emsxsub.service: Unit entered failed state.
Nov 07 08:18:45 irgprod2 systemd[1]: emsxsub.service: Failed with result 'exit-code'.
Nov 07 08:18:49 irgprod2 systemd[1]: emsxsub.service: Service hold-off time over, scheduling restart.
Nov 07 08:18:49 irgprod2 systemd[1]: Stopped EMSX API Subscription.

Output of systemctl show emsxsub.service | grep -i restart:

[trading@irgprod2 ~]$ systemctl show emsxsub.service | grep -i restart
Restart=always
RestartUSec=3s

systemctl version:

systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

Distro details:

[trading@irgprod2 ~]$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Best Answer

Removing After=multi-user.target solved the issue. Filipe's observation of a circular dependency seems to have been the correct call.

Related Question