Centos – Why does the systemd unit fail to run on server reboot

apache-httpdcentosrebootsystemd

I'm having troubles getting a systemd service unit file to run correctly when the server reboots. This server is a fresh CentOS 7.0.1406 install. I'm trying to set it up with multiple Apache httpd instances. On an older CentOS install I had the httpd instances working fine and so now I'm trying to port those configurations over to the new CentOS which uses systemd.

I can enable the service, start it, restart it and stop it without any issues. When I do a reboot it says "No such file or directory". If I reenable the service, everything works again until the next reboot.

Service file:

[Unit]
Description=Apache web server instance apache01
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/data/apacheinstances/apache01/logs/httpd.pid
ExecStart=/data/apacheinstances/apache01/bin/apachectl start
ExecStop=/data/apacheinstances/apache01/bin/apachectl stop
ExecReload=/data/apacheinstances/apache01/bin/apachectl reload
PrivateTmp=true
LimitNOFILE=infinity

[Install]
WantedBy=multi-user.target

How I enabled it:

$ sudo systemctl enable /data/apacheinstances/apache01/bin/apache01.service
ln -s '/data/apacheinstances/apache01/bin/apache01.service' '/etc/systemd/system/multi-user.target.wants/apache01.service'
ln -s '/data/apacheinstances/apache01/bin/apache01.service' '/etc/systemd/system/apache01.service'


$ sudo systemctl status apache01
apache01.service - Apache web server instance apache01
   Loaded: loaded (/data/apacheinstances/apache01/bin/apache01.service; enabled)
   Active: inactive (dead)

How I started it:

$ sudo systemctl start apache01


$ sudo systemctl status apache01
apache01.service - Apache web server instance apache01
   Loaded: loaded (/data/apacheinstances/apache01/bin/apache01.service; enabled)
   Active: active (running) since Wed 2014-10-08 14:37:56 PDT; 13s ago
  Process: 1740 ExecStart=/data/apacheinstances/apache01/bin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 1746 (httpd)
   CGroup: /system.slice/apache01.service
           +-1746 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1747 /usr/local/sbin/cronolog /data/apacheinstances/apache01/logs/error_log-%Y%m
           +-1748 /usr/local/sbin/cronolog /data/apacheinstances/apache01/logs/access_log-%Y%m
           +-1749 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1750 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1751 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start

Oct 08 14:37:56 server105 apachectl[1740]: Starting apache01: [  OK  ]
Oct 08 14:37:56 server105 systemd[1]: PID file /data/apacheinstances/apache01/logs/httpd.pid not readable (yet?) after start.
Oct 08 14:37:56 server105 systemd[1]: Started Apache web server instance apache01.

How I restarted it:

$ sudo systemctl restart apache01


$ sudo systemctl status apache01
apache01.service - Apache web server instance apache01
   Loaded: loaded (/data/apacheinstances/apache01/bin/apache01.service; enabled)
   Active: active (running) since Wed 2014-10-08 14:38:40 PDT; 12s ago
  Process: 1836 ExecStop=/data/apacheinstances/apache01/bin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 1844 ExecStart=/data/apacheinstances/apache01/bin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 1850 (httpd)
   CGroup: /system.slice/apache01.service
           +-1850 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1851 /usr/local/sbin/cronolog /data/apacheinstances/apache01/logs/error_log-%Y%m
           +-1852 /usr/local/sbin/cronolog /data/apacheinstances/apache01/logs/access_log-%Y%m
           +-1853 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1855 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1856 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start

Oct 08 14:38:40 server105 apachectl[1844]: Starting apache01: [  OK  ]
Oct 08 14:38:40 server105 systemd[1]: PID file /data/apacheinstances/apache01/logs/httpd.pid not readable (yet?) after start.
Oct 08 14:38:40 server105 systemd[1]: Started Apache web server instance apache01.

How I stopped it:

$ sudo systemctl stop apache01


$ sudo systemctl status apache01
apache01.service - Apache web server instance apache01
   Loaded: loaded (/data/apacheinstances/apache01/bin/apache01.service; enabled)
   Active: inactive (dead) since Wed 2014-10-08 14:39:44 PDT; 12s ago
  Process: 1940 ExecStop=/data/apacheinstances/apache01/bin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 1844 ExecStart=/data/apacheinstances/apache01/bin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 1850 (code=exited, status=0/SUCCESS)

Oct 08 14:38:40 server105 apachectl[1844]: Starting apache01: [  OK  ]
Oct 08 14:38:40 server105 systemd[1]: PID file /data/apacheinstances/apache01/logs/httpd.pid not readable (yet?) after start.
Oct 08 14:38:40 server105 systemd[1]: Started Apache web server instance apache01.
Oct 08 14:39:44 server105 systemd[1]: Stopping Apache web server instance apache01...
Oct 08 14:39:44 server105 apachectl[1940]: Stopping apache01: [  OK  ]
Oct 08 14:39:44 server105 systemd[1]: Stopped Apache web server instance apache01.

Status after reboot:

$ sudo systemctl status apache01
apache01.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

Oct 08 14:44:58 server105 systemd[1]: Cannot add dependency job for unit apache01.service, ignoring: Unit apache01.service failed to load: No such file or directory.

How I reenable and start it:

$ sudo systemctl reenable /data/apacheinstances/apache01/bin/apache01.service
rm '/etc/systemd/system/apache01.service'
rm '/etc/systemd/system/multi-user.target.wants/apache01.service'
ln -s '/data/apacheinstances/apache01/bin/apache01.service' '/etc/systemd/system/multi-user.target.wants/apache01.service'
ln -s '/data/apacheinstances/apache01/bin/apache01.service' '/etc/systemd/system/apache01.service'


$ sudo systemctl start apache01


$ sudo systemctl status apache01
apache01.service - Apache web server instance apache01
   Loaded: loaded (/data/apacheinstances/apache01/bin/apache01.service; enabled)
   Active: active (running) since Wed 2014-10-08 14:52:20 PDT; 5s ago
  Process: 1737 ExecStart=/data/apacheinstances/apache01/bin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 1743 (httpd)
   CGroup: /system.slice/apache01.service
           +-1743 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1744 /usr/local/sbin/cronolog /data/apacheinstances/apache01/logs/error_log-%Y%m
           +-1745 /usr/local/sbin/cronolog /data/apacheinstances/apache01/logs/access_log-%Y%m
           +-1746 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1747 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start
           +-1748 /usr/local/httpd-2.4.10-apache01/bin/httpd -f /data/apacheinstances/apache01/conf/httpd.conf -k start

Oct 08 14:52:20 server105 apachectl[1737]: Starting apache01: [  OK  ]
Oct 08 14:52:20 server105 systemd[1]: PID file /data/apacheinstances/apache01/logs/httpd.pid not readable (yet?) after start.
Oct 08 14:52:20 server105 systemd[1]: Started Apache web server instance apache01.

I'm not sure why it work without issue before a reboot but fail afterwards. If I don't do the reenable I continue to get the same error about "No such file or directory".

I've tried to research logging functionality within systemd to determine which file or directory it cannot find but have drawn a blank.

Has anyone else experienced this kind of behaviour before?

Best Answer

All services need to be loaded and parsed before starting them, and your service isn't available early on in the boot process when systemd parses all services since it's on a different partition.

To solve it either keep the local services in /etc/systemd/system/ or make a package for your distribution and place them in /usr/lib/systemd/system/

Related Question