Systemd – syslog-ng Service Not Starting but Command Works

startupsyslog-ngsystemd

I have a freshly installed version on CentOS 7 once which I have installed syslog-ng from the EPEL repositories.

~: yum list | grep syslog
syslog-ng.x86_64                        3.5.6-1.el7                    @epel

When I try to start it via systemctl, it fails as follows :

/usr/lib/systemd/system: systemctl start syslog-ng
Job for syslog-ng.service failed. See 'systemctl status syslog-ng.service' and 'journalctl -xn' for details.

When looking into the journals, we can see that their is a dependency on the socket which "starts" fine but that the process returns an error about the arguments being incorrect as shown below :

May 07 17:26:15 superserver.company.corp systemd[1]: Starting Syslog Socket.
May 07 17:26:15 superserver.company.corp systemd[1]: Listening on Syslog Socket.
May 07 17:26:15 superserver.company.corp systemd[1]: Starting System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 07 17:26:15 superserver.company.corp systemd[1]: Failed to start System Logger Daemon.
May 07 17:26:15 superserver.company.corp systemd[1]: Unit syslog-ng.service entered failed state.
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service holdoff time over, scheduling restart.
May 07 17:26:15 superserver.company.corp systemd[1]: Stopping System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: Starting System Logger Daemon...
May 07 17:26:15 superserver.company.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT

If we look into the service configuration file, we can confirm the dependency on the socket and the command that is used to start the service.

[Service]
Type=notify
Sockets=syslog.socket
ExecStart=/usr/sbin/syslog-ng -F -p /var/run/syslogd.pid

The problem is that if I run the above-mentionned command, it starts up just fine and it works as expected.

My question is : what is difference between me running the program startup command and systemd starting up the same program ? What can I do to find out what is actually wrong with it ?


Edit 1

I enabled the debug output as suggested by Raymond in the answers and the output doesn't teach us much more.

May 08 10:31:29 server.corp systemd[1]: Starting System Logger Daemon...
May 08 10:31:29 server.corp systemd[1]: About to execute: /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
May 08 10:31:29 server.corp systemd[1]: Forked /usr/sbin/syslog-ng as 3121
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service changed dead -> start
May 08 10:31:29 server.corp systemd[1]: Set up jobs progress timerfd.
May 08 10:31:29 server.corp systemd[1]: Set up idle_pipe watch.
May 08 10:31:29 server.corp systemd[3121]: Executing: /usr/sbin/syslog-ng -F -p /var/run/syslogd.pid
May 08 10:31:29 server.corp systemd[1]: Got notification message for unit syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: Got message
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: got STATUS=Starting up... (Fri May  8 10:31:29 2015
May 08 10:31:29 server.corp systemd[1]: Got notification message for unit syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: Got message
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: got STATUS=Starting up... (Fri May  8 10:31:29 2015
May 08 10:31:29 server.corp systemd[1]: Received SIGCHLD from PID 3121 (syslog-ng).
May 08 10:31:29 server.corp systemd[1]: Child 3121 (syslog-ng) died (code=exited, status=2/INVALIDARGUMENT)
May 08 10:31:29 server.corp systemd[1]: Child 3121 belongs to syslog-ng.service
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 08 10:31:29 server.corp systemd[1]: syslog-ng.service changed start -> failed
May 08 10:31:29 server.corp systemd[1]: Job syslog-ng.service/start finished, result=failed
May 08 10:31:29 server.corp systemd[1]: Failed to start System Logger Daemon. 

There are a few warnings that are displayed at the start of the syslog-ng processes (nothing that keeps it from starting properly) so I redirected all output to /dev/null but the end result is the same.

Also, as a side note, my entire system does not boot anymore if systemd is unable to syslog. This can be disabled with kernel options to log to kmesg.

Best Answer

We had the same problem on Debian 8.1, but fixed it by changing our syslog-ng local configuration to use unix-dgram instead of unix-socket.

I was clued in by this comment at RedHat Bugzilla:

Note about custom syslog-ng configurations files

People with custom syslog-ng configurations will most likely face upgrade problems due to the unix socket type mismatch between systemd and syslog-ng old configuration file:

  • systemd creates /dev/log as unix-dgram
  • syslog-ng < 3.2.5 expected /dev/log to be unix-stream (configuration file)

If you use 'unix-stream ("/dev/log")' in one of your log messages sources, you will need to manually change it to 'unix-dgram ("/dev/log")'.

Related Question