My use cases are what Fabrice A. Marie's Logs duplicates with systemd article (published by Kibin Labs) places into its "ugly" category, and its brief advice to add in an entire ELK stack (it's the "only recourse", it says) for "ugly" is overkill for my needs.
Fabrice A. Marie's categorization is that "good" applications can have standard output redirected to /dev/null
without resultant loss of information, only write very little to standard error and hence to /var/log/messages
(via the systemd journal), and send the bulk of their logs through per-application logging mechanisms (unmanaged by systemd or rsyslog) to private application-maintained log files. This is, if anything, the reverse of what I want, and not "good" by my criteria.
I have several applications (one of which, picked for the sake of example here, is phantomjs backend prerendering, but this is just one of my use cases) that are launched as services under systemd. My applications send their log to standard output and standard error (this being the normal default behaviour of console.log()
in node.js). It seems wrong that to make these qualify as "good" they should be rewritten to do their own private logging, instead of using systemd for that (as they are now). In any case, I'm not in a position to re-code these applications myself.
I appear, going by what Fabrice A. Marie says and by what else I have read, to have the stark choice of all of that standard output/standard error information going either to /dev/null
or to /var/log/messages
. Is there no happy median? Can I not set up systemd (and whatever else) so that my applications' logs, being written to standard output/error, are handled by systemd in the normal way, without an entire ELK stack in the middle, and yet end up in another place than /var/log/messages
?
I'm running both CentOS 7 and RHEL 7, and the service unit for my example service is currently this:
[Unit]
Description=prerender
After=network.target
[Service]
ExecStart=/usr/local/bin/node /var/www/prerender/node/server.js
Restart=always
User=prerender
Group=adm
Environment=PATH=/usr/bin:/usr/local/bin PORT=8900
WorkingDirectory=/var/www/prerender/node
[Install]
WantedBy=multi-user.target
Best Answer
That article looks like nonsense, putting the blame on other software for problems caused by systemd's hostile takeover of logging functions. It even mentions the solution (syslog) but only as a means to suggesting massive overkill like
logstash
.You don't need logstash to log a handful of daemons on a single host.
From the article:
It then completely ignores the obvious and sensible option of configuring
rsyslogd
and implies thatlogstash
andelasticsearch
are the only options if you want to do something that journald can't do.Modern syslog daemons, such as
rsyslog
andsyslog-ng
can be configured to filter syslog messages by all sorts of criteria, including daemon name and regex pattern match. The default may be/var/log/messages
, but it is trivially easy to change the default.So what you need are some
rsyslog
rules for filtering syslog messages from your service.Take whatever systemd advocates say about logging with a huge grain of salt - they like to pretend that
journald
was necessary because logging before that was primitive and inflexible, that they were solving a problem nobody had even thought about before let alone solved several times over.Also, BTW, the correct way for a program to do logging is neither stdout nor stderr, it's by using the syslog functions that are available in most languages.