At the moment I'm implementing the part of the monitoring system that is build around the rsyslog and auditd. I would like to include into the project a correct verification of the fact that remote rsyslog client facility is running. This check should be repeated on the rsyslog server at short time intervals.
The simpliest way to do that is to trace the timestamps of the latest log file modifications. If the log file (which one is used by rsyslog server to redirect incoming messages) have not been modified recently, we can conclude that the remote rsyslog went down and sending no more messages. But I doubt the correctness of this method.
Best Answer
That's not a bad idea. To it improve a little, perhaps each of the clients could be configured to send a message at a regular interval (providing a "heartbeat" or a "mark"). With Rsyslog, the directive is
$ActionWriteAllMarkMessages
[on|off]
. But heed the man page:What if there is an undiscovered configuration problem on the Syslog server that causes messages to be directed to an unexpected location, such as the file you monitor to show that the client is alive? Perhaps a more rigorous test might be to (grep/awk) for the "heartbeat" or "mark" messages provided by
$ActionWriteAllMarkMessages
, looking for the time of the message from a specific host.To verify that the remote Syslog server is running, you could use netcat (
nc
) with the-z
and-u
switches. From the manual:For example, with a five-second timeout (
-w5
):Or, if the Syslog server also uses TCP, then you might omit the
-u
switch to take advantage of the reliability of TCP, if only for this purpose. This checks that the Syslog server is listening and available from the location of the server performing the connectivity check. There's another facet that has equal importance: disk space. (If the Syslog server runs out of space to write messages, then the Syslog server may as well be considered down.)Considering that UDP can be "unreliable" by design, counting on the "mark" or "heartbeat" message might not always work in a congested network or Syslog server. Another approach might be to install a script on each client. The script (BASH, Python, or whatever works) could return whichever error code you devise (e.g.: Syslog process not running? return 1 or try to start the Syslog process first, or devise other tests like connectivity tests, et al). Use
xinetd
to start the script from/etc/xinetd.d/script_name
(on RHEL/CentOS):On RHEL the restart command is
service xinetd restart
. Edit/etc/services
to give a name to port 6777. I like to useiptables
to augment theonly_from
stanza. By the way, the port, 6777, was used just for illustration.