Inotifywatch: continue watch after file rotation

inotifylogrotate

I'm trying to write a script that uses inotifywatch to watch for changes in a log file. If a specific message is written to the log file, it's supposed to trigger a certain function. The script currently exists in this basic form:

while inotifywait -e modify /var/log/auth.log
do
  alert=$(tail -n1 /var/log/auth.log | grep -E -o ".{0,7}password")
  if [[ $alert == "Failed password" ]]
  then
   echo "FAILURE" >> test.log
  elif [[ $alert == "cepted password" ]]
  then
   echo "LOGIN" >> test.log
  fi
done

Everything works fine – up until the point where the logfile watched by inotifywatch is rotated. Then it ceases to function. I presume that's because during rotation the watched file is renamed and no longer written to afterwards and a new file with the old name is created in its place which inotify has never been told to watch in the first place.

I tried to circumvent this by switching from inotifywatch to using tail -f but the same problem seems to apply there as well.

Now I realize this could probably be solved by creating a huge if-structure in which inotifywatch not only watches for modify, but also for file creation and restarts a watch for modification. But I like to keep things simple, so does anyone know if there's a simpler way? (And please, no, I don't want to use prefabricated solutions like fail2ban etc. – the interesting part for me is to create stuff like this myself with simple tools.)

Best Answer

inotify is used to monitor files an directory by their inode, not by their name. When the file is rotated, its content does not change any longer (except for a short period, until the daemons are reloaded so that they use the newly created log file)

AFAIK, tail -f uses the inotify system, so it won't help. But if you have a working solution with tail -f then use tail --follow=name (or tail -F) if this is supported by your version of tail (POSIX tail does not support this). tail will then monitor the file identified by its filename. Here is an excerpt of the man page:

With --follow (-f), tail defaults to following the file descriptor, which means that even if a tail'ed file is renamed, tail will con‐ tinue to track its end. This default behavior is not desirable when you really want to track the actual name of the file, not the file descriptor (e.g., log rotation). Use --follow=name in that case. That causes tail to track the named file in a way that accommodates renaming, removal and creation.

[update]

Example of use:

tail -n0 -F my_file.log \
| while read -r log_line; do
    do_something_with "$log_line"
done

Because of the pipe, the while loop executes in a sub-process, which may cause you troubles if you want to modify variables outside the loop. If you use bash, you may want to use this alternative syntax which does not have this undesirable effect (but is less readable):

while read -r log_line; do
    do_something_with "$log_line"
done < <(tail -n0 -F my_file.log)

Related Solutions

Inotifywatch – Why Doesn’t It Detect Changes on Added Files?

This is due to the way you're using inotifywatch, and the way the tool itself works. When you run inotifywatch -r /tmp, you start watching /tmp and all the files that are already in it. When you create a file inside /tmp, the directory metadata is updated to contain the new file's inode number, which means that the change happens on /tmp, not /tmp/test-1. Additionally, since /tmp/test-1 wasn't there when inotifywatch started, there is no inotify watch placed on it. It means that any event which occurs on a file created after the watches have been placed will not be detected. You might understand it better if you see it yourself:

$ inotifywatch -rv /tmp &
Total of n watches.
$ cat /sys/kernel/debug/tracing/trace | grep inotifywatch | wc -l
n

If you have enabled the tracing mechanism on inotify_add_watch(2), the last command will give you the number of watches set up by inotifywatch. This number should the same as the one given by inotifywatch itself. Now, create a file inside /tmp and check again:

$ inotifywatch -rv /tmp &
Total of n watches.
$ touch /tmp/test1.txt
$ cat /sys/kernel/debug/tracing/trace | grep inotifywatch | wc -l
n

The number won't have increased, which means the new file isn't watched. Note that the behaviour is different if you create a directory instead :

$ inotifywatch -rv /tmp &
Total of n watches.
$ mkdir /tmp/test1
$ cat /sys/kernel/debug/tracing/trace | grep inotifywatch | wc -l
n + 1

This is due to the way the -r switch behaves:

-r, --recursive: [...] If new directories are created within watched directories they will automatically be watched.

^{Edit: I got a little confused between your two examples, but in the first case, the watches are correctly placed because the user calls inotifywatch on ~/* (which is expanded, see don_crissti's comment here). The home directory is also watched because ~/.* contains ~/.. Theoretically, it should also contain ~/.., which, combined with the -r switch, should result in watching the whole system.}

However, it is possible to get the name of the file triggering a create event in a watched directory, yet I'm guessing inotifywatch does not retrieve this information (it is saved a little deeper than the directory name). inotify-tools provides another tool, called inotifywait, which can behave pretty much like inotify-watch, and provides more output options (including %f, which is what you're looking for here) :

inotifywait -m --format "%e %f" /tmp

From the man page:

--format <fmt> Output in a user-specified format, using printf-like syntax. [...] The following conversions are supported:

%f: when an event occurs within a directory, this will be replaced with the name of the file which caused the event to occur.

%e: replaced with the Event(s) which occurred, comma-separated.

Besides, the -m option (monitor) will keep inotifywait running after the first event, which will reproduce a behaviour quite similar to inotifywatch's.

Best Answer

Related Solutions

Inotifywatch – Why Doesn’t It Detect Changes on Added Files?

Related Question