INotify – Difference Between max_user_instances and max_user_watches

Is it safe to raise that value and what would be the consequences of a too high value?

Yes, it's safe to raise that value and below are the possible costs [source]:

• Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit) [sources: 1, 2]
• This comes out of kernel memory, which is unswappable.
• Assuming you set the max at 524288 and all were used (improbable), you'd be using approximately 256MB/512MB of 32-bit/64-bit kernel memory.
  • Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design.

To check the max number of inotify watches:

cat /proc/sys/fs/inotify/max_user_watches

To set max number of inotify watches

Temporarily:

• Run sudo sysctl fs.inotify.max_user_watches= with your preferred value at the end.

Permanently (more detailed info):

• put fs.inotify.max_user_watches=524288 into your sysctl settings. Depending on your system they might be in one of the following places:
  • Debian/RedHat: /etc/sysctl.conf
  • Arch: put a new file into /etc/sysctl.d/, e.g. /etc/sysctl.d/40-max-user-watches.conf
• you may wish to reload the sysctl settings to avoid a reboot: sysctl -p (Debian/RedHat) or sysctl --system (Arch)

Check to see if the max number of inotify watches have been reached:

Use tail with the -f (follow) option on any old file, e.g. tail -f /var/log/dmesg: - If all is well, it will show the last 10 lines and pause; abort with Ctrl-C - If you are out of watches, it will fail with this somewhat cryptic error:

tail: cannot watch '/var/log/dmsg': No space left on device

To see what's using up inotify watches

find /proc/*/fd -lname anon_inode:inotify |
   cut -d/ -f3 |
   xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' |
   uniq -c |
   sort -nr

The first column indicates the number of inotify fds (not the number of watches though) and the second shows the PID of that process [sources: 1, 2].

I don't use inotifywatch, I use gidget, so my answer isn't specific to that tool, it's just a hopefully useful observation about inotify (which I heavily use).

Each inotify watch uses 540 bytes of kernel memory on 32-bit architectures, and 1080 bytes on 64-bit architectures. Kernel memory is unswappable. So there is a memory cost, certainly.

But in my experience it's not the number of watches that slows things down - the kernel checks them quickly. What tends to impact system performance, in real use, is whatever you are doing when the inotify events trigger. If you have (for example) ten to forty thousand HIPAA compliant 835 files arriving at wire speed over a gigabit or ten gig link, the user processes being fired up to deal with each one are going to hammer the system far harder than inotify itself.

To answer at least one of your questions, though: No, setting watches will not have any effect/cost for unwatched files or folders. The kernel always checks to see if there is a watch set, and the check will take the same amount of time and resources as long as there isn't one, no matter how many other filesystem objects are being watched. But again, if you are constantly generating vast numbers of processes (for any reason) that will definitely have an impact on total system performance.

Also, you mentioned that the tool you're using can recursively watch folders & subfolders. That is not something that inotify does, it's something your tool is doing. It's probably scanning the folder you've targeted for subfolders, setting up watches on all of them, and then triggering whenever a new one is created to set up another watch. So you've got some overhead activity going on there that is only indirectly related to the inotify system, and the impact it has on performance is going to be mostly due to the tool's behavior not the behavior of inotify. If the tool is sloppy and resource inefficient (I don't know, but I kind of doubt it, since inotify tools are usually written in C) this could be an issue.