Inotify and rsync on large number of files

inotifyrsync

I am using inotify to watch a directory and sync files between servers using rsync. Syncing works perfectly, and memory usage is mostly not an issue. However, recently a large number of files were added (350k) and this has impacted performance, specifically on CPU. Now when rsync runs, CPU usage spikes to 90%/100% and rsync takes long to complete, there are 650k files being watched/synced.

Is there any way to speed up rsync and only rsync the directory that has been changed? Or alternatively to set up multiple inotifywaits on separate directories. Script being used is below.

UPDATE: I have added the –update flag and usage seems mostly unchanged

#! /bin/bash

EVENTS="CREATE,DELETE,MODIFY,MOVED_FROM,MOVED_TO"

inotifywait -e "$EVENTS" -m -r --format '%:e %f' /var/www/ --exclude '/var/www/.*cache.*' | (
WAITING="";
while true; do
    LINE="";
    read -t 1 LINE;
    if test -z "$LINE"; then
        if test ! -z "$WAITING"; then
                echo "CHANGE";
                WAITING="";
                rsync --update -alvzr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/
        fi;
    else
        WAITING=1;
    fi;
done)

Best Answer

If the server has a slow processor avoid checksums and compression with rsync. I would remove ht "-z" option in the rsync command.

rsync --update -alvr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/

Note that it will not avoid rsync to compare the 650k files. You could rsync subdirectories of /var/www one by one to reduce the number of files checked at one time.

Related Solutions

Linux – Using Inotifywait for Large Number of Files in a Directory

Is there a particular reason you are using:

 | awk '($2=="CREATE"){print $3}' > ~/output.file

instead inotifywait options like --format and --outfile ?

If I run:

inotifywait -m --format '%f' -e create /home/don/folder/ --outfile /home/don/output.file

then open another tab, cd to ~/folder and run:

time seq -w 00001 50000 | parallel touch {}

real    1m44.841s
user    3m22.042s
sys     1m34.001s

(so I get much more than 500 files per minute) everything works fine and output.file contains all the 50000 file names that I just created.
Once the process has finished writing the files to disk you can append them to your test.out (assuming you are always in ~/folder):

xargs < /home/don/output.file cat >> final.file

Or use read if you want to process files as they are created. So, while in ~/folder you could run:

inotifywait -m --format '%f' -e create ~/folder | while read file; do cat -- "$file" >> ~/test.out; done

Note that in inotifywait stable, -m and -t cannot be used together. Support for usage of both switches has been recently added so if you build inotify-tools from git you should be able to use monitor with timeout (to specify how long it has to wait for an appropriate event to occur before exiting). I've tested the git version on my system (exit if no create events occur within 2 seconds) and it works fine:

inotifywait -m -t 2 --format '%f' -e create ~/folder | while read file; do cat -- "$file" >> ~/test.out; done

How does inotify work

Inotify is an internal kernel facility. There is no “inotify file”. There are dedicated system calls inotify_init, inotify_add_watch and inotify_rm_watch that allow processes to register themselves to be notified when certain filesystem events happen. When the event happens, the process receives a description of the event through the file descriptor returned by inotify_init.

The OS isn't “told” that a file has been changed: it knows, because it's doing the changing. It's the application that's told that a file has been changed instead of having to go looking.

The program inotifywait provides a simple way to use inotify from the command line.

Best Answer

Related Solutions

Linux – Using Inotifywait for Large Number of Files in a Directory

How does inotify work

Related Question