Customize Synology Hyper-Backup rsync options (and signification of HB parameters)

Well, I didn't think I was going to be able to accomplish this until I recently discovered a nifty trick you can do with rsync, and since nobody has answered in a while I'll present my solution.

The trick is when you use the following arguments:

rsync --suffix "" --backup-dir "." ...

This causes rsync to backup files before modifying them, but the backups turn out to be in-place, so you're actually making copies of the files before modifying them. This allows you to change files that were hard-linked with out changing the originals.

Then, the sequence to accomplish the desired behavior could be the following:

# locally hard-link the mutable files
rsync -ahv --link-dest C --include-from MUTABLE_FILES.filter C/* B

# copy locally + append remotely changed files 
# (also delete mutable files that disappeared at remote location A)
rsync -ahbv --suffix "" --backup-dir "." --append-verify \
      --include-from MUTABLE_FILES.filter --delete A/* B 

# now hard-link locally + transfer immutable files
rsync -ahv --link-dest C --include-from IMMUTABLE_FILES.filter A/* B

This probably could be solved with the first two steps without using filters but in my particular use case to guarantee coherence in the final destination I need the mutable files transfered before the immutable ones, and the default alphabetic ordering done by rsync does not guarantee this in my case. The reason why I need that is that the mutable files may get deleted and replaced by an immutable file. If I didn't transfer the immutable file because it didn't exist at the moment, but the mutable disappears before I get to it, I'm left with neither and I loose data.

Here ya go!

#!/bin/bash

# written by strobelight, you know who you are.
# license, MIT, go for it.

me=`basename $0`

EXCLUDES="\
    --exclude '*~'
    --exclude '.DS_Store'
"

CANDIDATES=/tmp/candidates

usage() {
    cat <<EOF

$me last_diff_dir new_diff_dir [ dir_to_copy ]

where:
    last_diff_dir  is the directory containing the last differential
    new_diff_dir   is the directory you want files saved to
    dir_to_copy    is optional and is the directory to copy from (default .)

cd directory_to_backup
Full backup: $me full_back full_back
Diff backup: $me full_back diff_1
Diff backup: $me full_back diff_2

EOF
    exit 1
}

get_dir() {
    HERE=`pwd`
    cd $1
    x=`pwd`
    cd $HERE
    echo $x
}

if [ $# -lt 2 ]; then
    usage
fi

LAST_DIR="$1"
NEW_DIR="$2"
DIR_TO_COPY="${3:-.}"

mkdir -p "$LAST_DIR" || exit 1
mkdir -p "$NEW_DIR" || exit 1

[ -d "$LAST_DIR" ] || usage
[ -d "$NEW_DIR" ] || usage
[ -d "$DIR_TO_COPY" ] || usage

LAST_DIR=`get_dir "$LAST_DIR"`
NEW_DIR=`get_dir "$NEW_DIR"`
DIR_TO_COPY=`get_dir "$DIR_TO_COPY"`

# get list of what's different
eval rsync -v --dry-run -axH --delete --update $EXCLUDES "$DIR_TO_COPY/" "$LAST_DIR" | awk '
    /building file list/ { next }
    /^$/ {next}
    /bytes.*received/ { nextfile }
    {
        for(i=5;i<NF;i++) {
            printf("%s ",$i)
        }
        printf("%s\n",$NF)
    }
    ' | sed 's:/$::' > $CANDIDATES
#cat $CANDIDATES

# use list to backup 
eval rsync --files-from=$CANDIDATES -lptgoDxH --delete $EXCLUDES ${DIR_TO_COPY}/ $NEW_DIR

For example, my current directory has 3 8k files:

$ ls -1sk
total 24
 8 seg1
 8 seg2
 8 seg3

My full backup doesn't yet exist, let's call that directory full_bak

ls ../full_bak
ls: ../full_bak: No such file or directory

First we need a full backup from which to do differentials. I've copied the script to my $HOME/bin directory as test123.sh. When both args are the same, that's essentially performing a full backup.

$HOME/bin/test123.sh ../full_bak ../full_bak

script outputs

.
seg1
seg2
seg3

Now look at ../full_bak

$ ls -1sk ../full_bak
total 24
 8 seg1
 8 seg2
 8 seg3

Make some changes

dd if=/dev/zero of=seg2 bs=512 count=11

Confirm there are differences:

$ diff -q . ../full_bak
Files ./seg2 and ../full_bak/seg2 differ

Now create a differential

$ $HOME/bin/test123.sh ../full_bak ../differential1
seg2

Look at differential having just the file thats different from the last full backup

$ ls -1sk ../differential1/
total 8
 8 seg2

Make another change

dd if=/dev/zero of=seg4 bs=512 count=10

Check what's different

diff -q . ../full_bak
Files ./seg2 and ../full_bak/seg2 differ
Only in .: seg4

and see we have a new file that's not in our full backup, and a changed file from before.

Do another differential to another directory

$ $HOME/bin/test123.sh ../full_bak ../differential2
.
seg2
seg4

and see the new differential has the 1st differential as well as the new file

$ ls -1sk ../differential2
total 16
 8 seg2
 8 seg4

Differential Backups

Here's a fullbackup wrapper using test123.sh:

#!/bin/bash

FULLDIR=/media/mydisk/home
SRCDIR=/home/user

$HOME/bin/test123.sh $FULLDIR $FULLDIR $SRCDIR

Here's a differential script creating sub directories based on the hour:

#!/bin/bash

FULLDIR=/media/mydisk/fullbackup/home
DIFFDIR=/media/mydisk/differentials/home
SRCDIR=/home/user
DIFFSUB=`date '+BAK_%H'`

$HOME/bin/test123.sh $FULLDIR $DIFFDIR/$DIFFSUB $SRCDIR