Here ya go!
#!/bin/bash
# written by strobelight, you know who you are.
# license, MIT, go for it.
me=`basename $0`
EXCLUDES="\
--exclude '*~'
--exclude '.DS_Store'
"
CANDIDATES=/tmp/candidates
usage() {
cat <<EOF
$me last_diff_dir new_diff_dir [ dir_to_copy ]
where:
last_diff_dir is the directory containing the last differential
new_diff_dir is the directory you want files saved to
dir_to_copy is optional and is the directory to copy from (default .)
cd directory_to_backup
Full backup: $me full_back full_back
Diff backup: $me full_back diff_1
Diff backup: $me full_back diff_2
EOF
exit 1
}
get_dir() {
HERE=`pwd`
cd $1
x=`pwd`
cd $HERE
echo $x
}
if [ $# -lt 2 ]; then
usage
fi
LAST_DIR="$1"
NEW_DIR="$2"
DIR_TO_COPY="${3:-.}"
mkdir -p "$LAST_DIR" || exit 1
mkdir -p "$NEW_DIR" || exit 1
[ -d "$LAST_DIR" ] || usage
[ -d "$NEW_DIR" ] || usage
[ -d "$DIR_TO_COPY" ] || usage
LAST_DIR=`get_dir "$LAST_DIR"`
NEW_DIR=`get_dir "$NEW_DIR"`
DIR_TO_COPY=`get_dir "$DIR_TO_COPY"`
# get list of what's different
eval rsync -v --dry-run -axH --delete --update $EXCLUDES "$DIR_TO_COPY/" "$LAST_DIR" | awk '
/building file list/ { next }
/^$/ {next}
/bytes.*received/ { nextfile }
{
for(i=5;i<NF;i++) {
printf("%s ",$i)
}
printf("%s\n",$NF)
}
' | sed 's:/$::' > $CANDIDATES
#cat $CANDIDATES
# use list to backup
eval rsync --files-from=$CANDIDATES -lptgoDxH --delete $EXCLUDES ${DIR_TO_COPY}/ $NEW_DIR
For example, my current directory has 3 8k files:
$ ls -1sk
total 24
8 seg1
8 seg2
8 seg3
My full backup doesn't yet exist, let's call that directory full_bak
ls ../full_bak
ls: ../full_bak: No such file or directory
First we need a full backup from which to do differentials. I've copied the script to my $HOME/bin directory as test123.sh. When both args are the same, that's essentially performing a full backup.
$HOME/bin/test123.sh ../full_bak ../full_bak
script outputs
.
seg1
seg2
seg3
Now look at ../full_bak
$ ls -1sk ../full_bak
total 24
8 seg1
8 seg2
8 seg3
Make some changes
dd if=/dev/zero of=seg2 bs=512 count=11
Confirm there are differences:
$ diff -q . ../full_bak
Files ./seg2 and ../full_bak/seg2 differ
Now create a differential
$ $HOME/bin/test123.sh ../full_bak ../differential1
seg2
Look at differential having just the file thats different from the last full backup
$ ls -1sk ../differential1/
total 8
8 seg2
Make another change
dd if=/dev/zero of=seg4 bs=512 count=10
Check what's different
diff -q . ../full_bak
Files ./seg2 and ../full_bak/seg2 differ
Only in .: seg4
and see we have a new file that's not in our full backup, and a changed file from before.
Do another differential to another directory
$ $HOME/bin/test123.sh ../full_bak ../differential2
.
seg2
seg4
and see the new differential has the 1st differential as well as the new file
$ ls -1sk ../differential2
total 16
8 seg2
8 seg4
Differential Backups
Here's a fullbackup wrapper using test123.sh:
#!/bin/bash
FULLDIR=/media/mydisk/home
SRCDIR=/home/user
$HOME/bin/test123.sh $FULLDIR $FULLDIR $SRCDIR
Here's a differential script creating sub directories based on the hour:
#!/bin/bash
FULLDIR=/media/mydisk/fullbackup/home
DIFFDIR=/media/mydisk/differentials/home
SRCDIR=/home/user
DIFFSUB=`date '+BAK_%H'`
$HOME/bin/test123.sh $FULLDIR $DIFFDIR/$DIFFSUB $SRCDIR
Here's the solution I came up to (as of Feb 2017), to be able to fully customize rsync options used by Hyper Backup's tasks.
Tweaking DiskStation's rsync executable
Difficult to say if tweaking HyperBackup configuration and tasks files is possible, but I could come to the conclusion that it was using the rsync binary present in /usr/bin/
. The rest is to set an intermediate script that tweaks the passed options up.
- connect to DiskStation server via SSH with the 'admin' user
sudo -i
et enter the same password as the 'admin' user
mv /usr/bin/rsync /usr/bin/rsync.orig
touch /usr/bin/rsync
chmod 4755 /usr/bin/rsync
echo '#!/bin/sh
exec /usr/bin/rsync.orig "$@" --option-you-want-to-add' > /usr/bin/rsync
Any other softer solution is welcome.
Making sure the modification will resist updates
I am not sure whether or not DSM manages the system's rsync executable, and if so, a DSM update could occur the modification we made to vanish into the void. Can anyone confirm that?
If so, I would then come up with a script, which I would program via Control Panel
> Task Scheduler
regularly (everyday at midnight for example), to ensure the modification will persist over updates, and that updates of the rsync binary itself will be taken into account.
First, I would set my modified rsync script to a path where I can make it evolve (if my modifications have to change over time):
/usr/local/bin/rsync_modified.sh :
#!/bin/sh
# This script is a custom modification script
# It calls the original binary rsync with modified options
# ordering to kill all child processes if receiving term signal
trap 'pkill -P $$' EXIT
# args to array
args2array () {
for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
echo " "
}
ARGS=$(args2array "$@")
# ...whatever modification you want to make on $ARGS...
# setting arguments again, from $ARGS
eval "set -- ${ARGS[@]}"
/usr/bin/rsync.orig "$@"
# Notes: the args2array call and the arguments setting in the end helps
# giving rsync.orig the arguments as they would be passed through direct
# call. Adding string or expanded arrays during the call of rsync.orig has
# revealed to fail in some cases, where rsync would ignore some of the
# added parameters.
then I would create this script which I could program in the Scheduled Tasks (with user 'root')
/usr/local/bin/rsyncUpdateManager.sh :
#!/bin/sh
# Make sure the modified version of rsync is not overwritten
# and that updates of the original rsync binary are taken into account.
# init
usageFile="/usr/bin/rsync"
origFile="/usr/bin/rsync.orig"
backupFile="/usr/bin/rsync.orig"
modificationScript="/usr/local/bin/rsync_modified.sh"
# check if usage file is a binary
grep -qI . $usageFile && TYPE="TEXT" || TYPE="BINARY"
if [ $TYPE == "BINARY" ]
then
# 1st installation or DSM updated rsync
if [ -f $origFile ]
then
# a original file already exists (probably created by this script)
# we back it up
NOW=$(date +"%Y%m%d_%H%M%S")
mv $origFile "${backupFile}.$NOW.bak"
fi
# rename binary file as original file
mv $usageFile $origFile
fi
# copy modification script in the place of usage file
cp $modificationScript $usageFile
# giving it the same rights as original file (on DiskStation server)
chmod 4755 $usageFile
Best Answer
The
-c
option forces rsync to calculate a checksum for all files. This means that it is reading the contents of all the video files before it starts deciding which ones to copy. That is obviously slow.Consider removing that option if possible. You only need it in rare cases. See the description on the manual page for further details: