How to include and exclude with rsync to back up the system

backuprsync

I have spent all day experimenting with rsync exclude-from and include-from, no success and completely lost. None of the examples or help I can find answer my exact, but quite simple requirement, and, either I can't make them work, or I do not get the result I want, nor can I understand why.

I am trying to set up an automatic daily backup, and after a few previous days attempts, based on this very good web site, which I have followed closely for incremental backups, and everything working well in tests, I have reached the stage of a dry-run of just my /home/Harry directory, without exclude-from and include-from, and it too works fine. I have also got the cron jobs sorted out that will run it for me.

Before anyone suggests using other than rsync and cron, note that I will be timing this to run daily at 11-30 p.m., with the computer shutting down a few minutes later. This too works fine in tests. If the computer is not on at that time it will not matter as there will be nothing to back up that day.

To progress further, and using the recommendations in "A Practical Guide to Fedora and Red Hat LInux", by Mark Sobell, 7th edition page 607, and other sources I want to refine it to include /home/, /var/, /usr/local/ and /etc/; and exclude /tmp/, /var/tmp/, /usr/tmp/, /proc/,/sys/ and /var/cache/.

Please: first of all are these sensible lists? and second: How do I specify this requirement in my rsync call?


Added after my question was answered.

Thanks to all who responded to my original question, I now have a working solution, listed in the script below. My experiences may be useful to anyone attempting something similar, and Michael Horner requested that I show the things that went wrong. Please refer to the website quoted for a complete description of how the cycle works, with a minimum of storage space needed.

My previous trials are, sort of, recorded in commented out lines in the script, which is what I used for my tests, originally to backup a few test files, and later to try to get the backup on to external hard drives. My practice is that, when I change something, I record the change by retaining and commenting out the original, so that I can go back to it again if necessary.

The long identifier starting CA6 is for a one Terabyte external hard drive on which I intend to store daily backups, probably with a 10 day cycle. I also have a 750 Gigabyte drive for weekly backups with probably a yearly cycle. You will see near the end of the script that this cycling has still to be attended to.

I am not concerned with any need to exactly restore the system if there is a crash. I have had three previous computers, none of them failed, but I kept manual backups on external hard drives and just use them, now, as a source for any material on them that I still need. So I would start with a bare installation and add what I need as I need it.

As hinted in my question, I experimented with using rsync exclude-from and include-from, but could not get the hang of the format of the necessary files. I might try to put the long list into a file, later.

#!/bin/bash
# Adapted from: http://webgnuru.com/linux/rsync_incremental.php
#  Website Backup Script
#======================================================================
# Define Variables
# Todays date in ISO-8601 format e.g. 2013-10-19:
DAY0=`date -I` 
# Yesterdays date in ISO-8601 format:
DAY1=`date -I -d "1 day ago"`
# The source directory:
SRC="/home/Harry/testrsync/bravo/"
#SRC="/home/Harry/"
# Filename of list of files to include
INC="/home/Harry/testrsync/include"
# Filename of file list to exclude
EXC="/home/Harry/testrsync/exclude"
# The target directory:
#TRG="/home/Harry/testrsync/backups/$DAY0"
TRG="/run/media/Harry/CA6C321E6C32062B/annals/$DAY0"
# The link destination directory:
#LNK="/home/Harry/testrsync/backups/$DAY1"
LNK="/run/media/Harry/CA6C321E6C32062B/$DAY1"
# The log file
#LOG="--logfile="/home/Harry/testrsync/backups/bulog --log-file-format="%t\n";""
LOG="--log-file=/home/Harry/testrsync/backups/bulog"
#The rsync options:
#OPT="--dry-run -avh --delete --link-dest=$LNK"
OPT="-aAXvh --delete --link-dest=$LNK"
#OPT="-ah --delete --link-dest=$LNK"
#======================================================================
#Execute the backup
#rsync -avvv $OPT --include-from=$INC --exclude-from=$EXC $SRC $TRG $LOG
#rsync -avv $OPT --include-from=$INC $SRC $TRG $LOG
#rsync $OPT--delete --stats --exclude-from=$EXC /home/Harry /var /usr/local /etc $TRG $LOG
rsync $OPT --exclude {"/dev/*","/proc/*","/sys/*","/tmp/*","/usr/tmp/*","/run/*","/mnt/*","/media/*","/var/cache/*","var/tmp/*","/","/lost+found"} /* $TRG $LOG
#Delete old backups cyclically
# for my tests I am going to use a three day cycle
DAY4=`date -I -d "4 days ago"`
#Delete the backup from 4 days ago, if it exists
if [ -d /home/Harry/testrsync/backups/$DAY4 ]
then
rm -r /home/Harry/testrsync/backups/$DAY4
#rm -r /run/media/HarryCA6C321E6C32062B/$DAY4
fi
# Cron does not output to a screen, so no result from next line
# zenity --info --text='Backup complete' --title="Backup Test"

Best Answer

So, the --exclude-from file, will be:

var/tmp/
var/cache/

and use rsync like this:

rsync -aAv --delete --stats --exclude-from /excludes /home /var /usr/local /etc /Backup/
Related Question