What keeps one side of an rsync so busy

btrfsiomdadmperformancersync

I've got a Debian machine on my LAN serving as a backup server for the others. It has four HDDs combined into a software RAID 5 md device, on that an LVM, and on that btrfs. Backups are made using rsync, and for a large filesystem take over an hour. For a long time I had thought there would be little I could do about this.

Recently, however, I noticed that the HDD activity was very different on both ends of the transfer. While the sending side, running Gentoo and mostly using ext4, had hardly any disk IO at all, the receiving side was constantly busy. As most data wouldn't change between transfers, I believe that metadata reads should make up the bulk of the data. But I'd be really surprised if reading inodes in btrfs is so much work than doing the same in ext4.

iotop confirmed disk reads of about 1-4 MB/s on the receiving side, whereas the sending side had only the occasional 0.5 MB/s burst.

My question is, can anyone explain what's going on here? Preferably with some indication how to work around the problem if possible.

Perhaps there is some btrfs tuning flag I could use, or something similar. I need a FS with snapshot capabilities on the backup server, and my attempt to use FreeBSD and ZFS quickly lead to an inconsistent FS, so I see little alternative to btrfs at the moment. Therefore answers telling me to use ext4 or zfs might receive upvotes but no check mark.

Rsync options in use, as requested by cjm:

--rsync-path='rsync --fake-super'
--archive               # -rlptgoD
--hard-links            # detect and preserve these
--acls
--xattrs
--sparse
--noatime               # based on patch from samba #7249c1
--delete
--delete-delay
--fuzzy
--human-readable        # size suffixes, base 1000
--stats

As well as a bunch of -f rules to omit some files.

The mount options of the btrfs are reported by mount as

rw,nosuid,noexec,noatime,nospace_cache

In particular, this includes the noatime flag, so there shouldn't be any writing involved unless there actually were differences in some files. I added this information in response to the answer by Kyle Jones.

Best Answer

One possible answer is that the remote filesystem is mounted by default with the "atime" option. Access time writes for everything that the remote rsync accesses combined with the write penalty you suffer with RAID 5 (computing parity means reading all the RAID disks before you write to one of them) could explain the I/O magnification on the remote side.

If I'm right, you can speed things up by mounting the remote filesystem with the "noatime" option.

Related Solutions

Rsync: How to Delete Files on Receiving Side with Exceptions

If you use --delete and --exclude together what is in the excluded location won't get deleted even if the source files are removed.

But that raises the issue that the folder won't be rsync'd at all. So you will need another rsync job to sync that folder.

Eg.

rsync -nav /home/richardjh/keepall/ /home/backup/richardjh/keepall/
rsync -nav --exclude=keepall --delete /home/richardjh /home/backup/richardjh

You could run these the other way around, but then it would delete all removed files and then replace them, which is not as efficient.

You can't do it as a one liner.

Rsync, delete files on receiving side that were deleted on sending side. (But don’t delete everything)

What you want to do is reasonable, but using rsync to do it on its own is not. So the answer is no.

The reason is simple: rsync keeps no history of what was in each directory and has no way of knowing what needs to be deleted and what not. Not without additional support.

You should ask yourself why you like to do this with rsync and make that more clear. There are other programs that use librsync1.so that are more intelligent.

With the relaxed constraints that you don't need rsync per se, you can take a look at rdiff-backup:

mkdir a
touch a/xx
touch a/yy
rdiff-backup a b
ls b

This shows xx and yy are in b.

touch b/zz
rm a/xx
rdiff-backup a b

This shows xx and zz are in b. rdiff-backup also keeps a directory rdiff-backup-data in b so you can rollback any changes, you should purge this on a regular basis using the rdiff-backup commands. (The example is with local files to show extra data in the target does not get deleted, but rdiff-backup works over a network as well).

Another alternative is to setup some distributed revision control system (mercurial, bazaar, git). With mercurial e.g. you can have a script (I use a Makefile for that), that pushes all the changes to the server and then does an update of the checked out files there, ignore any additional files that are on the remote server (but have not been put under revision control).

On the server you would do:

hg init
hg add file_list_excluding_that_should_not_should_be_deleted_if_not_on_client
hg commit -m "initial setup"

On the client:

hg clone ssh://username@server/dir_to_repository

Now if you remove a file on the client and do:

hg commit -m "removed file"
ssh username@server "cd dir_to_repository; hg update --clean"

Your removed file is removed on the server, but any other data (not added to the repository) does not get deleted.

Best Answer

Related Solutions

Rsync: How to Delete Files on Receiving Side with Exceptions

Rsync, delete files on receiving side that were deleted on sending side. (But don’t delete everything)

Related Question