Caveats with gzip –rsyncable – no speedup

gziprsync

I am using (trying) gzip to compress an SQL file in an rsyncable way in order to transfer the backup with minimal delay.

However it appears that this is not working, since the speedup is always 1.00.

The archive is created by dumping the database to a .sql file and then issuing gzip -f3 --rsyncable file.sql.

Next the remote machine does an rsync against the last backup with the following flags:

rsync -avhhiP --inplace

Why might my speedup be 1.00? Should I not be recreating the archive each time and instead updating it perhaps? I have seen no mention of this method from online guidance about the usage of the –rsyncable flag.

I am using:

# gzip -V
gzip 1.5

Best Answer

The question is already somewhat old, but maybe my answer still helps one or the other:

Debian Wheezy has the aforementioned bug in gzip which renders the --rsyncable-flag non-functional.

See the corresponding Debian bug.

You can use pigz as a replacement, which is a parallelizing gzip-replacement which uses multiple CPUs, can compress slightly more efficiently if you manually specify larger block sizes and officially provides an --rsyncable implementation which is supposed to be better than the one provided by the gzip patch.

Additionally, rsyncs --inplace parameter reduces the efficiency of the delta-transmission algorithm - to quote the manpage:

The efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file. This does not apply if you use --backup, since rsync is smart enough to use the backup file as the basis file for the transfer.

Related Question