Why does the rsync not allow block-size>128K

rsync

Running rsync with a large* --block-size like this:

rsync -avvz --rsh 'ssh -c arcfour' --block-size 1048576 --inplace --progress example.com:/big.file /big.file

I get the following error:

Invalid block length 1048576 [sender]

Both ends are running 64bit CentOS 6.4. From Googling I've seen --block-size used with much higher values, why is this not working for me?

* I'm using a large block size because I'm trying to work around a bug where rsync just spins CPU forever 44% into a 300GB file

Best Answer

From the source:

int32 max_blength = protocol_version < 30 ? OLD_MAX_BLOCK_SIZE : MAX_BLOCK_SIZE;

sum->blength = read_int(f);
if (sum->blength < 0 || sum->blength > max_blength) {
    rprintf(FERROR, "Invalid block length %ld [%s]\n",
        (long)sum->blength, who_am_i());
    exit_cleanup(RERR_PROTOCOL);
}

Where:

#define OLD_MAX_BLOCK_SIZE ((int32)1 << 29)
#define MAX_BLOCK_SIZE ((int32)1 << 17)

Which is 536870912 (512M) and 131072 (128k) respectively.


The change was made in version v3.0.0 and support for OLD_ was added in v3.0.3. (Links explain some of the rationale behind the changes.)

  • [PATCH] A patch to try to make really large files get handled without bogging down in sender-side hashtable searching.

  • [PATCH] Fixed the sending of large files with older rsync versions by handling the old block-size limit for protocols < 29.

Related Question