Safest way to copy files to AirPort Disk from the command-line

airdiskairportrsyncterminal

I have just started using an AirPort disk (a Drobo connected by USB to the AirPort Extreme). I am trying to decide what is the ‘safest’ way to copy files to the Drobo from my MacBook Pro, which connects to the AirPort’s Wi-Fi network (802.11n-only / 5GHz).

(By “safe” I mean “able to tell that something went wrong during the transfer and throw an error” especially considering that I’ll be transferring large quantities of data to a disk which might disappear over a Wi-Fi connection. Seems reasonable to take an extra precaution or two.)

It appears that rsync does a checksum to make sure that the transfer was successful, according to this section of man rsync:

-c, –checksum

…

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check.

Except that I am not sure (and cannot find out any official source to tell me) if that is true for copying files between different folders on the same computer, since (technically) there is no “receiving side” in the traditional sense of copying from one computer to another. It certainly does not seem like rsync is taking extra time to do a checksum after the transfer, but that’s mostly guesswork, and I’m not sure how to test it sufficiently to consider any decision I might make conclusive.

(In case anyone is tempted to suggest that I read the source code of rsync I’m not familiar enough with programming to do that. In fact, I’m not even sure where to find the actual rsync code that OS X uses. See also: links on this page.)

The relevant bit of scripting that I am intending to use is here:

for i in "$@"
do
    /usr/bin/rsync \
    --8-bit-output \
    --exclude='*/.DocumentRevisions-V100/*' \
    --exclude='*/.fseventsd/*' \
    --exclude='*/.Spotlight-V100/*' \
    --exclude='*/.TemporaryItems/*' \
    --exclude='*/.Trashes/*' \
    --extended-attributes \
    --human-readable \
    --inplace \
    --itemize-changes \
    --links \
    --partial \
    --progress \
    --recursive \
    --stats \
    --timeout=30 \ 
    --times \
    --verbose \
    -- "${i}" "${DEST}/"
done

(n.b. $DEST is previously defined as /Volumes/DroboUSB-Backups)

This will also skip over files which have previously been copied, but fill in any missing ones, much like the "merge" command we all wish the Finder had.

1) Did I miss any options to rsync that I should use?

(Note that I have intentionally not included --compress as most of files I’ll be copying will already be compressed.)

2) Are there any other programs that I ought to consider instead of rsync? (Alternatives have to be something I can script from the command line, so not Transmit or ChronoSync.)

Best Answer

Using rsync is likely your best option and the flags enumerated in your question are reasonable.

An alternative tool to use would be dd. This tool approaches the copying process at a lower level and is often promoted where exact copies of disks are required. The dd manual has all the options. I prefer rsync for its relative simplicity.

Post Copy Check

You may want to extend your script to perform a post-rsync check. Use another tool, such as diff to compare the source and destination folders are equivalent.

If diff does not provide the incremental control you seek, consider comparing the md5 hashes from the source and destination's contents. See How can I calculate an md5 checksum of a directory?

Related Question