Usage of –remove-source-files option of rsync

rsync

From the manpage of rsync

–remove-source-files

This tells rsync to remove from the sending side the files (meaning non-directories) that are a part of
the transfer and have been successfully duplicated on the receiving side.

  • Does it mean files on the sending side that are either part of the transfer or duplicated on the receiving side?

  • Can I also remove directories on the sending side?

Note that you should only use this option on source files that are quiescent.

  • What does "source files that are quiescent" mean?

If you are using this to
move files that show up in a particular directory over to another host, make sure that the finished
files get renamed into the source directory, not directly written into it, so that rsync can't possibly
transfer a file that is not yet fully written.

  • What does this mean?

If you can't first write the files into a different directory,
you should use a naming idiom that lets rsync avoid transferring files that are not yet finished (e.g.
name the file "foo.new" when it is written, rename it to "foo" when it is done, and then use the
option –exclude='*.new' for the rsync transfer).

  • What does this mean?

Starting with 3.1.0, rsync will skip the sender-side removal (and output an error) if the file's size or
modify time has not stayed unchanged.

  • What does this mean?

Thanks.

Best Answer

Q: Does it mean files on the sending side that are either part of the transfer or duplicated on the receiving side?

  • A: Both

Q: Can I also remove directories on the sending side?

  • A: Yes
    --remove-source-files then issue the command
    find <source_directory> -type d -empty -delete OR
    find <source_directory> -type l -type d -empty -delete (to include symlinks in the deletion)

(Was: --remove-source-files then issue the command rm -rf <source_directory>)

WARNING:

As mentioned in OrangeDog's comment, the rm -rf suggestion is unsafe. Specifically, any files that were for any reason not transferred (file changed between building the transfer list and starting to actually transfer that file, receiving side ran out of disk space, network connection dropped, etc.) will be left untouched in the source directory by rsync — but after your rm -rf invokation they're just gone. The find command above will recursively delete the empty source tree if all the source files have been successfully transferred and removed, but will leave alone any remaining files (and their containing directories, of course).

Q: What does "source files that are quiescent" mean?

  • A: It means files that have been written to and closed

Q: If you are using this to move files that show up in a particular directory over to another host, make sure that the finished files get renamed into the source directory, not directly written into it, so that rsync can't possibly transfer a file that is not yet fully written. What does this mean?

  • A: It means exactly what I said above

Q: If you can't first write the files into a different directory, you should use a naming idiom that lets rsync avoid transferring files that are not yet finished (e.g. name the file "foo.new" when it is written, rename it to "foo" when it is done, and then use the option --exclude='*.new' for the rsync transfer). What does this mean?

  • A: It means that RSYNC makes a list of files to be transferred first. Then it writes them into a different directory (Destination Directory), thus if you transfer a file that hasn't finished, it is best to rename it after it is done using the --exclude option

Q: Starting with 3.1.0, rsync will skip the sender-side removal (and output an error) if the file's size or modify time has not stayed unchanged. What does this mean?

  • A: If RSYNC detects that when its about to write the file to the destination directory that the file size has changed between the time it scanned it, to the time it actually writes it to the destination directory, then RSYNC will skip the file.
Related Question