Stopping rsync without leaving temporary files behind

rsync

When copying files from directory A to B with rsync using

$ rsync -a --backup --suffix=.$(date +"%Y%m%d%H%M%S")  A/ B/

a file called filename in A will temporarily be called something on the form .filename.9Tcfsa in B while being transferred. That is, rsync adds some random characters to the name until the transfer is complete.

If I interrupt rsync with Ctrl+c while transferring filename, the temporary file .filename.9Tcfsa is left in B. Since my rsync command never removes any files from B, each interruption of rsync will leave another temporary file in B. This becomes annoying litter.

Is it possible to stop rsync and have it also removing the temporary file?

UPDATE:
Since other people seem not to experience the issue above, I am providing a script with output to demonstrate what I see on my machine.

Script rsynctest.sh:

#/!bin/bash
mkdir -p A
mkdir -p B
echo "Creating a 1 GB file in A..."
dd if=/dev/zero of=A/bigfile bs=1M count=1000 >& /dev/null
echo "Now press CTRL-C to interrupt rsync."
rsync -a --backup --suffix=.$(date +"%Y%m%d%H%M%S")  A/ B/

Output when running the script twice:

$ ./rsynctest.sh 
Creating a 1 GB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.0]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(521) [generator=3.1.0]
$ ./rsynctest.sh 
Creating a 1 GB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.0]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(521) [generator=3.1.0]
$ ls -a B
.  ..  .bigfile.KvDV0T  .bigfile.MbalWJ
$

Notice that the last ls command reveals that B contains two temporary files.

Best Answer

Answer: It's a bug in rsync 3.1.0.

This is a note from the 3.1.1. release notes,

Fixed a failure to remove the partial-transfer temp file when interrupted (and rsync is not saving the partial files).

Upgrade your version of rsync.

Previous stuff

I've just tested this on my system.

tony@trinity:~$ uname -a
Linux trinity 3.2.0-4-686-pae #1 SMP Debian 3.2.73-2+deb7u2 i686 GNU/Linux
tony@trinity:~$ cat /etc/debian_version
7.10
tony@trinity:~$ rsync --version
rsync  version 3.0.9  protocol version 30

So not quite the same version of rsync, but the same major version.

As expected, when I hit Ctrl+c, rsync tidies up and does not leave any temporary files behind.

I created A/ and B/, filled A/ with some files, and then ran rsync once to populate B/. I then just ran touch in A/ and ran the rsync again.

I added a -v so I could see which file it was working on, but the behaviour was the same without the -v.

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S")  A/ B/
sending incremental file list
archives/floppies.tgz
archives/fromx1.tgz
archives/homestuff.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [sender=3.0.9]
tony@trinity:~$ cd B
tony@trinity:~/B$ cd archives/
tony@trinity:~/B/archives$ ls -l
total 78620
-rw-r----- 1 tony tony  7031885 Apr 15 14:44 floppies.tgz
-rw-r----- 1 tony tony  7031885 Apr 15 14:37 floppies.tgz.20160415144513
-rw-r--r-- 1 tony tony 13959801 Apr 15 14:44 fromx1.tgz
-rw-r--r-- 1 tony tony 13959801 Apr 15 14:37 fromx1.tgz.20160415144513
-rw-r--r-- 1 tony tony 26136212 Apr 15 14:37 homestuff.tgz
-rw-r----- 1 tony tony  5727535 Apr 15 14:37 legacy-x1-scripts.tgz
-rw-r----- 1 tony tony  6636756 Apr 15 14:37 olddos.tgz
drwxr-xr-x 2 tony tony     4096 Apr 15 14:44 oldsites
drwxr-xr-x 2 tony tony     4096 Apr 15 14:44 temp

No temporary files. So, maybe because the contents haven't changed, rsync doesn't need to create a temporary file.

This time, I have a single big file in A\.

tony@trinity:~$ ls -l A/
total 561528
-rw-r--r-- 1 tony tony 574996664 Apr 15 14:52 bigfile.tgz

I sync B/

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") A/ B/
sending incremental file list
bigfile.tgz

sent 575066959 bytes  received 31 bytes  10953656.95 bytes/sec
total size is 574996664  speedup is 1.00

Then, completely replace bigfile.tgz with something else.

tony@trinity:~$ ls -l A/
total 572576
-rw-r--r-- 1 tony tony 586311642 Apr 15 14:57 bigfile.tgz

That's a different tgz archive just copied over and over to make up the same file size.

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") A/ B/
sending incremental file list
bigfile.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [sender=3.0.9]
tony@trinity:~$ ls -l B/
total 561528
-rw-r--r-- 1 tony tony 574996664 Apr 15 14:52 bigfile.tgz

No temporary file.

Starting again, new file.

tony@trinity:~$ ls -l A/
total 433908
-rw-r--r-- 1 tony tony 444315604 Apr 15 15:02 bigfile.tgz

Sync to B/

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S")  A/ B/
sending incremental file list
bigfile.tgz

sent 444369947 bytes  received 31 bytes  32916294.67 bytes/sec
total size is 444315604  speedup is 1.00

Now, recreate A/bigfile.tgz with different content,

tony@trinity:~$ ls -l A/
total 545312
-rw-r--r-- 1 tony tony 558392040 Apr 15 15:04 bigfile.tgz

This time, run the rsync with --partial and see what changes. That switch is the one which usually forces rsync to leave partial files behind rather than clearing them up.

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") --partial A/ B/
sending incremental file list
bigfile.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [sender=3.0.9]
tony@trinity:~$ ls -l B/
total 464596
-rw-r--r-- 1 tony tony  31424512 Jan  1  1970 bigfile.tgz
-rw-r--r-- 1 tony tony 444315604 Apr 15 15:02 bigfile.tgz.20160415150558

As we can, this time, rsync has created a temporary file (called bigfile.tgz) with the old one already given the new extension.

Edit: one set of tests again using ls -la.

tony@trinity:~$ ls -la A/
total 510488
drwxr-xr-x  2 tony tony      4096 Apr 15 15:22 .
drwxr-xr-x 89 tony tony      4096 Apr 15 15:12 ..
-rw-r--r--  1 tony tony 522724240 Apr 15 15:22 bigfile.tgz

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") A/ B/
sending incremental file list
bigfile.tgz

sent 522788155 bytes  received 31 bytes  22246305.79 bytes/sec
total size is 522724240  speedup is 1.00

So B/ is synced,

tony@trinity:~$ ls -la B/
total 510484
drwxr-xr-x  2 tony tony      4096 Apr 15 15:23 .
drwxr-xr-x 89 tony tony      4096 Apr 15 15:12 ..
-rw-r--r--  1 tony tony 522724240 Apr 15 15:22 bigfile.tgz

Replace A/bigfile.tgz and resync.

tony@trinity:~$ ls -la A/
total 545320
drwxr-xr-x  2 tony tony      4096 Apr 15 15:24 .
drwxr-xr-x 89 tony tony      4096 Apr 15 15:12 ..
-rw-r--r--  1 tony tony 558392040 Apr 15 15:24 bigfile.tgz

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") A/ B/
sending incremental file list
bigfile.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [sender=3.0.9]
tony@trinity:~$ ls -la B/
total 510484
drwxr-xr-x  2 tony tony      4096 Apr 15 15:25 .
drwxr-xr-x 89 tony tony      4096 Apr 15 15:12 ..
-rw-r--r--  1 tony tony 522724240 Apr 15 15:22 bigfile.tgz

No temporary file.

I can't recreate the behaviour you describe using basic rsync.

Are you sure your rsync command is exactly as described, and are you sure it's not aliased somewhere to be something else?

Update:

Using your script, on a different machine,

tony@matrix:~$ ./rsynctest.sh
Creating a 500 MB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
rsync: [sender] write error: Broken pipe (32)
tony@matrix:~$ ls -la B/
total 8
drwxr-xr-x  2 tony users 4096 Apr 15 19:54 .
drwxr-xr-x 37 tony users 4096 Apr 15 19:54 ..
tony@matrix:~$ ./rsynctest.sh
Creating a 500 MB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(504) [generator=3.1.1]
tony@matrix:~$ ls -la B/
total 8
drwxr-xr-x  2 tony users 4096 Apr 15 19:54 .
drwxr-xr-x 37 tony users 4096 Apr 15 19:54 ..

tony@matrix:~$ uname -a
Linux matrix 3.12.46-guest-39-a97a54c-x86_64 #4 SMP Mon Aug 10 11:59:25 UTC 2015 x86_64 GNU/Linux
tony@matrix:~$ cat /etc/debian_version
8.3
tony@matrix:~$ rsync --version
rsync  version 3.1.1  protocol version 31

Another machine, this time Ubuntu rather than pure Debian.

tony@neo:/$ lsb_release -r
Release:        15.10
tony@neo:/$ uname -a
Linux neo 4.2.0-34-generic #39-Ubuntu SMP Thu Mar 10 22:13:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
tony@neo:/$ rsync --version
rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

tony@neo:~$ vi rsynctest.sh
tony@neo:~$ chmod 755 rsynctest.sh
tony@neo:~$ ./rsynctest.sh
Creating a 1 GB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
rsync: [receiver] write error: Broken pipe (32)
rsync: [sender] write error: Broken pipe (32)
tony@neo:~$ ls -la B/
total 8
drwxrwxr-x  2 tony tony 4096 Apr 15 20:09 .
drwxr-xr-x 20 tony tony 4096 Apr 15 20:09 ..
tony@neo:~$ ./rsynctest.sh
Creating a 1 GB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
tony@neo:~$ ls -la B/
total 8
drwxrwxr-x  2 tony tony 4096 Apr 15 20:10 .
drwxr-xr-x 20 tony tony 4096 Apr 15 20:09 ..
tony@neo:~$ ./rsynctest.sh
Creating a 1 GB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
tony@neo:~$ ls -la B/
total 8
drwxrwxr-x  2 tony tony 4096 Apr 15 20:10 .
drwxr-xr-x 20 tony tony 4096 Apr 15 20:09 ..
Related Question