Transferring millions of files from one server to another

file-transferperformancersyncscp

I have two servers. One of them has 15 million text files (about 40 GB). I am trying to transfer them to another server. I considered zipping them and transferring the archive, but I realized that this is not a good idea.

So I used the following command:

scp -r usrname@ip-address:/var/www/html/txt /var/www/html/txt

But I noticed that this command just transfers about 50,000 files and then the connection is lost.

Is there any better solution that allows me to transfer the entire collection of files? I mean to use something like rsync to transfer the files which didn't get transferred when the connection was lost. When another connection interrupt would occur, I would type the command again to transfer files, ignoring those which have already been transferred successfully.

This is not possible with scp, because it always begins from the first file.

Best Answer

As you say, use rsync:

rsync -azP /var/www/html/txt/ username@ip-address:/var/www/html/txt

The options are:

-a : enables archive mode, which preserves symbolic links and works recursively
-z : compress the data transfer to minimise network usage
-P : to display a progress bar and enables you to resume partial transfers

As @aim says in his answer, make sure you have a trailing / on the source directory (on both is fine too).

More info from the man page

Related Solutions

Stopping rsync without leaving temporary files behind

Answer: It's a bug in rsync 3.1.0.

This is a note from the 3.1.1. release notes,

Fixed a failure to remove the partial-transfer temp file when interrupted (and rsync is not saving the partial files).

Upgrade your version of rsync.

Previous stuff

I've just tested this on my system.

tony@trinity:~$ uname -a
Linux trinity 3.2.0-4-686-pae #1 SMP Debian 3.2.73-2+deb7u2 i686 GNU/Linux
tony@trinity:~$ cat /etc/debian_version
7.10
tony@trinity:~$ rsync --version
rsync  version 3.0.9  protocol version 30

So not quite the same version of rsync, but the same major version.

As expected, when I hit Ctrl+c, rsync tidies up and does not leave any temporary files behind.

I created A/ and B/, filled A/ with some files, and then ran rsync once to populate B/. I then just ran touch in A/ and ran the rsync again.

I added a -v so I could see which file it was working on, but the behaviour was the same without the -v.

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S")  A/ B/
sending incremental file list
archives/floppies.tgz
archives/fromx1.tgz
archives/homestuff.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [sender=3.0.9]
tony@trinity:~$ cd B
tony@trinity:~/B$ cd archives/
tony@trinity:~/B/archives$ ls -l
total 78620
-rw-r----- 1 tony tony  7031885 Apr 15 14:44 floppies.tgz
-rw-r----- 1 tony tony  7031885 Apr 15 14:37 floppies.tgz.20160415144513
-rw-r--r-- 1 tony tony 13959801 Apr 15 14:44 fromx1.tgz
-rw-r--r-- 1 tony tony 13959801 Apr 15 14:37 fromx1.tgz.20160415144513
-rw-r--r-- 1 tony tony 26136212 Apr 15 14:37 homestuff.tgz
-rw-r----- 1 tony tony  5727535 Apr 15 14:37 legacy-x1-scripts.tgz
-rw-r----- 1 tony tony  6636756 Apr 15 14:37 olddos.tgz
drwxr-xr-x 2 tony tony     4096 Apr 15 14:44 oldsites
drwxr-xr-x 2 tony tony     4096 Apr 15 14:44 temp

No temporary files. So, maybe because the contents haven't changed, rsync doesn't need to create a temporary file.

This time, I have a single big file in A\.

tony@trinity:~$ ls -l A/
total 561528
-rw-r--r-- 1 tony tony 574996664 Apr 15 14:52 bigfile.tgz

I sync B/

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") A/ B/
sending incremental file list
bigfile.tgz

sent 575066959 bytes  received 31 bytes  10953656.95 bytes/sec
total size is 574996664  speedup is 1.00

Then, completely replace bigfile.tgz with something else.

tony@trinity:~$ ls -l A/
total 572576
-rw-r--r-- 1 tony tony 586311642 Apr 15 14:57 bigfile.tgz

That's a different tgz archive just copied over and over to make up the same file size.

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") A/ B/
sending incremental file list
bigfile.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [sender=3.0.9]
tony@trinity:~$ ls -l B/
total 561528
-rw-r--r-- 1 tony tony 574996664 Apr 15 14:52 bigfile.tgz

No temporary file.

Starting again, new file.

tony@trinity:~$ ls -l A/
total 433908
-rw-r--r-- 1 tony tony 444315604 Apr 15 15:02 bigfile.tgz

Sync to B/

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S")  A/ B/
sending incremental file list
bigfile.tgz

sent 444369947 bytes  received 31 bytes  32916294.67 bytes/sec
total size is 444315604  speedup is 1.00

Now, recreate A/bigfile.tgz with different content,

tony@trinity:~$ ls -l A/
total 545312
-rw-r--r-- 1 tony tony 558392040 Apr 15 15:04 bigfile.tgz

This time, run the rsync with --partial and see what changes. That switch is the one which usually forces rsync to leave partial files behind rather than clearing them up.

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") --partial A/ B/
sending incremental file list
bigfile.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [sender=3.0.9]
tony@trinity:~$ ls -l B/
total 464596
-rw-r--r-- 1 tony tony  31424512 Jan  1  1970 bigfile.tgz
-rw-r--r-- 1 tony tony 444315604 Apr 15 15:02 bigfile.tgz.20160415150558

As we can, this time, rsync has created a temporary file (called bigfile.tgz) with the old one already given the new extension.

Edit: one set of tests again using ls -la.

tony@trinity:~$ ls -la A/
total 510488
drwxr-xr-x  2 tony tony      4096 Apr 15 15:22 .
drwxr-xr-x 89 tony tony      4096 Apr 15 15:12 ..
-rw-r--r--  1 tony tony 522724240 Apr 15 15:22 bigfile.tgz

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") A/ B/
sending incremental file list
bigfile.tgz

sent 522788155 bytes  received 31 bytes  22246305.79 bytes/sec
total size is 522724240  speedup is 1.00

So B/ is synced,

tony@trinity:~$ ls -la B/
total 510484
drwxr-xr-x  2 tony tony      4096 Apr 15 15:23 .
drwxr-xr-x 89 tony tony      4096 Apr 15 15:12 ..
-rw-r--r--  1 tony tony 522724240 Apr 15 15:22 bigfile.tgz

Replace A/bigfile.tgz and resync.

tony@trinity:~$ ls -la A/
total 545320
drwxr-xr-x  2 tony tony      4096 Apr 15 15:24 .
drwxr-xr-x 89 tony tony      4096 Apr 15 15:12 ..
-rw-r--r--  1 tony tony 558392040 Apr 15 15:24 bigfile.tgz

tony@trinity:~$ rsync -v -a --backup --suffix=.$(date +"%Y%m%d%H%M%S") A/ B/
sending incremental file list
bigfile.tgz
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [sender=3.0.9]
tony@trinity:~$ ls -la B/
total 510484
drwxr-xr-x  2 tony tony      4096 Apr 15 15:25 .
drwxr-xr-x 89 tony tony      4096 Apr 15 15:12 ..
-rw-r--r--  1 tony tony 522724240 Apr 15 15:22 bigfile.tgz

No temporary file.

I can't recreate the behaviour you describe using basic rsync.

Are you sure your rsync command is exactly as described, and are you sure it's not aliased somewhere to be something else?

Update:

Using your script, on a different machine,

tony@matrix:~$ ./rsynctest.sh
Creating a 500 MB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
rsync: [sender] write error: Broken pipe (32)
tony@matrix:~$ ls -la B/
total 8
drwxr-xr-x  2 tony users 4096 Apr 15 19:54 .
drwxr-xr-x 37 tony users 4096 Apr 15 19:54 ..
tony@matrix:~$ ./rsynctest.sh
Creating a 500 MB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(504) [generator=3.1.1]
tony@matrix:~$ ls -la B/
total 8
drwxr-xr-x  2 tony users 4096 Apr 15 19:54 .
drwxr-xr-x 37 tony users 4096 Apr 15 19:54 ..

tony@matrix:~$ uname -a
Linux matrix 3.12.46-guest-39-a97a54c-x86_64 #4 SMP Mon Aug 10 11:59:25 UTC 2015 x86_64 GNU/Linux
tony@matrix:~$ cat /etc/debian_version
8.3
tony@matrix:~$ rsync --version
rsync  version 3.1.1  protocol version 31

Another machine, this time Ubuntu rather than pure Debian.

tony@neo:/$ lsb_release -r
Release:        15.10
tony@neo:/$ uname -a
Linux neo 4.2.0-34-generic #39-Ubuntu SMP Thu Mar 10 22:13:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
tony@neo:/$ rsync --version
rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

tony@neo:~$ vi rsynctest.sh
tony@neo:~$ chmod 755 rsynctest.sh
tony@neo:~$ ./rsynctest.sh
Creating a 1 GB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
rsync: [receiver] write error: Broken pipe (32)
rsync: [sender] write error: Broken pipe (32)
tony@neo:~$ ls -la B/
total 8
drwxrwxr-x  2 tony tony 4096 Apr 15 20:09 .
drwxr-xr-x 20 tony tony 4096 Apr 15 20:09 ..
tony@neo:~$ ./rsynctest.sh
Creating a 1 GB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
tony@neo:~$ ls -la B/
total 8
drwxrwxr-x  2 tony tony 4096 Apr 15 20:10 .
drwxr-xr-x 20 tony tony 4096 Apr 15 20:09 ..
tony@neo:~$ ./rsynctest.sh
Creating a 1 GB file in A...
Now press CTRL-C to interrupt rsync.
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
tony@neo:~$ ls -la B/
total 8
drwxrwxr-x  2 tony tony 4096 Apr 15 20:10 .
drwxr-xr-x 20 tony tony 4096 Apr 15 20:09 ..

Best Answer

Related Solutions

Stopping rsync without leaving temporary files behind

Related Question