Freebsd – Understanding why such a discrepancy in network transfer

file-transferfreebsdnetworkingperformancersync

Summary:

I can't understand a peculiar discrepancy in network transfer.

  • Why is there such a discrepancy in syncing from one machine to the other and vice versa?

Also:

  • Given the maximum network transfer speed is about 110 M/sec, and the local disk speed for a similar operation is about 200 M/sec (so, no bottleneck there), why is there a much lower speed rsyncing between the two machines, than the theoretical 100M/sec?

Details:

First of all, server is

# uname -a                                                                                                                                                              
FreeBSD das 10.1-RELEASE-p8 FreeBSD 10.1-RELEASE-p8 #25 d0fb866(releng/10.1): Thu Nov     13 07:57:26 EST 2014     root@bellicose:/usr/obj/root/pcbsd-build-10-STABLE/git/freebsd/    sys/GENERIC  amd64

Client is:

# uname -a
Darwin compute.internal 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64

Both machines have 16GB of ram.

By doing a local rsync on the server, one kind of knows what to expect as disk speed, at least in that circumstance.

Using a binary file, test.bin, 732M, local rsync on FreeBSD server shows about 200 M/sec:

# rsync --stats -h test.bin copy.bin

[....]

sent 732.54M bytes  received 35 bytes  209.30M bytes/sec
total size is 732.36M  speedup is 1.00

That is about 200 M/sec.

On the mac mini client I have almost 70M/sec:

# rsync --progress --stats -h   test.bin copy.bin
test.bin
        732.36M 100%   70.06MB/s    0:00:09 (xfr#1, to-chk=0/1)

[....]

sent 732.54M bytes  received 35 bytes  69.77M bytes/sec
total size is 732.36M  speedup is 1.00

Now, doing a network speed test with iperf:

On the server (the FreeBSD server):

# iperf -f M -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.5 port 5001 connected with 192.168.1.226 port 50757
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  3356 MBytes   112 MBytes/sec

On the client (OS X mac mini):

# iperf -f M -M 9000 -c 192.168.1.5 -t 30 -w 80K
WARNING: attempt to set TCP maxmimum segment size to 9000 failed.
Setting the MSS may not be implemented on this OS.
------------------------------------------------------------
Client connecting to 192.168.1.5, TCP port 5001
TCP window size: 0.08 MByte (WARNING: requested 0.08 MByte)
------------------------------------------------------------
[  4] local 192.168.1.226 port 50757 connected with 192.168.1.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  3356 MBytes   112 MBytes/sec

So, I could assume that network connection (a straight cat 7 cable between the two nics) is about 110 M/sec.

Now, this is the puzzling situation:

If I rsync from the FreeBSD server to the mac mini, I get a transfer speed of about 50 M/sec:

# rsync --progress --stats -h test.bin -e "ssh -l gsl" '192.168.1.226:/tmp/'
Password:
test.bin
        732.36M 100%   57.10MB/s    0:00:12 (xfr#1, to-chk=0/1)

[....]

sent 732.45M bytes  received 46 bytes  50.51M bytes/sec
total size is 732.36M  speedup is 1.00

But rsync in the opposite direction gives a much lower transfer rate, 20M/sec:

# rsync --progress --stats -h   test.bin -e "ssh -l gsl" '192.168.1.6:/tmp/'
test.bin
        732.36M 100%   19.55MB/s    0:00:35 (xfr#1, to-chk=0/1)

[....]

sent 732.54M bytes  received 35 bytes  20.07M bytes/sec
total size is 732.36M  speedup is 1.00

My two questions:

  • Why is there such a discrepancy in syncing from one machine to the other and vice versa?

Also:

  • Given the maximum network transfer speed is about 110 M/sec, and the local disk speed for a similar operation is about 200 M/sec (so, no bottleneck there), why is there a much lower speed rsyncing between the two machines, than the theoretical 100M/sec?

Could somebody please help understand this, perhaps offering some advice on how to improve the transfer speed?

Update: Based on @dhag's answer, I tried to copy a file with netcat, ie using no compression:

On the "server" (pushing) side:

time cat test.bin | nc -l 2020
nc -l 2020  0.25s user 6.29s system 77% cpu 8.462 total

On the receiving side (FreeBSD):

time nc 192.168.1.226 2020 > test.bin
nc 192.168.1.226 2020 > test.bin  0.09s user 4.00s system 62% cpu 6.560 total

If I am not mistaken, that should mean 732M/6.29s = 117M/sec, which kind of exceed theiperf stats. Perhaps a caching issue?

Update 2: Using rsync with no encryption at all (only possible if using a daemon, and the rsync:// protocol):

# rsync  --progress --stats -h  test.bin rsync://gsl@192.168.1.5/share        ⏎
test.bin
     732.36M 100%  112.23MB/s    0:00:06 (xfer#1, to-check=0/1)

[....]

sent 732.45M bytes  received 38 bytes  112.69M bytes/sec
total size is 732.36M  speedup is 1.00

This also confirms @dhag's ideas.

Best Answer

I can only provide a guess, which is that the discrepancy is explained by varying computational, memory, caching or disk characteristics of the two hosts:

  • If we assume that CPU is a bottleneck, then it would make some sense if the slower machine were slower at sending (this assumes that encrypting is more computationally heavy than decrypting). This can be tested by switching to a cipher that is lighter to compute (try adding -c arcfour to your SSH command line; in this case, passing --rsh="ssh -c arcfour" to rsync).

  • If we assume that files are being read / written straight from / to the disk, then the disk could conceivably be a bottleneck; read speeds of 100 MBps are well within reach of more modern computers, but not of older ones, or computers running on laptop-class drives (such as, I believe, the Mac Mini).

  • If we further assume that the operating system uses filesystem caches, the situation could be complicated further:

    • If the source file is contained in the filesystem cache, in RAM, then it can be read much faster than 100 MBps;

    • if the target system applies write-caching and is able to fit a significant part of the file in RAM (in your case it should be, since RAM is much bigger than your test file), then it can claim the write is complete before it has actually reached the disk (this could mean that your measured 200MBps is).

The disk-versus-cache unknown can be tested by flushing the filesystem cache prior to reading (how to do so is OS-dependent): then sending the file will be at least as slow as the disk dictates. Conversely, by reading the file completely before sending (perhaps with cat file.bin >/dev/null), one can influence the OS into caching it.

To further investigate whether CPU is an issue, it would make sense to run top while the transfer is ongoing; if the rsync or ssh process is taking 100% of a core, then that would be a bottleneck.

Related Question