Networking – SCP Performance Over 1Gbit LAN

ethernetnetworkingscp

On my wired LAN, with 1GBit/s devices, I have two Linux machines (One Haswell, One Skylake Xeon) and when I do a secure copy of a large file, I see 38MB/s.

Seeing that this is 3 times below the 1000Mbit/s spec, I wonder if this performance is as expected?

Both machines use SSD for storage, both run 64bit Ubuntu.

During the transfer, both machine have approximately one core at 30% load.

The router that sits between the machines is a TP-Link Archer C7 AC1750. Both machines have Intel(R) Gigabit Ethernet Network devices that are in Full Duplex mode.

What is a normal scp transfer speed on 1Gbit LANs?

UPDATE

  • Using /dev/zero to rule out disk IO yielded the same results.
  • Using nc yielded slightly higher: 41MiB/s.
  • Paradoxically, UDP nc was slower than TCP nc, at 38MiB/s?
  • Switching to crossover cable: 112MB/s for scp.

CONCLUSION

The TP-Link router in between was the weak link in the network, and could not keep up.

Best Answer

It does seem slow from a theoretical stand point although I've not seen any transfers much quicker practically on home hardware.

Some experiments you might like to try to rule out possible limiting factors:

  • Assess your raw SSH speed by copying from /dev/zero to /dev/null. This rules out a HD bottle neck. 
    ssh remote_host cat /dev/zero | pv > /dev/null
  • Check other unencrypted protocols such as HTTP. HTTP actually sends files as with nothing but a header. Sending large files over HTTP is a reasonable measure of TCP speeds.
  • Check that you are not forcing traffic through the router but only it's ethernet switch. For example if your machine has a public IP and local IP, then scp to/from the local IP. This is because home routers often have to process WAN traffic through their CPU which creates a bottle neck. Even if both machines are on the LAN, using the public IP can force packets through the CPU as if it was going to the WAN.
  • Similarly I would use IPv4. Some home routers have a weird behaviour with IPv6 where they ask all local traffic to be forwarded to the router.
  • If at all possible try with a gigabit crossover cable instead of a router. This should rule out router.
Related Question