SSH Latency – Why SSH Freezes for Minutes While Other Traffic is Unaffected

latencyssh

I've noticed when I SSH into a remote machine over a slow link, SSH seems to "stick" after a relatively large amount of data is transferred.

For example, typing text over the link is fine, but running ls /etc will freeze the connection for a few minutes.

I would understand if the delay was because I was maxing out the connection speed and it returned to normal once the data had been transferred, but the connection freezes for far longer than you'd expect for the amount of data actually involved.

The weird thing is that if I open two SSH connections to and from the same machines, when one of them has frozen, the other one still works fine. So I can't be maxing out the connection speed, otherwise they would both freeze at the same time.

There is no traffic shaping active on either box or (as far as I can tell) the routers in between, so it shouldn't be anything dropping packets to keep the average transfer speed within a certain range.

Can anyone suggest anything that could cause this kind of behaviour, or anything further to check?

The same thing happens with scp and sshfs, with scp reporting a huge transfer rate (many MB/sec, then the speed slowly falls back to stalled for a few minutes, then if I'm lucky it'll repeat until the file finishes transferring.) sshfs works but often (not always) after a file is saved the mount point is non-responsive for a few minutes, temporarily blocking any program that tries to access it.

EDIT: I tried using iperf and get some interesting stats:

Local:  0.0- 0.9 sec   256 KBytes  2.25 Mbits/sec
Remote: 0.0- 7.0 sec   256 KBytes   302 Kbits/sec

Local:  0.0-15.9 sec  1.00 MBytes   529 Kbits/sec
Remote: 0.0-16.0 sec  1.00 MBytes   524 Kbits/sec

It looks like below a certain amout of data, the local end can send a lot faster than the remote can receive. I guess this isn't an SSH problem after all. I will investigate some TCP settings that might adjust this, but if anyone knows any for sure please advise!

Best Answer

Your SSH connection is exceeding the MTU size somewhere between client and server, and Path Maximum Transmission Unit Discovery is not working. (This is one of several reasons that blanket prevention of ICMP traffic in the name of security is a bad idea.)

Further reading

Related Question