Updates
- I've narrowed it down to only occurring on the wireless network at work. From home, I experience no lag/delay
- It also seems that after "heavy" SSH use and tunneling through one or two machines, the network drops me completely and I am forced to manually reconnect
- Looking at WireShark captures of the SSH session from OpenSSH versus JSch, it seems that OpenSSH has a handful of
TCP Dup Ack
,TCP Retransmission
, andTCP Spurious Retransmission
packets. But these should sort themselves out, being TCP..
Problem
Recently when connecting to remote machines using SSH, I've noticed that there is a distinct delay of around 1 second between me typing something and the characters actually appearing. If I hold down keys, they'll appear in "batches" of around 20 characters at ~1s intervals.
System Details
$ ssh -V
OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips 1 Mar 2016
$ echo $SHELL $TERM $DISPLAY /bin/zsh xterm-256color :0 $ uname -r; cat /proc/version 4.4.0-22-generic Linux version 4.4.0-22-generic (buildd@lgw01-41) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016
Observations
An odd observation is that the delay is there only when the allocation of a pseudo terminal occurs (i.e. the default, or the -t option). When the pseudo terminal is forcedly disabled, I can type commands and get their output without delay.
- My ~/.ssh/config file only contains "plain entries of the following format:
Host somename HostName 192.168.1.0 User someuser IdentityFile ~/.ssh/id_rsa
- There is no delay when I explicitly disable the pseudo terminal (i.e. using the -T option)
- Altering the settings mentioned in similar questions does not resolve the issue
- Ex: Top causes of slow ssh logins
- Most solutions I've found address the problem of slow login response. My issue occurs after I have logged in
- IntelliJ Idea Ultimate Edition can create SSH connections which do not have this issue. The IDE is running the Java SSH client JSch
- This has convinced me that it is a local issue and presumably not related to my network interface
- My coworkers on the same network do not have this issue, though they're using Macbooks
What I've Tried
- Enabling compressing, disabling X11 forwarding, various methods for speeding up the login process
- SSHing to localhost (no lag/delay here)
- SSHing to a variety of machines (i.e. not just one) — delay is present
- Downloading the latest stable version of OpenSSH and building it for my system
- The freshly compiled binary has the same issue, leading me to think it's caused by some wonky configs
- Ensuring my /etc/ssh/ssh_config looks "normal"
- I've included it at the end of the post just in case
- Purging SSH and re-installing using apt-get
- Dropping into a virtual TTY (I think that's what they're called…) using Ctrl + Alt + F{1-6} and SSHing from there
- Rebooting
- Switching to a different network
- The lag/delay only seems to be an issue on the network at work. At home, there's no noticeable delay/lag
- Some other stuff that I'm forgetting
tl;dr
My SSH binary/config/something seems to have a delay when typing and I've tried a lot of things with no success. Maybe it's some character buffer thing? Who knows! Hopefully you do.
ssh -vvv
OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips 1 Mar 2016 debug1: Reading configuration data /home/redacted/.ssh/config debug1: /home/redacted/.ssh/config line 6: Applying options for ops1 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: resolving "xx.xx.xx.xx" port 22 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to xx.xx.xx.xx [xx.xx.xx.xx] port 22. debug1: Connection established. debug1: identity file /home/redacted/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /home/redacted/.ssh/id_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to xx.xx.xx.xx:22 as 'redacted2' debug3: hostkeys_foreach: reading file "/home/redacted/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/redacted/.ssh/known_hosts:2 debug3: load_hostkeys: loaded 1 keys from xx.xx.xx.xx debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com,zlib debug2: compression stoc: none,zlib@openssh.com,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: host key algorithms: ssh-rsa,ssh-dss debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: MACs ctos: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: MACs stoc: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: compression ctos: none,zlib@openssh.com debug2: compression stoc: none,zlib@openssh.com debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: aes128-ctr MAC: umac-64@openssh.com compression: none debug1: kex: client->server cipher: aes128-ctr MAC: umac-64@openssh.com compression: none debug3: send packet: type 34 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048 closed debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close logout debug3: channel 0: will not send data after close debug2: channel 0: obuf empty debug2: channel 0: close_write debug2: channel 0: output drain -> closed debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1) debug3: send packet: type 1 debug1: fd 1 clearing O_NONBLOCK debug3: fd 2 is not O_NONBLOCK Connection to xx.xx.xx.xx closed. Transferred: sent 3376, received 3072 bytes, in 0.8 seconds Bytes per second: sent 3981.4, received 3622.9 debug1: Exit status 0
ssh -vvvT
OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips 1 Mar 2016 debug1: Reading configuration data /home/redacted/.ssh/config debug1: /home/redacted/.ssh/config line 6: Applying options for ops1 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: resolving "xx.xx.x.xx" port 22 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to xx.xx.x.xx [xx.xx.x.xx] port 22. debug1: Connection established. debug1: identity file /home/redacted/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /home/redacted/.ssh/id_rsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to xx.xx.x.xx:22 as 'redacted' debug3: hostkeys_foreach: reading file "/home/redacted/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/redacted/.ssh/known_hosts:2 debug3: load_hostkeys: loaded 1 keys from xx.xx.x.xx debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com,zlib debug2: compression stoc: none,zlib@openssh.com,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: host key algorithms: ssh-rsa,ssh-dss debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: MACs ctos: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: MACs stoc: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: compression ctos: none,zlib@openssh.com debug2: compression stoc: none,zlib@openssh.com debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: aes128-ctr MAC: umac-64@openssh.com compression: none debug1: kex: client->server cipher: aes128-ctr MAC: umac-64@openssh.com compression: none debug3: send packet: type 34 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048 drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug3: send packet: type 96 debug2: channel 0: input drain -> closed debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: close_write debug2: channel 0: output drain -> closed debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1) debug3: send packet: type 1 debug1: fd 1 clearing O_NONBLOCK debug3: fd 2 is not O_NONBLOCK Transferred: sent 3040, received 2800 bytes, in 1.9 seconds Bytes per second: sent 1598.6, received 1472.4 debug1: Exit status 0
/etc/ssh/ssh_config
# This is the ssh client system-wide configuration file. See # ssh_config(5) for more information. This file provides defaults for # users, and the values can be changed in per-user configuration files # or on the command line. # Configuration data is parsed as follows: # 1. command line options # 2. user-specific file # 3. system-wide file # Any configuration value is only changed the first time it is set. # Thus, host-specific definitions should be at the beginning of the # configuration file, and defaults at the end. # Site-wide defaults for some commonly used options. For a comprehensive # list of available options, their meanings and defaults, please see the # ssh_config(5) man page. Host * # ForwardAgent no # ForwardX11 no # ForwardX11Trusted yes # RhostsRSAAuthentication no # RSAAuthentication yes # PasswordAuthentication yes # HostbasedAuthentication no # GSSAPIAuthentication no # GSSAPIDelegateCredentials no # GSSAPIKeyExchange yes # GSSAPITrustDNS no # BatchMode no # CheckHostIP yes # AddressFamily any # ConnectTimeout 0 # StrictHostKeyChecking ask # IdentityFile ~/.ssh/identity # IdentityFile ~/.ssh/id_rsa # IdentityFile ~/.ssh/id_dsa # IdentityFile ~/.ssh/id_ecdsa # IdentityFile ~/.ssh/id_ed25519 # Port 22 # Protocol 2 # Cipher 3des # Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc # MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160 # EscapeChar ~ # Tunnel no # TunnelDevice any:any # PermitLocalCommand no # VisualHostKey no # ProxyCommand ssh -q -W %h:%p gateway.example.com # RekeyLimit 1G 1h # SendEnv LANG LC_* HashKnownHosts yes # GSSAPIAuthentication yes # GSSAPIDelegateCredentials no
Best Answer
Unfortunately I can't comment yet.
What comes to my mind on the client:
Have you had a look at
top
to see the CPU load? Maybe the SSH process uses up CPU for encryption.Have you had a look at
ssh -v[vv]
and inspected for some strangeness? Maybe server and client agree on a very secure cipher or MAC. Have a look for(where arcfour is really one of the weakest, lowest-CPU algorithms. I am connecting via an SSH proxy). Also, look for rekeying messages.
Also, compression might be an issue. However, ssh does not seem to be too explicit about the compression level here.
You didn't say explicitly, how 'remote' your remote servers are. In the same network, in a different network on the same campus, via the internet?
If via the internet, maybe your corporate firewall is deep-inspecting SSH traffic on port 22. Maybe a port change could help, if you can control the SSH servers
/etc/ssh/sshd_config
file.If via the internet, is SSH the 'only' tunnel you use, or are you using SSH via an additional VPN?
If on the same campus, also firewalling, routing, inspecting issues come to my mind. I once was blaming my internet provider for problems with my internet; it was damn slow. Went on for days. Until I found out that I enabled debugging on my Cisco router, which ate up resources there.
If local or remote, are you connecting directly or via an SSH proxy, gateway or jump host? In that case, like with the VPN, you have double encryption.