MacOS – ssh from the mac hangs after authenticating anywhere, regardless of app. It’s definitely the mac. Ideas

macosmojavesshterminal

I have a MacBook Pro running Mojave 10.14.1. When I try to ssh anywhere, after entering my password, the ssh session hangs immediately after authentication. Regardless of ssh app! And I'm baffled.

The facts:

  • This is just straight up ssh from the command line – no X forwarding or anything.
  • All other network on the Mac is fine (Safari, Outlook, Screen Sharing, etc.)
  • ssh does this regardless of destination. I did quite a lot of diagnosis, but just to completely eliminate the server side, I did this: (1) gen'd a new ssh key, (2) spun up a new Deb 9 VM at Digital Ocean with that key, (3) tried ssh with that key. Hung in the exact same spot with the exact same error. Used the key on another system and it worked fine, as expected.
  • It does this regardless of application. By that I mean I see the same behavior in macOS native Terminal, iTerm2, and RoyalTS. Weird! I assume they're all using some common library underneath but…what?
  • FileZilla using SFTP and the same credentials (and port) works just fine.
  • If I sudo to root on the Macbook and try the same ssh, I get the same result.
  • There is no VPN running on the mac (or the servers).
  • Below is a session example from a server on my LAN. I know I was able to ssh to this server on Monday morning. Still works coming from other systems. In this case I'm going over wifi to a server that is on Ethernet on the same subnet.
  • This is not a DNS timeout (i.e., fixed by UseDNS).
  • Exact same behavior regardless if password or key. Ends at the same "open confirm rwindow" spot with the same error.

Just to sex up the story:

  • I sent this system in to Apple via the Genius Bar for a screen replacement (the delaminating problem recall) on Monday. Got it back from their Houston, TX repair facility this morning (round trip: about 88 hours).
  • Same serial number but the lid's been completely replaced (missing a scratch), which is fine. I don't own a tinfoil hat but ssh worked perfectly immediately before I sent it to Apple…hmmm…;-)

Here is client side…the only changes I made to the log below are a global search/replace for my name (to MYUSER):

$ ssh -vv MYUSER@192.168.1.5
OpenSSH_7.8p1, LibreSSL 2.7.3
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.1.5 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.1.5 [192.168.1.5] port 22.
debug1: Connection established.
debug1: identity file /Users/MYUSER/.ssh/id_rsa type 0
debug1: identity file /Users/MYUSER/.ssh/id_rsa-cert type -1
debug1: identity file /Users/MYUSER/.ssh/id_dsa type -1
debug1: identity file /Users/MYUSER/.ssh/id_dsa-cert type -1
debug1: identity file /Users/MYUSER/.ssh/id_ecdsa type -1
debug1: identity file /Users/MYUSER/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/MYUSER/.ssh/id_ed25519 type -1
debug1: identity file /Users/MYUSER/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/MYUSER/.ssh/id_xmss type -1
debug1: identity file /Users/MYUSER/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u4
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.168.1.5:22 as 'MYUSER'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:oQ+J1YFF5ptE7oSYzQ6YXYGAOpwvzlfNAMsVlZsJIPI
debug1: Host '192.168.1.5' is known and matches the ECDSA host key.
debug1: Found key in /Users/MYUSER/.ssh/known_hosts:116
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /Users/MYUSER/.ssh/id_rsa (0x7f9e11513e00)
debug2: key: /Users/MYUSER/.ssh/id_dsa (0x0)
debug2: key: /Users/MYUSER/.ssh/id_ecdsa (0x0)
debug2: key: /Users/MYUSER/.ssh/id_ed25519 (0x0)
debug2: key: /Users/MYUSER/.ssh/id_xmss (0x0)
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:Jg8d5Nwt0kNN1geeAJpoNsyj6q7X1pJJ0JCXguxCEew /Users/MYUSER/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /Users/MYUSER/.ssh/id_dsa
debug1: Trying private key: /Users/MYUSER/.ssh/id_ecdsa
debug1: Trying private key: /Users/MYUSER/.ssh/id_ed25519
debug1: Trying private key: /Users/MYUSER/.ssh/id_xmss
debug2: we did not send a packet, disable method
debug1: Next authentication method: password
MYUSER@192.168.1.5's password: 
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
Authenticated to 192.168.1.5 ([192.168.1.5]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768

Long wait here, and then…

packet_write_wait: Connection to 192.168.1.5 port 22: Broken pipe

The only non-comment in the macbook's /etc/ssh/ssh_config is

Host *
    SendEnv LANG LC_*

…which is the same as another iMac I checked.

Best Answer

https://superuser.com/questions/1373697/ssh-fails-on-macos-mojave-10-14-1 has the solution.

In /etc/ssh/ssh_config:

Host *
     IPQoS lowdelay throughput

IIRC, the Genius Bar tech did a Mojave update before sending it in (as I had a Q about a hardware experience), which explains the worked-before-not-after repair scenario.