Linux – SSH Agent Forwarding not working even when using `ssh -A`

First of all, I've checked all the following

but none of them helped. So, here is my question:

I can ssh from A to B, or from A to C, but not from A to B, then B to C. When connecting from A to either B or C, I always use ssh -A to force SSH Agent Forwarding.
But why I'm still not able to connect A -> B -> C without being asked for passphrase?

UPDATE: nearly three years later, the same problem still haunts me, but now I've narrowed down the problem a bit:

Symptom: I can ssh A -> B, or A -> C, but not A -> B -> C, or A -> C -> B.

The problem is exactly described by the topic — SSH Agent Forwarding is not working.

From

Troubleshoot SSH issues
https://confluence.atlassian.com/bitbucket/troubleshoot-ssh-issues-271943403.html

it says:

To list your loaded keys, enter ssh-add -l. If you don't see the SSH key you want to use…

then there is problem — the SSH key you want to use is NOT loaded.

This is what's happening when I'm at A -> B or A -> C. I.e., right after I did ssh -A into an intermediate server. the SSH key is lost, not forwarded and not loaded.

$ ssh-add -l
The agent has no identities.

That's the reason why I cannot ssh further without the passphrase.

It does have SSH_AUTH_SOCK variable setup and several ssh-agent around though:

$ echo "$SSH_AUTH_SOCK"
/tmp/ssh-RtEuLOmFDBet/agent.3722

$ ps -e  | grep [s]sh-agent
 3723 ?        00:00:00 ssh-agent
 4613 ?        00:00:00 ssh-agent

It does not seem to be related to my own environment as they are identical, or the /etc/ssh/sshd_config file, as I've compared the ones from the intermediate servers that are working or not working.

UPDATE END.

More info: all three machines, are configured with the standard Ubuntu ssh configuration. I.e., the AllowAgentForwarding option is not in /etc/ssh/sshd_config, though I doubt whether it should, because I saw "Since agent forwarding defaults on, it should be enough to remove any AllowAgentForwarding line from sshd_config." from Extra configuration required for ssh-agent forwarding?.

Some say ssh-add will do, but when I do it on B or C, it asks me to Enter passphrase for my .ssh/id_rsa. Some say check the SSH_AUTH_SOCK, but I do have it on B or C (either from A to B, or from A to C):

$ env | grep SSH_AUTH_SOCK
SSH_AUTH_SOCK=/tmp/ssh-RTScJ5PZh9Mh/agent.2083

Is the Agent Forwarding not working because of missing the AllowAgentForwarding option ? Then which one (A, B, or C) should I put it in? Wouldn't ssh -A be enough? Also I have the
.ssh/id_rsa files on both B and C, is that the reason ssh-add ask for the passphrase for them?

EDIT:

Here is the log of -Avvv from B to C:

OpenSSH_6.2p2 Ubuntu-6ubuntu0.1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/myid/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to boxc.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/myid/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/myid/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-1024
debug1: identity file /home/myid/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/myid/.ssh/id_dsa" as a RSA1 public key
debug1: identity file /home/myid/.ssh/id_dsa type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: identity file /home/myid/.ssh/id_dsa-cert type -1
debug1: identity file /home/myid/.ssh/id_ecdsa type -1
debug1: identity file /home/myid/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2p2 Ubuntu-6ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2p2 Ubuntu-6
debug1: match: OpenSSH_6.2p2 Ubuntu-6 pat OpenSSH*
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: boxc
debug3: load_hostkeys: loading entries for host "boxc" from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:15
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5-etm@openssh.com
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com zlib@openssh.com
debug2: mac_setup: found hmac-md5-etm@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com zlib@openssh.com
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA ed:26:20:93:4c:88:ef:17:70:e3:d4:7a:42:4c:8e:69
debug3: put_host_port: [192.168.2.122]:21
debug3: put_host_port: boxc
debug3: load_hostkeys: loading entries for host "boxc" from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:15
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries from file "/home/myid/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:16
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'boxc' is known and matches the RSA host key.
debug1: Found key in /home/myid/.ssh/known_hosts:15
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/myid/.ssh/id_rsa (0x7f7e....e760),
debug2: key: /home/myid/.ssh/id_dsa (0x7f7e....e7a0),
debug2: key: /home/myid/.ssh/id_ecdsa ((nil)),
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/myid/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 149
debug2: input_userauth_pk_ok: fp 22:32:...:1d:e3
debug3: sign_and_send_pubkey: RSA 22:32:...:1d:e3
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
Enter passphrase for key '/home/myid/.ssh/id_rsa':

I've compared with my good session (A->C), and found all are none different, other than the last 3 lines, which starts with "key_parse_private_pem: PEM_read_PrivateKey failed". The good session instead has:

debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to boxc

Everything else are the same.

Again, my environment:

$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:6.2p2-6ubuntu0.1
  Candidate: 1:6.2p2-6ubuntu0.4
  Version table:
     1:6.2p2-6ubuntu0.4 0
        500 http://archive.ubuntu.com/ubuntu/ saucy-updates/main amd64 Packages
     1:6.2p2-6ubuntu0.3 0
        500 http://security.ubuntu.com/ubuntu/ saucy-security/main amd64 Packages

% sshd -v
sshd: illegal option -- v
OpenSSH_6.2p2 Ubuntu-6ubuntu0.1, OpenSSL 1.0.1e 11 Feb 2013

Thanks

Best Answer

I've been trying to troubleshoot this problem for nearly seven years, and finally it get solved -- I launch keychain in my ~/.profile, which starts its own 'ssh-agent', even on machine B & C. This is the source of the problem, because keychain's ssh-agent overshadow the sshd provided one.

Removing it (keychain) from my ~/.profile solved the problem.

Update, another possibility, ssh-agent etc. usually get started as part of starting up the GUI on the local system. e.g., in another case, the call's hidden in /etc/X11/xdm/sys.xsession!

I confirm my SSH Agent Forwarding is working by doing, in MachineA,

ssh -t MachineB ssh MachineC

while ssh MachineB then, within it ssh MachineC was failing.

I'll start it (ssh-agent from keychain etc) manually only from machine A from now on.

Best Answer

Related Solutions

Linux – Did I just send the private ssh key

Unable to connect to SSH after generated public key and private key

Related Question