The SSHFP records were generated on the ssh server as follows and then added to the zone in bind:
$ ssh-keygen -r www.test.us.
www.test.us. IN SSHFP 1 1 ad04dfaf343a93beeb939eed1612168f7eadbed7
www.test.us. IN SSHFP 2 1 432209c72c4f0e99546d601dd96c04ce804191f9
The required records can be grabbed from the ssh client via DNS as shown:
$ dig www.test.us any
;; QUESTION SECTION:
;www.test.us. IN ANY
;; ANSWER SECTION:
www.test.us. 120 IN SSHFP 1 1 AD04DFAF343A93BEEB939EED1612168F7EADBED7
www.test.us. 120 IN SSHFP 2 1 432209C72C4F0E99546D601DD96C04CE804191F9
www.test.us. 120 IN A 192.168.1.50
However ssh on the client fails to find them when connecting:
$ rm .ssh/known_hosts
$ ssh -vo VerifyHostKeyDNS=yes www
OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /Users/test/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 53: Applying options for *
debug1: Connecting to www [192.168.1.50] port 22.
debug1: Connection established.
debug1: identity file /Users/test/.ssh/id_rsa type 1
debug1: identity file /Users/test/.ssh/id_rsa-cert type -1
debug1: identity file /Users/test/.ssh/id_dsa type -1
debug1: identity file /Users/test/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.8p2_hpn13v11
debug1: match: OpenSSH_5.8p2_hpn13v11 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 69:dc:47:97:e1:a5:c9:07:4a:2b:9e:3c:a2:2b:c8:8c
DNS lookup error: name does not exist
The authenticity of host 'www (192.168.1.50)' can't be established.
RSA key fingerprint is 69:dc:47:97:e1:a5:c9:07:4a:2b:9e:3c:a2:2b:c8:8c.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?
Any ideas as to why this is failing? I know that DNSSEC is required to make it secure and that I should get a warning since DNSSEC is not currently enabled. I'm hoping to get this working without DNSSEC first before I start tackling that as an additional problem.
The ssh server is FreeBSD 9.1 with OpenSSH_5.8p2_hpn13v11 and is also hosting DNS using BIND 9.8.3-P4. I've tried connecting from OS X 10.8.2 with OpenSSH_5.9p1 as well as Arch Linux 3.6.10-1-ARCH with OpenSSH_6.1p1.
Update
In a further attempt to troubleshoot this, I stood up a new OpenBSD 5.2 VM which has OpenSSH_6.1 built into it as an ssh server. Since all other implementations of OpenSSH server are just ports of the OpenBSD one, surely this should work. On the server I generate the SSHFP records:
# ssh-keygen -r vm1.test.us.
vm1.test.us. IN SSHFP 1 1 419c5338920e11183380d81f002fc998389b944f
vm1.test.us. IN SSHFP 1 2 cb5bbbf5aef231f57a1a4dcf1e790f1be032b124d0d591023f33cfd5f91ec556
vm1.test.us. IN SSHFP 2 1 0fdf92ce946b5cfee5f96a3e1ef710edc50280ff
vm1.test.us. IN SSHFP 2 2 f2ee7334ee9f9a426f51f20af8f4bc7155d567c9d38a6bffaa6c643af405711e
vm1.test.us. IN SSHFP 3 1 b5e94320f0bc0b46cc6627ca7221679a65c79962
vm1.test.us. IN SSHFP 3 2 60704213a0bbd8dae813d113bfe4ae190a780b89836e6e1c567b7cfde89805f8
I add them to the FreeBSD bind server and reload named. Then test to see if I can access the records:
$ host -t any vm1
vm1.test.us has SSHFP record 1 1 419C5338920E11183380D81F002FC998389B944F
vm1.test.us has SSHFP record 1 2 CB5BBBF5AEF231F57A1A4DCF1E790F1BE032B124D0D591023F33CFD5 F91EC556
vm1.test.us has SSHFP record 2 1 0FDF92CE946B5CFEE5F96A3E1EF710EDC50280FF
vm1.test.us has SSHFP record 2 2 F2EE7334EE9F9A426F51F20AF8F4BC7155D567C9D38A6BFFAA6C643A F405711E
vm1.test.us has SSHFP record 3 1 B5E94320F0BC0B46CC6627CA7221679A65C79962
vm1.test.us has SSHFP record 3 2 60704213A0BBD8DAE813D113BFE4AE190A780B89836E6E1C567B7CFD E89805F8
vm1.test.us has address 192.168.1.60
$ dig -t any vm1.test.us
;; QUESTION SECTION:
;vm1.test.us. IN ANY
;; ANSWER SECTION:
vm1.test.us. 120 IN SSHFP 1 2 CB5BBBF5AEF231F57A1A4DCF1E790F1BE032B124D0D591023F33CFD5 F91EC556
vm1.test.us. 120 IN SSHFP 2 1 0FDF92CE946B5CFEE5F96A3E1EF710EDC50280FF
vm1.test.us. 120 IN SSHFP 2 2 F2EE7334EE9F9A426F51F20AF8F4BC7155D567C9D38A6BFFAA6C643A F405711E
vm1.test.us. 120 IN SSHFP 3 1 B5E94320F0BC0B46CC6627CA7221679A65C79962
vm1.test.us. 120 IN SSHFP 3 2 60704213A0BBD8DAE813D113BFE4AE190A780B89836E6E1C567B7CFD E89805F8
vm1.test.us. 120 IN SSHFP 1 1 419C5338920E11183380D81F002FC998389B944F
vm1.test.us. 120 IN A 192.168.1.60
The records are clearly being served over DNS, so I try to use ssh:
$ rm .ssh/known_hosts
$ ssh -vo VerifyHostKeyDNS=yes root@vm1
OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to vm1 [192.168.1.60] port 22.
debug1: Connection established.
debug1: identity file /Users/test/.ssh/id_rsa type 1
debug1: identity file /Users/test/.ssh/id_rsa-cert type -1
debug1: identity file /Users/test/.ssh/id_dsa type -1
debug1: identity file /Users/test/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.1
debug1: match: OpenSSH_6.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA d8:01:b5:b2:3e:c7:55:ce:19:c1:6d:77:39:92:7d:0f
DNS lookup error: name does not exist
The authenticity of host 'vm1 (192.168.1.60)' can't be established.
RSA key fingerprint is d8:01:b5:b2:3e:c7:55:ce:19:c1:6d:77:39:92:7d:0f.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?
At this point, I think it is safe to eliminate the ssh clients and servers as the point of failure. Instead I am going to focus the DNS server. Unless someone has a suggestion of where to look, I guess I'm stuck with taking packet captures and digging through them to find clues.
Update2
Okay, here are the results of my packet captures. ssh www; fails with the standard
No matching host key fingerprint found in DNS.
and the packet capture shows that DNS fails to return a record for the lookup.
mbp13.test.us www.test.us DNS Standard query 0x1c5e SSHFP www
www.test.us mbp13.test.us DNS Standard query response 0x1c5e No such name
Compare with ssh www.test.us; which also fails with the message
No matching host key fingerprint found in DNS.
however the packet capture shows that DNS actually returns the record.
mbp13.test.us www.test.us DNS Standard query 0x0ebd SSHFP www.test.us
www.test.us mbp13.test.us DNS Standard query response 0x0ebd SSHFP SSHFP`
First, it is a little disconcerting that the error message is the same for both cases. I can add some records to fix case 1 where no records are returned, but the big problem is case 2. DNS works and the SSHFP records are being returned to the ssh client. No packets are sent after the DNS query response and the ssh client immediately displays the no matching fingerprint message. This means either all of the ssh clients I'm testing with are broken or the fingerprint stored in DNS is wrong and doesn't match. I doubt it is the clients so why is the fingerprint in DNS wrong? The fingerprints were generated from the built in ssh tools ssh-keygen as described at the very beginning of this post. Also, the problem is not helped by the fact that the fingerprints are displayed in different formats depending on the context. If they were displayed in the same format it would be easy to see that the record stored in DNS is not the same as the fingerprint of the key returned by the ssh server being connected to.
DNS record format: ad04dfaf343a93beeb939eed1612168f7eadbed7
ssh client mesg format: 69:dc:47:97:e1:a5:c9:07:4a:2b:9e:3c:a2:2b:c8:8c
Does anyone have any suggestions as to why the fingerprints output from ssh-keygen -r don't match the public keys returned by the same ssh server?
Update3
I'm down to my last option. Unless someone points me in the right direction before the weekend, I'll spend my Saturday creating a duplicate environment using VMs that is completely OpenBSD based. Since OpenBSD owns OpenSSH, this would have to be ideal conditions for SSHFP over DNS to work. If an OpenBSD OpenSSH server with bind serving an OpenBSD OpenSSH client doesn't work, then SSHFP is broken as implemented and I'll move things to the OpenBSD forums and possibly file a bug report. I'm still hoping that I'm missing something obvious and that a helpful reply will save my weekend.
Best Answer
Apparently my issues were caused by two different problems.
Issue #1 SSHFP does not support using search paths. So if you add "domain example.com" to /etc/resolv.conf then you would expect ssh myhost to work with SSHFP since regular ssh will correctly resolve the name to myhost.example.com. Apparently the OpenBSD devs are aware of the issue since a patch was issued 2 years ago but it was never applied. Instead an ssh_config hack was suggested but that doesn't appear to work either. So the solution to the first issue is that FQDN must always be used with SSHFP.
Issue #2 Using FQDNs to solve the previous issue, everything works if I use the current version of the OpenSSH client which is OpenSSH_6.1. The OpenSSH_5.8p2 client on my FreeBSD system is able find the SSHFP records for a new OpenSSH_6.1 server, but it is unable to match the fingerprint it receives from DNS with the one it receives from the server. The OpenSSH_5.9p1 client on my OS X 10.8.2 machine is unable to even retrieve the SSHFP records for a new OpenSSH_6.1 server despite being a never version of the client than the FreeBSD machine. Obviously it is unable to match the non-existant SSHFP records with the fingerprint returned by the OpenSSH server. Lastly, ssh-keygen on the FreeBSD box produces bad SSHFP records according to the OpenSSH_6.1 clients which complain about a MITM attack since they don't match the fingerprint returned by the server. The solution appears to be that you must run the current version of both OpenSSH client and server for SSHFP to work. Using an older version of either the client or the server is asking for trouble.
Final Thoughts Using SSHFP with DNS is apparently too cutting edge to be used in a mixed OS environment and have everything "just work" since the non-OpenBSD OS's have to port OpenSSH portable which is out of date by the time it is ported. Perhaps in 3-5yrs, SSHFP will be stable enough that even the older versions which are ported to other OSs will also be stable and compatible with the latest version.