What causes SSH warning about ED25519 host key mismatch on new servers

Securitysshssh-keys

I frequently create and destroy virtual machines, and connect to them via SSH.

Operating system is RedHat Enterprise Linux versions 7 or 8 (happens with both) on both client and server side, and of course openssh as the SSH server.

The first SSH connection will always report host keys missing unless I use ssh-keyscan to pre-populate my known_hosts file. That is of course expected.

I then configure the system (usually using Ansible, but I've also observed the problem before Ansible even ran), and inevitably every time, when re-connecting, SSH reports that one of the host keys has changed.

Warning: the ED25519 host key for 'redacted' differs
from the key for the IP address 'redacted' Offending key for IP
in /redacted/.ssh/known_hosts:424 Matching host key in
/redacted/.ssh/known_hosts:362

I also checked the server-side /etc/ssh and found that all host keys have the same time stamp, so the ED25519 host key hasn't actually changed.

-rw-r-----.  1 root ssh_keys  480 Aug  6 17:26 ssh_host_ecdsa_key
-rw-r--r--.  1 root root      162 Aug  6 17:26 ssh_host_ecdsa_key.pub
-rw-r-----.  1 root ssh_keys  387 Aug  6 17:26 ssh_host_ed25519_key
-rw-r--r--.  1 root root       82 Aug  6 17:26 ssh_host_ed25519_key.pub
-rw-r-----.  1 root ssh_keys 2.6K Aug  6 17:26 ssh_host_rsa_key
-rw-r--r--.  1 root root      554 Aug  6 17:26 ssh_host_rsa_key.pub

I am looking to understand why known_hosts first gets populated with a different ED25519 key than what is used on subsequent connections.

I am not looking for fixes of the nature "edit your known_hosts file". I know how to do that (also see How to fix warning about ECDSA host key if you want to know more). On the large scale I'm sometimes working on (sometimes hundreds of systems) it's very tedious. In addition, simply manipulating the known_hosts file without understanding the reason why a key has changed can be a security concern.

What I'm instead looking for is a deeper understanding of what's going on, and how to prevent it from happening, or at least how to automatically address it (without compromising security!)

Update inspired by Frank Thomas:

The problem happens regardless of whether I reused a previously-used IP address, or a fresh one (most are fresh in my situation).

Another possibly relevant observation: in my known_hosts file, the rsa and edcsa host keys are associated with the FQDN. There are two entries for the ed25519 host keys. One is associated with the FQDN, while the other is associated with the IP address.

The problem appears to always refer to the version of the key associated with the IP address.

More details about how my known_hosts file gets populated:

First, delete any existing known_host entries. The getent hosts hostname will of course return both the IP address and the host name.

for host in $h $(getent hosts $h)
do
  /usr/bin/ssh-keygen -R $host 2>&1 >/dev/null
done

Then retrieve the new keys:

/usr/bin/ssh-keyscan $h 2>/dev/null >> ~/.ssh/known_hosts

This only populates the rsa and the ED25519 host key by the host name.

The ECDSA host key and the ED25519 key by IP address are populated by the first SSH connection. This seems to work fine for the ECDSA host key, but apparently not for the ED25519 key.

Best Answer

It appears that this may be a bug in openssh.

$ ssh <redacted FQDN>
Warning: the ED25519 host key for '<redacted FQDN>' differs from the key for the IP address '<redacted IP>'
Offending key for IP in /home/<redacted>/.ssh/known_hosts:450
Matching host key in /home/<redacted>/.ssh/known_hosts:447
Are you sure you want to continue connecting (yes/no)? no
Host key verification failed.

Lines 447 through 450 of my known_hosts file are:

<redacted FQDN> ssh-ed25519 AAAA<redacted>
<redacted FQDN> ssh-rsa AAAAB<redacted>
<redacted FQDN> ecdsa-sha2-nistp256 AAAAE<redacted>
<redacted IP> ecdsa-sha2-nistp256 AAAA<redacted>

The keys in the last two lines match, and all four lines refer to the same FQDN, or the corresponding IP address.

What leads me to conclude that this is a bug in openssh is that the "offending key" is not actually an ed25519 key. openssh should detect that and match it with the ecdsa-sha2-nistp256 instead.

After deleting the "offending key" and connecting with ssh again, the key is populated correctly with the right ssh-ed25519 key.

I'm still not sure why that key is initially populated with the wrong key type.

Update: I discovered that the initial population appears to be related to specifying my own ciphers in the sshd_config file, which is probably a bad idea anyway. Removing that, and removing the CRYPTO_POLICY= entry in /etc/sysconfig/sshd appears to have resolved the issue. This is on RedHat 8 (RedHat 7 does not have a global crypto policy).

Related Question