SSH – What are the keys accepted by openSSH

key-authenticationopensshssh

Question is simple:

What are the (private) key types and (their specification – format, syntax) accepted by current openSSH?

The vague answer "the keys generated by ssh-keygen" is not accepted – I know it. They are called PEM with such headers (RSA, DSA, ECDSA variations):

-----BEGIN RSA PRIVATE KEY-----

The other format is described in PROTOCOL.key and is extension by openssh. It is also used as a default for Ed25519 keys, as stated in manual and (regardless passphrase) marked with header:

-----BEGIN OPENSSH PRIVATE KEY-----

There are also legacy RSA1 keys that should be gone today with SSHv1 protocol, but can be identified by the header:

SSH PRIVATE KEY FILE FORMAT 1.1

Are there any others? I am searching for more verbose description of other keys. I didn't find it (so far) explicitly specified in any manual page nor in any RFC for SSH.

The bonus question (not needed to answer, but I would appreciate the insight):

What is the reason behind (openssh) decoding every dummy file and asking for a passphrase, even if it does not have the proper header?

Footnote: Question was originally posted on openssh-unix-dev list, but so far without any answer so I am trying this awesome community, if there is somebody able to answer it.

Best Answer

TL;DR

The most helpful comments from @forcefsck. Unfortunately he didn't fill the answer so I was unable to award the bounty. In short, the answer is PEM + RSA1 + new openSSH format which is described in the question and the main problem was with PEM.

The long one & Bonus

OpenSSH is using parser from openSSL (PEM_read_bio_PrivateKey()), which has the only return value for all the failures (NULL) and if it fails, openSSH expects it was because of wrong passphrase.

Without OpenSSL

I just tried to build OpenSSH without OpenSSL support (--without-openssl configure option) and the behaviour is "correct":

# ./ssh-add <(echo "")
Error loading key "/dev/fd/63": invalid format

Fix with OpenSSL

The other thing is how to fix it. The poke comes from ERR_get_error() function and their friends which should allow us to distinguish between different errors.

Wrong passphrase errors

# ./ssh-add /tmp/rsa
140480353842840:error:0906A068:lib(9):func(106):reason(104):pem_lib.c:457:
Enter passphrase for /tmp/rsa: 
140480353842840:error:06065064:lib(6):func(101):reason(100):evp_enc.c:592:
140480353842840:error:0906A065:lib(9):func(106):reason(101):pem_lib.c:482:

Reasons: PEM_R_BAD_PASSWORD_READ, PEM_R_BAD_BASE64_DECODE, PEM_R_BAD_DECRYPT.

Parse error codes

# ./ssh-add <(echo "")
139656018548376:error:0906D06C:lib(9):func(109):reason(108):pem_lib.c:701:Expecting: ANY PRIVATE KEY

or this:

140654301202072:error:0906D066:lib(9):func(109):reason(102):pem_lib.c:809:

Reason: PEM_R_NO_START_LINE, PEM_R_BAD_END_LINE, but there might be more possibilities.

Solution?

Adding some more checks for OpenSSL errors should give us an option to choose which error we want to mark as "format" error and which is "bad passphrase". This is located in function sshkey_parse_private_pem_fileblob() in sshkey.c on line around 3800.

unsigned long e = ERR_get_error();
if (ERR_GET_REASON(e) == PEM_R_NO_START_LINE ||
    ERR_GET_REASON(e) == PEM_R_BAD_END_LINE) {
        r = SSH_ERR_INVALID_FORMAT;
} else {
        r = SSH_ERR_KEY_WRONG_PASSPHRASE;
}
Related Question