You probably want to use the ServerAlive settings for this. They do not require any configuration on the server, and can be set on the command line if you wish.

ssh -o ServerAliveInterval=5 -o ServerAliveCountMax=1 $HOST

This will send a ssh keepalive message every 5 seconds, and if it comes time to send another keepalive, but a response to the last one wasn't received, then the connection is terminated.

The critical difference between ServerAliveInterval and TCPKeepAlive is the layer they operate at.

• TCPKeepAlive operates on the TCP layer. It sends an empty TCP ACK packet. Firewalls can be configured to ignore these packets, so if you go through a firewall that drops idle connections, these may not keep the connection alive.
• ServerAliveInterval operates on the ssh layer. It will actually send data through ssh, so the TCP packet has encrypted data in and a firewall can't tell if its a keepalive, or a legitimate packet, so these work better.

TL;DR

The most helpful comments from @forcefsck. Unfortunately he didn't fill the answer so I was unable to award the bounty. In short, the answer is PEM + RSA1 + new openSSH format which is described in the question and the main problem was with PEM.

The long one & Bonus

OpenSSH is using parser from openSSL (PEM_read_bio_PrivateKey()), which has the only return value for all the failures (NULL) and if it fails, openSSH expects it was because of wrong passphrase.

Without OpenSSL

I just tried to build OpenSSH without OpenSSL support (--without-openssl configure option) and the behaviour is "correct":

# ./ssh-add <(echo "")
Error loading key "/dev/fd/63": invalid format

Fix with OpenSSL

The other thing is how to fix it. The poke comes from ERR_get_error() function and their friends which should allow us to distinguish between different errors.

Wrong passphrase errors

# ./ssh-add /tmp/rsa
140480353842840:error:0906A068:lib(9):func(106):reason(104):pem_lib.c:457:
Enter passphrase for /tmp/rsa: 
140480353842840:error:06065064:lib(6):func(101):reason(100):evp_enc.c:592:
140480353842840:error:0906A065:lib(9):func(106):reason(101):pem_lib.c:482:

Reasons: PEM_R_BAD_PASSWORD_READ, PEM_R_BAD_BASE64_DECODE, PEM_R_BAD_DECRYPT.

Parse error codes

# ./ssh-add <(echo "")
139656018548376:error:0906D06C:lib(9):func(109):reason(108):pem_lib.c:701:Expecting: ANY PRIVATE KEY

or this:

140654301202072:error:0906D066:lib(9):func(109):reason(102):pem_lib.c:809:

Reason: PEM_R_NO_START_LINE, PEM_R_BAD_END_LINE, but there might be more possibilities.

Solution?

Adding some more checks for OpenSSL errors should give us an option to choose which error we want to mark as "format" error and which is "bad passphrase". This is located in function sshkey_parse_private_pem_fileblob() in sshkey.c on line around 3800.

unsigned long e = ERR_get_error();
if (ERR_GET_REASON(e) == PEM_R_NO_START_LINE ||
    ERR_GET_REASON(e) == PEM_R_BAD_END_LINE) {
        r = SSH_ERR_INVALID_FORMAT;
} else {
        r = SSH_ERR_KEY_WRONG_PASSPHRASE;
}