Ssh – Automatically unlock keys on a remote host accessed via SSH

As answered elsewhere, the trick is adding the option IdentitiesOnly yes which makes sure that only the configured keys will be used even if others are available from the agent.

You solved part of your issue with the use of keychain to act as storage for your passphrases for your SSH keys. As to your other question:

However, I am not sure if it is worth to encrypt a private key which is only used for ssh connection to a specific server. Is there any concrete security risk by not encrypting the private keys? In principle if one is able to access the private key he/she is also able to use the passwords stored in the keychain...so it seems that there might not be a big advantage in encrypting the private keys

It's not that your private keys are left unencrypted, it's that they're not protected with a passphrase. Therefore anyone that can gain access to your home directory's .ssh directory can get control of your SSH keys without issue.

If you're comfortable with leaving these keys on a system that you have full control over then I see no issue with leaving them as passphrase-less. I would also recommend generating a different set of keys that you can leave on secondary systems such as this, or even a unique set just for this server, so that you can completely abandon it without much issue if you ever discover its been compromised.

As to using keychain to manage your keys, I see not much to be gained with having passphrases on SSH keys if you're using keychain to manage them, except that keychain will only store them when you use them. If you don't use the keys often, and are diligent about cleaning up keychain when you're finished, then a passphrase would be an advantage, but IMO, it's a small one.