Check permissions. OpenSSH will not allow public-key authentication if a user's home directory, ~/.ssh, or ~/.ssh/authorized_keys is world writable. Also, ~/.ssh and ~/.ssh/authorized_keys should probably not be group-writable either. Leave IdentityFile at the default, it's has nothing to do with the server aspect. It points to the default private key file used by the ssh client on UNIX/Linux. ssh_config is used only by the ssh client, and not the OpenSSH sshd server. The contents of authorized_keys should be one line per public key that is authorized to log in and look something like this:
ssh-rsa AAA.../Us= My Public Key comment
Where ... is a bunch of base64 characters ([a-zA-Z/+=]). PuTTYGen can generate two forms of keys: PuTTY and OpenSSH compatible. The OpenSSH compatible will be one long line in the above format. If it's in PuTTY format, you can import it back into PuTTYGen and save as OpenSSH.
Yes, it is impossible to recover the private key from the public key. If it was possible, RSA would be fundamentally broken, and this would be major news (breaking RSA would not only break a lot of Internet communication security, but also allow all kinds of banking fraud, amongst others).
Logging in with a public key instead of a password in fact tends to increase security. If your password isn't strong enough, it can be brute-forced by an attacker with sufficient bandwidth. If the attacker doesn't have a copy of your private key file, the RSA key effectively cannot be brute-forced (a 1024-bit key is equivalent to something like a 160-character password made of random case-sensitive letters and digits). Someone who watches over your shoulder might be able to see your password and your key passphrase, but with a key they'd also need to get the key.
Private keys aren't always more secure than passwords. If the attacker obtains a copy of your private key files (for example by stealing your laptop or your backup media), she can try to brute-force the passphrase, and she can do it at high speed since you have no way to limit the rate (unlike password guesses that need to be made online). If your passphrase is good enough and you notice the theft immediately, you'll still have time to revoke the key.
A public key introduces an element of privacy exposure: if someone knows that you've used the same public key to log into A and to log into B, they know the same person logged into A and B. Merely possessing the public key makes you a suspect that you also have the private key, so you lose some anonimity. But that's usually minor, especially if you're just storing the key in ~/.ssh where only system administrators (who also know what IP address you logged in from) can see it.
These security considerations aside, a private key has many practical advantages. You don't need to type your password so often, and in particular can run automated scripts that don't prompt you once you've entered your key in ssh-agent or the like. You don't need to type your password so often, so you can afford to make it higher-entropy (longer, harder to type). You don't need to type your password so often, so there's less risk that it'll be snooped by a human observer or camera.
Best Answer
Your existing configuration seems very secure. However, there are additional things you can use to restrict access.
Port knocking can be used to keep the port closes most of the time. This is implemented using
iptables. There are daemons which can be used, or the rules can be implemented entirely iniptablesas described in the Shorewall documentation.If tcp wrappers is enabled. A couple of ruless like the following in /etc/hosts.allow will notify you whenever a remote connection is made to the
deamon. The first rule lets local connections work silently, adjust the ip address range as appropriate. The second rule prevents access from addresses which reverse to a number of country TLD, and emails a message for each successful connection. It could be noisy, if you don't use Port Knocking.fail2banrules can be used to temporarily blacklist hosts which are trying to brute force your server. I've seen occasional attempts when I have had ssh exposed to the Internet.