Ubuntu – How to solve ‘Connection refused’ errors in SSH connection

I have an Ubuntu Server 10.10 32-bit in my home. I'm making SSH connections to it from my PC via Putty.

The problem is, sometimes I'm able to login seamlessly. However, sometimes it gives me an error like this: Network error: Connection refused.

Then, I dont't change anything, try to login a few times more, wait a while and try again. Sometimes I can log in, sometimes I cannot. It seems pretty random to me.

What can I do to solve this?

Edit:

And Sometimes, Putty gives Network error: Software caused connection abort error after displaying login as: text.

Here is the ping -t output:

Pinging 192.168.2.254 with 32 bytes of data:
Reply from 192.168.2.254: bytes=32 time=6ms TTL=64
Reply from 192.168.2.254: bytes=32 time=65ms TTL=6
Reply from 192.168.2.254: bytes=32 time=88ms TTL=6
Reply from 192.168.2.254: bytes=32 time=1ms TTL=64
Reply from 192.168.2.254: bytes=32 time=3ms TTL=64
Reply from 192.168.2.254: bytes=32 time=1ms TTL=64
Reply from 192.168.2.254: bytes=32 time=1ms TTL=64
Reply from 192.168.2.254: bytes=32 time=1ms TTL=64
Reply from 192.168.2.254: bytes=32 time=1ms TTL=64

I turned off firewall of router, and everything seems to work now. Except for that, I still can't enter my web server by typing external IP from my PC.