SSH Server Not Answering Connection Requests – Troubleshooting

opensshssh

I'm attempting to set up an SSH server on my local machine using OpenSSH.
When I try to SSH from a remote host into my local SSH server, the SSH server doesn't respond and the request times out. I'm pretty sure there's a really obvious fix for this that I'm simply overlooking.

Here's what happens when I try to SSH in from a remote host:

yoshimi@robots:/$ ssh -vv volt@99.3.26.94
OpenSSH_6.7p1 Debian-5, OpenSSL 1.0.1k 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 99.3.26.94 [99.3.26.94] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: connect to address 99.3.26.94 port 22: Connection timed out
ssh: connect to host 99.3.26.94 port 22: Connection timed out

Where robots is my remote host, and 99.3.26.94 is my local SSH server.

SSH Is Running

volt@arnold:~$ ps -A | grep sshd
 5784 ?        00:00:00 sshd

Where arnold is my local SSH server.

Port Forwarding Is Set Up On the Router

I've got my home router set up to forward ports 80 and 22 to my SSH server. Interestingly, port 80 worked without a hitch — goes straight to the Apache web directory. Port 22 — not so much.

NMap Says It's Filtered

yoshimi@robots:/$ nmap -p 22 99.3.26.94

Starting Nmap 6.47 ( http://nmap.org ) at 2015-06-02 14:45 EDT
Nmap scan report for 99-3-26-94.lightspeed.bcvloh.sbcglobal.net (99.3.26.94)
Host is up (0.33s latency).
PORT   STATE    SERVICE
22/tcp filtered ssh

Nmap done: 1 IP address (1 host up) scanned in 7.59 seconds

Where robots is my remote host, and 99.3.26.94 is my local SSH server.

It's Not IPTables (I think)

volt@arnold:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
fail2ban-ssh  tcp  --  anywhere             anywhere             multiport dports ssh
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain fail2ban-ssh (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

…And I don't have any other firewalls in place — it's a relatively fresh Debian netinst.

So, then: What else could it be? It certainly appears to be a firewall-y sort of thing to just ignore traffic, but if it's not the router, it's not iptables, and it's not another firewall on the SSH server, …what the heck else is there??

EDIT: Connection Request from Internal Network Error

yoshimi@robots:/$ ssh volt@192.168.1.90
ssh: connect to host 192.168.1.90 port 22: No route to host

Best Answer

A Very Disappointing Self-Answer

Having set this problem aside for a day and come back to it, I was both relieved and perturbed (more perturbed than relieved) to find that everything was, mysteriously, working properly.

So, What Was the Issue?

No settings were changed or adjusted -- not on the router, not on the SSH server, and not on the SSH client's machine. It's fairly safe to say it was the router not handling the incoming traffic properly, in spite of proper settings. Given that dinky home router software isn't really designed to deal with port forwarding, it took the poor guy a while to implement the necessary changes.

But It's Been Like 6 Hours!!

Yeah dude, I know. I spent all day trying to figure out what was wrong -- and didn't ever find it because there wasn't anything wrong. Evidently, it can take 6 hours -- possibly more -- for the router settings to take effect.

So How Do I Know If This Is My Issue?

A nifty tool I came across during this escapade is tcpdump. This lean little guy sniffs traffic for you, offering valuable insight into what's actually going on. Plus, he's got some super filtering features that allow you to narrow down exactly what you want to look at/for. For example, the command:

tcpdump -i wlan1 port 22 -n -Q inout

Tells tcpdump to look for traffic via the wlan1 interface (-i = 'interface'), only through port 22, ignore DNS name resolution (-n = 'no name resolution'), and we want to see both incoming and outgoing traffic (-Q accepts in, out, or inout; inout is the default).

By running this command on your SSH server while attempting to connect via a remote machine, it quickly becomes clear where precisely the problem lies. There are, essentially, 3 possibilities:

If you're seeing incoming traffic from the remote machine, but no outgoing traffic from your local server, the problem lies with the server: there's probably a firewall rule that needs to be changed, etc.
If you're seeing both incoming and outgoing, but your remote machine isn't receiving the response, it's most likely the router: it's allowing the incoming traffic, but dropping your outgoing packets.
If there's no traffic at all, that's probably a router issue as well: the remote machine's SYN packets are being ignored and dropped by the router before they even reach your server.

And once you've discovered where the problem lies, a fix is (usually) trivial.

Related Solutions

SSH session through jumphost via remote port forwarding

It looks like you should be using local port-forwarding instead of remote port-forwarding. You may want to refer to the following helpful blog post by Dirk Loss:

SSH port forwarding visualized

It includes the following illustrative diagram:

In order to read the diagram you need to know that it describes the relationships between 4 different roles involved in creating and utilizing an SSH tunnel:

an ssh client (i.e. ssh - the OpenSSH command-line client) used to establish the tunnel;
an ssh server (i.e. sshd - the OpenSSH server daemon) used to maintain the other end of the tunnel;
an application server (e.g. another ssh server or an http server);
an application client (e.g. another ssh client or a web-browser) which wants to access the application server via the tunnel.

It's also important to understand that the two different types of forwarding correspond to two different use cases:

Local Forwarding: where the application client connects via the ssh client
Remote Forwarding: where the application client connects via the ssh server

Remote forwarding is so called because the forwarding is performed remotely (at the ssh server) rather than locally (at the ssh client). I also find "remote forwarding = reverse forwarding" to be a useful mnemonic.

So as you can see, in order to initiate a connection from an ssh client at the origin host through an sshd server on a proxy jumphost to a third target host you would have to use local port-forwarding. Remote port-forwarding is for the case in which you want the entry point to the tunnel to be located at the host running the sshd server rather than at the host running the ssh client.

In the man page the local port-forwarding syntax is written as follows:

ssh -L [bind_address:]port:host:hostport user@remote

This can be written more intuitively as the following:

ssh -L [local_bind_address:]local_port:remote_host:remote_host_port user@proxy_host

Or, using your naming conventions:

ssh -L [origin_bind_address:]origin_port:target_host:target_host_port user@jump_host

If we modify your command to use local port-forwarding instead then we end up with the following:

user@origin:~$ ssh -L *:1234:target:22 myusername@jumphost

Best Answer

Related Solutions

SSH session through jumphost via remote port forwarding

Related Question