Cannot SSH to AWS instance from Macbook.. routing

awsNetwork

This is driving me a bit nuts. I created an AWS instance and I can't ssh to it from my Macbook. I can ssh elsewhere. I can ssh from other machines on the same network to the AWS instance. So it's not my firewall, and it's not my AWS instance.

What on my Macbook might prevent me from being able to SSH outbound to a particular IP or range of IPs?

xxx@yyy:~/notes/Software/AWS $ tcpdump tcp port 22 &tcpdump:  verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
ssh 54.247.29.171
ssh: connect to host 54.247.29.171 port 22: Connection refused
xxx@yyy:~/notes/Software/AWS $ ssh 50.247.29.171
10:45:47.857206 IP 172.31.255.253.56694 > 50.247.29.171.ssh: Flags [S], seq 3668893593, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 128333370 ecr 0,sackOK,eol], length 0

It looks like a firewall or routing thing, but the routing is dead simple and my Macbook software firewall is disabled. I'm going to reboot soon to test that, but I really want to know more about how Macs work and why this connection might not work?

From tcpdump, you can see above that it's not even trying to connect to 54.247.29.171. But if I give it a bogus IP 50.247.29.171, at least it sends a packet.

Below, you can see that it's griping about routing somehow

xxx@yyy :~/notes/Software/AWS $ traceroute 54.247.29.171
traceroute to 54.247.29.171 (54.247.29.171), 64 hops max, 40 byte packets
traceroute: sendto: No route to host
 1 traceroute: wrote 54.247.29.171 40 chars, ret=-1
^C

But nothing jumps out at me:

xxx@yyy :~/notes/Software/AWS $ netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            172.31.255.1       UGSc           10       16     en0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              0    73859     lo0
169.254            link#4             UCS             0        0     en0
172.31.255/24      link#4             UCS             3        0     en0
172.31.255.1       0:e0:52:aa:bb:cc   UHLWI          12      231     en0   1176
172.31.255.18      0:26:ab:dd:ee:ff    UHLWI           0        0     en0   1182
172.31.255.253     127.0.0.1          UHS             0        0     lo0

Thoughts? This must be something really simple.


I had a thought that maybe it was my torrent program, transmission, but I disabled the peer blacklist and I'd expect it to use ipfw or something.

Not sure if this would cover everything, is there another way to block outbound connnections?:

 $ sudo ipfw list
65535 allow ip from any to any

Updates:

  • a reboot didn't fix it.
  • My VirtualBox Debian image on my Macbook can't reach it either

As requested

$ ssh -v 54.247.29.171
OpenSSH_5.2p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to 54.247.29.171 [54.247.29.171] port 22.
debug1: connect to address 54.247.29.171 port 22: Connection refused
ssh: connect to host 54.247.29.171 port 22: Connection refused

From the test below, it looks like port 80 is working… something is blocking outbound port 22 for this particular host.

I since found ppload in the task list, it's the peer guardian process. I forgot I had even installed it, but it would explain this behaviour (even the port 80). I killed it, but the block remained in place. I've uninstalled it since, but I have yet to reboot. (I was torrenting Libreoffice and didn't want to get a takedown notice 🙂 )

How could peerguardian perform such a block? In Linux, it's not so easy to see all firewall rules as "ipfw list", it will miss tables such as prerouting etc. Are there other 'tables' or equivalent in MacOS?

xxx@yyy:~ $ tcpdump host 54.247.29.171&
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
xxx@yyy:~ $ ssh -v 54.247.29.171&
OpenSSH_5.2p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to 54.247.29.171 [54.247.29.171] port 22.
debug1: connect to address 54.247.29.171 port 22: Connection refused
ssh: connect to host 54.247.29.171 port 22: Connection refused
xxx@yyy:~ $ telnet 54.247.29.171 22
Trying 54.247.29.171...
telnet: connect to address 54.247.29.171: Connection refused
telnet: Unable to connect to remote host
xxx@yyy:~ $ telnet 54.247.29.171 80
Trying 54.247.29.171...
telnet: connect to address 54.247.29.171: Connection refused
telnet: Unable to connect to remote host
xxx@yyy:~ $ 09:09:35.738770 IP 172.31.255.11.49874 > ec2-54-247-29-171.eu-west-1.compute.amazonaws.com.http: Flags [S], seq 3011923576, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 573636510 ecr 0,sackOK,eol], length 0
09:09:35.841506 IP ec2-54-247-29-171.eu-west-1.compute.amazonaws.com.http > 172.31.255.11.49874: Flags [R.], seq 0, ack 3011923577, win 0, length 0

Best Answer

Is the Elastic IP still bound to your instance? Under some circumstances it will be disassociated from the instance (but not released).