SSH port forwarding not working in macos catalina (10.15)

catalinassh

Something in macOS Catalina (10.15.1) is interfering with ssh port forwarding (needed for localhost debugging and developing against a web server system deployed in AWS).

I assume something with the enhanced security features, but I've been unable to figure out what from searching and poking around.

The tunnel seems to start up fine, but when a client tries to connect the connection is denied. The tunnel & app startup commands are the same as what was working fine in all previous releases of the os, afaik, but are failing on this new 10.15 install (nothing ported, fresh setup of tools. I did change my default shell back to bash.)

Network configuration: There is a redis server in an AWS VPC that's not publicly routable. We have a bastion server that is publically routable that can forward the connection via ssh. I can connect to the bastion server for an interactive session, so the keys and passphrases are all ok as far as I know. There's no VPN to AWS, so the bastion host is the only way into the private subnets.

Here's the command we use to start up the tunnel (prodbastion is defined in .ssh/config):

ssh -v -N -L 6399:redis.<HOSTNAME>:6379 prodbastion

This logs verbosely in the shell, here's the end of the supposedly successful authentication startup, followed by the repeated client connection failures


debug1: Authentication succeeded (publickey).
Authenticated to <bastionprod hostname> ([<bastion prod ip>]:22).
debug1: Local connections to LOCALHOST:6399 forwarded to remote address redis.<HOSTNAME>:6379
debug1: Local forwarding listening on ::1 port 6399.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 6399.
debug1: channel 1: new [port listener]
...
debug1: channel 2: free: direct-tcpip: listening port 6399 for redis.<HOSTNAME> port 6379, connect from 127.0.0.1 port 51454 to 127.0.0.1 port 6399, nchannels 3
debug1: Connection to port 6399 forwarding to redis.<HOSTNAME> port 6379 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: administratively prohibited: open failed
debug1: channel 2: free: direct-tcpip: listening port 6399 for redis.<HOSTNAME> port 6379, connect from 127.0.0.1 port 51457 to 127.0.0.1 port 6399, nchannels 3
debug1: Connection to port 6399 forwarding to redis.<HOSTNAME> port 6379 requested.
debug1: channel 2: new [direct-tcpip]
...

The client is a nodejs web app, needing data from redis. the logs there just say more or less cannot connect:

2019-12-09T17:13:51.286Z - info: Client Redis connection: {"host":"localhost","port":"6399","database":"4"}
2019-12-09T17:13:51.299Z - info: mongoose connected
2019-12-09T17:13:51.332Z - error: Redis error occured: AbortError: Ready check failed: Redis connection lost and command aborted. It might have been processed.
2019-12-09T17:13:51.473Z - error: Redis error occured: AbortError: Ready check failed: Redis connection lost and command aborted. It might have been processed.
...

It seems to have the same failure with the system ssh or openssh installed with brew.

Update in response to questions:

  • It was my error all along, the hostname in the tunnel command didn't exist but my bash history led me to believe it did
  • firewall was off
  • telnet could not connect to 6399 with the tunnel up to a bad hostname
  • telnet (and our redis-using app) could connect with the right hostname.
  • argh and sorry about the confusion!

Best Answer

From What's ssh port forwarding and what's the difference between ssh local and remote port forwarding:

I have drawn some sketches ssh tunnel starting from local

ssh tunnel starting from remote

Introduction

  1. local: -L Specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side.

    ssh -L sourcePort:forwardToHost:onPort connectToHost means: connect with ssh to connectToHost, and forward all connection attempts to the local sourcePort to port onPort on the machine called forwardToHost, which can be reached from the connectToHost machine.

  2. remote: -R Specifies that the given port on the remote (server) host is to be forwarded to the given host and port on the local side.

    ssh -R sourcePort:forwardToHost:onPort connectToHost means: connect with ssh to connectToHost, and forward all connection attempts to the remote sourcePort to port onPort on the machine called forwardToHost, which can be reached from your local machine.

Examples

Example for 1

ssh -L 8080:localhost:80 SUPERSERVER You specify that a connection made to the local port 80 is to be forwarded to port 8080 on

SUPERSERVER. That means if someone connects to your computer with a webbrowser, he gets the response of the webserver running on SUPERSERVER. You, on your local machine, have no webserver running.

Example for 2

ssh -R 80:localhost:8080 tinyserver You specify, that a connection made to the port 80 of tinyserver is to be forwarded to port 8080 on

your local machine. That means if someone connects to the small and slow server with a webbrowser, he gets the response of the webserver running on your local machine. The tinyserver, which has not enough diskspace for the big website, has no webserver running. But people connecting to tinyserver think so.

More examples

Other things could be: The powerful machine has five webservers running on five different ports. If a user connects to one of the five tinyservers at port 80 with his webbrowser, the request is redirected to the corresponding webserver running on the powerful machine. That would be

ssh -R 80:localhost:30180 tinyserver1
ssh -R 80:localhost:30280 tinyserver2
etc.

Or maybe your machine is only the connection between the powerful and the small servers. Then it would be (for one of the tinyservers that play to have their own webservers):

ssh -R 80:SUPERSERVER:30180 tinyserver1
ssh -R 80:SUPERSERVER:30280 tinyserver2
etc
Related Question