Ssh – Information/Assistance required on reverse SSH Tunneling (Naming Conventions etc)

sshssh-tunnelingvpn

I have multiple raspberry pi's running Arch Linux (No GUI) in which I need to access. These pi's are behind firewalls in each unique location. Currently I use openvpn to connect to these but the costs of that system is expensive per license. I use the access server from them.

As a result, I am trying to design and setup a system that allows me the ability to login to my VPN server (vps) and run a command to search for a specific name (OfficeDevice1991) such as: customcommandsearch "OfficeDevice1991" and then it then returns the IP address of the machine or something I can use to SSH into with. I am also looking for the ability to run a command to list every active connected device. It lists back the IP, name and perhaps how long it's been active for.

For this goal, I of course need to create something that includes the name of the device (in this case OfficeDevice1991) and then that pi will be able to hook into my vps public server. From the public server, I can log in and do a search of every device connected to it and return information required to ssh into.

I have been looking into reverse SSH and so far I got one of my test pi's connected and accessible from my vps using the following commands:

PI:

ssh -fN -R 12345:localhost:22 -i /publickeyfile useraccount@ip //Pi's command to connect to vpn

VPS:

ssh -p 12345 useraccount@localhost //command for vpn to connect to pi

This works great, but using this method, if I was to implement it, I would run into a few issues:

  1. I would need to set up unique unused ports
  2. Some way to keep these ports / tunnels opened
  3. I need to come up with a system to identify each device. I can log each port to a name in like a text file locally? It would be beneficial to be able to include that in the ssh setup for each device if possible. I would still need to make sure the ports I use are not being used by any other programs or any device already there.

What I don't want to have to do

  1. Check what ports are free to use for each RPI

  2. Have to manually edit .ssh/config to add a name to represent each port assigned to RPI from part 1 above.

I am writing this for information/assistance as to what to do for my goal.

Could anyone provide me with a suitable solution?

Best Answer

Here's a solution using OpenSSH >= 6.7 + socat:

  1. OpenSSH >= 6.7 can use Unix domain socket forwarding

    That means the reverse tunnel endpoint will be an UNIX listening socket instead of a traditional TCP listening socket. You can then manage more easily the flotilla of RPIs with an easy naming scheme: the socket's name will be the RPI's chosen (and fixed) name, like OfficeDevice1991. It could be even be an unique property from the RPI as long as it's a valid filename (since unix socket names adhere to filename conventions). For example its hostname, the MAC address of its ethernet or wifi card ...

    SSH can handle unix sockets for tunnels, not for connecting itself. It will need the help of a ProxyCommand to be able to work as unix-socket client. socat can handle many kind of connections, including unix sockets.

    UPDATE:
    There is also a specific issue to handle: the unix socket file is not deleted on clean exit, nor would it have been be deleted anyway for example after a crash. This require the option StreamLocalBindUnlink=yes. I didn't find initially that, as the name perhaps imply, this option must be set on the node creating the unix socket. So in the end it's set on the client with a local forward (-L) or else on the server (in sshd_config) with a remote forward (-R). OP found it there. This solution uses a remote forward.

    Configuration on VPS:

    mkdir /rpi-access

    (as root) edit the sshd_config file (/etc/ssh/sshd_config). It requires this additional option:

    StreamLocalBindUnlink yes

    Depending on default options it might also require AllowStreamLocalForwarding yes

    UPDATE2:
    Also set in sshd_config the parameters ClientAliveInterval and ClientAliveCountMax, thus allowing to detect a disconnect in a reasonable time, eg:

    ClientAliveInterval 300
ClientAliveCountMax 2

    Stale ssh connections should then be detected earlier on the VPS (~10mn with the example), and the corresponding sshd process will then exit.

    Usage on RPI:

    ssh -fN -R /rpi-access/OfficeDevice1991:localhost:22 -i /privatekeyfile useraccount@ip

    In a config file this would be similar to this:

    Host ip
User useraccount
RemoteForward /rpi-access/OfficeDevice1991:localhost:22
IdentityFile /privatekeyfile

    Repeating it again: the StreamLocalBindUnlink yes set on sshd on VPS side option is important: the socket just created is not removed, even upon normal exit. This option ensures that the socket is removed if it exists before use, thus allowing to be reused for further reconnections. This also means one can't consider the mere presence of the socket as meaning the RPI is connected (but see later).

    Now this allows to do on VPS:

    ssh -o 'ProxyCommand=socat UNIX:/rpi-access/%h -' rpiuseraccount@OfficeDevice1991

    As a configuration file, considering for example RPIs have all a name starting with OfficeDevice:

    Host OfficeDevice*
    User rpiuseraccount
    ProxyCommand socat UNIX:/rpi-access/%h -

  2. To keep the link, just use a loop

    The RPI can run a loop reconnecting ssh to the VPS whenever the connections ends. For this it musn't use the background mode (no -f). A keepalive mechanism should also be used. TCPKeepAlive (system level) or ServerAliveInterval (application level) are available. I think TCPKeepAlive is useful only on the server (the side receiving the connection), so let's rather use ServerAliveInterval.

    Its value (as well as ServerAliveCountMax) should probably be adapted depending on various criteria: a firewall dropping inactive connections after a certain time, the wished recovery delay, not generating useless traffic, ... let's say 300s here.

    OfficeDevice1991 RPI:

    #!/bin/sh
while : ; do
    ssh  -N -o ConnectTimeout=30 -o ServerAliveInterval=300 -R /rpi-access/OfficeDevice1991:localhost:22 -i /privatekeyfile useraccount@ip
    sleep 5 # avoid flood/DDoS in case of really unexpected issues
done

    Even if the remote side didn't yet detect the previous connectivity failure, and for a few more time has the old ssh connection still running, StreamLocalBindUnlink yes will anyway forcefully refresh the unix socket to the new connection.

  3. it's already handled by 1.

    There's no customcommandsearch needed. With the right settings set in 1. just using ssh OfficeDevice1991 will connect to OfficeDevice1991.

    If needed on the VPS, as root user only, this command:

    fuser /rpi-access/*

    can show which RPIs are currently connected (of course except those that recently lost the connection before detection). It won't show the stale unix socket files because there's no process tied to them.

Related Question