Shell – Determine dynamically allocated port for OpenSSH RemoteForward

opensshport-forwardingshell-scriptssh-tunneling

Question (TL;DR)

When assigning ports dynamically for remote forwarding (a.k.a. -R option), how can a script on the remote machine (for instance sourced from .bashrc) determine which ports were chosen by OpenSSH?

Background

I use OpenSSH (on both ends) to connect to our central server, that I share with multiple other users. For my remote session (for now) I would like to forward X, cups and pulseaudio.

The most trivial is forwarding X, using the -X option. The allocated X address is stored in the environmental variable DISPLAY and from that I can determine the corresponding TCP port, in most cases anyways. But I hardly ever need to, because Xlib honors DISPLAY.

I need a similar mechanism for cups and pulseaudio. The basics for both services exist, in the form of the environmental variables CUPS_SERVER and PULSE_SERVER, respectively. Here are usage examples:

ssh -X -R12345:localhost:631 -R54321:localhost:4713 datserver

export CUPS_SERVER=localhost:12345
lowriter #and I can print using my local printer
lpr -P default -o Duplex=DuplexNoTumble minutes.pdf #printing through the tunnel
lpr -H localhost:631 -P default -o Duplex=DuplexNoTumble minutes.pdf #printing remotely

mpg123 mp3s/van_halen/jump.mp3 #annoy co-workers
PULSE_SERVER=localhost:54321 mpg123 mp3s/van_halen/jump.mp3 #listen to music through the tunnel

The problem is setting CUPS_SERVER and PULSE_SERVER correctly.

We use port forwardings a lot and therefore I need dynamic port allocations. Static port allocations are not an option.

OpenSSH has a mechanism for dynamic port allocation on the remote server, by specifying 0 as bind-port for remote forwarding (the -R option). By using the following command, OpenSSH will dynamically allocate ports for cups and pulse forwarding.

ssh -X -R0:localhost:631 -R0:localhost:4713 datserver

When I use that command, ssh will print the following to STDERR:

Allocated port 55710 for remote forward to 127.0.0.1:4713
Allocated port 41273 for remote forward to 127.0.0.1:631

There is the information I want! Ultimately I want to generate:

export CUPS_SERVER=localhost:41273
export PULSE_SERVER=localhost:55710

However the "Allocated port …" messages are created on my local machine and sent to STDERR, which I can't access on the remote machine. Oddly enough OpenSSH does not seem to have means to retrieve Information about port forwardings.

How do I fetch that information to put it into a shell script to adequately set CUPS_SERVER and PULSE_SERVER on the remote host?

Dead Ends

The only easy thing I could find was increasing verbosity of the sshd until that information can be read from the logs. This is not viable as that information discloses a lot more information than is sensible to make accessible by non-root users.

I was thinking about patching OpenSSH to support an additional escape sequence which prints a nice representation of the internal struct permitted_opens, but even if that is what I want, I still can't script accessing the client escape sequences from the server side.

There must be a better way

The following approach seems very unstable and is limited to one such SSH session per user. However, I need at least two concurrent such sessions and other users even more. But I tried …

When the stars are aligned properly, having sacrificed a chicken or two, I can abuse the fact that sshd is not started as my user, but drops privileges after successful login, to do this:

get a list of port numbers for all listening sockets that belong to my user

netstat -tlpen | grep ${UID} | sed -e 's/^.*:$[0-9]\+$ .*$/\1/'
get a list of port numbers for all listening sockets that belong to processes my user started

lsof -u ${UID} 2>/dev/null | grep LISTEN | sed -e 's/.*:$[0-9]\+$ (LISTEN).*$/\1/'
All ports that are in the first set, but not in the second set have a high likelyhood to be my forwarding ports, and indeed subtracting the sets yields 41273, 55710 and 6010; cups, pulse and X, respectively.
6010 is identified as the X port using DISPLAY.
41273 is the cups port, because lpstat -h localhost:41273 -a returns 0.
55710 is the pulse port, because pactl -s localhost:55710 stat returns 0. (It even prints the hostname of my client!)

(To do the set substraction I sort -u and store the output from the above command lines and use comm to do the substraction.)

Pulseaudio lets me identify the client and, for all intents and purposes, this may serve as an anchor to separate SSH sessions that need separating. However, I haven't found a way to tie 41273, 55710 and 6010 to the same sshd process. netstat won't disclose that information to non-root users. I only get a - in the PID/Program name column where I would like to read 2339/54 (in this particular instance). So close …

Best Answer

Unfortunately I didn't find your question earlier, but I've just got a really good answer from kamil-maciorowski:

https://unix.stackexchange.com/a/584505/251179

In summary, establish a master connection first, keep that in the background, then issue a second command with -O *ctl_cmd* set to forward to request/setup port forwarding:

ssh -fNMS /path/to/socket user@server

port="$(ssh -S /path/to/socket -O forward -R 0:localhost:22 placeholder)"

This will give you $port on your local machine, and a connection in the background.

You can then use $port locally; or use ssh again to run a command on the remote server, where you can use the same control socket.

As to the flags, these are:

-f = Requests ssh to go to background.
-N = Do not execute a remote command.
-M = Places client into “master” mode, for connection sharing.
-S = Location of a control socket for connection sharing.
-O = Control an active connection multiplexing master process.

In my case I've added a bit more to keep checking the connection:

#!/bin/bash

#--------------------------------------------------
# Setup
#--------------------------------------------------

  set -u;

  tunnel_user="user";
  tunnel_host="1.1.1.1";

  local_port="22";
  local_name="my-name";

  path_key="$HOME/.ssh/tunnel_ed25519";

  path_lock="/tmp/tunnel.${tunnel_host}.pid"
  path_port="/tmp/tunnel.${tunnel_host}.port"
  path_log="/tmp/tunnel.${tunnel_host}.log"
  path_socket="/tmp/tunnel.${tunnel_host}.socket"

#--------------------------------------------------
# Key file
#--------------------------------------------------

  if [ ! -f "${path_key}" ]; then

    ssh-keygen -q -t ed25519 -f "${path_key}" -N "";

    /usr/local/bin/tunnel-client-key.sh
      # Sends the public key to a central server, also run via cron, so it can be added to ~/.ssh/authorized_keys
      # curl -s --form-string "pass=${pass}" --form-string "name=$(local_name)" -F "key=@${path_key}.pub" "https://example.com/key/";

  fi

#--------------------------------------------------
# Lock
#--------------------------------------------------

  if [ -e "${path_lock}" ]; then
    c=$(pgrep -F "${path_lock}" 2>/dev/null | wc -l);
      # MacOS 10.15.4 does not support "-c" to count processes, or the full "--pidfile" flag.
  else
    c=0;
  fi

  if [[ "${c}" -gt 0 ]]; then
    if tty -s; then
      echo "Already running";
    fi;
    exit;
  fi;

  echo "$$" > "${path_lock}";

#--------------------------------------------------
# Port forward
#--------------------------------------------------

  retry=0;

  while true; do

    #--------------------------------------------------
    # Log cleanup
    #--------------------------------------------------

      if [ ! -f "${path_log}" ]; then
        touch "${path_log}";
      fi

      tail -n 30 "${path_log}" > "${path_log}.tmp";

      mv "${path_log}.tmp" "${path_log}";

    #--------------------------------------------------
    # Exit old sockets
    #--------------------------------------------------

      if [ -S "${path_socket}" ]; then

        echo "$(date) : Exit" >> "${path_log}";

        ssh -S "${path_socket}" -O exit placeholder;

      fi

    #--------------------------------------------------
    # Lost lock
    #--------------------------------------------------

      if [ ! -e "${path_lock}" ] || ! grep -q "$$" "${path_lock}"; then

        echo "$(date) : Lost Lock" >> "${path_log}";

        exit;

      fi

    #--------------------------------------------------
    # Master connection
    #--------------------------------------------------

      echo "$(date) : Connect ${retry}" >> "${path_log}";

      ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes -fNTMS "${path_socket}" -i "${path_key}" "${tunnel_user}@${tunnel_host}" >> "${path_log}" 2>&1;

    #--------------------------------------------------
    # Setup and keep checking the port forwarding
    #--------------------------------------------------

      old_port=0;

      while ssh -S "${path_socket}" -O check placeholder 2>/dev/null; do

        new_port=$(ssh -S "${path_socket}" -O forward -R "0:localhost:${local_port}" placeholder 2>&1);

        if [[ "${new_port}" -gt 0 ]]; then

          retry=0;

          if [[ "${new_port}" -ne "${old_port}" ]]; then

            ssh -i "${path_key}" "${tunnel_user}@${tunnel_host}" "tunnel.port.sh '${new_port}' '${local_name}'" >> "${path_log}" 2>&1;
              # Tell remote server what the port is, and local_name.
              # Don't use socket, it used "-N"; if done, a lost connection keeps sshd running on the remote host, even with ClientAliveInterval/ClientAliveCountMax.

            echo "$(date) : ${new_port}" >> "${path_log}";

            echo "${new_port}" > "${path_port}";

            old_port="${new_port}";

            sleep 1;

          else

            sleep 300; # Looks good, check again in 5 minutes.

          fi

        else # Not a valid port number (0, empty string, number followed by an error message, etc?)

          ssh -S "${path_socket}" -O exit placeholder 2>/dev/null;

        fi

      done

    #--------------------------------------------------
    # Cleanup
    #--------------------------------------------------

      if [ ! -f "${path_port}" ]; then
        rm "${path_port}";
      fi

      echo "$(date) : Disconnected" >> "${path_log}";

    #--------------------------------------------------
    # Delay before next re-try
    #--------------------------------------------------

      retry=$((retry+1));

      if [[ $retry -gt 10 ]]; then
        sleep 180; # Too many connection failures, try again in 3 minutes
      else
        sleep 5;
      fi

  done

1 - general case

Suppose you have two server in a company network.

From local server "L1" you have access to a remote server "R1" via ssh on port 22.

The server R1 host a web service, but firewall/NAT/whatever block direct access to port 80.
"R1" can't access "L1" directly due to NAT issue.

from L1 you connect using (same user on both host)

ssh -L 10080:localhost:80 -R 10022:localhost:22 R1

now,

-L 10080:localhost:80 local browsing on 10080 will be forward to distant host (R1), localhost (e.g. distant 127.0.0.1) on port 80, thus you could browse distant host (R1), while blocked by firewall/NAT.
-R 10022:localhost:22 remote use of port 10022 will be forwarded to localhost (L1) port 22, you can initiate scp/ssh from remote to localhost. (using scp -P 10022 file localhost: (note upper -P ) )

This is actually a command line I use from an Ubuntu (14.04) laptop to go to production server (I can firefox to localhost:10080 and use scp from server to localhost:10022 to bring back file )

now your IP are

192.168.1.1 L1
172.17.17.1 R1

you use

ssh -L 10080:172.17.17.3:80 -R 10022:192.168.1.5:22 \
     -L 192.168.1.9:8888:172.17.17.9:9999 R1

now

distant access to port 10022 (any IP) will be forwarded to local 192.168.1.5 ssh/scp.
local access to port 10080 (any IP) will be forwarded to distant host 172.17.17.3 port 80.
local access to port 8888 from 192.168.1.9 will be forwarded to port 9999 on 172.17.17.9 (192.168.1.9 is a local address in L1, using alias mecanism, this is not a firewall rule).

those setting can be used on putty/bitwise client

Ask Ubuntu question

what about

 -R 8000:localhost:80

this read as

-R remote
8000 port 8000 (remote port 8000 ... )
localhost (remote port 8000 goest to localhost .. )
80 ( remote port 8000 goes to localhost port 80 )

SSH – Recording Dynamically Allocated Port Number with Remote Port Forwarding

You connect a master connection first without port forwarding:

ssh -NMS /path/to/socket user@server

/path/to/socket is up to you. A socket will be created there. In a script you want to use -f.

Then you use -S /path/to/socket -O … to control the master connection. E.g. you can check if it still works:

ssh -S /path/to/socket -O check placeholder

placeholder doesn't matter, it's /path/to/socket what identifies the master connection you want to control.

Now you can request port forwarding:

ssh -S /path/to/socket -O forward -R 0:localhost:22 placeholder

The command will tell you the port. It will exit right away, this means you were able to capture the output. Nothing is lost. Repeat the command and it will tell you the same port again instead of forwarding a second port to the same destination. So let's save the output to a variable:

port="$(ssh -S /path/to/socket -O forward -R 0:localhost:22 placeholder)"

And that's it, now you can use the variable.

For completeness, this is how you cancel the forwarding:

ssh -S /path/to/socket -O cancel -R 0:localhost:22 placeholder

although you don't need to cancel explicitly before terminating the master connection. You terminate it like this:

ssh -S /path/to/socket -O exit placeholder

Not all the above commands are necessary in a most basic script. You need these:

ssh -fNMS /path/to/socket user@server
port="$(ssh -S /path/to/socket -O forward -R 0:localhost:22 placeholder)"
   # anything you want, now you know the port
ssh -S /path/to/socket -O exit placeholder

possibly with some logic to handle connection refused etc.