Linux – How to release ports on the SSH server when a reverse ssh tunnel disconnects abruptly/uncleanly

linuxnetworkingsshtunnel

We have a some hardware we install at our customers' locations, that hardware connects to our ssh server and establishes a reverse ssh tunnel so we can gain access to several client systems for monitoring purposes.

Everything works fine until there is an unclean disconnect of the SSH session.

When that happens, on our SSH server the ports that were used by the reverse tunnel stay stuck in LISTENING mode and when our remote hardware eventually tries to auto reconnect and re-establish its tunnels it fails with the error

Warning: remote port forwarding failed for listen port XXXX

I tested if there was an issue with our SSH server or client by trying a clean disconnect and course that releases the ports just fine. When I simulate a connection failure (disconnect the Ethernet port of the client hardware for example) then we have the same issue I described above.

What is the proper way of handling this situation? Keep in mind these are reversed tunnels so whatever happens needs to be done on the SSH server. Ideally i need the ssh server to realize instantly that the SSH session hosting the tunnels is down and release the ports it was using. I guess the solution could involve killing the concerned SSH process but I need to be careful with that cause we have multiple clients connecting to the same ssh server and I wouldn't want to kick them offline.

Being so mature, I'm sure SSHD has some sort of built in feature to handle this but I just can't figure it out.

Please advise so I don't have to go back to administering Windows boxes …

FYI: I'm running this on a Debian based distro.

Best Answer

You have to use ClientAliveInterval in your sshd_config.

ClientAliveInterval 15

Ref: man sshd_config

ClientAliveCountMax
         Sets the number of client alive messages (see below) which may be
         sent without sshd(8) receiving any messages back from the client.
         If this threshold is reached while client alive messages are
         being sent, sshd will disconnect the client, terminating the
         session.  It is important to note that the use of client alive
         messages is very different from TCPKeepAlive (below).  The client
         alive messages are sent through the encrypted channel and
         therefore will not be spoofable.  The TCP keepalive option
         enabled by TCPKeepAlive is spoofable.  The client alive mechanism
         is valuable when the client or server depend on knowing when a
         connection has become inactive.

         The default value is 3.  If ClientAliveInterval (see below) is
         set to 15, and ClientAliveCountMax is left at the default,
         unresponsive SSH clients will be disconnected after approximately
         45 seconds.  This option applies to protocol version 2 only.

 ClientAliveInterval
         Sets a timeout interval in seconds after which if no data has
         been received from the client, sshd(8) will send a message
         through the encrypted channel to request a response from the
         client.  The default is 0, indicating that these messages will
         not be sent to the client.  This option applies to protocol
         version 2 only.
Related Question