If I run something like this:
ssh -4 -f -N -T -R "/home/dude/lol.socket:192.168.4.44:4444" dude@someserver -p 22 -i privatekey -o "ExitOnForwardFailure yes" -o ConnectTimeout=5 -o ConnectionAttempts=3 -o ServerAliveInterval=15 -o
And then lets say the connection is closed or dies for whatever reason.. say the computer reboots due to maintenance or error or there's internet connectivity issues or whatever -> we have a big problem. The created socket file /home/dude/lol.socket
on the someserver
does not get deleted by sshd. So as the reverse tunnel initiator is recovering and tries to recreate the tunnel it can't because:
Error: remote port forwarding failed for listen path /home/dude/lol.socket
On the server side you get something like:
error: bind: Address already in use
error: unix_listener: cannot bind to path: /home/dude/lol.socket
What would be the supported way / best hack to cleanup the socket after disconnect? Is this a bug in sshd, shouldn't it do that automatically if/when disconnects are noticed?
Backstory:
The idea behind using the sockets is simply that the server is going to handle n "dudes" creating reverse tunnels for m "lol" services in whatever ports and using sockets makes it much easier to ensure that a "dude" can only access and bind to his own sockets, but not other dudes sockets. It also frees me from having to keep record of which dude is using which port to expose which service. And when dude wants to connect to the service on some other server all he needs to know is the name of the service and bind it to some random local port (or socket if he wants to) i.e.
ssh -v -i -4 -N -T -L "127.0.0.1:3334:/home/dude/lol.sock" -p 22
dude@someserver -o "ExitOnForwardFailure yes" -o ConnectTimeout=5 -o
ConnectionAttempts=3 -o ServerAliveInterval=15 -o
ServerAliveCountMax=3
There's no need to know some magic port number that the reverse tunnel on the server is suppose to be running on. So, if you have better ideas how to solve this issue I'm all ears.
Tested with client/server both running Debian 9
(client actually on mac inside docker container) using openssh-client/server version 7.4p1-10+deb9u2
Best Answer
TL;DR;
The solution is to set the value of StreamLocalBindUnlink to yes in ssh configuration:
sudo echo "StreamLocalBindUnlink yes" >> /etc/ssh/sshd_config
.Long story
The reason this happens is because unix socket files are not automatically removed when the socket is closed. They need to be manually cleaned up when closing if this is desired by calling
remove
/unlink
with the filepath, but openssh does not do this. However, as I researched the subject further I came to the realisation that the "best practice" with unix sockets is tounlink
right before binding to it (Check this SO answer for more details). And this is exactly what theStreamLocalBindUnlink yes
tells sshd to do.Man page says:
The downside of this approach is that rebinding to the socket is now allowed even if the old connection is still there. Doing this seems to leave the old tunnel hanging in there so that any existing tcp connections going through that remain intact, but all new connections go to the new tunnel. Also the old tunnel seems to be permanently and irreversibly detached from the filesystem socket address and will not be able to receive anymore new connections even if the new tunnel is closed.
References