ssh systemd proxy socks socket-activation – Fix On-Demand SSH Socks Proxy with Systemd User Units

autosshPROXYsocket-activationsockssshsystemdsystemd-user

To reach an isolated network I use an -D .

In order to avoid having to type the details every time I added them to ~/.ssh/config:

$ awk '/Host socks-proxy/' RS= ~/.ssh/config
Host socks-proxy
  Hostname pcit
  BatchMode yes
  RequestTTY no
  Compression yes
  DynamicForward localhost:9118

Then I created a service unit definition file: 

$ cat ~/.config/systemd/user/SocksProxy.service 
[Unit]
Description=SocksProxy Over Bridge Host

[Service]
ExecStart=/usr/bin/ssh -Nk socks-proxy

[Install]
WantedBy=default.target

I let the daemon reload the new service definitions, enabled the new service, started it, checked its status, and verified, that it is listening:

$ systemctl --user daemon-reload
$ systemctl --user list-unit-files | grep SocksP
SocksProxy.service   disabled

$ systemctl --user enable SocksProxy.service
Created symlink from ~/.config/systemd/user/default.target.wants/SocksProxy.service to ~/.config/systemd/user/SocksProxy.service.

$ systemctl --user start SocksProxy.service 
$ systemctl --user status SocksProxy.service 
● SocksProxy.service - SocksProxy Over Bridge Host
   Loaded: loaded (/home/alex/.config/systemd/user/SocksProxy.service; enabled)
   Active: active (running) since Thu 2017-08-03 10:45:29 CEST; 2s ago
 Main PID: 26490 (ssh)
   CGroup: /user.slice/user-1000.slice/user@1000.service/SocksProxy.service
           └─26490 /usr/bin/ssh -Nk socks-proxy

$ netstat -tnlp | grep 118
tcp     0    0 127.0.0.1:9118        0.0.0.0:*             LISTEN     
tcp6    0    0 ::1:9118              :::*                  LISTEN

This works as intended. Then I wanted to avoid having to manually start the service, or running it permanently with , by using for on-demand (re-)spawning. That didn't work, I think (my version of) ssh cannot receive socket file-descriptors.

I found the documentation (1,2), and an example for using the systemd-socket-proxyd-tool to create 2 "wrapper" services, a "service" and a "socket":

$ cat ~/.config/systemd/user/SocksProxyHelper.socket 
[Unit]
Description=On Demand Socks proxy into Work

[Socket]
ListenStream=8118
#BindToDevice=lo
#Accept=yes

[Install]
WantedBy=sockets.target

$ cat ~/.config/systemd/user/SocksProxyHelper.service 
[Unit]
Description=On demand Work Socks tunnel
After=network.target SocksProxyHelper.socket
Requires=SocksProxyHelper.socket SocksProxy.service
After=SocksProxy.service

[Service]
#Type=simple
#Accept=false
ExecStart=/lib/systemd/systemd-socket-proxyd 127.0.0.1:9118
TimeoutStopSec=5

[Install]
WantedBy=multi-user.target

$ systemctl --user daemon-reload

This seems to work, until ssh dies or gets killed. Then it won't re-spawn at the next connection attempt when it should.

Questions:

  1. Can /usr/bin/ssh really not accept systemd-passed sockets? Or only newer versions? Mine is the one from up2date Debian 8.9.
  2. Can only units of root use the BindTodevice option?
  3. Why is my proxy service not respawning correctly on first new connection after the old tunnel dies?
  4. Is this the right way to set-up an "on-demand ssh socks proxy"? If, not, how do you do it?

Best Answer

  • Can /usr/bin/ssh really not accept systemd-passed sockets?

I think that's not too surprising, considering:

  • OpenSSH is an OpenBSD project
  • systemd only supports the Linux kernel
  • systemd support would need to be explicitly added to OpenSSH, as an optional/build-time dependency, so it would probably be a hard sell.
  • Can only units of root use the BindTodevice option?

User systemd instances are generally pretty isolated, and e.g. can not communicate with the main pid-0 instance. Things like depending on system units from user unit files are not possible.

The documentation for BindToDevice mentions:

Note that setting this parameter might result in additional dependencies to be added to the unit (see above).

Due to the above-mentioned restriction, we can imply that the option doesn't work from user systemd instances.

  • Why is my proxy service not respawning correctly on first new connection after the old tunnel dies?

As I understand, the chain of events is as follows:

  • SocksProxyHelper.socket is started.
  • A SOCKS client connects to localhost:8118.
  • systemd starts SocksProxyHelper.service.
  • As a dependency of SocksProxyHelper.service, systemd also starts SocksProxy.service.
  • systemd-socket-proxyd accepts the systemd socket, and forwards its data to ssh.
  • ssh dies or is killed.
  • systemd notices, and places SocksProxy.service into a inactive state, but does nothing.
  • SocksProxyHelper.service keeps running and accepting connections, but fails to connect to ssh, as it is no longer running.

The fix is to add BindsTo=SocksProxy.service to SocksProxyHelper.service. Quoting its documentation (emphasis added):

Configures requirement dependencies, very similar in style to Requires=. However, this dependency type is stronger: in addition to the effect of Requires= it declares that if the unit bound to is stopped, this unit will be stopped too. This means a unit bound to another unit that suddenly enters inactive state will be stopped too. Units can suddenly, unexpectedly enter inactive state for different reasons: the main process of a service unit might terminate on its own choice, the backing device of a device unit might be unplugged or the mount point of a mount unit might be unmounted without involvement of the system and service manager.

When used in conjunction with After= on the same unit the behaviour of BindsTo= is even stronger. In this case, the unit bound to strictly has to be in active state for this unit to also be in active state. This not only means a unit bound to another unit that suddenly enters inactive state, but also one that is bound to another unit that gets skipped due to a failed condition check (such as ConditionPathExists=, ConditionPathIsSymbolicLink=, … — see below) will be stopped, should it be running. Hence, in many cases it is best to combine BindsTo= with After=.

  • Is this the right way to set-up an "on-demand ssh socks proxy"? If, not, how do you do it?

There's probably no "right way". This method has its advantages (everything being "on-demand") and disadvantages (dependency on systemd, the first connection not getting through because ssh hasn't begun listening yet). Perhaps implementing systemd socket activation support in autossh would be a better solution.

Related Question