Can’t systemd service access user’s home dir, but works after service restart

systemd

I've created a systemd service that uses ssh to call home to a central server. 

[Unit]
Description=Create a tunnel in the cloud back to SSH on this machine
After=network-online.target

[Service]
User=mindhive
ExecStart=/usr/bin/ssh -o ServerAliveInterval=20 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=no -i /home/mindhive/.ssh/tunnel.id_rsa -N -T -R1822:localhost:22 tunnel@***server-hidden***
RestartSec=60
Restart=always

[Install]
WantedBy=multi-user.target

I use this on many of our servers, all Ubuntu 16.04. Suddenly one of them has stopped working. I can see from the logs (below) it's to do with access to .ssh in the homedir. The service is set to Restart=always and RestartSec=60, so after a reboot it sits there trying again every minute, failing each time. However, if I manually sudo systemctl restart ssh-tunnel.service then it starts no problem.

You can see from the logs below (output of journalctl) it failing and retrying but when started manually at 14:54:10 it starts fine.

From Googling so far I've tried adding both WorkingDirectory=~ and ProtectHome=off to the service. But that's made no difference.

Why can't ssh access the user's homedir when started by systemd after a reboot, but it can if started manually by systemctl? 

-- Reboot --
Feb 24 14:50:12 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:50:12 ***servername*** ssh[1252]: Warning: Identity file /home/mindhive/.ssh/tunnel.id_rsa not accessible: No such file or directory.
Feb 24 14:50:13 ***servername*** ssh[1252]: Could not create directory '/home/mindhive/.ssh'.
Feb 24 14:50:14 ***servername*** ssh[1252]: Failed to add the host to the list of known hosts (/home/mindhive/.ssh/known_hosts).
Feb 24 14:50:15 ***servername*** systemd[1]: ssh-tunnel.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:50:15 ***servername*** systemd[1]: ssh-tunnel.service: Unit entered failed state.
Feb 24 14:50:15 ***servername*** systemd[1]: ssh-tunnel.service: Failed with result 'exit-code'.
Feb 24 14:51:15 ***servername*** systemd[1]: ssh-tunnel.service: Service hold-off time over, scheduling restart.
Feb 24 14:51:15 ***servername*** systemd[1]: Stopped Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:51:15 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:51:15 ***servername*** ssh[1367]: Warning: Identity file /home/mindhive/.ssh/tunnel.id_rsa not accessible: No such file or directory.
Feb 24 14:51:16 ***servername*** ssh[1367]: Could not create directory '/home/mindhive/.ssh'.
Feb 24 14:51:17 ***servername*** ssh[1367]: Failed to add the host to the list of known hosts (/home/mindhive/.ssh/known_hosts).
Feb 24 14:51:18 ***servername*** systemd[1]: ssh-tunnel.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:51:18 ***servername*** systemd[1]: ssh-tunnel.service: Unit entered failed state.
Feb 24 14:51:18 ***servername*** systemd[1]: ssh-tunnel.service: Failed with result 'exit-code'.
Feb 24 14:52:18 ***servername*** systemd[1]: ssh-tunnel.service: Service hold-off time over, scheduling restart.
Feb 24 14:52:18 ***servername*** systemd[1]: Stopped Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:52:18 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:52:18 ***servername*** ssh[1370]: Warning: Identity file /home/mindhive/.ssh/tunnel.id_rsa not accessible: No such file or directory.
Feb 24 14:52:19 ***servername*** ssh[1370]: Could not create directory '/home/mindhive/.ssh'.
Feb 24 14:52:20 ***servername*** ssh[1370]: Failed to add the host to the list of known hosts (/home/mindhive/.ssh/known_hosts).
Feb 24 14:52:20 ***servername*** systemd[1]: ssh-tunnel.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:52:20 ***servername*** systemd[1]: ssh-tunnel.service: Unit entered failed state.
Feb 24 14:52:20 ***servername*** systemd[1]: ssh-tunnel.service: Failed with result 'exit-code'.
Feb 24 14:53:21 ***servername*** systemd[1]: ssh-tunnel.service: Service hold-off time over, scheduling restart.
Feb 24 14:53:21 ***servername*** systemd[1]: Stopped Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:53:21 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:53:21 ***servername*** ssh[1393]: Warning: Identity file /home/mindhive/.ssh/tunnel.id_rsa not accessible: No such file or directory.
Feb 24 14:53:21 ***servername*** ssh[1393]: Could not create directory '/home/mindhive/.ssh'.
Feb 24 14:53:22 ***servername*** ssh[1393]: Failed to add the host to the list of known hosts (/home/mindhive/.ssh/known_hosts).
Feb 24 14:53:23 ***servername*** systemd[1]: ssh-tunnel.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:53:23 ***servername*** systemd[1]: ssh-tunnel.service: Unit entered failed state.
Feb 24 14:53:23 ***servername*** systemd[1]: ssh-tunnel.service: Failed with result 'exit-code'.
Feb 24 14:54:10 ***servername*** systemd[1]: Stopped Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:54:10 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.

Best Answer

OK. Managed to figure it out myself... eventually. The issue is that Ubuntu had been installed with encrypted home dir. So, that's why the files in ~/.ssh weren't accessible to the service until I logged in (therefore decrypting the home dir) and restarted it.

Removing the encryption was not easy. I found these instructions best.

Related Question