I make frequent backups to a local drive which I want to sync daily to a remote server.
The target server is configured for SSH key (no password) access only. Since my primary SSH key for that server is passphrase-protected, I've created a second SSH key (not passphrase protected) + user to use for unattended backups – this way I do not have to be present to enter my passphrase when cron runs.
I'm using cron and rsync, and all of the commands work individually, but fail when combined.
The furthest I've got while troubleshooting is running
env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/"
which returns the error
Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]
Any tips on how to troubleshoot this further?
Here's what I've tried so far and I'm out of ideas:
- Cron is definitely running
ps aux | grep cron
-
Nothing unusual in /var/log/syslog
Sep 7 13:22:01 desktop CRON[6735]: (tom) CMD (sh /home/tom/Documents/Scripts/offsite-backup)
-
SSH in Terminal to remote server as the backup user works
ssh backups-user@XX.XX.XX.XX
- Running the command in Terminal works perfectly
rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/
-
Manually specifying the path to the backups-user key has no effect
rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/
-
Replacing the non-functioning command with a simple test command works
echo "Hello world" > ~/Desktop/test.txt
-
Shouting/swearing at the computer had no effect (but made me feel better temporarily).
Edit 1:
Here's my crontab file and the script it calls.
...
# m h dom mon dow command
MAILTO=""
* * * * * sh /home/tom/Documents/Scripts/offsite-backup
and
#!/bin/bash
rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/
Edit 2:
Just to clarify, /var/log/auth.log
on the target server contains the line Sep 11 08:23:01 <hostname> CRON[24421]: pam_unix(cron:session): session closed for user root
This is confusing because I'm no longer running cron every minute locally, but a new entry still appears every minute in the server logs. Crontab files for all users (including root) on the server are empty & do nothing.
Also, user 'backups-only' was created only on the server and with limited rights, with a dedicated SSH key copied to my desktop machine. I'm assuming this is the way to go because everything works when running the commands manually.
The crontab file posted above is for me, user 'tom' on my desktop machine. My intent is to have it call the script which should log in to the server as user 'backups-only'. I just tried running the backup script (rather than the command inside it) and it successfully connected & worked. I ran it on my desktop as user 'tom', same user who created the cron job that won't work. Here's the output from the server log corresponding with that successful login
Sep 11 08:35:31 <hostname> sshd[25071]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Sep 11 08:35:32 <hostname> sshd[25071]: Accepted publickey for backups-only from <desktop IP> port 54242 ssh2: RSA e2:e6:07:27:c1:continues...
Sep 11 08:35:32 <hostname> sshd[25071]: pam_unix(sshd:session): session opened for user backups-only by (uid=0)
Sep 11 08:35:32 <hostname> systemd-logind[638]: New session 12 of user backups-only.
Sep 11 08:36:00 <hostname> sshd[25133]: Received disconnect from <desktop IP>: 11: disconnected by user
Sep 11 08:36:00 <hostname> sshd[25071]: pam_unix(sshd:session): session closed for user backups-only
Best Answer
Since everything is working fine from the command line, the error
Permission denied (publickey)
means that the SSH part ofrsync
is using a different identity file than the specified username.From Jan's comment on the original question, we can specify the identity file in the
rsync
command using-e 'ssh -i /path/to/identity.file' ...
.Using the below command to start off with a fresh environment in cron and specifying the complete path to the file apparently solves the issue:
I'm still really interested in this finding. It probably has to do with cron, the fact that it starts with minimal environment variables, and the ssh-agent. I'll be setting up the same scenario ina a couple of days to test it out and report back.