Ubuntu – Why is this rsync + ssh cron job giving me ‘Permission denied (publickey)’ errors

backupcronpermissionsrsyncssh

I make frequent backups to a local drive which I want to sync daily to a remote server.

The target server is configured for SSH key (no password) access only. Since my primary SSH key for that server is passphrase-protected, I've created a second SSH key (not passphrase protected) + user to use for unattended backups – this way I do not have to be present to enter my passphrase when cron runs.

I'm using cron and rsync, and all of the commands work individually, but fail when combined.

The furthest I've got while troubleshooting is running

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/"

which returns the error

Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]

Any tips on how to troubleshoot this further?


Here's what I've tried so far and I'm out of ideas:

  1. Cron is definitely running ps aux | grep cron
  2. Nothing unusual in /var/log/syslog Sep 7 13:22:01 desktop CRON[6735]: (tom) CMD (sh /home/tom/Documents/Scripts/offsite-backup)

  3. SSH in Terminal to remote server as the backup user works ssh backups-user@XX.XX.XX.XX

  4. Running the command in Terminal works perfectly rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/
  5. Manually specifying the path to the backups-user key has no effect rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/

  6. Replacing the non-functioning command with a simple test command works echo "Hello world" > ~/Desktop/test.txt

  7. Shouting/swearing at the computer had no effect (but made me feel better temporarily).


Edit 1:

Here's my crontab file and the script it calls.

...
# m h  dom mon dow   command
MAILTO=""
* * * * * sh /home/tom/Documents/Scripts/offsite-backup

and

#!/bin/bash

rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/

Edit 2:

Just to clarify, /var/log/auth.log on the target server contains the line Sep 11 08:23:01 <hostname> CRON[24421]: pam_unix(cron:session): session closed for user root This is confusing because I'm no longer running cron every minute locally, but a new entry still appears every minute in the server logs. Crontab files for all users (including root) on the server are empty & do nothing.

Also, user 'backups-only' was created only on the server and with limited rights, with a dedicated SSH key copied to my desktop machine. I'm assuming this is the way to go because everything works when running the commands manually.

The crontab file posted above is for me, user 'tom' on my desktop machine. My intent is to have it call the script which should log in to the server as user 'backups-only'. I just tried running the backup script (rather than the command inside it) and it successfully connected & worked. I ran it on my desktop as user 'tom', same user who created the cron job that won't work. Here's the output from the server log corresponding with that successful login

Sep 11 08:35:31 <hostname> sshd[25071]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Sep 11 08:35:32 <hostname> sshd[25071]: Accepted publickey for backups-only from <desktop IP> port 54242 ssh2: RSA e2:e6:07:27:c1:continues...
Sep 11 08:35:32 <hostname> sshd[25071]: pam_unix(sshd:session): session opened for user backups-only by (uid=0)
Sep 11 08:35:32 <hostname> systemd-logind[638]: New session 12 of user backups-only.
Sep 11 08:36:00 <hostname> sshd[25133]: Received disconnect from <desktop IP>: 11: disconnected by user
Sep 11 08:36:00 <hostname> sshd[25071]: pam_unix(sshd:session): session closed for user backups-only

Best Answer

Since everything is working fine from the command line, the error Permission denied (publickey) means that the SSH part of rsync is using a different identity file than the specified username.

From Jan's comment on the original question, we can specify the identity file in the rsync command using -e 'ssh -i /path/to/identity.file' ....

Using the below command to start off with a fresh environment in cron and specifying the complete path to the file apparently solves the issue:

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/"

I'm still really interested in this finding. It probably has to do with cron, the fact that it starts with minimal environment variables, and the ssh-agent. I'll be setting up the same scenario ina a couple of days to test it out and report back.