Backup issue using rsync, SSH, and cron

Automating remote backups.

Two machines on LAN.

Rsync, SSH, and cron.

Root and user have ssh-keys in respective (/home/user/.ssh and /root/.ssh) directories.

SSH passwordless logins in terminal work for user and root.

Rsync from terminal works for user (home/user) and root (/ with exclusions).

Crontab scheduled backup works for user, but NOT for root.

I think this snippet is telling me the issue:

/var/log/messages
2022-09-13T09:20:01.714025-05:00 Basebox sshd[26640]: gkr-pam: unable to locate daemon control file
2022-09-13T09:20:01.765549-05:00 Basebox sshd[26640]: gkr-pam: stashed password to try later in open session
2022-09-13T09:20:01.765759-05:00 Basebox sshd[26640]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.58 user=root
2022-09-13T09:20:02.703392-05:00 Basebox org.xfce.ScreenSaver[2220]: Xlib: extension “DPMS” missing on display “:200.0”.
2022-09-13T09:20:03.479735-05:00 Basebox sshd[26638]: error: PAM: Authentication failure for root from 192.168.0.58
2022-09-13T09:20:03.504531-05:00 Basebox sshd[26647]: gkr-pam: unable to locate daemon control file
2022-09-13T09:20:03.504709-05:00 Basebox sshd[26647]: gkr-pam: stashed password to try later in open session
2022-09-13T09:20:03.506982-05:00 Basebox sshd[26647]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.58 user=root

From experience I know to look at configs, modes, and perms. Deep down I know PIBKAC.

I find myself in the weeds here. Any help is appreciated.

Hello and welcome to the openSUSE forums.

You say your crontab entry for root does not work. But you did not show anything. So please as root

crontab -l

And while you are new here, please:
There is an important, but not easy to find feature on the forums.

Please in the future use CODE tags around copied/pasted computer text in a post. It is the # button in the tool bar of the post editor. When applicable copy/paste complete, that is including the prompt, the command, the output and the next prompt.

An example is here: Using CODE tags Around your paste.

sudo crontab -l
40 10 * * * /usr/bin/rsync -n -r -t -p -o -g -x -v --progress --delete -l -z -s --exclude-from=/home/User/root-exclude-file / root@192.168.0.57:/home/User/test-backups/System

my guess is that there is no entry in /root/.ssh/authorized.keys just the /home/username/.ssh/authorized keys.

An alternative is to use /usr/bin/sshpass -p password before the /usr/bin/rsync.

Me - I would create a /root/myback.sh with the rsync command line and put the /usr/bin/bash /root/myback.sh >/root/myback.log 2>&1 in the crontab so you can see what happens.

Hello there. SSH keys exist for user and root on the client side. There are authorized_keys in the respective directories on the server side. Indeed, this all works (passwordless) for user files and root files from the command line.

Adding /usr/bin/sshpass -p password to the crontab entry doesn’t do the trick, since I’d much rather do this with SSH keys.

I checked the client side and root’s id_rsa.pub permission were set to 622. (That’s tripped me up in in past.) Changed to 600 and let crontab run again. Still no love.

Server side output:

/var/log/messages

2022-09-14T16:15:02.303369-05:00 Basebox sshd[17144]: gkr-pam: unable to locate daemon control file
2022-09-14T16:15:02.381317-05:00 Basebox sshd[17144]: gkr-pam: stashed password to try later in open session
2022-09-14T16:15:02.381912-05:00 Basebox sshd[17144]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.58  user=root
2022-09-14T16:15:02.731428-05:00 Basebox org.xfce.ScreenSaver[2187]: Xlib:  extension "DPMS" missing on display ":200.0".
2022-09-14T16:15:04.758025-05:00 Basebox sshd[17142]: error: PAM: Authentication failure for root from 192.168.0.58
2022-09-14T16:15:04.792900-05:00 Basebox sshd[17151]: gkr-pam: unable to locate daemon control file
2022-09-14T16:15:04.793173-05:00 Basebox sshd[17151]: gkr-pam: stashed password to try later in open session
2022-09-14T16:15:04.810965-05:00 Basebox sshd[17151]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.58  user=root
2022-09-14T16:15:06.732038-05:00 Basebox panel-7-pulseau[3271]: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.
2022-09-14T16:15:06.732400-05:00 Basebox panel-7-pulseau[3271]: pa_context_connect() failed: Access denied
2022-09-14T16:15:07.188354-05:00 Basebox sshd[17142]: error: PAM: Authentication failure for root from 192.168.0.58
2022-09-14T16:15:07.197873-05:00 Basebox sshd[17142]: Postponed keyboard-interactive for root from 192.168.0.58 port 51034 ssh2 [preauth]
2022-09-14T16:15:07.198639-05:00 Basebox sshd[17162]: gkr-pam: unable to locate daemon control file
2022-09-14T16:15:07.198857-05:00 Basebox sshd[17162]: gkr-pam: stashed password to try later in open session
2022-09-14T16:15:07.215804-05:00 Basebox sshd[17162]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.58  user=root
2022-09-14T16:15:09.005408-05:00 Basebox sshd[17142]: error: PAM: Authentication failure for root from 192.168.0.58
2022-09-14T16:15:09.006589-05:00 Basebox sshd[17142]: Failed none for root from 192.168.0.58 port 51034 ssh2
2022-09-14T16:15:09.015252-05:00 Basebox sshd[17142]: Failed password for root from 192.168.0.58 port 51034 ssh2
2022-09-14T16:15:09.023681-05:00 Basebox sshd[17142]: error: maximum authentication attempts exceeded for root from 192.168.0.58 port 51034 ssh2 [preauth]

There’s all sorts of PulseAudio stuff in there, but that’s a problem for another day.

It just says that authentication failed.

Compare server side output for a “good” case. Add “-vvv” to ssh options to get verbose information on client what it does. Start ssh server with verbose/debug option to see what happens on server side.

I don’t use cron. I have passwordless ssh working for root. And the following works too:

**erlangen:~ #** rsync -a /home/tester/ 6700k:/home/test/ 
**erlangen:~ #**

What about your system? Does rsync work as above?

**erlangen:~ #** ssh 6700k du -hd1 -t1k /home/test/ 
75M     /home/test/.cache 
512K    /home/test/.config 
28K     /home/test/.gnupg 
659M    /home/test/.local 
14M     /home/test/.mozilla 
4.0K    /home/test/.xemacs 
12K     /home/test/Schreibtisch 
747M    /home/test/ 
**erlangen:~ #** 

Karl:
“Does rsync work as above?” Yes. Uebrigens, Ich heisse auch Karl :slight_smile:

Arvid:
Here’s the output on the server side with verbosity raised to -vvv

/var/log/messages
2022-09-15T10:15:02.121276-05:00 Basebox sshd[4303]: gkr-pam: unable to locate daemon control file2022-09-15T10:15:02.164928-05:00 Basebox sshd[4303]: gkr-pam: stashed password to try later in open session
2022-09-15T10:15:02.165137-05:00 Basebox sshd[4303]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.58  user=root
2022-09-15T10:15:02.731331-05:00 Basebox org.xfce.ScreenSaver[2187]: Xlib:  extension "DPMS" missing on display ":200.0".
2022-09-15T10:15:03.731161-05:00 Basebox panel-7-pulseau[25773]: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.
2022-09-15T10:15:03.731512-05:00 Basebox panel-7-pulseau[25773]: pa_context_connect() failed: Access denied
2022-09-15T10:15:03.811186-05:00 Basebox sshd[4301]: error: PAM: Authentication failure for root from 192.168.0.58
2022-09-15T10:15:03.821805-05:00 Basebox sshd[4308]: gkr-pam: unable to locate daemon control file
2022-09-15T10:15:03.822220-05:00 Basebox sshd[4308]: gkr-pam: stashed password to try later in open session
2022-09-15T10:15:03.839069-05:00 Basebox sshd[4308]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.58  user=root
2022-09-15T10:15:05.655373-05:00 Basebox sshd[4301]: error: PAM: Authentication failure for root from 192.168.0.58
2022-09-15T10:15:05.664776-05:00 Basebox sshd[4301]: Postponed keyboard-interactive for root from 192.168.0.58 port 48810 ssh2 [preauth]
2022-09-15T10:15:05.665507-05:00 Basebox sshd[4315]: gkr-pam: unable to locate daemon control file
2022-09-15T10:15:05.665681-05:00 Basebox sshd[4315]: gkr-pam: stashed password to try later in open session
2022-09-15T10:15:05.682678-05:00 Basebox sshd[4315]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.58  user=root
2022-09-15T10:15:07.775357-05:00 Basebox sshd[4301]: error: PAM: Authentication failure for root from 192.168.0.58
2022-09-15T10:15:07.776580-05:00 Basebox sshd[4301]: Failed none for root from 192.168.0.58 port 48810 ssh2
2022-09-15T10:15:07.785058-05:00 Basebox sshd[4301]: Failed password for root from 192.168.0.58 port 48810 ssh2
2022-09-15T10:15:07.793482-05:00 Basebox sshd[4301]: error: maximum authentication attempts exceeded for root from 192.168.0.58 port 48810 ssh2 [preauth]
2022-09-15T10:15:07.793693-05:00 Basebox sshd[4301]: Disconnecting authenticating user root 192.168.0.58 port 48810: Too many authentication failures [preauth]



Raised where?

In the crontab/rsync command.

Did you mean for me to change the logging level in sshd_config?

Then why you posted log from server? This option will make verbose output on client, and it will at least show how client attempts to authenticate.

My bad. I’m kind of a newbie. Here’s client side:

sudo cat /var/log/messages
2022-09-15T12:30:01.833861-05:00 Bumfuzzle systemd[1]: Started User Manager for UID 0.2022-09-15T12:30:01.834364-05:00 Bumfuzzle systemd[1]: Started Session c340 of User root.
2022-09-15T12:30:01.838453-05:00 Bumfuzzle  cron[21085]: pam_unix(crond:session): session opened for user root(uid=0) by (uid=0)
2022-09-15T12:30:01.839281-05:00 Bumfuzzle  CRON[21100]: (root) CMD (/usr/bin/rsync -n -r -t -p -o -g -x -vvv --progress --delete -l -z -s --exclude-from=/home/JR/root-exclude-file / root@192.168.0.57:/home/JR/test-backups/Bumfuzzle/System)
2022-09-15T12:30:07.316082-05:00 Bumfuzzle  CRON[21085]: (root) CMDEND (/usr/bin/rsync -n -r -t -p -o -g -x -vvv --progress --delete -l -z -s --exclude-from=/home/JR/root-exclude-file / root@192.168.0.57:/home/JR/test-backups/Bumfuzzle/System)
2022-09-15T12:30:07.317716-05:00 Bumfuzzle  CRON[21085]: pam_unix(crond:session): session closed for user root
2022-09-15T12:30:07.318698-05:00 Bumfuzzle systemd[1]: session-c340.scope: Deactivated successfully.
2022-09-15T12:30:17.505046-05:00 Bumfuzzle systemd[1]: Stopping User Manager for UID 0...

Just to recap. Passwordless root file and user file backups work when using rsync from the terminal. The user file backup works when scheduled with (user’s) crontab, but root file backup does not work with (root’s) crontab.

No, that won’t work, it adds -vvv options to rsync, not to ssh. To add additional options to ssh you need to use --rsh (-e) rsync option, like

rsync --rsh "ssh -vvv" ...

See also “man rsync”.

Passwordless root file and user file backups work when using rsync from the terminal. The user file backup works when scheduled with (user’s) crontab, but root file backup does not work with (root’s) crontab.

Yes, I read it already. It means something happens differently when running from within cron and you need to find out what.

This is just a long shot because I did not read all the details above.

Differences between execution of a script/command from the terminbal vs. doing it from cron often are due to differences in the environment. Processes running from cron have less and different environment variables. Like PATH, but I saw you already use an absolute path to the executable. Worthwhile checking the others?

larryr:

“…my guess is that there is no entry in /root/.ssh/authorized.keys just the /home/username/.ssh/authorized keys.”

Closest so far to a solution. I DID have a /root/.ssh/authorized_keys. However, when I ran ssh-copy-id, I did it as my normal user, and NOT AS ROOT. Hence the authentication error. What’s even funnier is how this was discovered. I fired up a VM to see if I could replicate my problem. When I passed the SSH key to the VM, I typed the IP address of the server, not the VM! Why I even used the IP address instead of the hostname I don’t know. (That’s one of the reasons I’ve got dnsmasq caching.) LOLZ. I’m an idiot. In my defense, I did hint at PIBKAC in my original post.

Anyway, things are better, but still aren’t working. No files are transferred to the server.

Here’s the server side output:

/var/log/messages
2022-09-16T14:05:02.300158-05:00 Basebox sshd[18521]: Accepted publickey for root from 192.168.0.58 port 43492 ssh2: RSA SHA256:PLDTTWncbCMPae8lIKvC8h14CxyoxdB/bO6EroHxAoo
2022-09-16T14:05:02.354213-05:00 Basebox systemd[1]: Created slice User Slice of UID 0.
2022-09-16T14:05:02.354447-05:00 Basebox systemd[1]: Starting User Runtime Directory /run/user/0...
2022-09-16T14:05:02.354914-05:00 Basebox systemd-logind[764]: New session 174 of user root.
2022-09-16T14:05:02.355209-05:00 Basebox systemd[1]: Finished User Runtime Directory /run/user/0.
2022-09-16T14:05:02.355328-05:00 Basebox systemd[1]: Starting User Manager for UID 0...
2022-09-16T14:05:02.357086-05:00 Basebox systemd: pam_unix(systemd-user:session): session opened for user root(uid=0) by (uid=0)
2022-09-16T14:05:02.587885-05:00 Basebox systemd[18527]: Queued start job for default target Main User Target.
2022-09-16T14:05:02.589202-05:00 Basebox systemd[18527]: Created slice User Application Slice.
2022-09-16T14:05:02.589406-05:00 Basebox systemd[18527]: Started Daily Cleanup of User's Temporary Directories.
2022-09-16T14:05:02.589505-05:00 Basebox systemd[18527]: Reached target Paths.
2022-09-16T14:05:02.589597-05:00 Basebox systemd[18527]: Reached target Timers.
2022-09-16T14:05:02.589987-05:00 Basebox systemd[18527]: Starting D-Bus User Message Bus Socket...
2022-09-16T14:05:02.590146-05:00 Basebox systemd[18527]: PipeWire PulseAudio was skipped because of a failed condition check (ConditionUser=!root).
2022-09-16T14:05:02.590318-05:00 Basebox systemd[18527]: Listening on PipeWire Multimedia System Socket.
2022-09-16T14:05:02.591614-05:00 Basebox systemd[18527]: Starting Create User's Volatile Files and Directories...
2022-09-16T14:05:02.602880-05:00 Basebox systemd[18527]: Finished Create User's Volatile Files and Directories.
2022-09-16T14:05:02.604663-05:00 Basebox systemd[18527]: Listening on D-Bus User Message Bus Socket.
2022-09-16T14:05:02.604839-05:00 Basebox systemd[18527]: Reached target Sockets.
2022-09-16T14:05:02.604933-05:00 Basebox systemd[18527]: Reached target Basic System.
2022-09-16T14:05:02.605027-05:00 Basebox systemd[18527]: Reached target Main User Target.
2022-09-16T14:05:02.605113-05:00 Basebox systemd[18527]: Startup finished in 238ms.
2022-09-16T14:05:02.605211-05:00 Basebox systemd[1]: Started User Manager for UID 0.
2022-09-16T14:05:02.607262-05:00 Basebox systemd[1]: Started Session 174 of User root.
2022-09-16T14:05:02.614804-05:00 Basebox sshd[18521]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
2022-09-16T14:05:03.915854-05:00 Basebox panel-7-pulseau[9214]: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.
2022-09-16T14:05:03.916174-05:00 Basebox panel-7-pulseau[9214]: pa_context_connect() failed: Access denied
2022-09-16T14:05:04.959392-05:00 Basebox sshd[18521]: Received disconnect from 192.168.0.58 port 43492:11: disconnected by user
2022-09-16T14:05:04.959736-05:00 Basebox sshd[18521]: Disconnected from user root 192.168.0.58 port 43492
2022-09-16T14:05:04.962893-05:00 Basebox sshd[18521]: pam_unix(sshd:session): session closed for user root

Here’s the corresponding client side output:

/var/log/messages
2022-09-16T14:05:02.166129-05:00 Bumfuzzle cron[27195]: pam_unix(crond:session): session opened for user root(uid=0) by (uid=0)
2022-09-16T14:05:02.166978-05:00 Bumfuzzle CRON[27211]: (root) CMD (/usr/bin/rsync -n -r -t -p -o -g -x -v  --progress --delete -l -z -s --exclude-from=/home/JR/root-exclude-file / root@192.168.0.57:/home/JR/test-backups/Bumfuzzle/System)
2022-09-16T14:05:05.221559-05:00 Bumfuzzle CRON[27195]: (root) CMDEND (/usr/bin/rsync -n -r -t -p -o -g -x -v  --progress --delete -l -z -s --exclude-from=/home/JR/root-exclude-file / root@192.168.0.57:/home/JR/test-backups/Bumfuzzle/System)
2022-09-16T14:05:05.223606-05:00 Bumfuzzle CRON[27195]: pam_unix(crond:session): session closed for user root

Now I really feel stupid. Leaving the --dry-run flag in the options and wondering why this STILL isn’t working? Really? LOLZ!

cron is PITA, in my opinion of course. systemd is the better alternative and works like a charm: https://forums.opensuse.org/showthread.php/575437-backup-btrfs?p=3159034#post3159034

http://www.mistelberger.net/chillin.svg

Na sicher, Ich lerne das nach meine snapshots klappt!

I win! It works. Thanks for everyone’s help.

You stated multiple times that your rsync command worked in terminal as root which is impossible if server did not have the correct public key.