Does sftp work when started in cron?

Does sftp work when started in cron? Apparently not, since it does not here.

I created a script that uses sftp to download some files. It works as expected when run from the command line.

Our ISP has ssh set up to verify our credentials without using a password. I enter “ssh xxx@example.com” and two seconds later I’m logged in. It works the same in the script.

However, running it as a cron job produces this result:

[EMAIL="sohne@sohnen-moe.com"]xxx@example.com[/EMAIL]: Permission denied (publickey).
Connection closed

What is different when running the script from cron?

The (sanitized) script:

SFTP="/usr/bin/sftp"
FIND="/usr/bin/find"

OPTS1="-P 1022 -p -o GSSAPIAuthentication=no"
OPTS2="-i /home/xxx/.ssh/id_dsa"
OPTS3="-b /home/xxx/bin/sftp-sma-download-batch"
SITE="xxx@example.com"

DIR_DST="/u/websites/site-data"
DIR_ORG="$(pwd)"

cd $DIR_DST

# Get the latest files
#
txt="$SFTP $OPTS1 $OPTS2 $OPTS3 $SITE"
echo $txt
$txt

Does cron job run as root or your user? Is private key protected by passphrase?

Hi,

You should at least get an error message coming from cron.
or point the output of the script to a log file.

* * * * * mycron-script-entry > /path/to/logfile 2>&1

So you can examine what is inside the log file later.

also put an || operator and exit after the cd command

cd "$dest" || exit

Just in case the directory does not exists or you don’t have a proper permission to that directory the script will exit immediately.

User.
Yes.

As noted the script works when run from the command line. The cron job is for the same user.

I did. I posted it. The output is captured by the system and sent to me as email.

27 02  *  * 00 /home/xxx/bin/sftp-sma-download

Hi,

cron has a limited path and set of environmental variables, the ssh-agent might be one of them, that might explain the error you’re having.
though it got me curios if the latest and greatest systemd.timer can do that by default without any trickery.

I did not read all the details above, but noticed that this is about a common encountered problem: a command started using cron does not function as expected regardless if it functions when executed from a shell in a terminal (emulator).

This often encountered problem is based in the user thinking that a statement executed by cron runs in exactly the same environment as when executed (tested) in the users (default?) shell. This is not the case. The shell is no login shell, thus e.g. ~/.profile is not executed and all that the user has put there is thus not done/available. PATH is different. Environment variables may or may not be defined with the same or a different value as expected. A few hints:

  • read
man crontab
  • interactive programs (those that ask you to type some answer on the terminal) will of course not run without caring for that when run in the background.
  • use absolute paths instead of relative paths, the latter may use PATH to get an absolute path and thus fail;
  • check if the shell used by cron is the same you wrote your statement for (in most cases both will be bash, but it is easy to forget when you are e.g. a c-shell adept);
  • you may need to set environment variables either in the crontab (which will them be valid for all entries in that crontab) or in the statement itself;
  • and last, but not least, be careful when testing a command as user and then putting it into one of the crontabs that will be executed under root (root’s own crontab and the system crontabs).

On 09/02/2018 11:56 PM, jimoe666 wrote:
>
> arvidjaar;2879219 Wrote:
>> Does cron job run as root or your user? Is private key protected by
>> passphrase?
> User.
> Yes.

If the private key is protected, then you need to specify the passphrase
to un-protect it. Since you are not doing that in the script, it may
explain why you cannot authenticate.

To get more information out of the sftp (or ssh or scp) command, try
adding -vvv (three ‘v’ characters after a hyphen) and see what it reveals,
perhaps an inability to decrypt the private key, meaning you cannot use it.

If you want to use SSH keys for use in scripts you either need to be there
able to enter a passphrase, or else you need to store that passphrase
somewhere/somehow, or you need to have a key setup without a passphrase.

You may object to not having a passphrase, but then you will not be able
to script this; you want to trust cron to act as you, so cron must be able
to become you, meaning it needs access to a key, or password, or something
else to let the remote computer know it is actually you.

What I would do is setup a second key, get it to the target system and
make sure it works (name it something like id_rsa-script-key; stop using
DSA keys) with the following command, and then use it specifically:


SITE='xxx@example.com';  #fix this of course

ssh-keygen -t rsa -b 4096 -C 'key for scripting' -f id_rsa-script-key
ssh-copy-id -i ~/.ssh/id_rsa-script-key "${SITE}"

You should receive a message indicating your new key is copied over to the
target system, but now you need to modify it on that side, so SSH into
SITE and let’s do that:


ssh "${SITE}"

In your home directory on the target system should be your ./.ssh
directory (just like on your client system) with an authorized_keys file
inside. Inside this file your keys exist, one per line; the first one is
probably the one you already had which requires a passphrase, and the last
one is the one you just added with ssh-copy-id above, so now you need to
tell SSH EXACTLY what that key can be used for on the target system,
namely whatever command runs when you call sftp. Doing a tiny bit of
looking right now, I think what you need to do is prepend that last key’s
line with the following:


command="/usr/lib/ssh/sftp-server"

As a result your line ends up looking a bit like this, but with your key:


command="/usr/lib/ssh/sftp-server" ssh-rsa BLAHBLAHBLAH....

If you now test you should find that, when using that key, you can ONLY
run that one command, which is good if you are using sftp but otherwise it
is junk. This way your passphrase-less key is only useful for the file
transfer, or whatever else can be coaxed out of the sftp-server command,
meaning you have presumably reduced the risk of having that
passphrase-less key available. After you point your cron job to use this
new script hopefully that will also work since this key does not require a
passphrase.


Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.

“works from command line” can also mean “you have typed in passphrase to unlock private key”.

Was wondering where you got your binary “sftp-sma-download-batch” from and how it works…
Besides the fact that you’re calling two different sftp executables in your script…

And, wherever you got this “sftp-sma-download-batch” from, if it has documentation and/or examples.

TSU

“sftp-sma-download-batch” is a text file containing sftp commands. See the man page about the “-b” (batch) option.

To answer questions about a passphrase: My mistake; a passphrase is not required to log into the remote system.

I had thought that by providing the specific location of the public/private keys (the “-i” option) that issues like failure to log in would be resolved. All of the paths are fully qualified.

The consensus seems to be that there is a missing element in the cron environment that prevents sftp from being fully functional. What could it be?

Your batch command appears to be a custom creation so is a wildcard.
If it’s not a standard file from somewhere and your own creation, its contents should be posted on the chance something in it might be relevant.

TSU

$ cat bin/sftp-sma-download-batch 
#
cd /home/xxx/db
ls -lh
get xxx_*.sql.gz

I’d recommend that you instrument your rather complex sftp command (write some output to file since you’re trying to determine what happens when executed as a cron job)…
And, execute the two sftp commands (your main and then your batch) separately.

TSU

Okay. Here are the results of two runs, one interactive, the other by cron. The logs start where there is a difference in response. And even the different portion is almost the same until the end: “we did not send a packet…” “Agent” seems to be the thing that is different.

Interactive:

debug2: key: /home/xxx/.ssh/jumpline (0x55e90d1fb3a0), explicit, agent

debug1: Skipping ssh-dss key xxx@sma-station14l - not in PubkeyAcceptedKeyTypes
debug1: Skipping ssh-dss key xxx@sma-station14l - not in PubkeyAcceptedKeyTypes
debug1: Skipping ssh-dss key xxx@sma-station14l - not in PubkeyAcceptedKeyTypes
debug2: key: xxx@sma-station14l (0x55e90d1f6cd0), agent

debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: 
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:B1iu57Rkn5emB//MUP4YEipr4oRRmqZeBHMQWf0U+Mk /home/xxx/.ssh/jumpline
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:B1iu57Rkn5emB//MUP4YEipr4oRRmqZeBHMQWf0U+Mk
debug3: sign_and_send_pubkey: RSA SHA256:B1iu57Rkn5emB//MUP4YEipr4oRRmqZeBHMQWf0U+Mk

debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to sohnen-moe.com ([216.222.193.110]:1022).

Cron:

debug2: key: /home/xxx/.ssh/jumpline (0x55a2cb793ff0), explicit

debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: 
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:B1iu57Rkn5emB//MUP4YEipr4oRRmqZeBHMQWf0U+Mk /home/xxx/.ssh/jumpline
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:B1iu57Rkn5emB//MUP4YEipr4oRRmqZeBHMQWf0U+Mk
debug3: sign_and_send_pubkey: RSA SHA256:B1iu57Rkn5emB//MUP4YEipr4oRRmqZeBHMQWf0U+Mk

debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
sohne@sohnen-moe.com: Permission denied (publickey).
Connection closed

What permissions do you have set on the directories on the client side up
through the .ssh directory? Show the output from the following command:


namei -o -m ~/.ssh/


Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.

Yes. Interactive session has SSH agent which provides key. And this key must have been unlocked when you logged in. There are no miracles - it is impossible to both have key protected by passphrase and to use key non-interactively, without providing passphrase to unlock it.

$  namei -o -m ~/.ssh/
f: /home/jmoe/.ssh/
 drwxr-xr-x root root  /
 drwxr-xr-x root root  home
 drwxr-xr-x jmoe users jmoe
 drwx------ jmoe users .ssh

All private keys are mode 0600.

Do you see anything on the remote side that may be useful? I cannot
figure out why it would matter since everything looks fie to me (my system
has tighter permissions on ${HOME} but you could try to change that
temporarily, e.g. to drw------ (700) or something) but otherwise maybe
that log file will give you more data in case the remote side is involved
in dropping the connection, though I thought that was only valid if the
authorized_keys files permissions were wrong.

I do not suppose this is a system into which somebody can SSH and test
directly, is it? Last night I used my laptop and setup a scron job that
worked nicely with ssh and scp. Perhaps another option for you is to see
if you can use ssh or scp from cron rather than sftp, just in case
something about sftp or the batch use of it is interfering (unlikely, I know).


#This should call 'hostname' on the remote box and write it locally.
* * * * * ssh -i /home/XXX/.ssh/jumpline sohnen-moe.com 'hostname' >
/tmp/cron.txt


Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.