Bash script doesn't handle curly brackets correctly (I believe)

English Programming/Scripting
1

Hi geekos! And a happy new year to all of you!

I’m having a bit of a trouble with rsync. A working command I use for remote back-ups is this one:


rsync -razP --exclude={'*.git*','*.argh'} ./Data/ bar@foo.online:/home/42/precious_data

However, my plan was to have this as a (re-useable) shell script. And so I went and created this script:


options="-razP --exclude={'*.git*','*.argh'} --delete"
local_dir="./Data/"
remote_dir="bar@foo.online:/home/42/precious_data"

rsync ${options} ${local_dir} ${remote_dir}

It kind-of-works. However, all files (including those I nick-named ‘argh’ here), will also be copied. That’s huge binary files I don’t want to have!

Small aberrations, such as using {…} or .argh did not work. Even the seemingly straight-forward “–exclude .git --exclude *.argh” failed. I’m a bit puzzled by now.

Does anyone of you see the (probably stupid) mistake I made here?

2

To begin with, this is not a script at all, it is only some statmenets. A sxript should start with the so called “shebang” as first line

#!/usr/bin/bash

To skip some (a lot of) steps that are less important here:
Your rsync statement is first split by the shell into “words”. That is done on, amongst more, white space. Then variable expansipn is done, which will replace e.g. the word

${options}

with the string

-razP --exclude={'*.git*','*.argh'} --delete

. That new “word” is NOT split again. It will become ONE argument to the rsync command when it is executed on request of the shell. And the argument

-razP --exclude={'*.git*','*.argh'} --delete

will not be understood by rsync in the way you intended.

I have not tested this, but it may be that using eval will help you:

eval rsync ${options} ${local_dir} ${remote_dir}
3

I suggest you try changing the above line to:


options='-razP --exclude={"*.git*","*.argh"} --delete'

Please note that I have not tested this.

4

BTW, I will move this to Programming/Scripting.

5

Thanks for all the input.

Here are some remarks …

  • Using #!/usr/bin/env bash was done (my minimal example was just too minimal, sorry)
  • eval alone did not solve it
  • eval + using --exclude ‘a’ – exclude ‘b’ … worked, but is not very elegant to use
  • in combination with the exchange of " " ↔ ’ ', it seems to work

Here’s my (for now final) minimal script:


#!/usr/bin/env bash

options='-razP --exclude={"*.git*","*.svn*","*.argh"} --delete'
local_dir='./Data/'
remote_dir='bar@foo.online:/home/42/precious_data'

# copy changes to the server
eval rsync ${options} ${local_dir} ${remote_dir}
exit
6

Hm, I tried to reproduce in a test what I said in post #2, but I can not. :frowning:

7

Hi
Just an FYI from a packaging experience, ‘env’ is not used and stripped from any scripts, the shell is an explicit declaration… #!/usr/bin/bash if it’s not working then maybe your $PATH has an issue.

8

Yes, the OP’s shebang is better, but as there was none (shown), I went for the one that is often understood easiest.

9

You do not show your script so we have no way to comment on it.

eval + using --exclude ‘a’ – exclude ‘b’ … worked, but is not very elegant to use

In what way exactly? Your “elegant” solution is

  • less flexible as it is impossible to programmatically build exclude list incrementally
  • more error prone as now you have to take extra care to quote every possible unwanted shell expansion
  • more slow as it adds extra unneeded processing

Having some construct in a language does not mean it must be used by all means. Brace expansion was designed for interactive use to reduce amount of typing. Scripts must be robust and easy to understand unless you attempt to win obfuscated programming contest or prepare your home work.

in combination with the exchange of " " ↔ ’ ', it seems to work

Here’s my (for now final) minimal script:

Which works just fine if you swap ’ and ".

How can $PATH affect absolute pathname?

10

I would tend to avoid BASH brace expansion for the above example with this instead:

rsync -razP --exclude='*.git*' --exclude='*.argh' ./Data/ bar@foo.online:/home/42/precious_data

When you transition to a much longer list of exclusions, you might be better off with the more flexible --filter parameter that keeps your command line a bit more readable:


#!/bin/bash

cat <<EOF >tmpfilterfile
exclude *.git*
exclude *.argh
exclude *.bak
exclude *~
exclude .deleteme1
exclude .deleteme3
exclude .deleteme5
include .addme2
include .addme4
include .addme6
exclude .gitignore
exclude .git/
EOF

rsync -razP --filter ". tmpfilterfile" ./Data/ bar@foo.online:/home/42/precious_data
11

“You can also exclude multiple paths within curly braces”: command line - How to exclude multiple directories with rsync? - Ask Ubuntu

I tried “rsync --exclude={.xauth*,dup*} .”. It obviously does what I expect.

12

Hello,

In Bash, test and are shell builtins.

The double bracket, which is a shell keyword, enables additional functionality. For example, you can use && and || instead of -a and -o and there’s a regular expression matching operator =~.

Also, in a simple test, double square brackets seem to evaluate quite a lot quicker than single ones.

$ time for ((i=0; i<10000000; i++)); do “$i” = 1000 ]]; done

real 0m24.548s
user 0m24.337s
sys 0m0.036s
$ time for ((i=0; i<10000000; i++)); do “$i” = 1000 ]; done

real 0m33.478s
user 0m33.478s
sys 0m0.000s