rsync in stages per 'n' dirs at the time

Hi all,

I want to run rsync on very large directory with long path depth. Some dir names in the path are just ‘-’ or ‘_’, or contain these characters in the beginning or in the end of the dir name.
To be honest i don’t know how rsync will react to this directories. So i thought to run rsync with exclude option.
Another issue i have is i can’t just unleash rsync on dir that 740G large due to limited resources, i can only run it at night.
This is why i would love to run rsync in stages, like tell rsync to take 100 dirs from depth 1 etc.
would anyone can help me to work this out?
thnx

Start rsync in the evening and stop it in the morning, as you like. And let it make it’s thing in the sequence order it likes. It will get more and more files until everything is in place. Automate the process with crontab scripts.

Like vodoo says. When you do not change to much in the source, the destination will get synced after a few (or more nights).

And names with a _ or - should not be problematic (why should they?).

thnx guys for replies.
@vodoo
Please tell me what do you mean by stop rsync in the morning. Only way i know to stop running process is to kill it. And then if i restart rsync next evening wouldn’t it start from beginning all over again, or will it keep index of what ever it did already and go through it very fast and then go on with copying?

@Henk
well, i don’t know, ‘ls’ has sertanly problem with dir named ‘-’. So i thought rsync would have to, at least i didn’t want to risk it.

  1. when you kill rsync it will most probably allready have synced (copied) a lot. When restarted it wiill first find out what to do, and it will not sync what is allready synced. That is the whole trick of rsync, only to copy what is not the same.

  2. Look:

henk@boven:~/test> l
totaal 8
-rw-r--r--  1 henk wij    0 dec 16 21:27 -
drwxr-xr-x  2 henk wij 4096 dec 16 21:27 ./
drwxr-xr-x 77 henk wij 4096 dec 16 21:26 ../
-rw-r--r--  1 henk wij    0 dec 16 21:27 a
henk@boven:~/test> ls -l
totaal 0
-rw-r--r-- 1 henk wij 0 dec 16 21:27 -
-rw-r--r-- 1 henk wij 0 dec 16 21:27 a
henk@boven:~/test> ls -l -
-rw-r--r-- 1 henk wij 0 dec 16 21:27 -
henk@boven:~/test> ls -
-
henk@boven:~/test>

I do not see a problem here with ls.
But look at the next example:

henk@boven:~/test> ls -l
totaal 0
-rw-r--r-- 1 henk wij 0 dec 16 21:27 -
-rw-r--r-- 1 henk wij 0 dec 16 21:31 a
-rw-r--r-- 1 henk wij 0 dec 16 21:32 -b
henk@boven:~/test> ls
-  a  -b
henk@boven:~/test> ls -b
-  a  -b
henk@boven:~/test> ls -- -b
-b
henk@boven:~/test>

Many tools have problems with a filename starting with a - because they interprete that as an option (and then either work with that option or give an error about a non existing option). In most tools you can therefore end the option fields with --, after that vereything will be seen as filenames.

From the info coreutils:

--' Delimit the option list. Later arguments, if any, are treated as operands even if they begin with -‘. For example, sort -- -r' reads from the file named -r’.

Please tell me what do you mean by stop rsync in the morning. Only way i know to stop running process is to kill it. And then if i restart rsync next evening wouldn’t it start from beginning all over again, or will it keep index of what ever it did already and go through it very fast and then go on with copying?

Kill rsync with SIGINT. It will flush the buffers and terminate (more or less) gracefully. Next time rsync continues where it stopped.

Correction: try

kill -s SIGTERM

first. It is a bit more friendly than SIGINT = ^C

And as SIGTERM is the default

kill <PID-of-rsync-process>

will be sufficient.

On Sun, 19 Dec 2010 00:06:01 +0530, hcvv <hcvv@no-mx.forums.opensuse.org>
wrote:

> And as SIGTERM is the default
> Code:
> --------------------
> kill <PID-of-rsync-process>
> --------------------
> will be sufficient.

how does pressing <ctrl+c> stop the process being executed in that
terminal? one of the less friendly SIG… options i’d think, but i don’t
know.


phani.

I think we have some misunderstanding here.
I assume that you have the rsync process running in the background (batch) (e.g. by starting it from cron).
In that case there is no terminal connected (that is the definition of “background”).

You stop it by finding out it’s PID, e.g.

ps -ef|grep rsync

and then do the kill as mentioned above. This can of course be programmed in a small script and also started by cron. Thus, while you are eating, sleeping, sitting in the pub, working at your system, the rsync will start at e.g. 23:00 and stop at e.g. 08:00 without you even thinking about it.
Computers are about automagic!

EDIT: when you have it running all night from a terminal, you stop it simply by typing ^C in the morning. But who wants to have a termiinal session (and may be more
horrible that one running in a desktop session) all night without ay attandence?

On Sun, 19 Dec 2010 01:06:02 +0530, hcvv <hcvv@no-mx.forums.opensuse.org>
wrote:

> I think we have some misunderstanding here.
> I assume that you have the -rsync- process running in the background
> (batch) (e.g. by starting it from -cron-).
> In that case there is no terminal connected (that is the definition of
> “background”).

sorry for confusing the issue at hand. i’m not the OP, but reading your
previous reply, was wondering how pressing <ctrl+c> terminates any running
process–in the foreground, of course. nothing to do with the topic at
hand, except that i usually run rsync in the foreground, not regularly via
cron.


phani.

Misunderstanding again :\

Your questions was simply: what sort of interrupt does ^C do? And I indeed understood that wrong. Sorry for that.

I think it generates a SIGTERM, but I am still searching for a confirmation.

^C (actually whichever character is designated the interrupt character by stty) sends SIGINT.

On Sun, 19 Dec 2010 04:36:01 +0530, ken yap
<ken_yap@no-mx.forums.opensuse.org> wrote:

>
> ^C (actually whichever character is designated the interrupt character
> by stty) sends SIGINT.
>

thank you. i wanted to know that for a while, but not enough to search it
out in docs, or open a new thread. this one reminded me…


phani.

Thanks also. Saves me the time to search for it.

@OP: Have a look at

man pkill

too. This may make your life easier.

e.g.
ps -ef | awk ‘/rsync/ && !/awk/ {print $2}’ | xargs -r kill