I want to run rsync on very large directory with long path depth. Some dir names in the path are just ‘-’ or ‘_’, or contain these characters in the beginning or in the end of the dir name.
To be honest i don’t know how rsync will react to this directories. So i thought to run rsync with exclude option.
Another issue i have is i can’t just unleash rsync on dir that 740G large due to limited resources, i can only run it at night.
This is why i would love to run rsync in stages, like tell rsync to take 100 dirs from depth 1 etc.
would anyone can help me to work this out?
thnx
Start rsync in the evening and stop it in the morning, as you like. And let it make it’s thing in the sequence order it likes. It will get more and more files until everything is in place. Automate the process with crontab scripts.
thnx guys for replies. @vodoo
Please tell me what do you mean by stop rsync in the morning. Only way i know to stop running process is to kill it. And then if i restart rsync next evening wouldn’t it start from beginning all over again, or will it keep index of what ever it did already and go through it very fast and then go on with copying?
when you kill rsync it will most probably allready have synced (copied) a lot. When restarted it wiill first find out what to do, and it will not sync what is allready synced. That is the whole trick of rsync, only to copy what is not the same.
Look:
henk@boven:~/test> l
totaal 8
-rw-r--r-- 1 henk wij 0 dec 16 21:27 -
drwxr-xr-x 2 henk wij 4096 dec 16 21:27 ./
drwxr-xr-x 77 henk wij 4096 dec 16 21:26 ../
-rw-r--r-- 1 henk wij 0 dec 16 21:27 a
henk@boven:~/test> ls -l
totaal 0
-rw-r--r-- 1 henk wij 0 dec 16 21:27 -
-rw-r--r-- 1 henk wij 0 dec 16 21:27 a
henk@boven:~/test> ls -l -
-rw-r--r-- 1 henk wij 0 dec 16 21:27 -
henk@boven:~/test> ls -
-
henk@boven:~/test>
I do not see a problem here with ls.
But look at the next example:
henk@boven:~/test> ls -l
totaal 0
-rw-r--r-- 1 henk wij 0 dec 16 21:27 -
-rw-r--r-- 1 henk wij 0 dec 16 21:31 a
-rw-r--r-- 1 henk wij 0 dec 16 21:32 -b
henk@boven:~/test> ls
- a -b
henk@boven:~/test> ls -b
- a -b
henk@boven:~/test> ls -- -b
-b
henk@boven:~/test>
Many tools have problems with a filename starting with a - because they interprete that as an option (and then either work with that option or give an error about a non existing option). In most tools you can therefore end the option fields with --, after that vereything will be seen as filenames.
From the info coreutils:
--' Delimit the option list. Later arguments, if any, are treated as operands even if they begin with -‘. For example, sort -- -r' reads from the file named -r’.
Please tell me what do you mean by stop rsync in the morning. Only way i know to stop running process is to kill it. And then if i restart rsync next evening wouldn’t it start from beginning all over again, or will it keep index of what ever it did already and go through it very fast and then go on with copying?
Kill rsync with SIGINT. It will flush the buffers and terminate (more or less) gracefully. Next time rsync continues where it stopped.
I think we have some misunderstanding here.
I assume that you have the rsync process running in the background (batch) (e.g. by starting it from cron).
In that case there is no terminal connected (that is the definition of “background”).
You stop it by finding out it’s PID, e.g.
ps -ef|grep rsync
and then do the kill as mentioned above. This can of course be programmed in a small script and also started by cron. Thus, while you are eating, sleeping, sitting in the pub, working at your system, the rsync will start at e.g. 23:00 and stop at e.g. 08:00 without you even thinking about it.
Computers are about automagic!
EDIT: when you have it running all night from a terminal, you stop it simply by typing ^C in the morning. But who wants to have a termiinal session (and may be more
horrible that one running in a desktop session) all night without ay attandence?
> I think we have some misunderstanding here.
> I assume that you have the -rsync- process running in the background
> (batch) (e.g. by starting it from -cron-).
> In that case there is no terminal connected (that is the definition of
> “background”).
sorry for confusing the issue at hand. i’m not the OP, but reading your
previous reply, was wondering how pressing <ctrl+c> terminates any running
process–in the foreground, of course. nothing to do with the topic at
hand, except that i usually run rsync in the foreground, not regularly via
cron.