slow rsync backup

hi,

I'm fairly new to rsync, but I'm up and going with the program. I have
mounted a windows catalogue tree on my linux box. I run the command for
backup like this:

#rsync -rtv --progress --size-only --delete /source/folder /target/folder

The first time I ran the command, all the files were properly copied to the
target folder. The next time I ran the command, it seems from the time
consumed, that the files were copied once more. It took several hours to do
the second backup. I thought rsync only copied the files which were changed.
Should I have an -a in the line, or is it just like this rsync works?

--
Dag R
----newbie---
Norway

In article <tNrRg.2804$0w1.2142@prv-forum2.provo.novell.com>, Dag
Ringdal wrote:
> I thought rsync only copied the files which were changed.
>
correct, based on date/time stamps. have you verified that the stamps
are being preserved correctly?

bd
NSC Volunteer SysOp

"Brad Doster" <bd@NSCSysOps.net> skrev i melding
news:VA.00003b19.190d4df8@nscsysops.net...
> In article <tNrRg.2804$0w1.2142@prv-forum2.provo.novell.com>, Dag
> Ringdal wrote:
>> I thought rsync only copied the files which were changed.
>>
> correct, based on date/time stamps. have you verified that the stamps
> are being preserved correctly?
>
> bd
> NSC Volunteer SysOp
>
How do I check it? I get a message in the end of the 11909 files, that some
files could not be marked with a timestamp. I mean I read a list at the end
of the session, that some files could not be accessed and not put a viable
timestamp on. I think it was about 20 files. But how can I verify that the
stamps are being preserved correctly?

Dag

Dag Ringdal wrote:
> hi,
>
> I'm fairly new to rsync, but I'm up and going with the program. I have
> mounted a windows catalogue tree on my linux box. I run the command for
> backup like this:
>
> #rsync -rtv --progress --size-only --delete /source/folder /target/folder
>
> The first time I ran the command, all the files were properly copied to the
> target folder. The next time I ran the command, it seems from the time
> consumed, that the files were copied once more. It took several hours to do
> the second backup. I thought rsync only copied the files which were changed.
> Should I have an -a in the line, or is it just like this rsync works?
>
Rsync will only update the differences when it works as expected.


I don't know exactly how to do this since I don't use windows myself,
but I know it's a big problem with timestamps windows<->linux/unix.

Windows use localtime for everything, while linux use utc based times,
where the correct localtime is displayed for you depending on your TZ.
And in some cases TZ can be different at src <-> dest
So if I from Sweden rsync a file with date 2006-09-25 22:58 to Japan ,
they will see the date as 2006-09-26 05:58 , this is a big difference
compared with windows.

I was sure the --size-only option should skip checking timestamps when
finding candidates for update, but that was maybe wrong.

Here is omething I found on the net:
"
Some "gotchas" in working with rsync under Windows

1. Include/Exclude file lists
On windows, your --exclude-from or --include-from file must be in
Unix format (newlines rather that cr/lf). Put a dos2ux line in your
rsync script to ensure this.


2. Timestamps
Windows doesn't keep file modification times to better than two
seconds. Use the --modify-window=2 option to get around this.


3. Time zones
rsync tries to convert all file timestamps into UTC at both ends.
If seemingly identical files with the same timestamp seem to be
transferring, try this at both ends:
export TZ=UTC
ls -l --full-time file_in_question
If the remote and local files have the same timestamp, and they're
still transferring, then you have a different problem. Otherwise, try
--modify-window=3601 which means ignore time differences of less than an
hour. This can be very helpful after a daylight savings time change.
"


/birger

In article <XZSRg.249$0h7.226@prv-forum2.provo.novell.com>, Dag Ringdal
wrote:
> But how can I verify that the
> stamps are being preserved correctly?
>
If the source and target systems are both *nix, 'ls -l' displays the
time stamps. If another OS, then use whatever mechanism it provides
for viewing that info.

bd
NSC Volunteer SysOp

"Birger" <Birger.forums.only@norsborg.net> skrev i melding
news:sIXRg.566$0h7.147@prv-forum2.provo.novell.com...
> Dag Ringdal wrote:
>> hi,
>>
>> I'm fairly new to rsync, but I'm up and going with the program. I have
>> mounted a windows catalogue tree on my linux box. I run the command for
>> backup like this:
>>
>> #rsync -rtv --progress --size-only --delete /source/folder /target/folder
>>
>> The first time I ran the command, all the files were properly copied to
>> the
>> target folder. The next time I ran the command, it seems from the time
>> consumed, that the files were copied once more. It took several hours to
>> do
>> the second backup. I thought rsync only copied the files which were
>> changed.
>> Should I have an -a in the line, or is it just like this rsync works?
>>
> Rsync will only update the differences when it works as expected.
>
>
> I don't know exactly how to do this since I don't use windows myself,
> but I know it's a big problem with timestamps windows<->linux/unix.
>
> Windows use localtime for everything, while linux use utc based times,
> where the correct localtime is displayed for you depending on your TZ.
> And in some cases TZ can be different at src <-> dest
> So if I from Sweden rsync a file with date 2006-09-25 22:58 to Japan ,
> they will see the date as 2006-09-26 05:58 , this is a big difference
> compared with windows.
>
> I was sure the --size-only option should skip checking timestamps when
> finding candidates for update, but that was maybe wrong.
>
> Here is omething I found on the net:
> "
> Some "gotchas" in working with rsync under Windows
>
> 1. Include/Exclude file lists
> On windows, your --exclude-from or --include-from file must be in
> Unix format (newlines rather that cr/lf). Put a dos2ux line in your
> rsync script to ensure this.
>
>
> 2. Timestamps
> Windows doesn't keep file modification times to better than two
> seconds. Use the --modify-window=2 option to get around this.
>
>
> 3. Time zones
> rsync tries to convert all file timestamps into UTC at both ends.
> If seemingly identical files with the same timestamp seem to be
> transferring, try this at both ends:
> export TZ=UTC
> ls -l --full-time file_in_question
> If the remote and local files have the same timestamp, and they're
> still transferring, then you have a different problem. Otherwise, try
> --modify-window=3601 which means ignore time differences of less than an
> hour. This can be very helpful after a daylight savings time change.
> "
>
I put an -a in the command and then rsync went into archive mode. When I ran
it, rsync only replaced the changed files and the backup procedure went
rather fast. I ran the command once more and the backup was completed in a
few seconds.

Thanks for your help.

Dag

Dag Ringdal wrote:

> I put an -a in the command and then rsync went into archive mode.
> When I ran it, rsync only replaced the changed files and the backup
> procedure went rather fast. I ran the command once more and the
> backup was completed in a few seconds.
>
> Thanks for your help.
>
> Dag
>
>

Ok, but you may get problems after Oct 29

~> zdump -v Europe/Oslo |grep 2006
Sun Mar 26 00:59:59 2006 UTC =
Sun Mar 26 01:59:59 2006 CET isdst=0 gmtoff=3600

Sun Mar 26 01:00:00 2006 UTC =
Sun Mar 26 03:00:00 2006 CEST isdst=1 gmtoff=7200

Sun Oct 29 00:59:59 2006 UTC =
Sun Oct 29 02:59:59 2006 CEST isdst=1 gmtoff=7200

Sun Oct 29 01:00:00 2006 UTC =
Sun Oct 29 02:00:00 2006 CET isdst=0 gmtoff=3600

I guess Bergen has the same time as Oslo :-)

Well, if it fail after Oct 29 , then add --modify-window=3601

/birger