rsync copying all files instead of just what's changed

English Applications
#1

During my backups I’m finding that rsync is copying all files, instead of just what’s changed.

I’m rsyncing between 2 USB external hard drives. One hard drive is FAT32 and one is NTFS. I’ve examined some of the files and believe that the difference is that there’s a 1-second modtime difference developing in some of the files somehow.

Here’s an example. These duplicity files were synced from /media/BACKUPHD (the NTFS drive) to /media/VIDEOHD (the FAT32 drive) only a few hours ago this morning. They have not been touched or changed since then, but that 1-second difference in their time stamps has appeared:

tim@localhost:~> stat /media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg                                                                                        
  File: `/media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg'                                                                                                 
  Size: 681614072       Blocks: 1331280    IO Block: 4096   regular file                                                                                                                      
Device: 831h/2097d      Inode: 2902        Links: 1                                                                                                                                           
Access: (0777/-rwxrwxrwx)  Uid: (  500/     tim)   Gid: (    0/    root)                                                                                                                      
Access: 2011-01-09 12:38:04.478730000 +0100                                                                                                                                                   
Modify: 2011-01-07 16:28:39.000000000 +0100                                                                                                                                                   
Change: 2011-01-08 21:27:10.370064000 +0100      
                                                                                                                                             
tim@localhost:~> stat /media/VIDEOHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg                                                                                         
  File: `/media/VIDEOHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg'                                                                                                  
  Size: 681614072       Blocks: 1331296    IO Block: 16384  regular file                                                                                                                      
Device: 821h/2081d      Inode: 7398        Links: 1                                                                                                                                           
Access: (0755/-rwxr-xr-x)  Uid: (  500/     tim)   Gid: (    0/    root)                                                                                                                      
Access: 2011-01-11 13:20:14.000000000 +0100                                                                                                                                                   
Modify: 2011-01-07 16:28:38.000000000 +0100
Change: 2011-01-11 07:31:46.000000000 +0100

And here is a file from the same directory that for whatever reason (and it seems to be random) there’s no 1-second timestamp difference:

tim@localhost:~> stat /media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol11.difftar.gpg
  File: `/media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol11.difftar.gpg'
  Size: 681614777       Blocks: 1331280    IO Block: 4096   regular file
Device: 831h/2097d      Inode: 2903        Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (  500/     tim)   Gid: (    0/    root)
Access: 2011-01-11 07:31:45.667345000 +0100
Modify: 2011-01-07 16:31:24.000000000 +0100
Change: 2011-01-08 21:27:42.865310000 +0100


tim@localhost:~> stat /media/VIDEOHD/backups/duplicity/duplicity-full.20110107T145955Z.vol11.difftar.gpg
  File: `/media/VIDEOHD/backups/duplicity/duplicity-full.20110107T145955Z.vol11.difftar.gpg'
  Size: 681614777       Blocks: 1331296    IO Block: 16384  regular file
Device: 821h/2081d      Inode: 7399        Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  500/     tim)   Gid: (    0/    root)
Access: 2011-01-11 00:00:00.000000000 +0100
Modify: 2011-01-07 16:31:24.000000000 +0100
Change: 2011-01-11 07:32:24.000000000 +0100

Is there some kind of timestamp rounding that FAT32 does that the NTFS driver doesn’t do (or vice-versa)?

#2

One of the things that come to my mind: is rsync ever designed with non native Unix/Linux filesystems in mind? Are the time stamps as stored into these non Linux fses same as Linux timestamps? Are the timestamps as stored on VFAT32 the same as those on NTFS? The using of non Linux filesystems on Linux has it’s natural restrictions and this could be one of them.

Unix time, or POSIX time, is a system for describing points in time, defined as the number of seconds elapsed since midnight Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds.
(Wikipedia)

#3

tk83 wrote:
> During my backups I’m finding that rsync is copying all files, instead
> of just what’s changed.

should we assume you used the correct rsync command line switches to
effect that?

since i have no idea how windows may or may not record file times
without introducing one second errors, i can only suggest you find a
WinGuru for that help–in the mean time you could post the commands
you are using…

and, maybe consider using a Linux file system on your USB hard drives
(or, should we assume you are also working with MS-Windows and letting
it write to file on USB?)

–
DenverD
CAVEAT: http://is.gd/bpoMD [posted via NNTP w/openSUSE 10.3]

What if there were no hypothetical questions?

#4

IIRC the timestamp resolution on FAT filesystems is only within 2 seconds. Ah yes here we are, it always helps to read the man page:

–modify-window

When comparing two timestamps, rsync treats the timestamps as
being equal if they differ by no more than the modify-window
value. This is normally 0 (for an exact match), but you may
find it useful to set this to a larger value in some situations.
In particular, when transferring to or from an MS Windows FAT
filesystem (which represents times with a 2-second resolution),
–modify-window=1 is useful (allowing times to differ by up to 1
second).

You could also use the -c option which compares content instead of using timestamps, at a loss of speed.

#5

Thanks ken_yap, in the meantime I found this also: Time Stamps Change When Copying From NTFS to FAT
Yet more evidence of how sh*t FAT32 is and why it shouldn’t be the default for USB mass storage devices :slight_smile:

Unfortunately I’ve already re-formatted the FAT32 drive as NTFS and begun the process of syncing the data back, but I’ll keep in mind this rsync workaround for future use with FAT32 devices.

#6

It is debatable if vendors should create FAT32 file systems on their USB devices. But it is you who bought the device and did not partition it to your needs and did not put a filesystem on it to your needs. When I by a mass storage (disk or stick or whatever) device I normaly put an ext2/3/4 filesystem on it except when I want to use it for exchange with MS systems.

#7

It goes all the way back to DOS and FAT16. I remember a long time ago studying the DOS syscall for current time and discovered that it returned the number of seconds since midnight divided by two. (Date is returned by a separate register IIRC.) The reason is that 86400 doesn’t fit into 16 bits, but 43200 does. A little of stinginess a long time ago that cost hassles through the years.

#8

The device was partitioned to my needs as far as I could tell. There’re are plenty of other people (mostly Windows users) who’ve been caught unawares by this unpublicised limitation - their forum posts are how I found the answer to my own question so fast.

#9

On 2011-01-11 15:36, ken yap wrote:
>
> It goes all the way back to DOS and FAT16. I remember a long time ago
> studying the DOS syscall for current time and discovered that it
> returned the number of seconds since midnight divided by two. (Date is
> returned by a different syscall.) The reason is that 86400 doesn’t fit
> into 16 bits, but 43200 does. A little of stinginess a long time ago
> that cost hassles through the years.

It made a lot of sense at the time, storage was scarce.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)