Results 1 to 9 of 9

Thread: rsync copying all files instead of just what's changed

  1. #1

    Default rsync copying all files instead of just what's changed

    During my backups I'm finding that rsync is copying all files, instead of just what's changed.

    I'm rsyncing between 2 USB external hard drives. One hard drive is FAT32 and one is NTFS. I've examined some of the files and believe that the difference is that there's a 1-second modtime difference developing in some of the files somehow.

    Here's an example. These duplicity files were synced from /media/BACKUPHD (the NTFS drive) to /media/VIDEOHD (the FAT32 drive) only a few hours ago this morning. They have not been touched or changed since then, but that 1-second difference in their time stamps has appeared:
    Code:
    tim@localhost:~> stat /media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg                                                                                        
      File: `/media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg'                                                                                                 
      Size: 681614072       Blocks: 1331280    IO Block: 4096   regular file                                                                                                                      
    Device: 831h/2097d      Inode: 2902        Links: 1                                                                                                                                           
    Access: (0777/-rwxrwxrwx)  Uid: (  500/     tim)   Gid: (    0/    root)                                                                                                                      
    Access: 2011-01-09 12:38:04.478730000 +0100                                                                                                                                                   
    Modify: 2011-01-07 16:28:39.000000000 +0100                                                                                                                                                   
    Change: 2011-01-08 21:27:10.370064000 +0100      
                                                                                                                                                 
    tim@localhost:~> stat /media/VIDEOHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg                                                                                         
      File: `/media/VIDEOHD/backups/duplicity/duplicity-full.20110107T145955Z.vol10.difftar.gpg'                                                                                                  
      Size: 681614072       Blocks: 1331296    IO Block: 16384  regular file                                                                                                                      
    Device: 821h/2081d      Inode: 7398        Links: 1                                                                                                                                           
    Access: (0755/-rwxr-xr-x)  Uid: (  500/     tim)   Gid: (    0/    root)                                                                                                                      
    Access: 2011-01-11 13:20:14.000000000 +0100                                                                                                                                                   
    Modify: 2011-01-07 16:28:38.000000000 +0100
    Change: 2011-01-11 07:31:46.000000000 +0100
    And here is a file from the same directory that for whatever reason (and it seems to be random) there's no 1-second timestamp difference:
    Code:
    tim@localhost:~> stat /media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol11.difftar.gpg
      File: `/media/BACKUPHD/backups/duplicity/duplicity-full.20110107T145955Z.vol11.difftar.gpg'
      Size: 681614777       Blocks: 1331280    IO Block: 4096   regular file
    Device: 831h/2097d      Inode: 2903        Links: 1
    Access: (0777/-rwxrwxrwx)  Uid: (  500/     tim)   Gid: (    0/    root)
    Access: 2011-01-11 07:31:45.667345000 +0100
    Modify: 2011-01-07 16:31:24.000000000 +0100
    Change: 2011-01-08 21:27:42.865310000 +0100
    
    
    tim@localhost:~> stat /media/VIDEOHD/backups/duplicity/duplicity-full.20110107T145955Z.vol11.difftar.gpg
      File: `/media/VIDEOHD/backups/duplicity/duplicity-full.20110107T145955Z.vol11.difftar.gpg'
      Size: 681614777       Blocks: 1331296    IO Block: 16384  regular file
    Device: 821h/2081d      Inode: 7399        Links: 1
    Access: (0755/-rwxr-xr-x)  Uid: (  500/     tim)   Gid: (    0/    root)
    Access: 2011-01-11 00:00:00.000000000 +0100
    Modify: 2011-01-07 16:31:24.000000000 +0100
    Change: 2011-01-11 07:32:24.000000000 +0100
    Is there some kind of timestamp rounding that FAT32 does that the NTFS driver doesn't do (or vice-versa)?

  2. #2
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,163

    Default Re: rsync copying all files instead of just what's changed

    One of the things that come to my mind: is rsync ever designed with non native Unix/Linux filesystems in mind? Are the time stamps as stored into these non Linux fses same as Linux timestamps? Are the timestamps as stored on VFAT32 the same as those on NTFS? The using of non Linux filesystems on Linux has it's natural restrictions and this could be one of them.

    Unix time, or POSIX time, is a system for describing points in time, defined as the number of seconds elapsed since midnight Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds.
    (Wikipedia)
    Henk van Velden

  3. #3
    Join Date
    Jun 2008
    Location
    Earth - Denmark
    Posts
    10,730

    Default Re: rsync copying all files instead of just what's changed

    tk83 wrote:
    > During my backups I'm finding that rsync is copying all files, instead
    > of just what's changed.


    should we assume you used the correct rsync command line switches to
    effect that?

    since i have no idea how windows may or may not record file times
    without introducing one second errors, i can only suggest you find a
    WinGuru for that help--in the mean time you could post the commands
    you are using..

    and, maybe consider using a Linux file system on your USB hard drives
    (or, should we assume you are also working with MS-Windows and letting
    it write to file on USB?)

    --
    DenverD
    CAVEAT: http://is.gd/bpoMD [posted via NNTP w/openSUSE 10.3]

    What if there were no hypothetical questions?

  4. #4
    Join Date
    Jun 2008
    Location
    UTC+10
    Posts
    9,686
    Blog Entries
    4

    Default Re: rsync copying all files instead of just what's changed

    IIRC the timestamp resolution on FAT filesystems is only within 2 seconds. Ah yes here we are, it always helps to read the man page:

    --modify-window

    When comparing two timestamps, rsync treats the timestamps as
    being equal if they differ by no more than the modify-window
    value. This is normally 0 (for an exact match), but you may
    find it useful to set this to a larger value in some situations.
    In particular, when transferring to or from an MS Windows FAT
    filesystem (which represents times with a 2-second resolution),
    --modify-window=1 is useful (allowing times to differ by up to 1
    second).
    You could also use the -c option which compares content instead of using timestamps, at a loss of speed.

  5. #5

    Default Re: rsync copying all files instead of just what's changed

    Thanks ken_yap, in the meantime I found this also: Time Stamps Change When Copying From NTFS to FAT
    Yet more evidence of how ***** FAT32 is and why it shouldn't be the default for USB mass storage devices

    Unfortunately I've already re-formatted the FAT32 drive as NTFS and begun the process of syncing the data back, but I'll keep in mind this rsync workaround for future use with FAT32 devices.

  6. #6
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,163

    Default Re: rsync copying all files instead of just what's changed

    It is debatable if vendors should create FAT32 file systems on their USB devices. But it is you who bought the device and did not partition it to your needs and did not put a filesystem on it to your needs. When I by a mass storage (disk or stick or whatever) device I normaly put an ext2/3/4 filesystem on it except when I want to use it for exchange with MS systems.
    Henk van Velden

  7. #7
    Join Date
    Jun 2008
    Location
    UTC+10
    Posts
    9,686
    Blog Entries
    4

    Default Re: rsync copying all files instead of just what's changed

    It goes all the way back to DOS and FAT16. I remember a long time ago studying the DOS syscall for current time and discovered that it returned the number of seconds since midnight divided by two. (Date is returned by a separate register IIRC.) The reason is that 86400 doesn't fit into 16 bits, but 43200 does. A little of stinginess a long time ago that cost hassles through the years.

  8. #8

    Default Re: rsync copying all files instead of just what's changed

    Quote Originally Posted by hcvv View Post
    It is debatable if vendors should create FAT32 file systems on their USB devices. But it is you who bought the device and did not partition it to your needs and did not put a filesystem on it to your needs.
    The device was partitioned to my needs as far as I could tell. There're are plenty of other people (mostly Windows users) who've been caught unawares by this unpublicised limitation - their forum posts are how I found the answer to my own question so fast.

  9. #9
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: rsync copying all files instead of just what's changed

    On 2011-01-11 15:36, ken yap wrote:
    >
    > It goes all the way back to DOS and FAT16. I remember a long time ago
    > studying the DOS syscall for current time and discovered that it
    > returned the number of seconds since midnight divided by two. (Date is
    > returned by a different syscall.) The reason is that 86400 doesn't fit
    > into 16 bits, but 43200 does. A little of stinginess a long time ago
    > that cost hassles through the years.


    It made a lot of sense at the time, storage was scarce.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 11.2 x86_64 "Emerald" at Telcontar)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •