Page 3 of 7 FirstFirst 12345 ... LastLast
Results 21 to 30 of 61

Thread: Bit rot is real

  1. #21
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: Bit rot is real

    On 2014-01-30 20:55, Larry Finger wrote:

    > @ZStefan: I know your mind is probably made up and we should not bother
    > you with facts, but we may still influence some other readers of this
    > thread. Before you contribute to FUD, please read on how ECC works.


    Well, the btrfs devs certainly added this data recovery mechanism. They
    would not have bothered to if the hardware could always cope :-)

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 12.3 x86_64 "Dartmouth" at Telcontar)

  2. #22

    Default Re: Bit rot is real

    Quote Originally Posted by lwfinger View Post

    @ZStefan: I know your mind is probably made up and we should not bother you with
    facts, but we may still influence some other readers of this thread. Before you
    contribute to FUD, please read on how ECC works.
    My mind, even though so small (the smaller the brain, the happier the creature is), is not made up. I am trying to find a solution.

    I wouldn't even bother about high-level file verification tools shouldn't I have encountered the following cases, over about 8 years, that I remember. In all cases, I checked or tried to check the disk for errors, and couldn't find any with reasonable effort. Most likely, the data corruption ocurred during the years of storage.

    Once I had to discard a very large zip file. It became unzippable when it was needed months later. I usually check backups, through 'diff -r', and probably have checked the file right after creation. Data was lost forever, since the original was deleted to use the disk space for other purposes.

    I had to discard, because of data corruption, two large data files, containing acquired data. As the data was acquired, the files were reviewed and tested and nothing wrong was noticed. This was painful, I had to explain to colleagues. I have kept one of the corrupted files.

    I have a relatively short, old video file, in which the distortion described in the original article occurs. When it was played after acquisition years ago, I don't remember seeing the distortion.

    I had to edit a few audio files which contained the violent chirp described in the article. The durations of chirps were short and the edits were successful. I don't remember when the chirps appeared but I have the impression that they came in gradually.

    While there can be other explanations, bit rot is on my mind, considering the way the corruption appeared.


    Quote Originally Posted by lwfinger View Post
    any procedure that could be devised to "refresh" the files would greatly increase the probability of error.
    I agree with this. An simple automatic refresh may be disastrous. A more complex refresh, with comparison and late deletion of original, might be better though.

    But I am not thinking about refreshing the file. I worry about corruption gone undetected. Namely, I would like to use an atomatic tool to compare the checksums of the source file and of the backup file, to alarm me if they are different. Otherwise, if the source is corrupted, by the next backup, which will likely take place on the same HD, the data will become irreparably corrupted.

  3. #23

    Default Re: Bit rot is real

    Quote Originally Posted by robin_listas View Post
    On 2014-01-25 06:46, ZStefan wrote:
    > Now, I wonder whether there are tools to reveal and to fight the bit rot
    > without switching to btrfs? Maybe some utilities that would calculate
    > the md5sums of important files automatically, and compare them from time
    > to time?


    par2.
    I read about and tried to install.

    The goal is exactly what I want, but the usability is not there. The implementation has problems with large files. The last substantial update was in 2006.

    However, I downloaded and tried to compile. Of course didn't work. There were multiple warnings and an error stopped the compilation, even though ./configure was successful. I short, not usable.

    Someone is working on PAR3. Their forum shows some activity in 2011-2012.

    However, every effort in the Parchive project is abandoned, buggy, experimantal or incomplete.

    It looks like I have to create the checksums manually...

  4. #24

    Default Re: Bit rot is real

    ZStefan wrote:
    > But I am not thinking about refreshing the file. I worry about
    > corruption gone undetected. Namely, I would like to use an atomatic tool
    > to compare the checksums of the source file and of the backup file, to
    > alarm me if they are different.


    Rsync will happily do that for you.

  5. #25
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: Bit rot is real

    On 2014-01-31 13:21, Dave Howorth wrote:
    > ZStefan wrote:
    >> But I am not thinking about refreshing the file. I worry about
    >> corruption gone undetected. Namely, I would like to use an atomatic tool
    >> to compare the checksums of the source file and of the backup file, to
    >> alarm me if they are different.

    >
    > Rsync will happily do that for you.


    Mmm, not really.

    It compares the source with the backup, yes. If the source was damaged,
    as described, the backup will be faithful to the source; the checksum
    would match. But the backup would have the damaged file.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 12.3 x86_64 "Dartmouth" at Telcontar)

  6. #26

    Default Re: Bit rot is real

    Carlos E. R. wrote:
    > On 2014-01-31 13:21, Dave Howorth wrote:
    >> ZStefan wrote:
    >>> But I am not thinking about refreshing the file. I worry about
    >>> corruption gone undetected. Namely, I would like to use an atomatic tool
    >>> to compare the checksums of the source file and of the backup file, to
    >>> alarm me if they are different.

    >> Rsync will happily do that for you.

    >
    > Mmm, not really.
    >
    > It compares the source with the backup, yes. If the source was damaged,
    > as described, the backup will be faithful to the source; the checksum
    > would match. But the backup would have the damaged file.


    Duh, no. You check the checksums before you make the backup!

    Now how you know whether the checksums are different because of 'bit
    rot' or because you changed the file and really do want to make a new
    backup, that is an exercise for the reader.

  7. #27
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: Bit rot is real

    On 2014-01-31 14:41, Dave Howorth wrote:

    >>> Rsync will happily do that for you.

    >>
    >> Mmm, not really.
    >>
    >> It compares the source with the backup, yes. If the source was damaged,
    >> as described, the backup will be faithful to the source; the checksum
    >> would match. But the backup would have the damaged file.

    >
    > Duh, no. You check the checksums before you make the backup!


    Of course we do.

    That's not the issue. The problem described here is when the existing
    copy of the file gets damaged while stored.

    · You have the original, assumed correct.
    · You make a backup, with rsync with checksum verification.
    · The backup will be correct.
    · You store it.
    · The original is erased.
    · At some time, some small corruption happens on the backup (or rather,
    archive), and the hardware does not detect it. The chance is very small,
    but as the stored data is huge, the possibility becomes real. Small, but
    real.
    · The backup is now bad.
    · Any other backup will be correct compared to the previous backup, but
    not to the original.


    Notice that rsync with checksums does not store the checksums. It simply
    calculates the checksums of both original and backup; if they differ,
    the file is copied again. But those checksum are not stored, so it is
    not possible to verify the integrity of the backup if you do not have
    the original files.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 12.3 x86_64 "Dartmouth" at Telcontar)

  8. #28

    Default Re: Bit rot is real

    Carlos E. R. wrote:
    > On 2014-01-31 14:41, Dave Howorth wrote:
    >
    >>>> Rsync will happily do that for you.
    >>> Mmm, not really.
    >>>
    >>> It compares the source with the backup, yes. If the source was damaged,
    >>> as described, the backup will be faithful to the source; the checksum
    >>> would match. But the backup would have the damaged file.

    >> Duh, no. You check the checksums before you make the backup!

    >
    > Of course we do.
    >
    > That's not the issue. The problem described here is when the existing
    > copy of the file gets damaged while stored.


    No. The problem described was:

    "I would like to use an atomatic tool to compare the checksums of the
    source file and of the backup file, to alarm me if they are different."

  9. #29
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: Bit rot is real

    On 2014-01-31 07:16, ZStefan wrote:

    >> par2.


    > I read about and tried to install.
    >
    > The goal is exactly what I want, but the usability is not there. The
    > implementation has problems with large files. The last substantial
    > update was in 2006.


    I have used it with DVDs, holding files under 500MB each.

    > However, I downloaded and tried to compile. Of course didn't work. There
    > were multiple warnings and an error stopped the compilation, even though
    > ./configure was successful. I short, not usable.


    No need to build, it is available on the distro, from some home repos.
    Version 0.4

    I built it myself years ago, version 0.4. I would have to find out if it
    still builds.

    There is also "kde3-kpar2" on the kde3 repo.

    I see there is a library with this description:

    +++··········
    libpar2-0 - Library for Performing Common Tasks Related to PAR Archives

    LibPar2 allows for the generation, modification, verification, and
    repairation of PAR v1.0 and PAR v2.0 (PAR2) recovery sets. It contains
    the basic functions needed for working with these sets and is the basis
    for GUI applications such as KPar2 and GPar2.
    ··········++-

    So the next step is finding that Gpar2 (KPar2 is the kde3-kpar2 above).
    I don't see it on the distro repos. [...] Hum. It is on parchive from
    sourceforge.


    I also see "par2cmdline" on some home repos, not the same repos as for
    par2, even though they could be the same program.


    > Someone is working on PAR3. Their forum shows some activity in
    > 2011-2012.


    Oh.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 12.3 x86_64 "Dartmouth" at Telcontar)

  10. #30
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: Bit rot is real

    On 2014-01-31 15:03, Dave Howorth wrote:
    > Carlos E. R. wrote:



    > No. The problem described was:
    >
    > "I would like to use an atomatic tool to compare the checksums of the
    > source file and of the backup file, to alarm me if they are different."


    Well, the paragraph then is confusing, because what he really means is
    what I explained :-) Surely he will confirm.

    I'll try to rewrite that paragraph.

    Create checksums of files on source and/or backups, so that if later
    (after the backup is made and verified as correct) a file in any of the
    two changes, the change can be detected using the checksum on that same
    side. To alarm the user if a file changes, any time (months, years)
    after the backup was made and verified.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 12.3 x86_64 "Dartmouth" at Telcontar)

Page 3 of 7 FirstFirst 12345 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •