Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 23

Thread: Safest Long Term Storage

  1. #11
    Join Date
    Aug 2010
    Location
    Chicago suburbs
    Posts
    13,569
    Blog Entries
    3

    Default Re: Safest Long Term Storage

    Quote Originally Posted by PeterChz View Post
    Trying to avoid bit-rot or crashy smashy.
    Don't forget that other risk.

    You store your data on reliable media, guaranteed to have no bit rot.

    And then you discover that technology has changed and you can no longer find devices that can read that media.
    openSUSE Leap 15.2 Beta; KDE Plasma 5;

  2. #12
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    2,972

    Default Re: Safest Long Term Storage

    Quote Originally Posted by tsu2 View Post
    then choose something that is less volatile like SSD or optical media...
    Data archived on CD-ROM was supposed to have a “shelf-life” of more than 30 years … Experience has shown that, it depends on the manufactured quality and, the burning …

    Yes, there are special “glass” drives for archiving data but, AFAIK, not a consumer product …

  3. #13
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    2,972

    Talking Re: Safest Long Term Storage

    @PeterChz:

    The NAS I prefer to use, because it's “Linux friendly”, has this statement regarding their filesystem choice: <https://www.qnap.com/solution/qnap-ext4/en-us/>.
    Yes, using an Internet search tool to try and find out which is “better” – ext4, XFS or Btrfs – currently doesn't seem to find anything which one could take as being “current” – there's information which is at least 6 years old which, IMHO, is currently not the actual “state of affairs” …

    XFS:
    • It has features such as “striped RAID arrays” and native backup/restore utilities which, IMHO, are “very nice to have” for archives …

  4. #14

    Default Re: Safest Long Term Storage

    Thanks everyone for all the responses.
    I just realized my typo, ZFS not XFS. sorry.
    XFS does seem like the solid alternative. But my fear was, even though OpenSuse or other OS CAN implement ZFS manually. I don't know if they should. Was curious if anyone rolls their own ZFS storage setup.

    From my research and your guys help, Tape, if I can find a machine cheap enough would be the best archival-able copy. Though I had it recommended if I go with tape, it's best to use the same machine. Some have had issues with same model but different machines not reading tapes 100%. Again might be one person's bad experience.

    Having one system semi live running as a NAS is a good standby, we have a small non RAID version of this going currently, I'll have to check that link @dcurtisfra for a more robust setup. Curious if there are setups that put the RAID card or server to sleep unless woken up to use. Like Wake On Lan? That way periodic tests of systems can be done or even automated. Reduce use.

    Interesting to hear that bit rot is FUD. I've felt the pain on old floppies, CD's and some drives have died just due to old age and use causing data loss. But low use, properly stored drives, I can see have a chance of survival.

    I find this such an interesting topic too, there are so many levels of choice that need to harmonize.

    Software:
    | NAS software: SAMBA, NFS, etc.
    | Operating System
    | Logical Formatting of disks: XFS, ZFS, ReiserFS
    -----------------------------------------------------
    Hardware:
    | Hardware Raid of Disk
    | Size and Type of Hard Disk

    Advice on any level is appreciated, or a level I am not considering.

    Having a hard disk copy stored offsite somewhere is a good idea. More ambitious ideas like, building an openstack cloud between servers built across the country at friends or family houses or CoLos have crossed my mind. But that maybe enterprise work day seeping into my skull. Overkill for keeping family photo's safe for 20-50 years. Cheaper to just create a handful of TB drive backups.

    ... how about writing data to 35mm film... and other mad scientist ideas. . . .

  5. #15

    Default Re: Safest Long Term Storage

    Interesting 35mm film data storage exists.
    https://www.cpclondon.com/
    Using QR codes to save data on each frame. yikes 300~400bytes per frame. Not data dense enough for most things, but talk about long term storage.

  6. #16
    Join Date
    Jul 2018
    Location
    Loma Linda, Mo
    Posts
    214

    Default Re: Safest Long Term Storage

    As someone that backed up for a major fortune company - they started with high density tape cartridges - after 6 months when we needed to extract some files - both the primary and backup tapes had read errors that we had to get around.

    I recommended a different backup scheme - that they implemented - it runs on SLES - the file systems are LVM - each file system has 3 mirrors - each backup device in in a different city to prevent a disaster from making the data unavailable.

    The primary set of backups are on the live servers - each live server has an addition set of backups that are offline except for the nightly rsync to make them a good backup and them they are put offline again.

    The storage is large EMC, HP, Fujitsu storage arrays that also mirror to other sites in other states. Different brands to protect from a possible design problem.

    Disk failures still happen but no data is lost. Just a drive rebuild via LVM.

    The backups have a weekly low priority dd for each drive to look for failures - It is sad that almost every week we see a drive go bad. Makes no difference if Seagate, WD, Toshiba, IBM - they all will expires sometime - I have an old system with an MFM drive that is 30 years old and still works (64MB - takes 30 minutes to read it all - on 386sx Mandrake Linux from 1994)

    My hone backups are the same except they are on USB3 4 TB drives on 3 separate systems that rsync after I confirm that nothing has be compromised. I use Opensuse 15.0 and LVM to mirror and ext4 file systems under LVM.

    Personally, I dislike btrfs as too complex with too many jobs running and no real benefits - I backup and con restore my computers to any one of the last 30 weeks of backups. I test on another system every so often to make sure that the backups are complete and the stuff to reset grub2 and /boot worked fine on new iron (unformatted disk booted off live USB key).

    I have been using Unix/Xenix/Linux since 1973 - I almost know what I am doing. The sad thing is every time I almost know everything to do they change it again. But I don't miss the 4 hour boot times when the power went off. (fsck everything).
    Opensuse 15.1 with VirtualBox VM's (Windows 98, XP, 7, 8.1, 10 & OpenSUSE 15.0)

    Unix since 1974 (pdp-11 in "B" , Interdata 7/32 in "C") (AT&T, Tandy, Convergent, IBM, NCR, HP flavors)
    Linux since 1995 (mandrake, redhat, fedora, centos, now OpenSUSE)

  7. #17
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    26,187

    Default Re: Safest Long Term Storage

    Quote Originally Posted by larryr View Post
    But I don't miss the 4 hour boot times when the power went off. (fsck everything).
    At least you got time then to go for a good dinner.
    Henk van Velden

  8. #18
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    12,131
    Blog Entries
    2

    Default Re: Safest Long Term Storage

    Rotating hard drives are typically an important part of any backup system today, and is supposed to be even reliable long term storage if done the right way. Perhaps more importantly for many, reliable storage can be the prime way to avoid needing to do a recovery from backup at all.

    Backblaze has been releasing statistical results of the drives it goes through annually, and as a major backup company goes through enormous numbers of drives annually.

    The following article includes Backblaze statistics for the year 2018

    https://www.extremetech.com/extreme/...le-hard-drives

    It should be no surprise that the Backblaze reports have a major impact on the street price of drives, in some years I've seen HGST drives (best rated) twice the price of Seagate (for a few years had really poor reliability statistics). It's probably also a primary reason why laggards in some years like Seagate have greatly improved their reliability since.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  9. #19

    Default Re: Safest Long Term Storage

    Rotating drives is a really good suggestion. I wasn't familiar with that technique but failing hard drives as many have mentioned is the worst fear. With an enterprise contract it's not such a big deal, they must make tons on these flaky drives at the very least make us feel dependent on their support contracts. But with some raid inplace, like Raid 1 on primary device has been a life saver to me. I was able to recover twice in 5 years from disastrous failures.

    larryr, that is a fantastic setup. Presuming these are user files, how was access control enforced? Local System Users, Samba or NFS to supply the rest of the office? Curious if there is a better way to provide network attached storage. At home a SAMBA share running one shared user is what connects our phones, TV, Computers to view photos, documents etc. I know SAMBA is more powerful that what I use it for, and I should probably dive in a little deeper. I never jumped into offerings like plex, owncloud, etc. They seem to change and update too quickly for my taste. Just looking for a reliable standby that can integrate with just about any other O.S. to provide greatest functionality.

    Rotating the three drives for your home setup is great too. I appreciate the ext4 with LVM setup. Pretty minimal but robust. Never had trouble with that myself.

    This discussion really makes me realize all the issues with my current setup. While I have a little bit extra in place, not nearly enough if I value my data. I have to look into a safety lock box for some offsite storage, where I can rotate a drive every few months. I don't know that Ill be lucky enough to setup something at a distant relatives or the like. Someone needs to start a low cost co-lo in their garage, or a few people and we can store running computers to archive off site storage.

  10. #20
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    12,131
    Blog Entries
    2

    Default Re: Safest Long Term Storage

    Quote Originally Posted by dcurtisfra View Post
    Data archived on CD-ROM was supposed to have a “shelf-life” of more than 30 years … Experience has shown that, it depends on the manufactured quality and, the burning …

    Yes, there are special “glass” drives for archiving data but, AFAIK, not a consumer product …
    Never use CDs for archiving, I learned the hard way.
    Look it up, the ink that is hit by the laser in a Writer is completely exposed to air, humidity, physical scratching, anything you can think of.
    A DVD though is different, the ink layer is sandwiched between plastic layers on both sides so is less exposed.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

Page 2 of 3 FirstFirst 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •