Page 1 of 3 123 LastLast
Results 1 to 10 of 21

Thread: large file cp or diff error

  1. #1

    Default large file cp or diff error

    openSUSE 13.1 fresh install...

    Copied some vmware vmdk files with vmware not installed, i.e., an archive copy to disk) from eSATA to SATA diskto different places (install (1st) - files copied after 1.5TB other data, then errors; then re-install clean, copy with no other data, to a different place on disk, then check again);

    Copying mix of small and 17G or greater files, then have diff or cmp errors on the large files (not sure how many, as diff/cmp seem to halt after first difference)

    Only having the problem on these large files, many smaller files seem okay (e.g., up to 4 or 5 GB). Ran it twice as above. Have not seen this with previous versions (11.3, 11.4, 12.1, 12.2, 12.3) (I do this each time for a new disk)

    source disk is a 3000GB, destination is a 2000GB, not a disk overflow; running luks on both disks, single root (encrypted) and /boot, ext4.

    Ran badblocks (no errors in r/w) on the destination disk; disk has been used before in this manner with no problems.

    Has anyone else seen this type of problem?

  2. #2
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    13,295
    Blog Entries
    2

    Default Re: large file cp or diff error

    Quote Originally Posted by ThomasZMitchell View Post
    openSUSE 13.1 fresh install...

    Copied some vmware vmdk files with vmware not installed, i.e., an archive copy to disk) from eSATA to SATA diskto different places (install (1st) - files copied after 1.5TB other data, then errors; then re-install clean, copy with no other data, to a different place on disk, then check again);

    Copying mix of small and 17G or greater files, then have diff or cmp errors on the large files (not sure how many, as diff/cmp seem to halt after first difference)

    Only having the problem on these large files, many smaller files seem okay (e.g., up to 4 or 5 GB). Ran it twice as above. Have not seen this with previous versions (11.3, 11.4, 12.1, 12.2, 12.3) (I do this each time for a new disk)

    source disk is a 3000GB, destination is a 2000GB, not a disk overflow; running luks on both disks, single root (encrypted) and /boot, ext4.

    Ran badblocks (no errors in r/w) on the destination disk; disk has been used before in this manner with no problems.

    Has anyone else seen this type of problem?
    When I installed my first 13.1,
    I immediately noticed that by default the /tmp (and other tmp) directories are deployed as tmpfs, which means that they run in memory and don't write to the hard drive.

    When I was experimenting with this in 12.3, I quickly found that although there are some performance advantages to deploying /tmp to tmpfs, it also meant that file transfers and downloads which place a temporary copy in /tmp would be a problem in "memory challenged" machines when transferring very large files. <Some> apps like bittorent don't suffer from this problem because they transfer only tiny pieces at a time and write <everything> generally to the same directory as the app and don't write to /tmp but in general everything else will write to /tmp.

    So, if I were to guess you have no more than 4GB RAM in your machine.

    The solution is to re-point /tmp to a location on disk. In my case, this was easy because I typically don't allocate the whole disk for these (and other) little emergencies, I simply created a partition in my empty space and pointed /tmp there. But, you can point /tmp anywhere you have sufficient space.

    I expect that for a distro like openSUSE, it's probably safer to deploy /tmp to disk by default.
    And, if the machine has <8GB RAM, I probably recommend choosing "custom layout" and re-configuring this option during initial install.

    TSU

  3. #3

    Default Re: large file cp or diff error

    Quote Originally Posted by tsu2 View Post
    When I installed my first 13.1,
    I immediately noticed that by default the /tmp (and other tmp) directories are deployed as tmpfs, which means that they run in memory and don't write to the hard drive.

    When I was experimenting with this in 12.3, I quickly found that although there are some performance advantages to deploying /tmp to tmpfs, it also meant that file transfers and downloads which place a temporary copy in /tmp would be a problem in "memory challenged" machines when transferring very large files. <Some> apps like bittorent don't suffer from this problem because they transfer only tiny pieces at a time and write <everything> generally to the same directory as the app and don't write to /tmp but in general everything else will write to /tmp.

    So, if I were to guess you have no more than 4GB RAM in your machine.

    The solution is to re-point /tmp to a location on disk. In my case, this was easy because I typically don't allocate the whole disk for these (and other) little emergencies, I simply created a partition in my empty space and pointed /tmp there. But, you can point /tmp anywhere you have sufficient space.

    I expect that for a distro like openSUSE, it's probably safer to deploy /tmp to disk by default.
    And, if the machine has <8GB RAM, I probably recommend choosing "custom layout" and re-configuring this option during initial install.

    TSU
    3 GB RAM; I don't see /tmp deployed anywhere (tmpfs entries in 'df -a' for /var/run, /var/lock, etc. but no mount point for /tmp; On a 12.3 machine, same thing. One thing, I use scripts to install and my disk setup is 16G for swap, whether the machine has 3GB (like this one) or 4, 6 or 8GB. Machine in question is hp-dc5700MT, E6700 Core2 Duo and 3GB RAM. I don't make seperate partitions for /home, /var, or /tmp anymore, just /boot (plain) and / and swap (encrypted) because it's just easier with LUKS (only one password at boot time). Just wondering what is different about 13.1; I have not had this problem in previous versions of openSUSE.

    Thanks for the reply, btw.

  4. #4
    Join Date
    Nov 2009
    Location
    West Virginia Sector 13
    Posts
    16,287

    Default Re: large file cp or diff error

    Type mount

    in 13.1 some mounts are done by systemd using tmpfs and are no longer in fstab

  5. #5

    Default Re: large file cp or diff error

    Quote Originally Posted by gogalthorp View Post
    Type mount

    in 13.1 some mounts are done by systemd using tmpfs and are no longer in fstab
    Not there, either -- thanks

  6. #6

    Default Re: large file cp or diff error

    Quote Originally Posted by ThomasZMitchell View Post
    Not there, either -- thanks
    And even noticed that /tmp is persistent when another disk was pulled from a machine and mounted as a backup device.

    Well, for now, problem solved by using dd to copy the huge files: "dd if=bigfile1 of=bigfile2 bs=16M"

    It's quick enough and "diff -q bigfile1 bigfile2" produces no error messages. Could experiment with different block sizes (bs=) but that's for another day.

    Thanks to Tsu and gogalthorp (did not know about the tmpfs mounts from systemd).

  7. #7
    Join Date
    Nov 2009
    Location
    West Virginia Sector 13
    Posts
    16,287

    Default Re: large file cp or diff error

    It really depends on how you got to 13.1. If you did an online upgrade and the old fstbl is kept then the mounts still maybe in there. But as I understand those should be removed and let systemd handle it.



    .

  8. #8

    Default Re: large file cp or diff error

    No. /tmp is _not_ setup as tmpfs in openSUSE, not even on 13.1.
    By default it is just a directory on the / partition, that's why it is not in the fstab either.

    /run, /var/run, /var/lock, and /dev are tmpfs's however, f.e.

  9. #9
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: large file cp or diff error

    On 2014-01-15 19:26, tsu2 wrote:

    > When I installed my first 13.1,
    > I immediately noticed that by default the /tmp (and other tmp)
    > directories are deployed as tmpfs, which means that they run in memory
    > and don't write to the hard drive.


    Not on any of my installs.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 12.3 x86_64 "Dartmouth" at Telcontar)

  10. #10
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: large file cp or diff error

    On 2014-01-16 06:16, ThomasZMitchell wrote:

    > Well, for now, problem solved by using dd to copy the huge files: "dd
    > if=bigfile1 of=bigfile2 bs=16M"


    I now use dd_rescue instead for disk images.

    Code:
    dd_rhelp source destination
    It can restart a copy half made, and is way much reliable in case of
    read errors.

    13.1 ships with a different version of dd_rescue, the gnu version, with
    a slightly different syntax. I can not check it now.


    Alternatively, to copy directories for backup, I use rsync with checksums:

    Code:
    rsync --archive --acls --xattrs --hard-links --stats  \
    --human-readable  --checksum  \
    /source_directory/  /destination_directory
    If there are errors, it should copy again the bad parts. You can even
    run it twice for rechecks.


    > It's quick enough and "diff -q bigfile1 bigfile2" produces no error
    > messages. Could experiment with different block sizes (bs=) but that's
    > for another day.


    If 'cp' copies with errors, I would bugzilla about it. Seems serious.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 12.3 x86_64 "Dartmouth" at Telcontar)

Page 1 of 3 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •