Page 1 of 3 123 LastLast
Results 1 to 10 of 28

Thread: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

  1. #1

    Default coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    With the current 32-bit Tumbleweed (i586) copying or moving a large file (7GiB) with "cp" or "mv"
    leads to the error message "value too large for defined data type".
    Despite of this the file is correctly copied or moved.
    The version of the coreutils is "coreutils-8.25-3.3.i586".
    The filesystem type is ext4 with the features "has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize".

    Is this just a cosmetic error (the files are ok after the operation) or is it a time bomb?
    Did I configure something wrongly?

  2. #2
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,299
    Blog Entries
    2

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    Yeah,
    Although I haven't touched a large 32-bit system in a long time,
    I wouldn't be surprised if such a system had file system limitations of 4GB and in some cases 2GB.

    It would be a fundamental limitation of 32-bit.

    You can use the command "split" to split into smaller files and later "cat" to re-integrate.
    The following also describes using 7zip which I haven't done and looks interesting
    http://askubuntu.com/questions/54579...-smaller-parts



    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  3. #3
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,299
    Blog Entries
    2

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    FWIW
    Here is an example using split and cat...

    7GB file named "BigFile"
    You know that you want to avoid a 4GB file size limit.
    As a bonus, you know that your file transfer method isn't very good at transferring files larger than 10MB, so you want to move individual pieces smaller than that. Transferring files locally from one part of your file system to another might not have a problem with large files, but some protocols like HTTP/HTTPS/XML have this performance limitation.

    Split your large file into smaller files
    You should know that by default your file chunks will be named "xa*" by default, make sure there are no existing files in the same directory that start with "xa"
    Now, run the following command which specifies each file chunk is 10MB in size
    Code:
    split --bytes=10MB BigFile
    You can inspect the result
    Code:
    ls xa*
    Move your pieces to their new location
    Code:
    mv xa* target_destination
    Run the following to re-integrate the pieces
    Code:
    cat xa* > BigFile
    If you'd rather have only a few (say, 3) big chunks instead of a large number of little chunks, you can run the following instead
    Code:
    split --number=3 BigFile
    HTH,
    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  4. #4

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    Quote Originally Posted by tsu2 View Post
    Yeah,
    Although I haven't touched a large 32-bit system in a long time,
    I wouldn't be surprised if such a system had file system limitations of 4GB and in some cases 2GB.

    It would be a fundamental limitation of 32-bit.
    No, that only depends on the filesystem, and is independent of whether you have a 32bit or 64bit systems.
    E.g. FAT16 has indeed a 2 GiB file size limit, and FAT32 has 4 GiB.
    The standard btrfs has a maximum file size of 16 EiB (also on 32bit systems) though, ext4's limit ranges from 16 GiB to 16 TiB.

    See https://en.wikipedia.org/wiki/Compar...systems#Limits.

    And the Linux kernel's file access functions use 64bit integers also on 32bit systems.

    So it may be a "bug" in the coreutils, if an application uses int variables for some size value that would be 2 or 4 GiB max on 32bit.
    Sounds rather "cosmetic" though, if the files are copied/moved properly.
    And to repeat, ext4 does support 7 GiB big files in any case.
    Last edited by wolfi323; 21-Oct-2016 at 11:35.

  5. #5
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,299
    Blog Entries
    2

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    Quote Originally Posted by wolfi323 View Post
    No, that only depends on the filesystem, and is independent of whether you have a 32bit or 64bit systems.
    E.g. FAT16 has indeed a 2 GiB file size limit, and FAT32 has 4 GiB.
    The standard btrfs has a maximum file size of 16 EiB (also on 32bit systems) though, ext4's limit ranges from 16 GiB to 16 TiB.

    See https://en.wikipedia.org/wiki/Compar...systems#Limits.

    And the Linux kernel's file access functions use 64bit integers also on 32bit systems.

    So it may be a "bug" in the coreutils, if an application uses int variables for some size value that would be 2 or 4 GiB max on 32bit.
    Sounds rather "cosmetic" though, if the files are copied/moved properly.
    And to repeat, ext4 does support 7 GiB big files in any case.
    I started to write, then deleted a paragraph I was about to write that a file operation would be independent of whatever a file system supports, because although a 32-bit file system can have "extensions" installed to support addressing beyond 32-bit, a file operation like a "move" likely isn't a streaming operation where the bits would be transferred in smaller chunks (eg standard 256k disk blocks) but would have to be defined as a single "big" file... And, a 32-bit array has a 4GB hard limit. That's why I stated the 4GB limit, but for other reasons there may be a smaller (eg 2GB) limit.

    That said,
    My reasoning on why 32-bit core-utils may have a 4GB limit is only a reasonable analysis on general principles on my part, and in no way was based on a special inspection and knowledge exactly how 32-bit core-utils works.

    Hence, as observed there is no problem with a 7GB file residing on the file system but a problem if it's moved, more likely off the existing partition (a move within the same partition may bypass certain restrictions and simply re-makes the file system pointer).

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  6. #6

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    Quote Originally Posted by tsu2 View Post
    a file operation like a "move" likely isn't a streaming operation where the bits would be transferred in smaller chunks (eg standard 256k disk blocks) but would have to be defined as a single "big" file...
    Seriously?
    Loading a 7GiB file completely into memory wouldn't be possible on 32bit, yes, but it would likely also cause big problems on 64bit systems (especially with less RAM...).

    Of course the bits will be transferred in smaller chunks.

    A "file copy" basically means reading data from the input file and writing to the output file until there's no more data left.

    My reasoning on why 32-bit core-utils may have a 4GB limit is only a reasonable analysis on general principles on my part, and in no way was based on a special inspection and knowledge exactly how 32-bit core-utils works.
    As I said, the Linux kernel uses 64bit integers for file sizes and such stuff (even on 32bit), so there should be no limit (at least no 2/4 GiB limit) unless the file system has one.

    And I very much doubt that cp or mv would have such a limit, they do not even have to care about the file size if they only read chunks of data (and I'm sure they do).
    I really think this is just a cosmetic problem.
    I don't find this exact error message anywhere in the coreutils source code though, so hard to say what it really means.

  7. #7
    Join Date
    Oct 2011
    Location
    Germany (Ore Mountains)
    Posts
    427

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    I just tested this after today's updates with coreutils-8.25-3.4.i586, kernel 4.8.3 and the file systems ext4 +XFS. I don't see such messages.

    Hendrik

  8. #8

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    I have been testing again and found out:
    The error message is displayed only,
    if "-a" is used with "cp"
    or
    if "mv" is used across file system boundaries.

  9. #9
    Join Date
    Oct 2011
    Location
    Germany (Ore Mountains)
    Posts
    427

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    With "cp -a" I get this message too and some attributes are missing:

    Code:
     ls -l test*
    -rw-r--r-- 1 hendrik users 7516192768 23. Okt 20:32 test.img
    -rw------- 1 hendrik users 7516192768 23. Okt 20:32 test.copy
    Do you care to open a bug report?

    Hendrik

  10. #10

    Default Re: coreutils-8.25-3.3.i586 "cp", "mv", etc.: "value too large for defined data type"

    Sure.
    If you could please tell me the right place for this report.
    "glibc" for example was at sourceware.org.

Page 1 of 3 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •