Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: interpreting ls output

  1. #1
    Join Date
    Feb 2010
    Location
    nh,usa
    Posts
    161

    Default interpreting ls output

    I'm comparing the directory contents of 2 disks after an rsync.

    The following command

    Code:
    ls  -X  -1  --recursive  --size   --all --almost-all -l
    Code:
    produces output like this.
    
    36 -rw------- 1 david users 34779 May 26 07:56 30154f2a9d7d1c220d3b883f82ab76f2.png
    Doing this on each disk, sometimes produces slightly different results.

    Q1) Whats the 36 refer to?

    Q2) Whats the best way to compare the contents of 2 directories?
    Regards,

    RIH5342

  2. #2
    Join Date
    Aug 2010
    Location
    Chicago suburbs
    Posts
    12,875
    Blog Entries
    3

    Default Re: interpreting ls output

    Quote Originally Posted by rih5342 View Post
    Q1) Whats the 36 refer to?
    I believe that is the size (the number of data blocks allocated to the file).

    Q2) Whats the best way to compare the contents of 2 directories?
    That depends on what you are looking for.

    I think I have sometimes used like this:
    Code:
    ls -l directoryA | sort > file1
    ls -l directoryB | sort > file2
    diff file1 file2
    openSUSE Leap 15.1; KDE Plasma 5;
    testing Leap 15.2Alpha

  3. #3
    Join Date
    Feb 2010
    Location
    nh,usa
    Posts
    161

    Default Re: interpreting ls output

    I'm trying to make an exact copy of a directory onto a second disk.

    Why is this number sometimes different on the second disk?

    Thank you.
    Regards,

    RIH5342

  4. #4
    Join Date
    Aug 2010
    Location
    Chicago suburbs
    Posts
    12,875
    Blog Entries
    3

    Default Re: interpreting ls output

    The blocksize could be different, though that is unlikely.

    More likely, you are dealing with a sparse file. In a sparse file, you might have a large file size, but a small number of disk blocks. For example, if you created a new empty file, and then you did a seek to position 1mb in to write, you would get a sparse file. Reading that file would return bytes containing zero, until you reached the actual data. But there aren't any disk blocks for those zeros.

    When you copy the file, that copies the zeros. So the destination file won't be sparse and will have more blocks assigned. The "cp" command and several other commands actually have options for handling sparse files.
    openSUSE Leap 15.1; KDE Plasma 5;
    testing Leap 15.2Alpha

  5. #5
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,378

    Default Re: interpreting ls output

    Quote Originally Posted by rih5342 View Post
    I'm trying to make an exact copy of a directory onto a second disk.

    Why is this number sometimes different on the second disk?

    Thank you.
    you mean the number at the beginning of the output line? I have also no idea, never saw it before. It is not in a simple
    Code:
    ls -l
    It must be the result of one of the many options you use. Did you tray to add them one by one util you see this number appear?

    BTW, I do not think the -1 option is usefull here. It is when using
    Code:
    ls
    because that only shows file names and without the -1 makes several columns if possible. I do not think the longer lines of an ls -l need this.

    Also I am not sure why you use the --size. Why not simply let it as it is in bytes?

    Another remark from me is that rsync is designed to copy files. I doubt it guarantees that copying a tree of files results in somethng that is exactly a byte by byte equal to the original. It does so for the contents of the individual files. But directories may become organised different I assume, and when the results are on a file system of a different type that is almost sure IMO.
    Henk van Velden

  6. #6
    Join Date
    Feb 2010
    Location
    nh,usa
    Posts
    161

    Default Re: interpreting ls output

    I use rsync to copy from 1 disk to another, identical disks, same file system.

    Based on my experience, I know that rsync can be fussy at times. I've set the file permissions to be identical on both disks before running rsync.

    I run a script where I pipe the output of ls into a file for each disk,then use diff to compare those 2 files.

    Code:
    ls --version
    echo "  " 
    cd $DIR1
    ls  -X  -1  --recursive  --size   --all --almost-all ___David  &>$FILE1
    cd $DIR2
    ls  -X  -1  --recursive  --size   --all --almost-all ___David  &>$FILE2
    cd $DIR3
    diff  $FILE1  $FILE2  &>$FILE3
    I feel pretty good when the number of files is exact, but the amount of disk space is usually different.

    Any suggestions for a better way to get an exact file copy for a 2Tb directory?

    Is there any way to massage a file before hand, so that when it is copied, there are no changes to the copies file?

    Thank you.
    Regards,

    RIH5342

  7. #7
    Join Date
    Aug 2010
    Location
    Chicago suburbs
    Posts
    12,875
    Blog Entries
    3

    Default Re: interpreting ls output

    Quote Originally Posted by rih5342 View Post
    I use rsync to copy from 1 disk to another, identical disks, same file system.
    It turns out that "rsync" does have a "--sparse" option.

    Based on my experience, I know that rsync can be fussy at times. I've set the file permissions to be identical on both disks before running rsync.
    I often use
    Code:
    rsync -an -v --delete source-directory/. destination-directory/.
    and that works pretty well for me. But it does not get hard links right, though there's actually an option for that.

    I feel pretty good when the number of files is exact, but the amount of disk space is usually different.
    The disk space used by a directory will be a variable. And the existence of sparse files will also cause variation.

    Any suggestions for a better way to get an exact file copy for a 2Tb directory?
    Exactly what do you mean by "exact"?
    openSUSE Leap 15.1; KDE Plasma 5;
    testing Leap 15.2Alpha

  8. #8
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,378

    Default Re: interpreting ls output

    I am still a bit unsure about your goal (see also http://www.catb.org/~esr/faqs/smart-questions.html#goal).

    May be it is just about the terms you use. Sometimes it looks as if you use the terms "directory" and "disk" as synonyms, which they are not. E.g. when you talk about a 2Tb directory, which would be beyond any practical meaning (think of the many Tb all those millions of files administered within such a directory would take alone).

    So, is this about a directory with all the files administered by it (the complete tree of directories and files starting from this directory), a file system (mounted at that directory you talk of), a disk partition, a whole disk? And yes, some of these options may turn out te be (allmost) the same. But nevertheless it is better to be sure we all mean the same.
    Henk van Velden

  9. #9
    Join Date
    Feb 2010
    Location
    nh,usa
    Posts
    161

    Default Re: interpreting ls output

    Code:
    Exactly what do you mean by "exact"?

    For me in this case, exact means identical, so that if I were to delete the source directory, I would not lose any data.

    Code:
    Also I am not sure why you use the --size. Why not simply let it as it is in bytes?
    I'll try this.


    Code:
    It turns out that "rsync" does have a "--sparse" option.
    I'll try this too.

    Code:
    May be it is just about the terms you use. Sometimes it looks as if you  use the terms "directory" and "disk" as synonyms, which they are not.  E.g. when you talk about a 2Tb directory, which would be beyond any  practical meaning (think of the many Tb all those millions of files  administered within such a directory would take alone).
    Thank you for letting me know my word choice could be better.

    By 2Tb directory, I mean a directory with 2Tb of data in it.

    This is how my data is organized, everything under one directory. This make the rsync script simple.

    Thank you.
    Regards,

    RIH5342

  10. #10
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,378

    Default Re: interpreting ls output

    Quote Originally Posted by rih5342 View Post
    [CODE]
    For me in this case, exact means identical, so that if I were to delete the source directory, I would not lose any data.

    [
    Well, rsync is around for several tens of years. I would say you could now trust that it is able to do this basic functionality.
    For all the years I have used rsync for all sorts of goals (backup of course amongst that), it never failed to copy files to a place and state where could be recovered.

    But everyone to his own hobby of course. It is only I am afraid that you will be busy with all sorts of side effects of the test you are doing (which in themselves may be interesting subjects, like "what is the meaning of that extra number in front of the output lines of ls in certain circumstances").


    BTW, quoting people's text is done with the QUOTE tags: the button with "speaking cloud" just left of the CODE tags # button.
    Henk van Velden

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •