Page 1 of 2 12 LastLast
Results 1 to 10 of 15

Thread: Find Duplicate files in multiple disks

  1. #1

    Default Find Duplicate files in multiple disks

    Hello I am looking for an application (better kde one) that can search two external hard disks I have and find any duplicate files. I did some backups before to one disk which i copied few years ago to the other disk. Right now I would like some program to check files and tell me if there are the same.

    I would like to thank you in advance for your help


    Regards
    Alex

  2. #2
    Join Date
    Jan 2011
    Location
    Slovenija
    Posts
    3

    Default Re: Find Duplicate files in multiple disks

    I don't know about guis, but here's something that should work from the console (untested):
    # change these two paths to point to your mount points
    MYDISK1=/media/something
    MYDISK2=/media/somethingelse

    TMP1=/tmp/blablabla123142-disk1.tmp
    TMP2=/tmp/blablabla123142-disk2.tmp
    TMP3=/tmp/blablabla123142-hashes.tmp

    find "$MYDISK1" -type f -exec md5sum {} + > "$TMP1"
    find "$MYDISK2" -type f -exec md5sum {} + > "$TMP2"
    sort "$TMP1" "$TMP2" | uniq -d -w 32 | awk '{print $1}' > "$TMP3"
    # this will print all the duplicate files
    grep -F -f "$TMP3" "$TMP1" "$TMP2"
    # if you want to delete them add the following to the grep line: | awk '{$1=""; print $0}' | xargs rm -i
    # making it
    #grep -F -f "$TMP3" "$TMP1" "$TMP2" | awk '{$1=""; print $0}' | xargs rm -i
    # remove the -i if you don't want to be asked about each file
    rm "$TMP3" "$TMP1" "$TMP2"

  3. #3

    Default Re: Find Duplicate files in multiple disks

    Quote Originally Posted by lynxlynxlynx View Post
    I don't know about guis, but here's something that should work from the console (untested):
    a gui please i am really bad with konsole

  4. #4

    Default Re: Find Duplicate files in multiple disks

    Quote Originally Posted by lynxlynxlynx View Post
    Code:
    find "$MYDISK1" -type f -exec md5sum {} + > "$TMP1"
    find "$MYDISK2" -type f -exec md5sum {} + > "$TMP2"
    That would take for ever. You should list all the files, sort by size and checksum only the files which have the same size (and/or the same name, depending on what you're looking for).

  5. #5
    Join Date
    Jan 2011
    Location
    Slovenija
    Posts
    3

    Default Re: Find Duplicate files in multiple disks

    I didn't say it was fast, but it should get the job done. Since the OP didn't specify exactly what he wants to do, maybe a rsync is enough.

  6. #6
    Join Date
    Jun 2008
    Location
    Berlin
    Posts
    2,061

    Default Re: Find Duplicate files in multiple disks

    No, he does not what to sync, but check for dublicates.

    I recommend 'dupf', which is a very easy, yet effective console-tool which also can be used with a GUI. The package is called 'dupfinder' and is offered via Packman. In case you cannot find it in the menu, you can start it with

    Code:
    dupfgui

  7. #7
    Join Date
    Jan 2011
    Location
    Slovenija
    Posts
    3

    Default Re: Find Duplicate files in multiple disks

    rsync doesn't necessarily sync, but you found something simpler anyway.

  8. #8

    Default Re: Find Duplicate files in multiple disks

    Quote Originally Posted by gropiuskalle View Post
    No, he does not what to sync, but check for dublicates.

    I recommend 'dupf', which is a very easy, yet effective console-tool which also can be used with a GUI. The package is called 'dupfinder' and is offered via Packman. In case you cannot find it in the menu, you can start it with

    Code:
    dupfgui
    \
    Thanks a lot for that.
    I am trying it right now

  9. #9

    Default Re: Find Duplicate files in multiple disks

    I tried the program and it works! I would still your help with one more thing.
    I have two folders that are quite alike. Both have roughly 900+ files and only 20 of these 900 files are NOT duplicates. The program finds succseefully all the duplicates but I want to find the non-duplicated files.

    CAn you help me do that with the program u suggested?
    Regards

  10. #10

    Default Re: Find Duplicate files in multiple disks

    If the directories you want to compare don't include subdirectories - or you don't need to compare recursively - , you can use a two pane file manager, such as Midnight Commander (mc) in a terminal or the GUI program krusader. In krusader, with one directory in each pane, you would do <Edit-Compare Directories> and then <Edit - Invert Selection>.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •