Page 1 of 3 123 LastLast
Results 1 to 10 of 24

Thread: How to create a database for the contents of folder containing 11000 mp3 files.

  1. #1
    Join Date
    Aug 2011
    Location
    India
    Posts
    249

    Default How to create a database for the contents of folder containing 11000 mp3 files.

    Hi I am Rupesh from India and I have downloaded huge mp3 files from internet and I want to create a database of these files.

    Recently I have seen a website which contains some important lectures and decided to download the site. Actually the website consists of up to 60000 mp3 files and in that I need 13000 files and all the files I need are contained in a separate directory I mean with in a web page and it's sub web pages. I have downloaded what I need ie., 11000 mp3 files of size 185 gb using offline browser called extreme picture finder and remaining 2000 files are not downloaded. The extreme picture finder downloaded these files fastly and accurately but for remaining 2000 files it is taking lot of time and so I have stopped the program. I want to download remaining 2000 files manually I mean not through any offline browser.

    The website from which I have downloaded mp3 file's is very nicely organized. The website administrators arranged webpages nicely I mean suppose if we examine a webpage they have specified that all the sub pages under that webpage consists of say 50 directories and 1000 files etc.,. I want to create a database file similar to the above.

    The extreme picture finder application in Windows downloaded the files to the folder e:/downloads/ with exact file system structure as source website. Is there any possibility to create a database of all the folders and files which are under this folder. I think that we can create a ms access database so that I can copy the resulted access database file called copied.mdb to my tab and open the file in android office applications like kingsoft etc.,.

    Actually what I am expecting is suppose I open the database file in android application like kingsoft and examine the properties of a folder and suppose it shows that it contains 100 file's and then I will open the website address related to that directory in androids Firefox and if that web address shows that it contains 160 files then I will try to download remaining 60 file's. With 20 to 30 such attempts I can download all the remaining 2000 file's.

    For Windows and Linux there are programs such as mediainfo,mediaconch which are somewhat useful for this particular need but in Windows I have found an application called playtime which will create an excel file of all the mp3 file's and their properties.

    In Linux there's a command line tool called ls (list) with option -R (recursive). Is it possible to create a database using such command line utilities.

    Please try to suggest a way to create a database like ms access or atleast any other sql database.


    Regards,
    Rupesh.

  2. #2
    Join Date
    Jun 2008
    Location
    West Yorkshire, UK
    Posts
    3,450

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    A lot depends on the naming convention they have used; if they have used underscore (_) in all the filenames rather than space:
    Code:
    ls -l > list.txt
    will create a list of all the files in a folder; this will have some multiple spaces in it; reduce these by using Search for two spaces and replace with one until you only have single spaces in the file. Then Search for space and replace with comma (,) and you have a CSV file which you can import into any database program.
    You can list them all with:
    Code:
    ls -lR > list.txt
    which will give you a recursive list of the files in each folder; however, unless you decide to use this as a quick way to get them all and then deal with each folder separately, you will have to decide how you want to distinguish the folders in this long file.

  3. #3
    Join Date
    Sep 2013
    Location
    Norfolk, UK
    Posts
    1,262

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    Quote Originally Posted by rupeshforu3 View Post
    ... Recently I have seen a website which contains some important lectures and decided to download the site. ... I have downloaded what I need ie., 11000 mp3 files ... I want to download remaining 2000 files ...
    No copyright infringement there I hope
    Regards, Paul

    2x Tumbleweed (Snapshot: 20191012) KDE Plasma 5
    2x Leap 15.1 KDE Plasma 5

  4. #4
    Join Date
    Aug 2011
    Location
    India
    Posts
    249

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    The files which I have downloaded from the website are not copyrighted and they are distributing those files freely. In the website they have clearly mentioned that "if anyone finds any file which is copyrighted then immediately report then we will remove them".

    Can you suggest how to create a database from the output which is coming from ls -lr or atleast an excel file.
    Regards,
    Rupesh.

  5. #5
    Join Date
    Sep 2008
    Posts
    2,997

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    Quote Originally Posted by rupeshforu3 View Post
    The files which I have downloaded from the website are not copyrighted and they are distributing those files freely. In the website they have clearly mentioned that "if anyone finds any file which is copyrighted then immediately report then we will remove them".

    Can you suggest how to create a database from the output which is coming from ls -lr or atleast an excel file.
    do what john_hudson told you and create a csv file, then startup libreoffice Calc do a File->Open and select that csv file
    https://help.libreoffice.org/Calc/Im...ting_CSV_Files
    after that you can save that spreadsheet as ods or xls

    ps tannington was just making a joke

  6. #6
    Join Date
    Aug 2011
    Location
    India
    Posts
    249

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    One small doubt is it possible to get properties of audio file like playback time using ls command.

  7. #7
    Join Date
    Oct 2008
    Location
    Glasgow, Scotland
    Posts
    1,134

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    Rupesh,
    It would help both yourself and those wishing to help if you would explain what it is that you are trying to achieve and why.

    In the last few months you have asked for, and been given, advise on using “mediainfo” and a java csv file comparison tool. You are now asking about a “database”. That normally implies a database manager (e.g. MariaDb, Berkeleydb) to operate on more than one type of data (e.g. correlating audio files with keywords, artists, etc.). You have given no indication of either.

    At its simplest a text file “database” of your mp3 filenames could be created with:
    Code:
    >  ls -r "path totop level mp3 directory" > LectureFilenames.text
    but I do not expect this would be very useful.

    If you just want to have a local copy of these lectures for off-line private study, and they are well laid out on the source web-site, why not consider making a local mirror of (part of) the original site using “wget” or, if the site is designed for it, “rsync”?
    ~Thank you for sharing an interesting problem.
    --
    slàinte mhath,
    rayH

  8. #8
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,278
    Blog Entries
    2

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    Generally speaking,
    Nowadays people expect more than just storing media content in a database.

    They want the files searchable by metadata special to the type of media, like author, time, titles, lyrics, album, more.
    They often want to be able to create custom playlists.
    They often want to create custom categories, ie "tagging"

    To do the above,
    You need to look first at User apps that support the features you want, not how the content is stored.

    And, consider how your content will be accessed. Are you interested only in a locally running app? Or, perhaps a Home Entertainment system might be better as a centralized repository for all your devices?

    As for storing your files in a database,
    You'd have to state your reasons for doing so, and perhaps more specifically objectives.
    For instance,
    Less storage used
    Faster search
    Security

    But, of course you should consider how you want to retrieve content from your database, if an app isn't already available can a very generic interface be sufficient? Do you have to skill or initiative to learn how to write the code?

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  9. #9
    Join Date
    Aug 2011
    Location
    India
    Posts
    249

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    I love to listen to music and also love to listen lectures given by someone. I am very serious about how many music files does I have and how many lectures dies I have etc., If I have downloaded some music or lectures from internet I have the habit of checking how many files I have downloaded and are the files downloaded properly or downloaded only half in size etc., . Previously I have downloaded some music files from internet and I have compared my downloaded files and the files on source website and they are different in size.

    The files from website I am trying to download are important to me and so I am struggling a lot to download all the files I want but surprisingly the total size of website is 1.5 tb approximately and in those I want to download upto 230 GB. It is not possible to download 230 GB of files within one day and even offline browser's are also taking lot of time.

    If I can obtain a way to check how many folders and files are present in the root directory of my downloaded directory it is possible to download all the files I want very easily. At present I have Android phone but it doesn't have 250 GB of memory and so I think if I can obtain database it will be easier to compare local files and the source website.

    At present I got an idea if we can obtain list of files using ls -r command and store the result in an text file then we can use touch to create files with directory structure same. I think that touch command will just create file with zero file size.

    By using ls and touch commands I can recreate total files with directory structure and I can zip the resulting folder and then copy 1 mb zip to Android phone and so I can examine which files are missing. Please give some suggestions on my idea.

  10. #10
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,278
    Blog Entries
    2

    Default Re: How to create a database for the contents of folder containing 11000 mp3 files.

    Use something like HTTrack or even the basic wget command to download your files.
    As you might imagine wget is about as fast as you can get with minimal overhead, but unlike HTTrack has limited download options.
    IIRC HTTrack website copying can be interrupted and resumed at a later time.

    Once downloaded,
    You can use an app like Calibre for organization and access.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

Page 1 of 3 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •