Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: Perl Mechanize: how to process a simple loop [plz review my approach]

  1. #1

    Default Perl Mechanize: how to process a simple loop [plz review my approach]

    hello CommunitY!

    i am heading for Perl-programming. I want to learn something


    Well i am currently working on a small solution: I have tried various tutorials (examples of Mechanize - that i have found on the CPAN) not oll of them work - some of them are broken!

    Now i try t o get some real-world-task!

    Especially interesting for me as a PHP/Perl-beginner is this site in Switzerland:
    see this link - click on it and see more

    Which has a dataset of 2700 foundations. All the data are free to use with no limitations copyrights on it.

    what we have so far: Well the harvesting task should be no problem if i take WWW::Mechanize - particularly for doing the form based search and selecting the individual entries.
    Hmm - i guess that the algorithm would be basically 2 nested loops: the outer loop runs the form based search, the inner loop processes the search results.

    The outer loop would use the select() and the submit_form() functions
    on the second search form on the page. Can we use DOM processing here.

    Well - how can we get the get the selection values.

    The inner loop through the results would use the follow link function to get to the actual entries using the following call.

    PHP Code:
    $mech->follow_link(url_regex => qr/webgrab_path=http:\/\/evs2000.*\?
    Id=\d+$/, => $result_nbr); 
    This would forward our mechanic browser to the entry page. Basically the URL query looks for links that have the webgrap_path to Id pattern, which is unique for each database entry. The $result_nbr variable tells mecha which one of the results it should follow next.

    If we have several result pages we would also use the same trick to traverse through the result pages.
    For the semantic extraction of the entry information,we could parse the content of the actual entries with XML:LibXML's html parser (which works fine on this page), because it gives you some powerful DOM selection (using XPath) methods.
    Well the actual looping through the pages should be doable in a few lines of perl of max. 20 lines - likely less.

    But wait: the processing of the entry pages will then be the most complex part
    of the script.

    Approaches: In principle we could do the same algorithm with a single while loop
    if we use the back() function smartly.

    Can you give me a hint for the beginning - the processing of the entry pages - doing this in Perl:: Mechanize


    Look forward to hear from you

    regards
    db1
    Last edited by dilbertone; 18-May-2011 at 12:05. Reason: more infos

  2. #2
    Join Date
    Nov 2010
    Location
    Ελλάδα(Αθήνα)-Россия (г. Красноярск)
    Posts
    1,711

    Default Απ: Perl Mechanize: how to process a simple loop [plz review my approach]

    Hello Unfortunately I can not help you so much. I am programmer but I develop in C++, Fortran, Qt and with linux Kernel.
    Anyway I will try to find something for you.
    Read this first The Perl Programming Language - www.perl.org
    Beginning Perl (free) - www.perl.org
    Perl - Wikipedia, the free encyclopedia
    Πάντα Φιλικά, Στάμος.
    Desktop: openSUSE Leap 15.3||x86_64||Gnome 3.14 & KDE 5||AMD Ryzen 7 8 core||Nvidia GTX 1080 Ti Gaming Edition
    Γλώσσες Προγραμματισμού: C++, Qt developing.
    http://bit.ly/fT8Hfi

  3. #3

    Default Re: Απ: Perl Mechanize: how to process a simple loop [plz review my approach]

    hello dear stamostolias - many many thanks for the quick reply. Great to hear from you!

    Well i will dig deeper into those readinds... Besides this good readings.. if you can give me some additional hints i would be more than happy...

    many many greetings to Greece!!
    dilbertone!!

  4. #4
    Join Date
    Nov 2010
    Location
    Ελλάδα(Αθήνα)-Россия (г. Красноярск)
    Posts
    1,711

    Default Απ: Re: Απ: Perl Mechanize: how to process a simple loop [plz review my approach]

    I will try to find something good for you.
    Thank you.

    If you want something about programming and developing. Send me PM or if you want to learn a programming language: C++, Fortran, Qt. Do not worry.
    Πάντα Φιλικά, Στάμος.
    Desktop: openSUSE Leap 15.3||x86_64||Gnome 3.14 & KDE 5||AMD Ryzen 7 8 core||Nvidia GTX 1080 Ti Gaming Edition
    Γλώσσες Προγραμματισμού: C++, Qt developing.
    http://bit.ly/fT8Hfi

  5. #5

    Default Re: Απ: Perl Mechanize: how to process a simple loop [plz review my approach]

    Good moring dear stamostolias! καλημέρα!

    many thanks to you! This is such a great place to be! I am very happy!

    Have a great day.

    Quote Originally Posted by stamostolias View Post
    Hello Unfortunately I can not help you so much. I am programmer but I develop in C++, Fortran, Qt and with linux Kernel.
    Anyway I will try to find something for you. Read this first The Perl Programming Language - www.perl.org
    Beginning Perl (free) - www.perl.org
    Perl - Wikipedia, the free encyclopedia
    Look forward to hear from you!

    Many greetings to you dear friend in Athens /Greece !!

    dilbertone

  6. #6
    Join Date
    Nov 2010
    Location
    Ελλάδα(Αθήνα)-Россия (г. Красноярск)
    Posts
    1,711

    Default Απ: Re: Απ: Perl Mechanize: how to process a simple loop [plz review my approach]

    Quote Originally Posted by dilbertone View Post
    Good moring dear stamostolias! καλημέρα!

    many thanks to you! This is such a great place to be! I am very happy!

    Have a great day.



    Look forward to hear from you!

    Many greetings to you dear friend in Athens /Greece !!

    dilbertone
    So and I am happy to give help and this problem has solved. I will send you another files from that. Anyway how did you decide to learn Perl programming language?? It would be better to start with a object oriented programming language.
    Πάντα Φιλικά, Στάμος.
    Desktop: openSUSE Leap 15.3||x86_64||Gnome 3.14 & KDE 5||AMD Ryzen 7 8 core||Nvidia GTX 1080 Ti Gaming Edition
    Γλώσσες Προγραμματισμού: C++, Qt developing.
    http://bit.ly/fT8Hfi

  7. #7

    Default Re: Απ: Re: Απ: Perl Mechanize: how to process a simple loop [plz review my approach]

    hi Stamos -

    thx for replying!! quite busi at the moment - come back later today.

    warm regards
    matze / aka db1

  8. #8

    Default Re: Perl Mechanize: how to process a simple loop [plz review my approach]

    stamostolias wrote:
    > So and I am happy to give help and this problem has solved. I will send
    > you another files from that. Anyway how did you decide to learn Perl
    > programming language?? It would be better to start with a object
    > oriented programming language.


    Lets have a religious war

    Perl *is* an object-oriented language

    http://perldoc.perl.org/search.html?q=object

    Cheers, Dave

    PS dilbert1, PHP is not the same thing as Perl. Don't confuse the two.

  9. #9
    Join Date
    Nov 2010
    Location
    Ελλάδα(Αθήνα)-Россия (г. Красноярск)
    Posts
    1,711

    Default Απ: Re: Perl Mechanize: how to process a simple loop [plz review my approach]

    Quote Originally Posted by djh-novell View Post
    stamostolias wrote:
    > So and I am happy to give help and this problem has solved. I will send
    > you another files from that. Anyway how did you decide to learn Perl
    > programming language?? It would be better to start with a object
    > oriented programming language.


    Lets have a religious war

    Perl *is* an object-oriented language

    Search results - perldoc.perl.org

    Cheers, Dave

    PS dilbert1, PHP is not the same thing as Perl. Don't confuse the two.
    It is a object oriented language, but in my opinion it will be better to start with the basis. I mean C,C++ etc.

    What do you mean Religious War??
    Πάντα Φιλικά, Στάμος.
    Desktop: openSUSE Leap 15.3||x86_64||Gnome 3.14 & KDE 5||AMD Ryzen 7 8 core||Nvidia GTX 1080 Ti Gaming Edition
    Γλώσσες Προγραμματισμού: C++, Qt developing.
    http://bit.ly/fT8Hfi

  10. #10

    Default Re: Perl Mechanize: how to process a simple loop [plz review my approach]

    hello you both! hello Stamos and djh-Novell,


    great to read your posts;-) Well i guess that djh-novell does not want to start a _real-religious war! ;-) Programmers like their languages for specific choices and in regard of specific pros (and cons).

    I have had a look at Python. i like it because it is very easy to read. Stamos you are right - or let me say: i can understand you in your ideas regarding learning programming languages...

    i had a look at some languages: Well i think, Python is quite a very nice language: one thing I've noticed between programming in perl and python. Its some times easy to write a perl program, which is why they're pretty much a dime a dozen. But reading one on the other hand...it can be so painful. So if i have time left - i am start to learn Python. Python seems like a good choice: it's easy to read,
    fairly powerful and, compared to some other languages (such as C, C++ and Java),
    and it doesn't seem too complicated.


    But back to the problem - written above. I have some difficulties writing the code in Mechanize to get the entrypage...

    see here see this link here - and the page - a switzerland page... with a database with more than 2700 records

    Can you (both) give me a hint for the beginning - the processing of the entry pages - doing this in Perl::Mechanize or any other language...?

    Look forward to hear from you both...

    greetings from Germany
    dillbertone / matze

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •