GET request on a given $url runned in LWP::UserAgent?

English Programming/Scripting
#1

good evening dear communty, hello all OpenSuse-fans!

I looked up articles about using LWP however I am still lost What is aimed: Here on this following site we find a list of many shools:

see [b]this site - with a big server! http://www-db.sn.schule.de/index.php?id=25

i want to parse the sites - and therefore i want to use LWP::UserAgent;
and for the Parsing i want to use either

  • use HTML::TreeBuilder::XPath; or
  • HTML::TokeParser

But this is another question: Well at the moment i have some issues with the LWP::Useragent Note the subsite of the overview can be reached via direct links…
but -note: each site has content. eg the following URLs

Datenblatt der Einrichtungeid=709
Datenblatt der Einrichtungeid=789
Datenblatt der Einrichtungeid=1297
Datenblatt der Einrichtungeid=761

…and so forth. The question is : how to i run LWP::UserAgent?
I want to have all the - 1000 sites.

Perhaps we have to count up form zero to 10000 with there

extern_eid=709 -(count from zero to 100000) here

What do you suggest!?

Here the ideas and specs for LWP User Agent;

cpan.org/~gaas/libwww-perl-5.837/lib/LWP/UserAgent.pm

REQUEST METHODS
The methods described in this section are used to dispatch requests via the user agent. The following request methods are provided:

$ua->get( $url )
$ua->get( $url , $field_name => $value, … )

This method will dispatch a GET request on the given $url. Further arguments can be given to initialize the headers of the request. These are given as separate name/value pairs. The return value is a response object. See HTTP::Response for
a description of the interface it provides. There will still be a response object returned when LWP can’t connect to the server specified in the URL or when
other failures in protocol handlers occur.

The question is: How to use LWP::UserAgent on the above mentioned site the right way - effectively!?

I look forward to any and all help!

dilbertone!

#2 
$ua->get('http://www-db.sn.schule.de/index.php', id => 21, extern_uid => 709);
#3

good morning ken_yap

many thanks for the quick answer!

I am trying to use LWP::UserAgent on the same URLs [see below] with different query arguments, and i am wondering if LWP::UserAgent provides a way for us to loop through the query arguments? I am not sure that LWP::UserAgent has a method for us to do that.

I tried to figure it out. And i digged deeper in the Manpages and Howtos.

KenYap, we can have a loop constructing the URLs and use LWP::UserAgent repeatedly:


for my $id (0 .. 100000)
{
    $ua->get($url."?id=21&extern_eid=".(709-$id))
    //rest of the code
}

Alternatively we can add a request_prepare handler that computes and add the query arguments before we send out the request.

**Question: **Do you think, that this fits the needs?

Again: What is aimed: Here on this following site we find a list of many schools:
Auswahl einer Einrichtung

i want to parse the sites - and therefore i want to use LWP::UserAgent;

Note the subsite of the overview can be reached via direct links… but -note: each site has content. eg the following URLs

http://www-db.sn.schule.de/index.php?id=21&extern_eid=789
http://www-db.sn.schule.de/index.php?id=21&extern_eid=789
http://www-db.sn.schule.de/index.php?id=21&extern_eid=1297

#4 
for my $i (0..10000) {
  $ua->get('http://www-db.sn.schule.de/index.php', id => 21, extern_uid => $i);
  # process reply
}
#5

hello Ken_yap

that looks good.

many many thanks!

i will try this !

regards
dilbertone;-)