good evening dear OpenSuse-fans,
well i have a running LWP::UserAgent that should be applied on following URL
see this site here: http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=5503
Note: a big public server with lots of power.
**btw **this runs with many many similar targets see the following endings:
html?show_school=5503
html?show_school=9002
html?show_school=5512
note: i ve applied some advices from ken_yap and others that i got here:
**btw: ** you see that this is a Perl-job. Truely. I cannot inmage any other language that does this as well as Perl. I want to do this with use LWP::UserAgent; Well agreed t his can also be done with MECHANIZE which might be a bit easier but i will try with LWP::UserAgent first. well in order to fetch the above mentioned pages i have to do a loop.
i wanted to loop over the results - and therefore i tried to apply the corresponding URLS but i got a bunch of errors…:
suse-linux:/usr/perl # perl perl_mecha_example_two.pl
Global symbol “$pagecontent” requires explicit package name at perl_mecha_example_two.pl line 24.
Global symbol “$url” requires explicit package name at perl_mecha_example_two.pl line 29.
Execution of perl_mecha_example_two.pl aborted due to compilation errors (#1)
(F) You’ve said “use strict” or “use strict vars”, which indicates
that all variables must either be lexically scoped (using “my” or “state”),
declared beforehand using “our”, or explicitly qualified to say
which package the global variable is in (using “::”).Uncaught exception from user code:
Global symbol “$pagecontent” requires explicit package name at perl_mecha_example_two.pl line 24.
Global symbol “$url” requires explicit package name at perl_mecha_example_two.pl line 29.
Execution of perl_mecha_example_two.pl aborted due to compilation errors.
at perl_mecha_example_two.pl line 86
See the code - some errors… can you advice!?
#!/usr/bin/perl -W
use strict;
use warnings; # give out some warnings if something does not run well
use diagnostics; # tell me when something is wrong
use DBI;
use LWP::UserAgent;
use HTTP::Request::Common;
use HTML::TreeBuilder::XPath;
# first get a list of all schools
my $ua = LWP::UserAgent->new;
$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7");
#pretending to be firefox on linux.
for my $i (0..10000) {
my $request = HTTP::Request->new(GET => sprintf(" here to put the URL into =%d", $i));
$request->header('Accept' => 'text/html');
my $response = $ua->request($request);
if ($response->is_success) {
$pagecontent = $response -> content;
}
# now we can do whatever with the $pagecontent
}
my $request = POST $url,
order => "schule_ort",
schulname => undef,
Basisdaten => undef,
Profil => undef,
Schulort => undef,
typid => "11",
Fax =>
Homepage => undef,
verbinder => "AND"
];
print "getting all schools - this could take some time
";
my $response = $ua->request($request);
# extract the ids
my @ids = $response->content =~ /getSchoolDetail\((\d+)/gs;
print "found " . scalar @ids . " schools
";
# for this demo we only do the first 5
my @ids_to_do = @ids[0..4];
# use your own user and password
my $dbh = DBI->connect("DBI:mysql:database=schulen", "user", "pass", { AutoCommit => 0 }) or die $!;
my $sth = $dbh->prepare(<<sqlend);
insert into schulen ( name , plz , ort, strasse , tel, fax , mail, quelle , original_id )
values ( ?, ?, ?, ?, ?, ?, ?, ?, ? )
sqlend
# now loop over ids
for my $id (@ids_to_do) {
# get detail information for id
my $res = $ua->get(" here to put the URL into => &gid=$id");
# parse the response
my $tree = HTML::TreeBuilder::XPath->new;
$tree->parse($res->content);
my $xpath = q|//div@id='MCinhview']//div@class='floatbox']//table|;
my ($adress_table, $tel_table) = $tree->findnodes($xpath);
my ($adr) = $adress_table->find("td");
my ($name, $city, $street) = map { s/^\s*//; s/\s*$//; $_ } ($adr->content_list)[2,4,6];
my($plz, $ort) = $city =~ /^(\d+)\s*(.*)/;
my ($tel, $fax, $mail) = map { s/^\s*//; s/\s*$//; $_ } map { ($_->content_list)[1] } $tel_table->find("td");
$sth->execute($name, $plz, $ort, $street, $tel, $fax, $mail, "SA", $id);
$dbh->commit;
$tree->delete;
print "$name done
";
}
well i have :
- to apply the right URL in the above code. (…with the loops)
- furthermore i have to fix all that causes warnings…