[28560] in Perl-Users-Digest
Perl-Users Digest, Issue: 9924 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Thu Nov 2 21:05:47 2006
Date: Thu, 2 Nov 2006 18:05:09 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Thu, 2 Nov 2006 Volume: 10 Number: 9924
Today's topics:
Re: Abysmal performance of Net::SFTP (and I have GMP & xhoster@gmail.com
Re: accessing main:: vars in required file <tadmc@augustmail.com>
Re: File Permissions using NIS and NFS and copying via <No_4@dsl.pipex.com>
Re: how do i update one section of a page leaving rest? <bootiack@yahoo.com>
Re: Masking/Hiding a password in Perl Source <tadmc@augustmail.com>
Re: Masking/Hiding a password in Perl Source <veatchla@yahoo.com>
Re: Perl equivalent to unix script <dmercer@mn.rr.com>
Re: Sorting and Writing Effecient Code <tadmc@augustmail.com>
Re: Sorting and Writing Effecient Code <tadmc@augustmail.com>
Re: Sorting and Writing Effecient Code <tadmc@augustmail.com>
submitting a form with no name attribute with mechanize <nospam@home.com>
WWW::Mechanize - simulating complete page download odigity@gmail.com
Re: WWW::Mechanize - simulating complete page download <tadmc@augustmail.com>
Re: WWW::Mechanize - simulating complete page download odigity@gmail.com
Re: WWW::Mechanize - simulating complete page download odigity@gmail.com
Re: WWW::Mechanize - simulating complete page download odigity@gmail.com
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: 02 Nov 2006 23:18:38 GMT
From: xhoster@gmail.com
Subject: Re: Abysmal performance of Net::SFTP (and I have GMP & Pari)
Message-Id: <20061102181900.132$ua@newsreader.com>
usenet@DavidFilmer.com wrote:
...
>
> Here is my test script:
>
> #!/usr/bin/perl
> use strict; use warnings;
> use Net::SFTP;
>
> #Create with: dd if=/dev/urandom of=/tmp/1mb.junk bs=1024 count=1024
> my $testfile = "/tmp/1mb.junk";
>
> my %comm = (
> user => 'myuserid',
> password => 'mypassword',
> ssh_args => [ cipher => 'blowfish-cbc' ],
> debug => 0,
> );
>
> my $sftp = Net::SFTP->new('localhost', %comm) or die "oops: $@";
>
> $sftp->put ($testfile, "$testfile.test")
> or die("ftp put failed ", $sftp->message);
>
> END{
> print map{"$_\n"} sort keys %INC; #check for GMP/Pari
> }
>
> __END__
>
> The blowfish cipher is indeed the "fastest" (as the docs indicate). I
> tried the other available ciphers with inferior results:
> arcfour 14 seconds
> 3des-cbc 22 seconds
>
> The "interesting" parts of the %INC dump (in the END block of the test
> script) look like this (and validate that GMP/Pari are being called):
So it appears that GMP/Pari are installed, but from the profile it appears
they aren't doing anything.
> I ran the script thorough Devel::Profile (perl -d:Profile test.pl); the
> top results (> 1.0%) are:
Can you get Profile to give you *all* of the subs and look through that
for GMP/Pari calls?
> %Time Sec. #calls sec/call name
> 28.05 2.714 131970 0.000021 Crypt::Blowfish::encrypt
> 23.32 2.256 139 0.016228 Net::SSH::Perl::Cipher::CBC::encrypt
> 10.08 0.975 282 0.003458 IO::Select::select
> 5.53 0.535 32 0.016733 IO::Select::can_read
> 4.51 0.437 132840 0.000003 Crypt::Blowfish::crypt
Crypt::Blowfish::encrypt doesn't do anything other than rearrange
it's args and pass them on to the XS code Crypt::Blowfish::crypt.
Yet it takes over 6 times longer than the code that does the real
work? My gut feeling is that the bottleneck here is simply the fact
that you make 2 function calls for every 8 bytes to be encrypted.
I don't see how this could be possibly be fast. Could someone who *does*
find this to be fast on their machines please DProf this example and
post the results?
> I'm happy to entertain any "try this and see what happens" ideas.
The first two things I'd try, would be to use DProf rather than Profile,
and see if you get similar results. And to "strace" the run (I think you
are on Linux, right?), and see if anything pops out that way.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
------------------------------
Date: Thu, 2 Nov 2006 17:23:58 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: accessing main:: vars in required file
Message-Id: <slrnekkvge.qo2.tadmc@tadmc30.august.net>
Stephen O'D <stephen.odonnell@gmail.com> wrote:
> I thought (wrongly) that if I required a file
> that wasnt a module that it would be in the same lexial scope.
That wrong thought can easily be avoided by remembering that
lexical variable are never visible across file boundaries.
> I know what I am doing is really 'good practice' - i am trying to
> achive something similar to MJD's dispatch tables described in Higher
> Order Perl - i think I need to go back and re-read it ...
Perhaps you should read this too:
"Coping with Scoping":
http://perl.plover.com/FAQs/Namespaces.html
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Thu, 02 Nov 2006 23:20:43 +0000
From: Big and Blue <No_4@dsl.pipex.com>
Subject: Re: File Permissions using NIS and NFS and copying via perl
Message-Id: <Pf2dnb6BsrzP4NfYRVnygw@pipex.net>
alerman@gmail.com wrote:
>
> I have written a script that copies files from a test server to a
> production server. The userID's and groupID's are the same on both
> servers and we are using NIS.
>
> I set the permissions on the test server to alerman:www-data (I own it,
> it is in the web users group) so that the web server can read and
> execute it.
>
> The files copy over to the new server, but are always alerman:alerman .
> This means that my web server cant read them because of permissions
> issues.
>
> Any suggestions?
>
> I have tried both File::Copy and using `cp -p from to`
If cp -p isn't setting the group then either it is broken or the process
doing it is *not* in the www-data group.
After a copy which has set the group to alerman try:
chgrp www-data <1-file>
and see what happens.
However, it's not clear where the Web server comes into this. I'm
assuming it isn't involved in doing any of the actual copying.
--
Just because I've written it doesn't mean that
either you or I have to believe it.
------------------------------
Date: 2 Nov 2006 17:56:00 -0800
From: "gavino" <bootiack@yahoo.com>
Subject: Re: how do i update one section of a page leaving rest?
Message-Id: <1162518960.488100.305990@f16g2000cwb.googlegroups.com>
Michele Dondi wrote:
> On 1 Nov 2006 22:44:23 -0800, "gavino" <bootiack@yahoo.com> wrote:
>
> >> > How Do i updae one section of a page leaving the rest?
> >>
> >> You could open that page in your favourite editor, delete the section you
> >> don't like, and type in the new content.
> [snip]
> >thanks dude
> >The idea is to make cool web apps that are fast
> >A statik site wouldn't do well.
>
> Oh, you should have specified these details in the first place. Then
> it would have been easier to help you. So here's another try: open the
> sources for that web app in your favourite editor and write some code
> that will delete the section you don't like, and update it with the
> new content.
>
> >Remember to vote republican.
>
> I will remember. I won't do it. I couldn't anyway. So, it doesn't make
> a difference.
>
>
> Michele
> --
> {$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
> (($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
> .'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
> 256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
You realise you come off as an idiot when you talk like this?
vote republican.
------------------------------
Date: Thu, 2 Nov 2006 17:27:02 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: Masking/Hiding a password in Perl Source
Message-Id: <slrnekkvm6.qo2.tadmc@tadmc30.august.net>
l v <veatchla@yahoo.com> wrote:
> do not use variable names like $password to advertise what the variable
> holds.
My boss never notices when I put passwords in programs:
$this_is_not_a_password = '!@#$%^&*()';
works like a charm!
:-)
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Thu, 02 Nov 2006 19:33:34 -0600
From: l v <veatchla@yahoo.com>
Subject: Re: Masking/Hiding a password in Perl Source
Message-Id: <12kl73617r47i08@news.supernews.com>
Tad McClellan wrote:
> l v <veatchla@yahoo.com> wrote:
>
>> do not use variable names like $password to advertise what the variable
>> holds.
>
>
> My boss never notices when I put passwords in programs:
>
> $this_is_not_a_password = '!@#$%^&*()';
>
> works like a charm!
>
> :-)
>
>
Neither does mine, but like the OP, our corporate security folks and SOx
auditors go nuts over it. But have little to offer as to a solution.
--
Len
------------------------------
Date: Fri, 03 Nov 2006 00:36:59 GMT
From: "Dan Mercer" <dmercer@mn.rr.com>
Subject: Re: Perl equivalent to unix script
Message-Id: <Law2h.52883$F7.41789@tornado.rdc-kc.rr.com>
"Dr.Ruud" <rvtol+news@isolution.nl> wrote in message news:eiccbo.8o.1@news.isolution.nl...
: Dan Mercer schreef:
: > usenet:
: >> Mike:
:
: >>> cat tempfile1 | sort > newfile2;
: >>
: >> That's rather convoluted even for UNIX scripting. Why not:
: >> sort tempfile1 > newfile2;
: >
: > Even that's too convoluted - why not just
: > sort -o file file
:
: Because that is not equivalent. For example you can't tell from what is
: presented whether tempfile1 is still needed for something else, after
: the sort.
The original pipeline was:
cat tempfile1 | sort > newfile2; rm tempfile1
Dan Mercer
:
: --
: Affijn, Ruud
:
: "Gewoon is een tijger."
:
------------------------------
Date: Thu, 2 Nov 2006 17:08:55 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: Sorting and Writing Effecient Code
Message-Id: <slrnekkuk7.qi7.tadmc@tadmc30.august.net>
banker123 <bradbrockman@yahoo.com> wrote:
> Question
> 1. How do I sort the variables in a scalar context not an array?
That question makes no sense.
"scalar" means "a single thing".
You cannot sort a single thing, you need more than one thing
before you can impost an order (sorting) on the things.
> Should I be reading the variables I have extracted into an array?
Probably.
> 2. This program executes every 15 seconds to output the contents of
> the directory. The output is used to control work flow and meet crucial
> deadlines. Is this effeciently written?
"premature optimization is the root of all evil"
(google it)
> Any suggestions?
Don't concern yourself much with efficiency until _after_ it has
been demonstrated that what you already have is too slow.
Is it taking close to 15 seconds to execute?
If not, then you are done without having to spend any time tweeking stuff.
> My first "production" perl program, after reading and learning perl for
> about a month.
>
> #Open directory
> my $dir="G:/Formware/files/";
I would suggest leaving the trailing slash out of there.
It looks confusing when you use it later.
> opendir DH, $dir or die "Cannot open$!";
It would be nice to know the name of the directory that you were
attempting to open, so you should include that info (along with
some delimiters) in your diagnostic message:
opendir DH, $dir or die "Cannot open '$dir' $!";
> #Get system date in the format mmddyy
> @months = qw(01 02 03 04 05 06 07 08 09 10 11 12);
You should always put these:
use warnings;
use strict;
at the top of every Perl program. They will catch many common
mistakes for you.
> @days = qw(01 02 03 04 05 06 07 08 09 10..31);
You never use the @days array in your subsequent code, so why is it there?
> ($second, $minute, $hour, $dayOfMonth, $month, $yearOffset, $dayOfWeek,
> $dayOfYear, $daylightSavings) = localtime();
If you use a "Slice" (see that section in perldata.pod) you can get
just the ones you want without a bunch of dummy variables that you
never use:
my($day, $month, $year) = (localtime)[3,4,5];
> $dayOfMonth=sprintf("%02d", $dayOfMonth);
$day = sprintf '%02d', $day;
$month = sprintf '%02d', $month + 1;
> $date = "$months[$month]$dayOfMonth06";
^^
^^
Your program will break in about two months...
$year = sprintf '%02d', ($year + 1900) % 100;
my $date = "$month$day$year";
> #Read each file beginning with system date from directory
> @files=grep(/^$date/,readdir(DH));
Whitespace is not a scarce resource. Feel free to use as much of it
as you like to make your code easier to read and understand:
@files = grep(/^$date/, readdir(DH));
Even better, eliminate unnecessary parenthesis:
@files = grep /^$date/, readdir DH;
or
@files = grep { /^$date/ } readdir DH;
> closedir (DH);
>
>
> foreach $file (@files) {
Here you read things into an array when you do not need to. Below
you do not read things into an array when you do need to. Get
the logic right first, then worry about performance.
foreach my $file ( grep /^$date/, readdir DH ) { # no @files temp array
> open (data, "$dir$file") or die "Cannot open file: $!";
Your working program will become a non-working program when you update
your perl and it includes a new function named data().
It looks like you are missing a directory separator, but you're not.
It would be better if it didn't look like a mistake. Leave the
trailing slash out of $dir, and put it in explicitly when it is needed.
You should report the filename in the diagnostic message again.
The 3-argument form of open() is much much better, so you should get
used to using that form all of the time.
open my $data, '<', "$dir/$file" or die "Cannot open '$dir/$file' $!";
> while ( <data> ) {
while ( <$data> ) {
> if ( /JOBNAME/ ) {
> $job = substr($_,8,8);
> chomp($job);
Why do you give substr args that capture a newline only to remove the newline?
Just grab one less character, then you won't need a chomp:
my $job = substr($_, 8, 7);
> }
> elsif ( /\.TIF\s{12}/ ) {
> my $box = substr($_,73,6);
> printf "$box $job $file\n";
> last;
> }
You should line up your closing brace with the keyword that goes with
the opening brace, and indent the code to show its structure:
elsif ( /\.TIF\s{12}/ ) {
my $box = substr($_,73,6);
printf "$box $job $file\n";
last;
}
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Thu, 2 Nov 2006 17:16:13 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: Sorting and Writing Effecient Code
Message-Id: <slrnekkv1t.qi7.tadmc@tadmc30.august.net>
banker123 <bradbrockman@yahoo.com> wrote:
>> > Question
>> > 1. How do I sort the variables in a scalar context not an array?
>> > Should I be reading the variables I have extracted into an array?
>>
>> If you want to sort them, then you need to store them. Either an array,
>> or maybe a hash.
>
> How do I store the variables into an array, push or pop command,
Either one is fine, since you are going to rearrange their order
later when you get to the sorting part anyway.
> does
> this have much overhead?
It is a difference of a few microseconds, and below you say wasting
thousands of microseconds is acceptable, so asking this is stepping
over dollars in order to pick up pennies.
> I woul like the saort to point to a sort
> order file that containas priorites. Example:
>
> Variables
> Box 1
> Box 2
> Box 3
Those are not variables.
Perhaps you meant values instead?
If you show it in Perl code, it will be much less ambiguous.
Have you seen the Posting Guidelines that are posted here frequently?
>> Every time you execute this script, it reprocesses the same files it
>> already processed the last time. That is the fundamental inefficiency,
>> but how to fix it depends on external factors.
>
> Understood this is running real-time therefore the re-processing is
> acceptable.
Huh?
If you have time constraints, then wasting time is UNacceptable.
If you do not have time constraints, why do you keep harping on efficieny?
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Thu, 2 Nov 2006 17:21:39 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: Sorting and Writing Effecient Code
Message-Id: <slrnekkvc3.qo2.tadmc@tadmc30.august.net>
Mirco Wahab <wahab@chemie.uni-halle.de> wrote:
> Thus spoke banker123 (on 2006-11-02 20:36):
>
>> 2. This program executes every 15 seconds to output the contents of
>> the directory. The output is used to control work flow and meet crucial
>> deadlines. Is this effeciently written? Any suggestions?
>>
>> My first "production" perl program, after reading and learning perl for
>> about a month.
>
> This is really good for a "first time program" ;-)
>
> I don't know your files, but I straightened the
> program a bit (all programmers do so with other
> peoples code ;-), maybe it won't work (in the third
> part) because I had to guess what your data is like.
>
> ...
> # date stuff
> my ($d, $m, $y) = (localtime)[3..5];
> my $date = sprintf "%02d"x3, $m+1, $d, $y-100;
>
> # directory stuff
> my $dir = shift or die "no dir given";
> opendir(DH, $dir) || die "$dir: $!";
> my @files = grep /^$date/, readdir(DH);
> closedir DH;
>
> # file stuff
> foreach my $fname (@files) {
> open(my $fh, "$dir$fname") or die "$fname: $!";
> my ($job, $box);
> while( <$fh> ) {
> # extract something
> ($job) = /.{8}(.{8})/ if /JOBNAME/;
> # extract something else
> ($box) = /.{73}(.{6})/
> and printf "$box $job %s $fname\n", ' 'x20
> and last if /\.TIF\s{12}/
> }
> close $fh;
> }
>
>
>
> Don't forget to close your files, otherwise
> your program will fail sensational ...
How will it fail?
Perl closes any open filehandles when it exits anyway.
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Fri, 03 Nov 2006 00:44:51 GMT
From: "Nospam" <nospam@home.com>
Subject: submitting a form with no name attribute with mechanize
Message-Id: <7iw2h.40777$iq4.35071@newsfe1-gui.ntli.net>
Is there a way to submit a form with no name attribute with mechanize, when
the html is:
<INPUT TYPE=submit VALUE="Submit your ad >> ">
<input type="hidden" name="posting_id" value="">
<input type="hidden" name="current_live_date" value="">
------------------------------
Date: 2 Nov 2006 15:14:03 -0800
From: odigity@gmail.com
Subject: WWW::Mechanize - simulating complete page download
Message-Id: <1162509243.843320.88080@h54g2000cwb.googlegroups.com>
Hey, all.
I'm using WWW::Mechanize for the first time because I have been tasked
with benchmarking the complete download time for my company's intranet
homepage from various branch offices. The problem is that this is a
very complex (horrible, ugly) page - multiple frames, images, external
CSS files, external javscript files, etc.
I've got the frames and images part working successfully, but it seems
WWW::Mechanize doesn't have the ability to detect the link and script
tags that point to external resources.
What's the best way to solve this problem? Do I need to switch to a
more powerful stand-alone HTML parsing module? Do I need to resort to
nasty page-specific regexs? Or is there an elegant solution I'm
overloooking?
Thanks!
-ofer
------------------------------
Date: Thu, 2 Nov 2006 18:30:18 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <slrnekl3cq.qv8.tadmc@tadmc30.august.net>
odigity@gmail.com <odigity@gmail.com> wrote:
> Hey, all.
>
> I'm using WWW::Mechanize for the first time because I have been tasked
> with benchmarking the complete download time for my company's intranet
> homepage from various branch offices. The problem is that this is a
> very complex (horrible, ugly) page - multiple frames, images, external
> CSS files, external javscript files, etc.
>
> I've got the frames and images part working successfully, but it seems
> WWW::Mechanize doesn't have the ability to detect the link
A simple search of the modules docs shows 5 methods with "link"
in their names:
grep ^= Mechanize.pod | grep link
=head2 $mech->links()
=head2 $mech->follow_link(...)
=head2 $mech->find_link()
=head2 $mech->find_all_links( ... )
=head2 $mech->_extract_links()
I'd read up on what those do...
> and script
> tags that point to external resources.
If you mean JavaScript, then yes, Mechanize cannot find those.
> What's the best way to solve this problem?
If it was my task, I'd let a Real Browser figure out what all
needs to be fetched by using the
Web Scraping Proxy
http://www.research.att.com/~hpk/wsp/
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: 2 Nov 2006 16:47:09 -0800
From: odigity@gmail.com
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <1162514829.002090.237900@e3g2000cwe.googlegroups.com>
I've done more poking around, and this is more difficult than I
thought.
WWW::Mechanize uses both HTML::Tree and HTML::Parser under the hood to
do the parsing work. I've read up on both, so I'm now familiar with
what both packages are capable of.
I ran HTML::LinkExtor against my web document for fun, and discovered
that there are also background images in various 'table' and 'td'
elements that I need to grab as well. LinkExtor seems to do a good job
of finding the links to all the resources that a browser would normally
download. Here's a list of all of the cases I've discovered so far, in
the form of element.attribute, with a note about typical resource type:
frame.src # html
iframe.src # html
img.src # image
link.href # css (usually)
script.src # js (usually)
table.background # image
td.background # image
So it looks like I'm going to have drop WWW::Mechanize in favor of my
own custom solution combining LWP::UserAgent and HTML::LinkExtor, plus
some careful manipulation of base urls.
If this works out, it seems like a good candidate for a new module
dedicated to executing a complete download of a page as a browser
would. My test will be if the output matches the contents of a File ->
Save Page As operation in Firefox (type: Web Page, complete).
-ofer
odigity@gmail.com wrote:
> Hey, all.
>
> I'm using WWW::Mechanize for the first time because I have been tasked
> with benchmarking the complete download time for my company's intranet
> homepage from various branch offices. The problem is that this is a
> very complex (horrible, ugly) page - multiple frames, images, external
> CSS files, external javscript files, etc.
>
> I've got the frames and images part working successfully, but it seems
> WWW::Mechanize doesn't have the ability to detect the link and script
> tags that point to external resources.
>
> What's the best way to solve this problem? Do I need to switch to a
> more powerful stand-alone HTML parsing module? Do I need to resort to
> nasty page-specific regexs? Or is there an elegant solution I'm
> overloooking?
>
> Thanks!
>
> -ofer
------------------------------
Date: 2 Nov 2006 16:53:55 -0800
From: odigity@gmail.com
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <1162515235.772024.62540@h48g2000cwc.googlegroups.com>
That's a good outside-the-box idea. Only problem is the WSP tarball
link requires authorization. Do you know the username/password, or an
alternative download location?
-ofer
Tad McClellan wrote:
> odigity@gmail.com <odigity@gmail.com> wrote:
> > Hey, all.
> >
> > I'm using WWW::Mechanize for the first time because I have been tasked
> > with benchmarking the complete download time for my company's intranet
> > homepage from various branch offices. The problem is that this is a
> > very complex (horrible, ugly) page - multiple frames, images, external
> > CSS files, external javscript files, etc.
> >
> > I've got the frames and images part working successfully, but it seems
> > WWW::Mechanize doesn't have the ability to detect the link
>
>
> A simple search of the modules docs shows 5 methods with "link"
> in their names:
>
> grep ^= Mechanize.pod | grep link
>
> =head2 $mech->links()
> =head2 $mech->follow_link(...)
> =head2 $mech->find_link()
> =head2 $mech->find_all_links( ... )
> =head2 $mech->_extract_links()
>
> I'd read up on what those do...
>
>
> > and script
> > tags that point to external resources.
>
>
> If you mean JavaScript, then yes, Mechanize cannot find those.
>
>
> > What's the best way to solve this problem?
>
>
> If it was my task, I'd let a Real Browser figure out what all
> needs to be fetched by using the
>
> Web Scraping Proxy
>
> http://www.research.att.com/~hpk/wsp/
>
>
> --
> Tad McClellan SGML consulting
> tadmc@augustmail.com Perl programming
> Fort Worth, Texas
------------------------------
Date: 2 Nov 2006 17:29:39 -0800
From: odigity@gmail.com
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <1162517378.935053.46520@b28g2000cwb.googlegroups.com>
Arg!
Some of the URLs I need to benchmark contain instances of resources
being downloaded as the result of javascript code being executed. One
of them actually starts out as a small document containing a form and
some JS that determines the local time zone, inserts the value in a
hidden form variable, then submits the form. Lunatics.
I don't think there's any combination of Perl modules or clever tricks
that will yield a 100% correct solution to the problem of behaving the
way a browser behaves given an initial URL. What I need is a real
browser. Ideally, a Perl interface to the Firefox engine that will
allow me to feed the engine the initial URL, tell it "go get 'em, tiger
- er, fox", and when it comes to a full stop, have it dump a list of
all HTTP requests it generated, the download time for each, and ideally
the MIME type (to differentiate between html, js, css, and images in
the benchmark report).
Considering I've never used XS, and considering this is supposed to be
a limited scope project (as in, due soon), I'm going to have to give up
the hope of delivering a 100% correct solution. The Web Scraping Proxy
is still good idea, but I can't download it (I emailed the author about
that), and even if I could, the script it would generate would break as
soon as one of the sites changed.
-ofer
odigity@gmail.com wrote:
> I've done more poking around, and this is more difficult than I
> thought.
>
> WWW::Mechanize uses both HTML::Tree and HTML::Parser under the hood to
> do the parsing work. I've read up on both, so I'm now familiar with
> what both packages are capable of.
>
> I ran HTML::LinkExtor against my web document for fun, and discovered
> that there are also background images in various 'table' and 'td'
> elements that I need to grab as well. LinkExtor seems to do a good job
> of finding the links to all the resources that a browser would normally
> download. Here's a list of all of the cases I've discovered so far, in
> the form of element.attribute, with a note about typical resource type:
>
> frame.src # html
> iframe.src # html
> img.src # image
> link.href # css (usually)
> script.src # js (usually)
> table.background # image
> td.background # image
>
> So it looks like I'm going to have drop WWW::Mechanize in favor of my
> own custom solution combining LWP::UserAgent and HTML::LinkExtor, plus
> some careful manipulation of base urls.
>
> If this works out, it seems like a good candidate for a new module
> dedicated to executing a complete download of a page as a browser
> would. My test will be if the output matches the contents of a File ->
> Save Page As operation in Firefox (type: Web Page, complete).
>
> -ofer
>
> odigity@gmail.com wrote:
> > Hey, all.
> >
> > I'm using WWW::Mechanize for the first time because I have been tasked
> > with benchmarking the complete download time for my company's intranet
> > homepage from various branch offices. The problem is that this is a
> > very complex (horrible, ugly) page - multiple frames, images, external
> > CSS files, external javscript files, etc.
> >
> > I've got the frames and images part working successfully, but it seems
> > WWW::Mechanize doesn't have the ability to detect the link and script
> > tags that point to external resources.
> >
> > What's the best way to solve this problem? Do I need to switch to a
> > more powerful stand-alone HTML parsing module? Do I need to resort to
> > nasty page-specific regexs? Or is there an elegant solution I'm
> > overloooking?
> >
> > Thanks!
> >
> > -ofer
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 9924
***************************************
