[28565] in Perl-Users-Digest
Perl-Users Digest, Issue: 9929 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Fri Nov 3 18:10:22 2006
Date: Fri, 3 Nov 2006 15:10:14 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Fri, 3 Nov 2006 Volume: 10 Number: 9929
Today's topics:
Re: understanding an error <noreply@gunnar.cc>
Re: understanding an error <justin.0611@purestblue.com>
Re: understanding an error <justin.0611@purestblue.com>
Re: understanding an error <tadmc@augustmail.com>
Re: understanding an error <tadmc@augustmail.com>
Re: understanding an error <hjp-usenet2@hjp.at>
Re: Wrapper application s1037989@gmail.com
Re: Wrapper application <bik.mido@tiscalinet.it>
Re: Wrapper application <hjp-usenet2@hjp.at>
Re: WWW::Mechanize - simulating complete page download odigity@gmail.com
Re: WWW::Mechanize - simulating complete page download odigity@gmail.com
Re: WWW::Mechanize - simulating complete page download odigity@gmail.com
Re: WWW::Mechanize - simulating complete page download odigity@gmail.com
Re: WWW::Mechanize - simulating complete page download (reading news)
Re: WWW::Mechanize - simulating complete page download odigity@gmail.com
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Fri, 03 Nov 2006 20:36:07 +0100
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: understanding an error
Message-Id: <4r1k54FnukogU1@individual.net>
Michele Dondi wrote:
> Justin C wrote:
>>
>>print scalar ( %hash ), "\n" ;
> [snip]
>>The output, when this is run, is:
>>4/8
>
> As somebody put it, "the value of a hash in scalar context is of
> interest only to a perl programmer, not to a Perl programmer".
Sometimes I do:
if ( %hash ) {
# do something...
}
and I'm certainly not a perl programmer, so "somebody" must be wrong. ;-)
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: Fri, 03 Nov 2006 19:43:21 +0000
From: Justin C <justin.0611@purestblue.com>
Subject: Re: understanding an error
Message-Id: <justin.0611-1AD131.19432103112006@stigmata>
In article <4r0mibFov98sU1@news.dfncis.de>,
anno4000@radom.zrz.tu-berlin.de wrote:
> perldoc perldata, look for the paragraph starting
>
> If you evaluate a hash in scalar context,
Hmmm... I prefer Michele's answer!
--
Justin C, by the sea.
------------------------------
Date: Fri, 03 Nov 2006 19:47:43 +0000
From: Justin C <justin.0611@purestblue.com>
Subject: Re: understanding an error
Message-Id: <justin.0611-B58799.19474303112006@stigmata>
In article <2vamk2ppfov1afje9vm67svoooaubphb1g@4ax.com>,
Michele Dondi <bik.mido@tiscalinet.it> wrote:
> On Fri, 03 Nov 2006 10:17:23 -0000, Justin C
> <justin.0611@purestblue.com> wrote:
>
> >I was trying to count the number of elements in a hash,
>
> It would be important to know what you mean with "the number of
> elements in a hash": the number of keys()? If so just use that. The
> number of values? Well, that's the same. But if you mean the number of
> *distinct* values, i.e. the cardinality of the image, then you more or
> less fall back on the very FAQ about how to find the distinct elements
> from a *list*.
I did mean number of keys, thank you.
>
> As somebody put it, "the value of a hash in scalar context is of
> interest only to a perl programmer, not to a Perl programmer".
Clear and to the point, nice. Thanks.
--
Justin C, by the sea.
------------------------------
Date: Fri, 3 Nov 2006 14:24:39 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: understanding an error
Message-Id: <slrnekn9c7.va2.tadmc@tadmc30.august.net>
Michele Dondi <bik.mido@tiscalinet.it> wrote:
> As somebody put it, "the value of a hash in scalar context is of
> interest only to a perl programmer, not to a Perl programmer".
I lay claim to that one.
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Fri, 3 Nov 2006 14:26:12 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: understanding an error
Message-Id: <slrnekn9f4.va2.tadmc@tadmc30.august.net>
Gunnar Hjalmarsson <noreply@gunnar.cc> wrote:
> Michele Dondi wrote:
>> Justin C wrote:
>>>
>>>print scalar ( %hash ), "\n" ;
>> [snip]
>>>The output, when this is run, is:
>>>4/8
>>
>> As somebody put it, "the value of a hash in scalar context is of
>> interest only to a perl programmer, not to a Perl programmer".
>
> Sometimes I do:
>
> if ( %hash ) {
> # do something...
> }
>
> and I'm certainly not a perl programmer, so "somebody" must be wrong. ;-)
"somebody" would instead write:
if ( keys %hash ) {
just to avoid having a maintenance programmer have to go look up
what a hash in scalar context does.
:-)
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Fri, 3 Nov 2006 23:16:18 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: understanding an error
Message-Id: <slrneknfti.plj.hjp-usenet2@yoyo.hjp.at>
On 2006-11-03 20:26, Tad McClellan <tadmc@augustmail.com> wrote:
> Gunnar Hjalmarsson <noreply@gunnar.cc> wrote:
>> Michele Dondi wrote:
>>> As somebody put it, "the value of a hash in scalar context is of
>>> interest only to a perl programmer, not to a Perl programmer".
>>
>> Sometimes I do:
>>
>> if ( %hash ) {
>> # do something...
>> }
>>
>> and I'm certainly not a perl programmer, so "somebody" must be wrong. ;-)
Note, that you you aren't interested in the value itself, just whether
the value is true.
> "somebody" would instead write:
>
> if ( keys %hash ) {
>
> just to avoid having a maintenance programmer have to go look up
> what a hash in scalar context does.
That also turned out to be quite a bit faster in an application where I
needed to check the (approximate) size of a hash rather frequently.
Getting the number of elements in a hash is fast, counting used buckets
isn't. (or at least wasn't in the version of perl I used then)
hp
--
_ | Peter J. Holzer | > Wieso sollte man etwas erfinden was nicht
|_|_) | Sysadmin WSR | > ist?
| | | hjp@hjp.at | Was sonst wäre der Sinn des Erfindens?
__/ | http://www.hjp.at/ | -- P. Einstein u. V. Gringmuth in desd
------------------------------
Date: 3 Nov 2006 10:21:57 -0800
From: s1037989@gmail.com
Subject: Re: Wrapper application
Message-Id: <1162578117.603412.269830@e3g2000cwe.googlegroups.com>
Michele Dondi wrote:
> On 2 Nov 2006 12:26:50 -0800, s1037989@gmail.com wrote:
>
> >(Please ignore the uselessness and redunancy... This is a work in
> >progress and I just wanted to provide a real-life, working example for
>
> Do not worry! We recommend people to prepare minimal but functional
> examples all the time. Which I and many others simply regard as an act
> of common sense. If only everybody shared your common sense...
Great! :)
> >the benfit of others. Again, this code base can be used to extend the
> >functionality of squidGuard -- not recommended outside of prototyping.)
> >
> >use URI::Split qw(uri_split uri_join);
> >use IPC::Open2;
>
> (Do a favour to yourself and)
>
> use strict;
> use warnings;
Always!!
> as well. Even in a minimal example, if it's not really, say, a
> oneliner or so.
>
> >our $request;
> >our $response;
>
> BTW: why do you want them as package variables?
Well, for the more fully functional application, I was intending a ton
of subroutines and I didn't want to have to pass those variables into
the subroutine on EVERY call. It makes for messy code... Is this very
frowned upon?
> >my $pid = open2(\*CHLD_OUT, \*CHLD_IN, '/usr/bin/squidGuard', '-c',
> >'/etc/squid/squidGuard.conf');
>
> I would use lexical handles as is now commonly recommended with
> standard open().
Please explain? Does that mean to just drop the \*?
> >while ( chomp($request = <STDIN>) ) {
> > print CHLD_IN "$request\n";
> > chomp($response = <CHLD_OUT>);
> > $response = $request unless $response;
>
> Also
>
> $response ||= $request;
Yeah... Sometimes the logic growing in my mind doesn't let me do the
extremely obvious things! Ha ha!
> > message(0, $request eq $response ? "+++ OK: $request" : "---
> >ERR: $response");
>
> Err... message()? I'm afraid it's not a minimal, but *complete*, and
> working example, after all... ;-)
I thought about this one right after I sent it. All I CnP'd was the
main while loop as that is what the focus of this thread is. But I
figure people, as you have demonstrated, are smart enough to figure
that one out if they do want to use this code sample. :D
> (But since you actually did solve your problem, it shouldn't be a
> problem itself now.)
>
>
> Michele
> --
> {$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
> (($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
> .'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
> 256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: Fri, 03 Nov 2006 20:47:27 +0100
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: Wrapper application
Message-Id: <jv6nk2lvanpe8nuppullu8d5407lfe8pti@4ax.com>
On 3 Nov 2006 10:21:57 -0800, s1037989@gmail.com wrote:
>> >our $request;
>> >our $response;
>>
>> BTW: why do you want them as package variables?
>
>Well, for the more fully functional application, I was intending a ton
>of subroutines and I didn't want to have to pass those variables into
>the subroutine on EVERY call. It makes for messy code... Is this very
>frowned upon?
Well, yes and no: if you're really sure about what you're doing, then
just go on! But they can still be (file scoped) lexicals.
>> >my $pid = open2(\*CHLD_OUT, \*CHLD_IN, '/usr/bin/squidGuard', '-c',
>> >'/etc/squid/squidGuard.conf');
>>
>> I would use lexical handles as is now commonly recommended with
>> standard open().
>
>Please explain? Does that mean to just drop the \*?
my $pid = open2 my $chld_out, my $chld_in, @rest;
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: Fri, 3 Nov 2006 20:57:37 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: Wrapper application
Message-Id: <slrnekn7ph.l3s.hjp-usenet2@yoyo.hjp.at>
On 2006-11-03 18:21, s1037989@gmail.com <s1037989@gmail.com> wrote:
> Michele Dondi wrote:
>> On 2 Nov 2006 12:26:50 -0800, s1037989@gmail.com wrote:
>> >our $request;
>> >our $response;
>>
>> BTW: why do you want them as package variables?
>
> Well, for the more fully functional application, I was intending a ton
> of subroutines and I didn't want to have to pass those variables into
> the subroutine on EVERY call. It makes for messy code... Is this very
> frowned upon?
They can still be lexically scoped:
my $request;
my $response;
if you only access them in the same file.
>> >my $pid = open2(\*CHLD_OUT, \*CHLD_IN, '/usr/bin/squidGuard', '-c',
>> >'/etc/squid/squidGuard.conf');
>>
>> I would use lexical handles as is now commonly recommended with
>> standard open().
>
> Please explain? Does that mean to just drop the \*?
No. It means to use lexically scoped scalar variables:
my $pid = open2(my $chld_out, my $chld_in,
'/usr/bin/squidGuard',
'-c', '/etc/squid/squidGuard.conf');
hp
--
_ | Peter J. Holzer | > Wieso sollte man etwas erfinden was nicht
|_|_) | Sysadmin WSR | > ist?
| | | hjp@hjp.at | Was sonst wäre der Sinn des Erfindens?
__/ | http://www.hjp.at/ | -- P. Einstein u. V. Gringmuth in desd
------------------------------
Date: 3 Nov 2006 13:26:16 -0800
From: odigity@gmail.com
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <1162589176.534615.69410@h54g2000cwb.googlegroups.com>
Mumia W. (reading news) wrote:
> On 11/02/2006 08:07 PM, odigity@gmail.com wrote:
> > I'm an idiot. There are instructions as to how to obtain the
> > username/password on the page. :)
> >
> > -ofer
> >
>
> Bottom-post in this newsgroup.
> http://www.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
>
>
> --
> paduille.4060.mumia.w@earthlink.net
Thanks for the reminder. I just left a two year stint at a company
where everyone top-posted when replying to emails (drove me mad at
first). Now I have to unlearn a bad habit...
-ofer
------------------------------
Date: 3 Nov 2006 13:32:02 -0800
From: odigity@gmail.com
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <1162589522.578529.224150@m7g2000cwm.googlegroups.com>
Mark Clements wrote:
> odigity@gmail.com wrote:
> > Arg!
> >
> > Some of the URLs I need to benchmark contain instances of resources
> > being downloaded as the result of javascript code being executed. One
> > of them actually starts out as a small document containing a form and
> > some JS that determines the local time zone, inserts the value in a
> > hidden form variable, then submits the form. Lunatics.
> >
> > I don't think there's any combination of Perl modules or clever tricks
> > that will yield a 100% correct solution to the problem of behaving the
> > way a browser behaves given an initial URL. What I need is a real
> > browser. Ideally, a Perl interface to the Firefox engine that will
>
> Win32::IE::Mechanize
> Mozilla::Mechanize
>
> Have never used the latter, but the IE one works well.
>
> Mark
Fantastic! Will give them both a spin. Thanks!
-ofer
------------------------------
Date: 3 Nov 2006 13:53:11 -0800
From: odigity@gmail.com
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <1162590791.450318.292590@m7g2000cwm.googlegroups.com>
odigity@gmail.com wrote:
> Mark Clements wrote:
> > odigity@gmail.com wrote:
> > > Arg!
> > >
> > > Some of the URLs I need to benchmark contain instances of resources
> > > being downloaded as the result of javascript code being executed. One
> > > of them actually starts out as a small document containing a form and
> > > some JS that determines the local time zone, inserts the value in a
> > > hidden form variable, then submits the form. Lunatics.
> > >
> > > I don't think there's any combination of Perl modules or clever tricks
> > > that will yield a 100% correct solution to the problem of behaving the
> > > way a browser behaves given an initial URL. What I need is a real
> > > browser. Ideally, a Perl interface to the Firefox engine that will
> >
> > Win32::IE::Mechanize
> > Mozilla::Mechanize
> >
> > Have never used the latter, but the IE one works well.
> >
> > Mark
>
> Fantastic! Will give them both a spin. Thanks!
>
> -ofer
Doh.
Unfortunately, neither of these solve my problem. They are merely
reimplementations of the WWW::Mechanize functionality using IE and
Mozilla as user agents instead of LWP::UserAgent (not even sure why
that's useful...), and therefore they lack the abilities I need in the
same way WWW::Mechanize does.
-ofer
------------------------------
Date: 3 Nov 2006 13:58:09 -0800
From: odigity@gmail.com
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <1162591089.734475.27280@i42g2000cwa.googlegroups.com>
Tad McClellan wrote:
> odigity@gmail.com <odigity@gmail.com> wrote:
> > Hey, all.
> >
> > I'm using WWW::Mechanize for the first time because I have been tasked
> > with benchmarking the complete download time for my company's intranet
> > homepage from various branch offices. The problem is that this is a
> > very complex (horrible, ugly) page - multiple frames, images, external
> > CSS files, external javscript files, etc.
> >
> > I've got the frames and images part working successfully, but it seems
> > WWW::Mechanize doesn't have the ability to detect the link
>
>
> A simple search of the modules docs shows 5 methods with "link"
> in their names:
>
> grep ^= Mechanize.pod | grep link
>
> =head2 $mech->links()
> =head2 $mech->follow_link(...)
> =head2 $mech->find_link()
> =head2 $mech->find_all_links( ... )
> =head2 $mech->_extract_links()
>
> I'd read up on what those do...
>
>
> > and script
> > tags that point to external resources.
>
>
> If you mean JavaScript, then yes, Mechanize cannot find those.
>
>
> > What's the best way to solve this problem?
>
>
> If it was my task, I'd let a Real Browser figure out what all
> needs to be fetched by using the
>
> Web Scraping Proxy
>
> http://www.research.att.com/~hpk/wsp/
>
>
> --
> Tad McClellan SGML consulting
> tadmc@augustmail.com Perl programming
> Fort Worth, Texas
The fun doesn't stop.
I fired up the Web Scraping Proxy and took it 'round the block for a
test spin. It works great... except for NTLM protected sites, which
are the kind of sites I'm trying to benchmark.
I have perl code working that accesses those sites with LWP::UserAgent,
LWP::UserAgent::Ntlm, and Authen::NTLM. Unfortunately, this method is
incompatible with the WSP because it uses a completely custom solution
involving sockets - in other words, it doesn't truly speak HTTP. I'm
not sure how to begin hacking NTLM support onto it.
I'll email the author and see what he says. In the meantime, I'll
continue to work on my LWP/HTML::LinkExtor-based 90% solution.
-ofer
------------------------------
Date: Fri, 03 Nov 2006 22:55:47 GMT
From: "Mumia W. (reading news)" <paduille.4060.mumia.w@earthlink.net>
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <TNP2h.2031$l25.1706@newsread4.news.pas.earthlink.net>
On 11/03/2006 03:53 PM, odigity@gmail.com wrote:
> odigity@gmail.com wrote:
>> Mark Clements wrote:
>>> odigity@gmail.com wrote:
>>> > Arg!
>>> >
>>> > Some of the URLs I need to benchmark contain instances of resources
>>> > being downloaded as the result of javascript code being executed. One
>>> > of them actually starts out as a small document containing a form and
>>> > some JS that determines the local time zone, inserts the value in a
>>> > hidden form variable, then submits the form. Lunatics.
>>> >
>>> > I don't think there's any combination of Perl modules or clever tricks
>>> > that will yield a 100% correct solution to the problem of behaving the
>>> > way a browser behaves given an initial URL. What I need is a real
>>> > browser. Ideally, a Perl interface to the Firefox engine that will
>>>
>>> Win32::IE::Mechanize
>>> Mozilla::Mechanize
>>>
>>> Have never used the latter, but the IE one works well.
>>>
>>> Mark
>> Fantastic! Will give them both a spin. Thanks!
>>
>> -ofer
>
> Doh.
>
> Unfortunately, neither of these solve my problem. They are merely
> reimplementations of the WWW::Mechanize functionality using IE and
> Mozilla as user agents instead of LWP::UserAgent (not even sure why
> that's useful...), and therefore they lack the abilities I need in the
> same way WWW::Mechanize does.
>
> -ofer
>
I think that Mozilla (Firefox) is scriptable, but you have to get into
the developer documents to learn how to do that.
http://developer.mozilla.org/
--
paduille.4060.mumia.w@earthlink.net
------------------------------
Date: 3 Nov 2006 15:01:52 -0800
From: odigity@gmail.com
Subject: Re: WWW::Mechanize - simulating complete page download
Message-Id: <1162594912.614114.90050@h54g2000cwb.googlegroups.com>
odigity@gmail.com wrote:
> Tad McClellan wrote:
> > odigity@gmail.com <odigity@gmail.com> wrote:
> > > Hey, all.
> > >
> > > I'm using WWW::Mechanize for the first time because I have been tasked
> > > with benchmarking the complete download time for my company's intranet
> > > homepage from various branch offices. The problem is that this is a
> > > very complex (horrible, ugly) page - multiple frames, images, external
> > > CSS files, external javscript files, etc.
> > >
> > > I've got the frames and images part working successfully, but it seems
> > > WWW::Mechanize doesn't have the ability to detect the link
> >
> >
> > A simple search of the modules docs shows 5 methods with "link"
> > in their names:
> >
> > grep ^= Mechanize.pod | grep link
> >
> > =head2 $mech->links()
> > =head2 $mech->follow_link(...)
> > =head2 $mech->find_link()
> > =head2 $mech->find_all_links( ... )
> > =head2 $mech->_extract_links()
> >
> > I'd read up on what those do...
> >
> >
> > > and script
> > > tags that point to external resources.
> >
> >
> > If you mean JavaScript, then yes, Mechanize cannot find those.
> >
> >
> > > What's the best way to solve this problem?
> >
> >
> > If it was my task, I'd let a Real Browser figure out what all
> > needs to be fetched by using the
> >
> > Web Scraping Proxy
> >
> > http://www.research.att.com/~hpk/wsp/
> >
> >
> > --
> > Tad McClellan SGML consulting
> > tadmc@augustmail.com Perl programming
> > Fort Worth, Texas
>
> The fun doesn't stop.
>
> I fired up the Web Scraping Proxy and took it 'round the block for a
> test spin. It works great... except for NTLM protected sites, which
> are the kind of sites I'm trying to benchmark.
>
> I have perl code working that accesses those sites with LWP::UserAgent,
> LWP::UserAgent::Ntlm, and Authen::NTLM. Unfortunately, this method is
> incompatible with the WSP because it uses a completely custom solution
> involving sockets - in other words, it doesn't truly speak HTTP. I'm
> not sure how to begin hacking NTLM support onto it.
>
> I'll email the author and see what he says. In the meantime, I'll
> continue to work on my LWP/HTML::LinkExtor-based 90% solution.
>
> -ofer
Follow-up:
The author of WSP (Howard Katseff) informed me that WSP v2 uses
LWP::UserAgent under the hood. I downloaded v2 and modified it to
support NTLM for the domains I need to access, and it seems to be
working.
-ofer
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 9929
***************************************
