[24126] in Perl-Users-Digest
Perl-Users Digest, Issue: 6320 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sun Mar 28 18:25:49 2004
Date: Sun, 28 Mar 2004 15:25:12 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sun, 28 Mar 2004 Volume: 10 Number: 6320
Today's topics:
LWP cookies <rbell01824@comcast.net>
Re: LWP cookies <postmaster@castleamber.com>
Re: LWP cookies <rbell01824@comcast.net>
Re: LWP cookies <uri@stemsystems.com>
Re: LWP cookies <rbell01824@comcast.net>
Re: LWP cookies <flavell@ph.gla.ac.uk>
Perl Serial io on linux (T.Paakki)
Re: Perl Serial io on linux <wherrera@lynxview.com>
Re: problem/help with using LWP <rbell01824@earthlink.net>
Re: Quality control for CPAN? <m@lteubl.de>
Re: shared libraries compatibility <noreply@gunnar.cc>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sun, 28 Mar 2004 17:41:46 GMT
From: Richard Bell <rbell01824@comcast.net>
Subject: LWP cookies
Message-Id: <u7E9c.24921$K91.69732@attbi_s02>
I'm using LWP to scrape some data from a DB over the web. Eventually I
end up executing the following code:
#!/usr/bin/perl
use warnings;
use strict;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
<much code deleted>
### request number 18 ###
my $req18 = HTTP::Request->new(POST => '
http://www2.fdic.gov/sdi/download_exect.asp ');
$req18->header('Host' => 'www2.fdic.gov');
$req18->header('User-Agent' => 'Mozilla/5.0 (X11; U; Linux i686; en-US;
rv:1.4.1) Gecko/20031030');
$req18->header('Accept' =>
'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1');
$req18->header('Accept-Language' => 'en-us,en;q=0.5');
$req18->header('Accept-Encoding' => 'gzip,deflate');
$req18->header('Accept-Charset' => 'ISO-8859-1,utf-8;q=0.7,*;q=0.7');
$req18->header('Keep-Alive' => '300');
$req18->header('Proxy-Connection' => 'keep-alive');
$req18->header('Referer' =>
'http://www2.fdic.gov/sdi/download_FindInsts.asp');
$req18->header('Cookie' => 'CustRepUserID=; InstSelected=Yes;
HasCustReps=Yes; RepSelChanged=No; ActiveRepName=;
rep3x=9000=&8000=&7000=&6000=&5000=&99=namehcr%2Coffdom%2Cofffor%2Cstmult%2Cwebaddr%2C&30=nimy%2Croa%2Croe%2Ceeffr%2Castempm%2C&4000=&3000=;
rep1x=16000=&12100=&16=&15=&16300=&14=dep%2Cdepfor%2Cdepdom%2Ctrn%2Cddt%2Cdepsmamt%2Cdepsmb%2Cdeplgamt%2Cdeplgb%2C&16100=&12200=&1010=&13=&12000=&12=&11000=&11=&1000=&16200=&12300=&10000=&1110=&10=numemp%2Casset%2C;
rep2x=23=&26000=&27500=&24000=&24=&27100=&22=&27400=&22000=&21=&27200=&21000=&25=&27300=&25000=&20=&27000=&20000=&2000=;
CPG=HASCPG=No&CPGUSERID=64EEB2F8EDBEFB37;
rep0=USERREP7=B%26As&USERREP6=pdpa&USERREP5=EPGSales02&USERREP4=aa&USERREP3=bh10%2D28report&USERREP2=Test+MS+Security+Patches&USERREP8=BOCCPeers&USERREP1=random;
SetCookie=OK; ASPSESSIONIDSDADQBAB=BALONNDDJHGCHFNNEELFLCMD');
$req18->header('Content-Type' => 'application/x-www-form-urlencoded');
$req18->header('Content-Length' => '384');
$req18->content('SQL=SELECT+stru.Cert+from+risstru+stru++where+stru.Active+%3D+1+++++++++++and+stru.repdte+%3D+%276%2F30%2F2003%27++order+by+stru.NAME+&inFlags=&selections=10%2Cnumemp%2Casset%2C14%2Cdep%2Cdepfor%2Cdepdom%2Ctrn%2Cddt%2Cdepsmamt%2Cdepsmb%2Cdeplgamt%2Cdeplgb%2C30%2Cnimy%2Croa%2Croe%2Ceeffr%2Castempm%2C99%2Cnamehcr%2Coffdom%2Cofffor%2Cstmult%2Cwebaddr&Period1=6%2F30%2F2003&IncomeBasis=');
print "<-------------------------------------------->";
print $ua->request($req18)->as_string;
Amongst other things, this code defines a number of headers including
one for cookies. I'm unclear on several things and could use some help
from someone who actually knows how LWP does its magic with regard to
cookes.
I assume the form is not posted until the line
print $ua->request($req18)->as_string;
executes. Is this correct?
The cookies header contains a bunch of stuff, including a "session
cookie" that expires. While I'm not entirely sure, I think it's this bit:
ASPSESSIONIDSDADQBAB=BALONNDDJHGCHFNNEELFLCMD
That is the cookie and value when this code was originally got (some
days ago) using wsp.pl and template.pl. But the cookie, of course,
expires. If I revisit the page that issues the cookie (a new value for
ASPSESSIONIDSDADQBAB) will the new value be sent along as part of the
'request' or do I need to modify the Cookie header? Does it 'override'
the value specified in the header?
This is causing me considerable confusion as when I run a perl script
that goes through all 18 pages/forms based on wsp and template run some
days ago (with a now expired session cookie), I'm able to fetch the data
OK. Clearly some current correct session cookie is being sent and all
is well. Unfortunately, when I run a perl script that only visits the
page that seems to be issuing ASPSESSIONIDSDADQBAB and then run the
above code (just $req18 stuff), I get a return page that indicates that
the session has expired. Any clues?
Thanks
R
------------------------------
Date: Sun, 28 Mar 2004 11:53:34 -0600
From: John Bokma <postmaster@castleamber.com>
Subject: Re: LWP cookies
Message-Id: <4067111e$0$24340$58c7af7e@news.kabelfoon.nl>
Richard Bell wrote:
[cookies]
IIRC lwp can perfectly handle cookies. RTFM. I wrote a script some time
ago, and no problems with the cookies. It seems you are trying to
reinvent the wheel.
--
John personal page: http://johnbokma.com/
Freelance Perl / Java developer available - http://castleamber.com/
------------------------------
Date: Sun, 28 Mar 2004 18:18:55 GMT
From: Richard Bell <rbell01824@comcast.net>
Subject: Re: LWP cookies
Message-Id: <jGE9c.25254$JO3.26898@attbi_s04>
John Bokma wrote:
> Richard Bell wrote:
>
> [cookies]
>
> IIRC lwp can perfectly handle cookies. RTFM. I wrote a script some time
> ago, and no problems with the cookies. It seems you are trying to
> reinvent the wheel.
>
John,
Thanks for the response. As the original post clearlly indicates, I'm
aware that LWP does indeed handle cookies. The issue has to do with
cookies that expire and those that are forced (including expired
cookies) via the header v. those that are reloaded as a result of
visiting a page with cookies enabled. Can you provide any insight in
this regard?
R
------------------------------
Date: Sun, 28 Mar 2004 20:28:01 GMT
From: Uri Guttman <uri@stemsystems.com>
Subject: Re: LWP cookies
Message-Id: <x74qs8vhri.fsf@mail.sysarch.com>
>>>>> "RB" == Richard Bell <rbell01824@comcast.net> writes:
RB> Thanks for the response. As the original post clearlly indicates,
RB> I'm aware that LWP does indeed handle cookies. The issue has to
RB> do with cookies that expire and those that are forced (including
RB> expired cookies) via the header v. those that are reloaded as a
RB> result of visiting a page with cookies enabled. Can you provide
RB> any insight in this regard?
with lwp you maintain a cookie jar object which you can stuff with
cookies from the fetched pages or your own. so you can manage all the
cookies you want. you can create a jar full of all the cookies you have
been sent and extract a subset into another jar for any pages you
want. the header calls you had there made my head spin!! major wheel
invention going on. i doubt you need any of them at all and lwp will
send the cookies you want without trouble. and with the http::cookie
object you can examine and set all the fields including timestamps and
such.
uri
--
Uri Guttman ------ uri@stemsystems.com -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
------------------------------
Date: Sun, 28 Mar 2004 21:09:16 GMT
From: Richard Bell <rbell01824@comcast.net>
Subject: Re: LWP cookies
Message-Id: <%9H9c.25358$w54.166857@attbi_s01>
Uri Guttman wrote:
>>>>>>"RB" == Richard Bell <rbell01824@comcast.net> writes:
>
>
> RB> Thanks for the response. As the original post clearlly indicates,
> RB> I'm aware that LWP does indeed handle cookies. The issue has to
> RB> do with cookies that expire and those that are forced (including
> RB> expired cookies) via the header v. those that are reloaded as a
> RB> result of visiting a page with cookies enabled. Can you provide
> RB> any insight in this regard?
>
> with lwp you maintain a cookie jar object which you can stuff with
> cookies from the fetched pages or your own. so you can manage all the
> cookies you want. you can create a jar full of all the cookies you have
> been sent and extract a subset into another jar for any pages you
> want. the header calls you had there made my head spin!! major wheel
> invention going on. i doubt you need any of them at all and lwp will
> send the cookies you want without trouble. and with the http::cookie
> object you can examine and set all the fields including timestamps and
> such.
>
> uri
>
Uri,
That is helpful. As I understand it, notwithstanding any cookie I
specify in my header, LWP will send its own most recent version
(assuming it has one). Is this correct? Presumably, my header
specified cookie is simply ignored. Is this correct?
As to the headers cookies et al. They are the result of what I would
get if I actually visited 18 pages of forms in turn to build the query I
post. Moreover, the server is VERY picky about what it will dance with.
In deference to reality I just send them all along. It is worth
noting that many of the cookies hold the context associated with what
would happen if I in fact visited all 18 pages (so they are in that
sense necessary for the POST to work, hardly, as you mentioned, wheel
invention). Since I don't in fact visit the pages and since LWP starts
its cookie jar fresh (BTW, it wouldn't matter if I saved the cookies as
the session cookie would expire in any case) I force their value and
revisit the page that issues the current session cookie to freshen it.
None the less, I remained concern that the current session cookie is
being sent given that I force what is surely a bogus value in my header.
It seems to be working OK, but I'm concerned to understand why and to
reassure myself that it will continue to do so.
Thanks.
R
------------------------------
Date: Sun, 28 Mar 2004 23:05:21 +0100
From: "Alan J. Flavell" <flavell@ph.gla.ac.uk>
Subject: Re: LWP cookies
Message-Id: <Pine.LNX.4.53.0403282256220.12330@ppepc56.ph.gla.ac.uk>
On Sun, 28 Mar 2004, Richard Bell wrote:
> As to the headers cookies et al. They are the result of what I would
> get if I actually visited 18 pages of forms in turn to build the query I
> post. Moreover, the server is VERY picky about what it will dance with.
If it works with a browser, I'm not sure why it wouldn't work with
LWP. The theory is that you just have to call it in the right way
(unless the server is misbehaving - in which case you're on entirely
the wrong group for discussing WWW server problems).
> In deference to reality I just send them all along.
If they're in scope for the request, and they're in the cookie jar
already, the module should know what to do with them. Duplicating
that function does indeed look like re-inventing the wheel, and the
code was anything -but- aesthetic - which is often a warning sign.[1]
> It is worth noting that many of the cookies hold the context
> associated with what would happen if I in fact visited all 18 pages
> (so they are in that sense necessary for the POST to work, hardly,
> as you mentioned, wheel invention).
But surely the point is that you want to present the POST request 'as
if' you had visited those pages - and that's done by pre-loading the
cookie jar with the appropriate cookies.
good luck
[1] even if, speaking as a physicist, I can "write FORTRAN in any
language" ;-)
------------------------------
Date: 28 Mar 2004 07:30:04 -0800
From: trp24@hotmail.com (T.Paakki)
Subject: Perl Serial io on linux
Message-Id: <6358a14b.0403280730.2ee72a6b@posting.google.com>
I recently bought a serial port attached touchscreen LCD display and
wanted to control it via perl on Linux (Zaurus).
Has anyone done this before? I was thinking it would go something
like this:
Display Splash on touchscreen
Wait for tap on touchscreen
read from local file containing temperature values
clear display
wait for response
display temp screen
loop until done is pressed
The only examples I've seen detail modem communications or just a
terminal, I'm looking for automated control.
any help would be appreciated!
Thanks
Travis
------------------------------
Date: Sun, 28 Mar 2004 12:54:32 -0700
From: Bill <wherrera@lynxview.com>
Subject: Re: Perl Serial io on linux
Message-Id: <B-2dnbBlo7A0sPrdRVn-tA@adelphia.com>
T.Paakki wrote:
> I recently bought a serial port attached touchscreen LCD display and
> wanted to control it via perl on Linux (Zaurus).
>
> Has anyone done this before? I was thinking it would go something
> like this:
...
> read from local file containing temperature values
> clear display
> wait for response
> display temp screen
> loop until done is pressed
>
> The only examples I've seen detail modem communications or just a
> terminal, I'm looking for automated control.
Can you run Expect on a Zaurus?
------------------------------
Date: Sun, 28 Mar 2004 16:50:34 GMT
From: Richard Bell <rbell01824@earthlink.net>
Subject: Re: problem/help with using LWP
Message-Id: <gsvd60poj86o2ij0lbrefpi3h0nl81ce9v@4ax.com>
Thanks for responding.
As I mentioned in my original post, I've already used wsp (that's how
I got this far). The issue now is how to find out of a list of
roughly 50 or so cookies, which one is the session cookie that times
out. I've spent some time reading wsp's -v output and can see the
cookie traffic. What I'm not clear on is how to tell which one is
getting me in trouble.
BTW, when I installed wsp, I got some test errors and only got it to
run by forcing the make install. I'm running Fedora. Is this
expected behavior? Is forcing the make install an appropriate
solution?
I could also use some clues as to how to get a current session cookie
(I assume LWP to the page that originally dispenses it) and then place
it in the appropriate header line of the POST to actually get the data
I really want. Has anyone had any experience in this regard? What
are the problems/issues/techniques that I should be aware of?
I'll take another look at mechanize. I'd hoped to avoid working my
way through the entire form process as the form to POST to actually
get the data I want always has exactly the same content. The issue
seems to be purely one of the session cookie, not the form (or for
that matter the navigation).
BTW, I should add that I've recently returned to *nix and perl after
many years away, so I'm not experienced what most of what is new.
Please, forgive what may seem nieve questions.
Thanks again.
R
On Sun, 28 Mar 2004 04:39:59 GMT, Uri Guttman <uri@stemsystems.com>
wrote:
>>>>>> "BW" == Bob Walton <invalid-email@rochester.rr.com> writes:
>
> BW> Richard Bell wrote:
> >> I'm trying to use LWP to scrape some data from a web site fronting a
>
>
> BW> Then you should consider using a web scaping proxy:
> BW> http://www.research.att.com/~hpk/wsp/
>
>and check out WWW::Mechanize and its related modules. they make
>following all those links and filling forms much easier. it is based on
>LWP so you get all of that with many extra features designed to make
>scraping and such very easy.
>
>uri
------------------------------
Date: Sun, 28 Mar 2004 15:05:00 +0200
From: Malte Ubl <m@lteubl.de>
Subject: Re: Quality control for CPAN?
Message-Id: <c46lid$gs$1@news.dtag.de>
Garry Heaton wrote:
> I've had problems getting some CPAN modules to compile in the past. Is there
> any way of deciding which CPAN modules are more or less reliable other than
> trial 'n error?
On http://search.cpan.org/ distribution can be rated and reviewed (Those
features are rather new) and you can see how the test suite performed
for others.
malte
------------------------------
Date: Sun, 28 Mar 2004 13:42:36 +0200
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: shared libraries compatibility
Message-Id: <c46dna$2fqrnt$1@ID-184292.news.uni-berlin.de>
Tassilo v. Parseval wrote:
> Also sprach Gunnar Hjalmarsson:
>> HTML::Parser is both a standard module,
>
> HTML::Parser is not a standard module.
Sorry, I draw a conclusion from the fact that it's included at
http://www.perldoc.com/perl5.8.0/lib/HTML/Parser.html
> The fact that it is written in C will indeed make distributing the
> script very difficult if he intends to include all prerequisite
> modules. It's therefore easier to have the end user install it on
> its own. Maybe providing some detailed explanations on how to do
> that on the various platforms is better.
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 6320
***************************************
