[26269] in Perl-Users-Digest
Perl-Users Digest, Issue: 8452 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sat Sep 24 14:10:15 2005
Date: Sat, 24 Sep 2005 11:10:08 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sat, 24 Sep 2005 Volume: 10 Number: 8452
Today's topics:
Re: Website scraper <tadmc@augustmail.com>
Re: Website scraper <steve@uptime.org.uk>
Re: Website scraper <1usa@llenroc.ude.invalid>
Re: Website scraper <steve@uptime.org.uk>
Re: Website scraper <1usa@llenroc.ude.invalid>
Re: Website scraper <steve@uptime.org.uk>
Re: Website scraper <matthew.garrish@sympatico.ca>
Re: Website scraper <matthew.garrish@sympatico.ca>
Re: Website scraper <steve@uptime.org.uk>
Re: Website scraper <matthew.garrish@sympatico.ca>
Re: Website scraper <dvh@dvhdvhdvh.dvh>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 24 Sep 2005 09:24:43 -0500
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: Website scraper
Message-Id: <slrndjaodb.s3v.tadmc@magna.augustmail.com>
[ Newsgroups trimmed. I don't do the alt.* hierarchy ]
Stephen Hildrey <steve@uptime.org.uk> wrote:
> DVH wrote:
>> $headline = $stream->get_trimmed_text('/b') \
>> if ($tag->[1]{class} =~ /^h[12]$/);
>>
>> I don't understand what that backslash is doing at the end of the first
>> line.
>
> I think the author got mixed up between Perl and shell scripting - where
> '\' is used to continue across newlines.
So, the backslash at the end of the line is escaping the newline that
follows it (but there is no need to escape that newline, so it does
not do anything that is useful).
> $headline = $stream->get_trimmed_text('/b')
> if ($tag->[1]{class} =~ /^h[12]$/);
>
> > And I don't see where the loop following the "if" in the second line
^^^^^^^^
> > actually begins - shouldn't it begin with a curly bracket?
>
> It's an example of Perl's "statement if (cond)" syntax.
Note that there is no "loop" in the code the OP showed.
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Sat, 24 Sep 2005 16:11:14 +0100
From: Stephen Hildrey <steve@uptime.org.uk>
Subject: Re: Website scraper
Message-Id: <1127574674.36019.0@demeter.uk.clara.net>
Tad McClellan wrote:
> Stephen Hildrey <steve@uptime.org.uk> wrote:
>>DVH wrote:
>>>$headline = $stream->get_trimmed_text('/b') \
>>> if ($tag->[1]{class} =~ /^h[12]$/);
>>>
>>>I don't understand what that backslash is doing at the end of the first
>>>line.
>>
>>I think the author got mixed up between Perl and shell scripting - where
>>'\' is used to continue across newlines.
>
> So, the backslash at the end of the line is escaping the newline that
> follows it (but there is no need to escape that newline, so it does
> not do anything that is useful).
No. This is Perl - the backslash is a syntax error:
$ cat > backslash.pl << _EOF && perl backslash.pl
> use strict;
> use warnings;
> my $foo = "foo" \
> if (1);
> _EOF
syntax error at backslash.pl line 3, near "my ="
Execution of backslash.pl aborted due to compilation errors.
>>$headline = $stream->get_trimmed_text('/b')
>> if ($tag->[1]{class} =~ /^h[12]$/);
>>
>>>And I don't see where the loop following the "if" in the second line
>
> ^^^^^^^^
>
>>>actually begins - shouldn't it begin with a curly bracket?
>>
>>It's an example of Perl's "statement if (cond)" syntax.
>
> Note that there is no "loop" in the code the OP showed.
Good spot. I assume he meant "block".
OP: if you are still experiencing difficulties with the code, do post
back - I'm sure we'll be able to help :-)
Steve
--
Stephen Hildrey
E-mail: steve@uptime.org.uk / Tel: +442071931337
Jabber: steve@jabber.earth.li / MSN: foo@hotmail.co.uk
------------------------------
Date: Sat, 24 Sep 2005 15:25:12 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: Website scraper
Message-Id: <Xns96DB742C5CD26asu1cornelledu@127.0.0.1>
"DVH" <dvh@dvhdvhdvh.dvh> wrote in
news:dh34hg$d0i$1@nwrdmz01.dmz.ncs.ea.ibs-infra.bt.com:
> The script is meant to scrape headlines from the BBC website and put
> them into an RSS feed.
http://news.bbc.co.uk/rss/newsonline_world_edition/americas/rss.xml
--
A. Sinan Unur <1usa@llenroc.ude.invalid>
(reverse each component and remove .invalid for email address)
comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: Sat, 24 Sep 2005 16:36:23 +0100
From: Stephen Hildrey <steve@uptime.org.uk>
Subject: Re: Website scraper
Message-Id: <1127576183.17734.0@damia.uk.clara.net>
A. Sinan Unur wrote:
> "DVH" <dvh@dvhdvhdvh.dvh> wrote in
> news:dh34hg$d0i$1@nwrdmz01.dmz.ncs.ea.ibs-infra.bt.com:
>>The script is meant to scrape headlines from the BBC website and put
>>them into an RSS feed.
> http://news.bbc.co.uk/rss/newsonline_world_edition/americas/rss.xml
A valid point, well made :-)
Still, I think scraping is a useful technique to be aware of - I read
that same article myself, and since have found numerous uses for
scraping [1]:
1. Being paid to write news aggregators,
2. Getting text-message notifications in response to various ebay
events,
3. Being able to enjoy a night in the pub, despite $airline having
lost my luggage (scrape the lost-luggage tracking site, send SMS
hourly :-) )
The possibilities are endless!
Steve
[1] - yes, this may be a bit of a "grey area" in some AUPs/ToSs - YMMV.
--
Stephen Hildrey
E-mail: steve@uptime.org.uk / Tel: +442071931337
Jabber: steve@jabber.earth.li / MSN: foo@hotmail.co.uk
------------------------------
Date: Sat, 24 Sep 2005 15:46:16 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: Website scraper
Message-Id: <Xns96DB77BEEAD2Fasu1cornelledu@127.0.0.1>
Stephen Hildrey <steve@uptime.org.uk> wrote in news:1127576183.17734.0
@damia.uk.clara.net:
> A. Sinan Unur wrote:
>> "DVH" <dvh@dvhdvhdvh.dvh> wrote in
>> news:dh34hg$d0i$1@nwrdmz01.dmz.ncs.ea.ibs-infra.bt.com:
>
>>>The script is meant to scrape headlines from the BBC website and put
>>>them into an RSS feed.
>
>> http://news.bbc.co.uk/rss/newsonline_world_edition/americas/rss.xml
>
> A valid point, well made :-)
>
> Still, I think scraping is a useful technique to be aware of - I read
> that same article myself,
Agreed. I was doing it before I knew it was called scraping. Even MS did
it (in the form of being able to import data from HTML tables into Excel
given a page URL).
If the RSS feed exists in the first place, why not go ahead and use it
without mucking about with the internals of some HTML code?
Sinan
--
A. Sinan Unur <1usa@llenroc.ude.invalid>
(reverse each component and remove .invalid for email address)
comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: Sat, 24 Sep 2005 16:58:55 +0100
From: Stephen Hildrey <steve@uptime.org.uk>
Subject: Re: Website scraper
Message-Id: <1127577535.10927.0@nnrp-t71-02.news.clara.net>
A. Sinan Unur wrote:
> If the RSS feed exists in the first place, why not go ahead and use it
> without mucking about with the internals of some HTML code?
1. If the HTML exposes some information not present in the RSS.
2. Note - there wasn't a BBC RSS feed at the time that article
was written (November 2001).
3. The OP says he wants to "adapt the example script", so I don't
know that he is even going to use it to scrape the BBC.
But I agree in principle with your point - in-house RSS feeds generated
from back-end data sources are far more robust than a bespoke solution
that is based on data acquired through an attempt to reverse a
data-to-presentation transform.
Steve
--
Stephen Hildrey
E-mail: steve@uptime.org.uk / Tel: +442071931337
Jabber: steve@jabber.earth.li / MSN: foo@hotmail.co.uk
------------------------------
Date: Sat, 24 Sep 2005 12:22:20 -0400
From: "Matt Garrish" <matthew.garrish@sympatico.ca>
Subject: Re: Website scraper
Message-Id: <_2fZe.307$Bi.106745@news20.bellglobal.com>
"Stephen Hildrey" <steve@uptime.org.uk> wrote in message
news:1127574674.36019.0@demeter.uk.clara.net...
> Tad McClellan wrote:
>> Stephen Hildrey <steve@uptime.org.uk> wrote:
>>>DVH wrote:
>>>>$headline = $stream->get_trimmed_text('/b') \
>>>> if ($tag->[1]{class} =~ /^h[12]$/);
>>>>
>>>>I don't understand what that backslash is doing at the end of the first
>>>>line.
>>>
>>>I think the author got mixed up between Perl and shell scripting - where
>>>'\' is used to continue across newlines.
>>
>> So, the backslash at the end of the line is escaping the newline that
>> follows it (but there is no need to escape that newline, so it does
>> not do anything that is useful).
>
> No. This is Perl - the backslash is a syntax error:
>
> $ cat > backslash.pl << _EOF && perl backslash.pl
> > use strict;
> > use warnings;
> > my $foo = "foo" \
> > if (1);
> > _EOF
> syntax error at backslash.pl line 3, near "my ="
> Execution of backslash.pl aborted due to compilation errors.
>
You shouldn't make claims you can't substantiate...
my $time = localtime(\
time);
print $time;
The interpreter can usually understand what you're trying to do, but in your
case you broke the line before the conditional "if" and it's not going to
assume that's what you meant so it gives an error. You can, however, break
it anywhere else:
my $foo = \
"foo" if (1);
or
my $foo = "foo" if
(1);
Matt
------------------------------
Date: Sat, 24 Sep 2005 12:24:14 -0400
From: "Matt Garrish" <matthew.garrish@sympatico.ca>
Subject: Re: Website scraper
Message-Id: <N4fZe.308$Bi.106896@news20.bellglobal.com>
"Matt Garrish" <matthew.garrish@sympatico.ca> wrote in message
news:_2fZe.307$Bi.106745@news20.bellglobal.com...
>
> my $foo = "foo" if
my $foo = "foo" if \
That's what I get for typing...
Matt
------------------------------
Date: Sat, 24 Sep 2005 17:45:34 +0100
From: Stephen Hildrey <steve@uptime.org.uk>
Subject: Re: Website scraper
Message-Id: <1127580334.17298.1@ersa.uk.clara.net>
Matt Garrish wrote:
> "Stephen Hildrey" <steve@uptime.org.uk> wrote in message
> news:1127574674.36019.0@demeter.uk.clara.net...
>
>>Tad McClellan wrote:
>>
>>>Stephen Hildrey <steve@uptime.org.uk> wrote:
>>>
>>>>DVH wrote:
>>>>
>>>>>$headline = $stream->get_trimmed_text('/b') \
>>>>> if ($tag->[1]{class} =~ /^h[12]$/);
>>>>>
>>>>>I don't understand what that backslash is doing at the end of the first
>>>>>line.
>>>>
>>>>I think the author got mixed up between Perl and shell scripting - where
>>>>'\' is used to continue across newlines.
>>>
>>>So, the backslash at the end of the line is escaping the newline that
>>>follows it (but there is no need to escape that newline, so it does
>>>not do anything that is useful).
>>
>>No. This is Perl - the backslash is a syntax error:
>>
>> $ cat > backslash.pl << _EOF && perl backslash.pl
>> > use strict;
>> > use warnings;
>> > my $foo = "foo" \
>> > if (1);
>> > _EOF
>> syntax error at backslash.pl line 3, near "my ="
>> Execution of backslash.pl aborted due to compilation errors.
>>
>
>
> You shouldn't make claims you can't substantiate...
>
> my $time = localtime(\
> time);
> print $time;
>
> The interpreter can usually understand what you're trying to do, but in your
> case you broke the line before the conditional "if" and it's not going to
> assume that's what you meant so it gives an error. You can, however, break
> it anywhere else:
Sorry if I was ambiguous - I was trying to maintain the structure of the
code in the OP's example, and not talking about the general case.
Steve
--
Stephen Hildrey
E-mail: steve@uptime.org.uk / Tel: +442071931337
Jabber: steve@jabber.earth.li / MSN: foo@hotmail.co.uk
------------------------------
Date: Sat, 24 Sep 2005 13:21:43 -0400
From: "Matt Garrish" <matthew.garrish@sympatico.ca>
Subject: Re: Website scraper
Message-Id: <FWfZe.325$Bi.109635@news20.bellglobal.com>
"Stephen Hildrey" <steve@uptime.org.uk> wrote in message
news:1127580334.17298.1@ersa.uk.clara.net...
> Matt Garrish wrote:
>>
>> The interpreter can usually understand what you're trying to do, but in
>> your case you broke the line before the conditional "if" and it's not
>> going to assume that's what you meant so it gives an error. You can,
>> however, break it anywhere else:
>
> Sorry if I was ambiguous - I was trying to maintain the structure of the
> code in the OP's example, and not talking about the general case.
>
Clarity is key. I read your comment as a reference to Perl syntax in
general. The OP's does cause a compilation error, as you were alluding to.
Matt
------------------------------
Date: Sat, 24 Sep 2005 17:41:20 +0000 (UTC)
From: "DVH" <dvh@dvhdvhdvh.dvh>
Subject: Re: Website scraper
Message-Id: <dh433v$ho7$1@nwrdmz03.dmz.ncs.ea.ibs-infra.bt.com>
Stephen Hildrey <steve@uptime.org.uk> wrote in message
news:1127574674.36019.0@demeter.uk.clara.net...
>
> OP: if you are still experiencing difficulties with the code, do post
> back - I'm sure we'll be able to help :-)
Thanks Stephen.
I removed the backslash, and tidied up a couple of other obvious bugs. My
script now runs through the HTML and successfully creates a well-formatted
RSS file. It's an empty file though, so I think the next stage is to look at
the order of the tags and make sure the script can actually find what it's
looking for.
It isn't immediately obvious how to do this, so I may indeed come back...
thanks for the offer.
[I'm doing this because I want to scrape other sites which don't have an RSS
feed - as you mention elsewhere in the thread, there are numerous uses for
this sort of scraping. But it seemed logical to start with the technique
described in the tutorial].
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 8452
***************************************
