[19152] in Perl-Users-Digest
Perl-Users Digest, Issue: 1347 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sat Jul 21 00:10:34 2001
Date: Fri, 20 Jul 2001 21:10:14 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Message-Id: <995688613-v10-i1347@ruby.oce.orst.edu>
Content-Type: text
Perl-Users Digest Fri, 20 Jul 2001 Volume: 10 Number: 1347
Today's topics:
Request Peformance Advice (Buck Turgidson)
Re: Request Peformance Advice (Clinton A. Pierce)
Re: Request Peformance Advice <uri@sysarch.com>
special characters in regular expressions (Shang-Lin Chen)
Re: special characters in regular expressions <bwalton@rochester.rr.com>
SSLeay 1.7 installation help... PLEASE! (Corey)
String processing . . . (John Almberg)
Re: String processing . . . <james@zephyr.org.uk>
web-site statistical capture method? (Mike)
What is perl good for? (None)
Who can help me about the confused (..) operator? <no@mail.addr>
Re: Who can help me about the confused (..) operator? (Clinton A. Pierce)
Re: writing user agent (Nipa)
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: 20 Jul 2001 05:45:09 -0700
From: jc_va@hotmail.com (Buck Turgidson)
Subject: Request Peformance Advice
Message-Id: <f98999c8.0107200445.240c6d5c@posting.google.com>
I wrote my first real Perl with a lot if input from this group. It
works great, but is very slow. I have a file containing database
column names that are changing and I need to update a directory of
hundreds of SQL modules and change the column names.
I read the conversion mappings from the file into an array, and for
each file in a directory, I apply each regex derived from the mappings
array (about 60) to each line. It is very, very slow. I would like
to speed it up a bit, and was hoping someone could suggest some
efficiency improvements.
Much appreciated.
Buck
<code>
#!/usr/bin/perl -w
%readsuffixes = ("sql","yes");
$convertfile = 'c:\mydir\convert.txt';
$sourcedirname = 'c:\temp\sqlin';
$outdirname = 'c:\temp\sqlout';
#--------------------------------------------------
# Load array with conversion pairs
# from conversion file
#
# file format: EMPLOYEENAME=EMPL_NAME
# Approximately 60 column conversions
#--------------------------------------------------
open(CONVERT, $convertfile);
while ( $line = <CONVERT> ) {
$pairs = $line;
chomp($pairs);
push @convertpairs, uc($pairs);
push @convertpairs, lc($pairs);
$pairs = lc($pairs);
$pairs =~ s/([a-z]+)/\u$1/g; # proper case
push @convertpairs, $pairs;
}
#--------------------------------------------------
# Read input directory, apply all regexes in array
# to each line in each file.
#
#--------------------------------------------------
opendir(DIR,$sourcedirname) or die "Can't open $sourcedirname: $! \n";
@files = readdir(DIR);
foreach $file (@files) {
if ((-e "$sourcedirname$file") || (-e "$sourcedirname\\$file")) {
print "$sourcedirname\\$file\n";
$file =~ /([^.]*)\.(\w*)/;
$suffix = lc($2);
if (exists($readsuffixes{$suffix})) {
$filename = "$sourcedirname\\$file";
open(INFILE,$filename) or die "Can't open $filename\n";;
$outfilename = ">$outdirname\\$file";
open(OUTFILE,$outfilename) or die "Can't open
$filename\n";;
print $outfilename;
$fileswritten += 1;
while ($line = <INFILE>) {
foreach $arrayelement(@convertpairs) { # slow
@pair = split /=/, $arrayelement; # slow
$line =~ s/$pair[0]/$pair[1]/g; # slow
}
print OUTFILE $line;
}
}
close(INFILE);
close(OUTFILE);
}
}
closedir(DIR);
print "\n Files written: $fileswritten\n";
</code>
------------------------------
Date: Sat, 21 Jul 2001 02:35:53 GMT
From: clintp@geeksalad.org (Clinton A. Pierce)
Subject: Re: Request Peformance Advice
Message-Id: <dU567.34606$JN6.6754252@news1.rdc1.mi.home.com>
[Posted and mailed]
In article <f98999c8.0107200445.240c6d5c@posting.google.com>,
jc_va@hotmail.com (Buck Turgidson) writes:
> while ($line = <INFILE>) {
> foreach $arrayelement(@convertpairs) { # slow
> @pair = split /=/, $arrayelement; # slow
> $line =~ s/$pair[0]/$pair[1]/g; # slow
> }
> print OUTFILE $line;
> }
Hey! Ya found the slow part without too much trouble. If you can keep from
doing things within a loop, especially another loop, and especially other slow
things, then do it. @convertpairs doesn't change within the foreach/while loops.
Do all of that work on the outside.
So something like this (untested, and highly abbreviated):
$es="";
foreach $arrayelement (@convertpairs) {
@pair = split /=/, $arrayelement;
$es.="\$line=~s/$pair[0]/$pair[1]/g;";
}
#--------------------------------------------------
# Read input directory, apply all regexes in array
# to each line in each file.
#
#--------------------------------------------------
foreach $file (@files) {
# ....
if (exists....) {
while($line=<INFILE>) {
eval $es;
}
}
}
In fact, it might be quicker to eval the string once, create a coderef and
call that inside of the while loop:
$es="sub { ";
foreach $arrayelement (@convertpairs) {
@pair = split /=/, $arrayelement;
$es.="\$line=~s/$pair[0]/$pair[1]/g;";
}
$es.="}";
$changeline=eval $es;
#--------------------------------------------------
# Read input directory, apply all regexes in array
# .....
while($line=<INFILE>) {
&$changeline;
}
But you'd have to benchmark it to be sure.
--
Clinton A. Pierce Teach Yourself Perl in 24 Hours *and*
clintp@geeksalad.org Perl Developer's Dictionary
"If you rush a Miracle Man, for details, see http://geeksalad.org
you get rotten Miracles." --Miracle Max, The Princess Bride
------------------------------
Date: Sat, 21 Jul 2001 03:23:42 GMT
From: Uri Guttman <uri@sysarch.com>
Subject: Re: Request Peformance Advice
Message-Id: <x7g0brufr5.fsf@home.sysarch.com>
>>>>> "CAP" == Clinton A Pierce <clintp@geeksalad.org> writes:
CAP> In article <f98999c8.0107200445.240c6d5c@posting.google.com>,
CAP> jc_va@hotmail.com (Buck Turgidson) writes:
>> while ($line = <INFILE>) {
>> foreach $arrayelement(@convertpairs) { # slow
>> @pair = split /=/, $arrayelement; # slow
>> $line =~ s/$pair[0]/$pair[1]/g; # slow
>> }
>> print OUTFILE $line;
>> }
CAP> $es="";
CAP> foreach $arrayelement (@convertpairs) {
CAP> @pair = split /=/, $arrayelement;
CAP> $es.="\$line=~s/$pair[0]/$pair[1]/g;";
CAP> }
CAP> foreach $file (@files) {
CAP> # ....
CAP> if (exists....) {
CAP> while($line=<INFILE>) {
CAP> eval $es;
CAP> }
CAP> }
CAP> }
if the file is not too big, you can slurp it in as a single string and
then loop over all the replacement pairs. you won't need the eval or
even 2 loops.
the building up a code block and eval trick may still be wanted only
because of the multiple files. but it can be applied to a whole file and
not just a line at a time.
you could also build up an array of qr// and the replacement strings and
use that which will speed it up and not need the eval trick.
uri
--
Uri Guttman --------- uri@sysarch.com ---------- http://www.sysarch.com
SYStems ARCHitecture and Stem Development ------ http://www.stemsystems.com
Search or Offer Perl Jobs -------------------------- http://jobs.perl.org
------------------------------
Date: 20 Jul 2001 13:40:05 -0700
From: schen@caltech.edu (Shang-Lin Chen)
Subject: special characters in regular expressions
Message-Id: <ced2ab2d.0107201240.4f4c3f0b@posting.google.com>
I'm trying to write a function that will remove the tags from an SGML
file. Many of the tags in my file contain quotation marks, i.e.
<!Doctype refentry "-//OASIS//DTD DocBook V3.1//EN">
or
<arg choice="req">.
I tried to remove the tags using the line:
$file =~ s/<\W*\w*\W*>//g;
which removed all tags except for the ones containing quotation marks.
How can I remove all the tags?
------------------------------
Date: Sat, 21 Jul 2001 03:32:40 GMT
From: Bob Walton <bwalton@rochester.rr.com>
Subject: Re: special characters in regular expressions
Message-Id: <3B58F7F5.D6F91AC2@rochester.rr.com>
Shang-Lin Chen wrote:
>
> I'm trying to write a function that will remove the tags from an SGML
> file. Many of the tags in my file contain quotation marks, i.e.
>
> <!Doctype refentry "-//OASIS//DTD DocBook V3.1//EN">
> or
> <arg choice="req">.
>
> I tried to remove the tags using the line:
> $file =~ s/<\W*\w*\W*>//g;
> which removed all tags except for the ones containing quotation marks.
> How can I remove all the tags?
To do this right, you will need to fully parse the file (maybe something
on CPAN would help). A quick-and-dirty that might work in some cases
is:
$file=~s/<[^>]*>//sg;
But that will fail in many cases, such as if the quoted string contains
a > character, for example. It probably won't deal with commentary
correctly either.
--
Bob Walton
------------------------------
Date: 20 Jul 2001 16:45:24 -0700
From: spbid@netscape.net (Corey)
Subject: SSLeay 1.7 installation help... PLEASE!
Message-Id: <fcbdbb38.0107201545.4004e3bb@posting.google.com>
Hello,
I am tryiny to build/install SSLeay 1.7, on my UNIX FreeBSD server.
After running ./Makefile.PL -t, it detects my OpenSSL 9.5a fine, but
after a few lines of the build process this is the error messge I am
getting...
.....PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib
-I/usr/libdata/perl/5.00503/mach -I/usr/libdata/perl/5.00503 test.pl
1..16
Can't locate Errno.pm in @INC (@INC contains: blib/arch blib/lib
/usr/libdata/perl/5.00503/mach /usr/libdata/perl/5.00503
/usr/libdata/perl/5.00503/mach /usr/libdata/perl/5.00503
/usr/local/lib/perl5/site_perl/5.005/i386-freebsd
/usr/local/lib/perl5/site_perl/5.005 .) at blib/lib/Net/SSLeay.pm line
21.
BEGIN failed--compilation aborted at blib/lib/Net/SSLeay.pm line 21.
BEGIN failed--compilation aborted at test.pl line 19.
not ok 1
*** Error code 2
could anyone steer me in the right direction as to what I am doing
wrong??
Thanks
Corey
------------------------------
Date: 19 Jul 2001 08:35:54 -0700
From: jalmberg@identry.com (John Almberg)
Subject: String processing . . .
Message-Id: <c396bdba.0107190735.39a8ffb8@posting.google.com>
I'm sure there's an easy answer to this one, but I can't figure it out
. . .
How can I break a long string (say 500 characters) into multiple
strings of at most n characters (say 60), breaking only on word
boundries?
I'm doing this now using substr to chew up the long string, one
character at a time, but there's got to be a better way!
Thanks!
John
------------------------------
Date: Sat, 21 Jul 2001 02:31:56 +0100
From: James Coupe <james@zephyr.org.uk>
Subject: Re: String processing . . .
Message-Id: <g5tojMmMuNW7Ewyd@gratiano.zephyr.org.uk>
In message <c396bdba.0107190735.39a8ffb8@posting.google.com>, John
Almberg <jalmberg@identry.com> writes
>How can I break a long string (say 500 characters) into multiple
>strings of at most n characters (say 60), breaking only on word
>boundries?
use Text::Wrap;
--
James Coupe PGP Key: 0x5D623D5D
EBD690ECD7A1F
She twirled a lock of hair around her forefinger and smiled B457CA213D7E6
faintly. "Actually, I'd settle for a small Greek." 68C3695D623D5D
------------------------------
Date: 20 Jul 2001 15:03:27 -0700
From: michael_of_neb@yahoo.com (Mike)
Subject: web-site statistical capture method?
Message-Id: <5a10590e.0107201403.513e7b8a@posting.google.com>
Does anybody have a written-in-perl web-site statistical capture
method?
Either a known combination of modules or an off-the-shelf perl mong
would be okay, so this message goes out to the perl community in
general.
The idea is to get the statistics on a client's site concerning most
of
the various differentiations of such things as the origination ip of
the
browser, which search engine thay may have come from, what type of
browser they are using, and the like. Much of this information is
collected by the typical logfile that should be supplied by the
client's
hosting company, but that is not what we want.
This is something that we would like to provide to our clients as a
service,
collecting the information ourselves and passing some combination of
the raw
information, the statistical information and the graphical
representations of
the data on to the client.
Historical and real-time/on-the-fly statistical analysis are
eventually required, but whatever gets the ball rolling would be cool.
Since we want to provide the service ourselves, of course using
somebody
else's service is out-of-the-question.
As far as I can tell, all that is required is a small cgi perl
combination
javascript method installed on any given page that gets a single pixel
gif
for the page by going to our collection site to get it, and then
simply
drops off at this site the salient information from the hit on the
client.
The data is then stored in some way that further differentiates it
among
the various clients, and allows selective statistics to be maintained
and
displayed. This would be the perl part.
The whole thing aught to be pretty simple.
Mike
------------------------------
Date: 19 Jul 2001 21:58:11 -0700
From: pohanl@aol.com (None)
Subject: What is perl good for?
Message-Id: <12124e47.0107192058.798acec7@posting.google.com>
http://www.edepot.com/cgi-bin/minesweeper.pl
source:
http://cpan.valueclick.com/authors/id/P/PO/POHANL/minesweeper.pl
------------------------------
Date: Sat, 21 Jul 2001 10:12:08 +0800
From: MMX166+2.1G HD <no@mail.addr>
Subject: Who can help me about the confused (..) operator?
Message-Id: <050glt41plvq4pa3s8t2f2h0m1l06pnua0@4ax.com>
Terrible range operator (..) which in the scalar context...I had read
the perldoc many times, but I can't understand it at all. Can any
kindly angel help me about this operator (in the scalar context)?
------------------------------
Date: Sat, 21 Jul 2001 02:24:51 GMT
From: clintp@geeksalad.org (Clinton A. Pierce)
Subject: Re: Who can help me about the confused (..) operator?
Message-Id: <TJ567.34576$JN6.6747917@news1.rdc1.mi.home.com>
In article <050glt41plvq4pa3s8t2f2h0m1l06pnua0@4ax.com>,
MMX166+2.1G HD <no@mail.addr> writes:
> Terrible range operator (..) which in the scalar context...I had read
> the perldoc many times, but I can't understand it at all. Can any
> kindly angel help me about this operator (in the scalar context)?
Not a really easy operator to get your head wrapped around. Here's an
explanation extracted from the Perl Developer's Dictionary:
******
In scalar context, the .. operator returns a boolean value. The rules
for how the .. (scalar) operator work are as follows:
* Each .. operator is independent of any other operators in
the program.
* When the left operand is false, the operator returns false.
* When the left operand is true, the operator returns true.
* The operator continues returning true until the right operand
is true (on the next evaluation)
* The right operand will be tested again, and the operator can
alternate from true back to false on the same evaluation. Use
the ... (3-dots) version of the range operator if you don't want
this behavior.
The right operand is not evaluated when the operator is in its false state,
and the left operand is not evaluated when the operator is in its true
state. A quick demo of the operator might be:
$a=$b=0;
while (1) {
if ($seq=$a..$b) {
print "The operator is true ($seq).\n";
$b++;
} else {
print "The operator is false ($seq).\n";
$a++;
}
last if $a && $b > 1;
}
# Prints: (see below for explanation of "2E0")
# The operator is false ().
# The operator is true (1).
# The operator is true (2E0).
If an operand is a numeric literal, the operand is implicitly compared to the
variable $. (line input number). This allows you to mimic the behavior
of Unix utilities such as sed and awk.
The false value returned by the range operator is the empty string. The true
value returned is the sequence number (starting with 1). The final sequence
number has the string E0 appended to it -- it's still a numeric value (1e0 is
just 1) but can be parsed to tell if you've run off the end of the list.
Further examples:
# Very sed-ish way to print the first 10 lines
# of a filehandle
while( <FH> ) {
print if (1..10);
}
# From the first blank line to the end of file,
# print the lines
while( <OFH> ) {
if (/^$/ .. eof()) {
print;
}
}
*****
Hope that helps!
--
Clinton A. Pierce Teach Yourself Perl in 24 Hours *and*
clintp@geeksalad.org Perl Developer's Dictionary
"If you rush a Miracle Man, for details, see http://geeksalad.org
you get rotten Miracles." --Miracle Max, The Princess Bride
------------------------------
Date: 20 Jul 2001 13:56:40 -0700
From: npatel@webley.com (Nipa)
Subject: Re: writing user agent
Message-Id: <24d1a4c3.0107201256.5235e4c5@posting.google.com>
rene.nyffenegger@gmx.ch (Rene Nyffenegger) wrote in message news:<Xns90E151D2E3964gnuegischgnueg@130.133.1.4>...
> >I have not done any work with UA so far. and have no idea where to
> >start from. so please if some one can point out some info or could
> >provied some help.
> >
> >thanks
> >Nipa
>
>
Thanks Rene,
I appriciate that.
However, I am not sure about the scripting of user agent. If you could
provide me some reference where I can find related examples or if you
could provide me some insight on how to go about writing a script
relating UserAgent.
help appreciate.
Thanks
nipa
> use LWP::RobotUA;
>
> or
>
> use LWP::UserAgent;
>
> (to be downloaded from www.cpan.org)
>
> Hth
>
> Rene
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 1347
***************************************
