[30817] in Perl-Users-Digest
Perl-Users Digest, Issue: 2062 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Wed Dec 17 16:09:46 2008
Date: Wed, 17 Dec 2008 13:09:12 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Wed, 17 Dec 2008 Volume: 11 Number: 2062
Today's topics:
Re: FAQ 9.4 How do I remove HTML from a string? <brian.d.foy@gmail.com>
Help with DOS file names <soup_or_power@yahoo.com>
Re: herding ones and zeroes into bytes <jimsgibson@gmail.com>
mathematical operation in a perl one liner in substitut <hmida3bida@google.com>
Re: mathematical operation in a perl one liner in subst <jurgenex@hotmail.com>
Re: mathematical operation in a perl one liner in subst <someone@example.com>
Re: mathematical operation in a perl one liner in subst <tadmc@seesig.invalid>
Re: mathematical operation in a perl one liner in subst <hmida3bida@google.com>
Max Execution Time for mod_perl <howachen@gmail.com>
Rounding up in perl <david.groff@noaa.gov>
Re: Rounding up in perl <jurgenex@hotmail.com>
Re: Rounding up in perl <hjp-usenet2@hjp.at>
Re: Rounding up in perl <mdudley@king-cart.com>
Re: Rounding up in perl <jurgenex@hotmail.com>
Re: Rounding up in perl <sbryce@scottbryce.com>
speeding up cgi perl <skeldoy@gmail.com>
Re: speeding up cgi perl <smallpond@juno.com>
Re: speeding up cgi perl <hjp-usenet2@hjp.at>
Re: speeding up cgi perl xhoster@gmail.com
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Wed, 17 Dec 2008 14:55:08 -0600
From: brian d foy <brian.d.foy@gmail.com>
Subject: Re: FAQ 9.4 How do I remove HTML from a string?
Message-Id: <171220081455084510%brian.d.foy@gmail.com>
In article <20081216121445.630$Lc@newsreader.com>, <xhoster@gmail.com>
wrote:
> dkcombs@panix.com (David Combs) wrote:
> > In article <lq0206-4c9.ln1@blue.stonehenge.com>,
> > PerlFAQ Server <brian@stonehenge.com> wrote:
>
> > > Here's one "simple-minded" approach, that works for most files:
> > >
> > > #!/usr/bin/perl -p0777
> Knowing why they were assembled in the manner they were is more about
> understanding the structure of HTML than it is about Perl, so I don't know
> that dwelling on that in a Perl FAQ is warranted. If the regex had been
> presented scattered over a dozen lines, using /x, I know that would have
> annoyed me quite a bit.
It's not that, it's just that the writing is poor. You should never
assume that code speaks for itself, so the answer should also say it in
words such as "This regex looks for an opening angle bracket....". With
a couple of sentences you reinforce the point in a different way.
However, the whole answer needs to be fixed up, and a could answer can
take a couple of whole evenings to write and edit.
------------------------------
Date: Wed, 17 Dec 2008 13:03:01 -0800 (PST)
From: "souporpower@gmail.com" <soup_or_power@yahoo.com>
Subject: Help with DOS file names
Message-Id: <c6822172-b4d0-4d2e-a1b6-e914432aabdd@40g2000prx.googlegroups.com>
This is my code to print the URL's from the browser favorites:
my @dirs = `dir`;
my $f;
foreach $f (@dirs) {
next if ($f =~ /url/);
`cd $f`;
print "Dir=" . $f . "\n";
# if ($f =~ /url/) {
# print "File=" . $f . "\n";
# open(FIN2, $f);
# while(<FIN2>) {
# print $_ if $_ =~ /http/;
# }
# close(FIN2);
# next;
# }
my @files=`dir *.url`;
my $f2;
foreach $f2 (@files) {
#chop $f2;
#chop $f2;
$f2 =~ s/\x0D//g;
print "AFile=" . "$f" . $f2 . "\n";
#my $file="$f\\$f2";
open(FIN, $f2);
while(<FIN>) {
print $_;
next if ($_ =~ /ico/);
print $_ if $_ =~ /http/ ;
}
close(FIN);
}
#`cd ..`;
}
The problem is each file returned by the DOS dir command has carriage
returns at the beginning of the file name (can't use chop or chomp).
Can someone please help me
out? Tried $f =~ s/^\\n//g;
------------------------------
Date: Wed, 17 Dec 2008 10:24:24 -0800
From: Jim Gibson <jimsgibson@gmail.com>
Subject: Re: herding ones and zeroes into bytes
Message-Id: <171220081024247926%jimsgibson@gmail.com>
In article <1o25rqlh4p3lu.16t09wglznhar$.dlg@40tude.net>, George
<george@example.invalid> wrote:
> On Fri, 12 Dec 2008 18:48:39 -0800, Jürgen Exner wrote:
>
> > George <george@example.invalid> wrote:
> >> For diplomacy to be effective, words must be credible - and no one can now
> >> doubt the word of America.
> >> George W. Bush
> >
> > It is better to keep quiet and let people wonder if you are an idiot
> > than to open your mouth and remove any doubt.
> > -Unknown author
>
> I thought that was attributed to Mark Twain.
Nobody knows (according to Ask Yahoo):
<http://ask.yahoo.com/20010115.html>
--
Jim Gibson
------------------------------
Date: 17 Dec 2008 17:00:58 GMT
From: kevin kitenik <hmida3bida@google.com>
Subject: mathematical operation in a perl one liner in substitute
Message-Id: <49493049$0$18364$ba4acef3@news.orange.fr>
Hi everybody,
i need to make an arithmetic operation ine one field in a csv file.
exp:
cat file
a ; 123
b ; 432
c ; 45
i need a one liner perl code to substitute the second filed by
its value divided by 10
to have
a ; 12.3
b ; 43.2
c ; 4.5
i need this sample of code:
perl -i -pe 's/;(\d+)/ $1/10 /' file
this can't work because i dont know how to tell perl
to execute $1/10 (or even $1 - 10 )
please, help, i know i can do it onether way,
but want to know if this is possible
because it's so easy (one line)
--
thanks a lot.
------------------------------
Date: Wed, 17 Dec 2008 09:15:24 -0800
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: mathematical operation in a perl one liner in substitute
Message-Id: <nncik4t9oim9qq50i58vvc0tf1387g3lmn@4ax.com>
kevin kitenik <hmida3bida@google.com> wrote:
>i need a one liner perl code to substitute the second filed by
> its value divided by 10
>to have
>a ; 12.3
>b ; 43.2
>c ; 4.5
>
>i need this sample of code:
>
>perl -i -pe 's/;(\d+)/ $1/10 /' file
See "perldoc perlop", section "Regexp Quote-Like Operators", subsection
"s/PATTERN/REPLACEMENT/egimosx":
Options are:
e Evaluate the right side as an expression.
jue
------------------------------
Date: Wed, 17 Dec 2008 09:53:32 -0800
From: "John W. Krahn" <someone@example.com>
Subject: Re: mathematical operation in a perl one liner in substitute
Message-Id: <v0b2l.61911$zQ3.32012@newsfe12.iad>
kevin kitenik wrote:
>
> i need to make an arithmetic operation ine one field in a csv file.
>
> exp:
> cat file
> a ; 123
> b ; 432
> c ; 45
>
> i need a one liner perl code to substitute the second filed by
> its value divided by 10
> to have
> a ; 12.3
> b ; 43.2
> c ; 4.5
>
> i need this sample of code:
>
> perl -i -pe 's/;(\d+)/ $1/10 /' file
>
> this can't work because i dont know how to tell perl
> to execute $1/10 (or even $1 - 10 )
$ echo "a ; 123
b ; 432
c ; 45
" | perl -pe's{(;\s*)(\d+)}{$1.($2/10)}e'
a ; 12.3
b ; 43.2
c ; 4.5
John
--
Those people who think they know everything are a great
annoyance to those of us who do. -- Isaac Asimov
------------------------------
Date: Wed, 17 Dec 2008 11:58:16 -0600
From: Tad J McClellan <tadmc@seesig.invalid>
Subject: Re: mathematical operation in a perl one liner in substitute
Message-Id: <slrngkifdo.1rt.tadmc@tadmc30.sbcglobal.net>
kevin kitenik <hmida3bida@google.com> wrote:
> Hi everybody,
>
> i need to make an arithmetic operation ine one field in a csv file.
>
> exp:
> cat file
> a ; 123
> b ; 432
> c ; 45
>
> i need a one liner perl code to substitute the second filed by
> its value divided by 10
> to have
> a ; 12.3
> b ; 43.2
> c ; 4.5
>
> i need this sample of code:
>
> perl -i -pe 's/;(\d+)/ $1/10 /' file
>
> this can't work because i dont know how to tell perl
> to execute $1/10 (or even $1 - 10 )
You have 3 more problems than you are aware of.
First, lets deal with the problem that you are aware of:
you want an s///e modifier.
As to the 3 problems you have not gotten to yet:
1) your pattern will not match your data as it does not
allow spaces after the semicolon
2) even if your pattern matched, there would be no semicolons in
the resulting file since you failed to put them back in
3) you have an unescaped slash character in your replacement code
(I'll use curly braces as an alternate delimiter to fix this one).
So:
perl -i -pe 's{(;\s*)(\d+)}{ $1 . $2/10 }e' file
--
Tad McClellan
email: perl -le "print scalar reverse qq/moc.noitatibaher\100cmdat/"
------------------------------
Date: 17 Dec 2008 20:17:22 GMT
From: kevin kitenik <hmida3bida@google.com>
Subject: Re: mathematical operation in a perl one liner in substitute
Message-Id: <49495e52$0$18364$ba4acef3@news.orange.fr>
Thanks a lot to all your replies.
very usefuls.
il really appreciate yours helps
have a nice day.
--
thanks a lot.
------------------------------
Date: Wed, 17 Dec 2008 08:04:02 -0800 (PST)
From: howa <howachen@gmail.com>
Subject: Max Execution Time for mod_perl
Message-Id: <9dce7145-9537-4bd0-b45e-a1878e5031c2@l33g2000pri.googlegroups.com>
Hello,
Is it possible to set a Max Execution Time for each mod_perl request,
so if a client hit a page which need more than a certain limit of time
to execute, the request will be killed automatically after that limit.
In PHP, there is such a limit of 30 seconds, don't know if it is
possible to set in mod_perl.
Thanks.
------------------------------
Date: Wed, 17 Dec 2008 10:37:22 -0500
From: David Groff <david.groff@noaa.gov>
Subject: Rounding up in perl
Message-Id: <gib6bi$68m$1@news.nems.noaa.gov>
Is there more than one function for rounding a
number up in perl?
I tried the ceil() function but get an undefined subroutine
message.
------------------------------
Date: Wed, 17 Dec 2008 08:18:59 -0800
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: Rounding up in perl
Message-Id: <9i9ik4p9fokcgnv12enj3gnvi880t77vh8@4ax.com>
David Groff <david.groff@noaa.gov> wrote:
>
>
>Is there more than one function for rounding a
>number up in perl?
>I tried the ceil() function but get an undefined subroutine
>message.
See
perldoc -q round
"Does Perl have a round() function? What about ceil() and floor()?"
jue
------------------------------
Date: Wed, 17 Dec 2008 17:42:46 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: Rounding up in perl
Message-Id: <slrngkib06.u7t.hjp-usenet2@hrunkner.hjp.at>
On 2008-12-17 15:37, David Groff <david.groff@noaa.gov> wrote:
> Is there more than one function for rounding a
> number up in perl?
For rounding - yes. For rounding up - no.
> I tried the ceil() function but get an undefined subroutine
> message.
You forgot "use POSIX"?
hp
------------------------------
Date: Wed, 17 Dec 2008 17:57:08 GMT
From: Marshall Dudley <mdudley@king-cart.com>
Subject: Re: Rounding up in perl
Message-Id: <U3b2l.359$xY.166@newsreading01.news.tds.net>
David Groff wrote:
>
>
> Is there more than one function for rounding a
> number up in perl?
> I tried the ceil() function but get an undefined subroutine
> message.
Simply add 0.5 to the number and round, or convert to an integer then
add 1 or add 1 then convert to an integer.
Marshall
------------------------------
Date: Wed, 17 Dec 2008 11:52:54 -0800
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: Rounding up in perl
Message-Id: <o2mik413lm2uadu9t26kis1gfa73ot8dfu@4ax.com>
Marshall Dudley <mdudley@king-cart.com> wrote:
>David Groff wrote:
>> I tried the ceil() function but get an undefined subroutine
>> message.
>[...] convert to an integer then
>add 1 or add 1 then convert to an integer.
Bad idea if the number is a natural number already.
jue
------------------------------
Date: Wed, 17 Dec 2008 13:08:27 -0700
From: Scott Bryce <sbryce@scottbryce.com>
Subject: Re: Rounding up in perl
Message-Id: <gibm8l$ic3$1@news.motzarella.org>
Marshall Dudley wrote:
> David Groff wrote:
>>
>>
>> Is there more than one function for rounding a
>> number up in perl?
>> I tried the ceil() function but get an undefined subroutine
>> message.
> Simply add 0.5 to the number and round, or convert to an integer then
> add 1 or add 1 then convert to an integer.
Neither of these methods is bullet proof. Accuracy of the results will
depend on the number and/or the method used to do the rounding.
use strict;
use warnings;
while (my $i = <DATA>)
{
chomp $i;
my $j = $i + 1;
my $k = $i + 0.5;
print "$i rounds to ", sprintf('%d', $j), ' or ', sprintf('%.0f', $k),
"\n";
}
__DATA__
2
3
----
2 rounds to 3 or 2
3 rounds to 4 or 4
------------------------------
Date: Wed, 17 Dec 2008 08:04:27 -0800 (PST)
From: skeldoy <skeldoy@gmail.com>
Subject: speeding up cgi perl
Message-Id: <8a0c5375-4169-47f7-b89d-3becce9e5c3b@z6g2000pre.googlegroups.com>
Hey! I am working on an application that features a huge database
(mysql) and some cgi (perl) for listing, sorting, searching,
dupechecking and more. I see that the configuration for the mysql is
pretty much spot on - most of the data is cached and so the mysqld-
process isn't really doing much in terms of causing bottlenecks. But I
believe that the perl-code may be the bottleneck here. I have turned
off buffering completely and I render pretty much just the things that
are needed. Still it can take up to a minute to print (in html) a
query that returns in a second on the mysql-console.
The output from the cgi is around 15MB for every operation the user
does, so I see the potential for slowness right there, in the sheer
amount of data that has to be produced and transmitted over the net,
but still I don't really understand what I have done to make the cgi
so slow.
The cgi is mostly taking a parameter like $query and doing a "select
from db where value like "%$query%"" and returning that in pretty <td>
$_</td>-form. That seems to work reasonably fast. But when I do a
"select * from db" things tend to get really slow when dealing with
15000-entries++ (even though mysql has it all cached and spits it out
in a split second). The cgi-process sits there, spitting out html to
the client, using up 95% of the cpu-time of one of the cores and using
50MBs of memory or so. I have no idea what it does. I have replaced
most of the " with '. And I can't really see that I am doing something
that needn't be done. Is there an issue with creating multiple
database-connections (DBD::mysql) that I should be aware of?
If somebody has experience in doing huge db's with perl, can you
please give me some pointers? Is this a code issue or is it a network-
issue or is it a browser issue? Does anyone have any tips for doing
huge databases with perl?
------------------------------
Date: Wed, 17 Dec 2008 08:44:48 -0800 (PST)
From: smallpond <smallpond@juno.com>
Subject: Re: speeding up cgi perl
Message-Id: <3acea530-1567-4fd8-9495-0e77f4674bfe@s9g2000prm.googlegroups.com>
On Dec 17, 11:04 am, skeldoy <skel...@gmail.com> wrote:
"select * from db"
If you're sending back everything, why not just render the page once
and
return static html? You can schedule updates when the db changes.
------------------------------
Date: Wed, 17 Dec 2008 18:04:36 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: speeding up cgi perl
Message-Id: <slrngkic94.u7t.hjp-usenet2@hrunkner.hjp.at>
On 2008-12-17 16:04, skeldoy <skeldoy@gmail.com> wrote:
> Hey! I am working on an application that features a huge database
> (mysql) and some cgi (perl) for listing, sorting, searching,
> dupechecking and more. I see that the configuration for the mysql is
> pretty much spot on - most of the data is cached and so the mysqld-
> process isn't really doing much in terms of causing bottlenecks. But I
> believe that the perl-code may be the bottleneck here. I have turned
> off buffering completely and I render pretty much just the things that
> are needed. Still it can take up to a minute to print (in html) a
> query that returns in a second on the mysql-console.
>
> The output from the cgi is around 15MB for every operation the user
> does, so I see the potential for slowness right there, in the sheer
> amount of data that has to be produced and transmitted over the net,
> but still I don't really understand what I have done to make the cgi
> so slow.
>
> The cgi is mostly taking a parameter like $query and doing a "select
> from db where value like "%$query%""
You should use placeholders instead for security, but that won't affect
speed much.
> and returning that in pretty <td> $_</td>-form. That seems to work
> reasonably fast. But when I do a "select * from db" things tend to get
> really slow when dealing with 15000-entries++ (even though mysql has
> it all cached and spits it out in a split second). The cgi-process
> sits there, spitting out html to the client, using up 95% of the
> cpu-time of one of the cores and using 50MBs of memory or so. I have
> no idea what it does.
To find out where it spends the time, use Devel::NYTProf.
The simplest method to use it is usually to just invoke it from the
commandline:
export REQUEST_METHOD=GET
export QUERY_STRING='whatever'
perl -d:NYTProf yourscript.cgi
This creates a file nytprof.out in the current directory which you can
convert to a nice HTML report with the nytprofhtml command.
There are also other ways for invoking it, see the docs.
CGI does have considerable overhead per row. However, for just 15000
rows I'd still expect that to be finished in a few seconds at most.
> If somebody has experience in doing huge db's with perl, can you
> please give me some pointers?
Is 700 GB and 6 billion rows in the largest table huge enough? I don't
use mysql for that one, though.
> Is this a code issue or is it a network-
> issue or is it a browser issue?
Since you wrote that your CGI script uses 95% CPU time during the time
it runs it is unlikely to be a browser problem. But browsers in general
are rather slow at rendering tables. For 15000 rows, the browser may
easily take a few dozen seconds to render the table (depending on the
speed of your workstation, how much memory you have, etc.) Try accessing
the CGI with wget or a similar tool - is it much faster than with the
browser?
hp
------------------------------
Date: 17 Dec 2008 17:20:33 GMT
From: xhoster@gmail.com
Subject: Re: speeding up cgi perl
Message-Id: <20081217121917.228$IY@newsreader.com>
skeldoy <skeldoy@gmail.com> wrote:
> Hey! I am working on an application that features a huge database
> (mysql) and some cgi (perl) for listing, sorting, searching,
> dupechecking and more.
Isn't dupechecking more a back-office thing best done from the command
line, not CGI?
> I see that the configuration for the mysql is
> pretty much spot on - most of the data is cached and so the mysqld-
> process isn't really doing much in terms of causing bottlenecks. But I
> believe that the perl-code may be the bottleneck here. I have turned
> off buffering completely
What does that mean? $|=1? mysql_use_result?
I wouldn't know how to identify *all* sources of buffering present in
a complex system, let alone know how to disable them completely.
> and I render pretty much just the things that
> are needed. Still it can take up to a minute to print (in html) a
> query that returns in a second on the mysql-console.
It takes less than a second to stream the entire <big_num> of data
to your screen on the mysql-console? Or do you somehow tell it to
run the query but suppress the actual results?
> The output from the cgi is around 15MB for every operation the user
> does, so I see the potential for slowness right there, in the sheer
> amount of data that has to be produced and transmitted over the net,
> but still I don't really understand what I have done to make the cgi
> so slow.
Does the CGI have to be invoked by POST, or will GET work?
Use wget to fetch the page. See how long it takes when the results just
have to fetched and stored and not rendered.
And once you do that and have the results stored as a static file, you
might as well put that in a location that is served up via http and open it
with the client browser. See how long does it take to pull the data over
the net and render it in a browser when it is coming from a static file
rather than a CGI.
>
> The cgi is mostly taking a parameter like $query and doing a "select
> from db where value like "%$query%"" and returning that in pretty <td>
> $_</td>-form. That seems to work reasonably fast. But when I do a
> "select * from db" things tend to get really slow when dealing with
> 15000-entries++ (even though mysql has it all cached and spits it out
> in a split second). The cgi-process sits there, spitting out html to
> the client, using up 95% of the cpu-time of one of the cores
You can try adding -d:DProf or -d:SmallProf (or you favorite profiler) to
the CGI shebang line. Or come up with an adapter run the CGI in a
command-line environment, and profile it from there.
> and using
> 50MBs of memory or so.
On most modern computers, 50MB is a rounding error. Is that the case for
your computer?
> I have no idea what it does. I have replaced
> most of the " with '.
???
> And I can't really see that I am doing something
> that needn't be done. Is there an issue with creating multiple
> database-connections (DBD::mysql) that I should be aware of?
*Anything* can be done incorrectly, in which case there are things to be
aware of. Are you using multiple MySQL connections in your CGI? If so,
how are you doing it?
> If somebody has experience in doing huge db's with perl, can you
> please give me some pointers?
Profile, profile, profile. If that doesn't work, comment out chunks
of code and see if that affects speed (for example, do all the work except
for the actual printing of the html table.)
> Is this a code issue or is it a network-
> issue or is it a browser issue?
If the CGI script itself is using 95% of a CPU, then it probably isn't
a browser or network issue. Unless your CPU utilization reporting tool
accounts for I/O waits as CPU usage (I've seen some that do).
> Does anyone have any tips for doing
> huge databases with perl?
15,000 records is not huge. It is just barely even medium. Unless you
have giant blob or text fields. If you do have a large database with
DBD::mysql, it is important to know the benefits and drawbacks of
mysql_use_result.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 2062
***************************************
