[28761] in Perl-Users-Digest
Perl-Users Digest, Issue: 5 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sun Jan 7 18:05:46 2007
Date: Sun, 7 Jan 2007 15:05:04 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sun, 7 Jan 2007 Volume: 11 Number: 5
Today's topics:
Re: comparing large number of data <mark.clementsREMOVETHIS@wanadoo.fr>
Re: comparing large number of data <nospam@somewhere.com>
Re: comparing large number of data <DJStunks@gmail.com>
Re: Elegant equivalent to this regex? <cwilbur@chromatico.net>
Re: File::Temp and external editor <hjp-usenet2@hjp.at>
Iterator special variable $_ (Mark Hobley)
Re: Iterator special variable $_ <mritty@gmail.com>
Re: Iterator special variable $_ <noreply@gunnar.cc>
PERL5LIB variable does not work as expected <stahl.karl@gmail.com>
Re: regex <tadmc@augustmail.com>
Re: regex <asandstrom@accesswave.ca>
Re: substitutions in a large file <tadmc@augustmail.com>
Re: system command won't run with -T option on <hjp-usenet2@hjp.at>
Re: Unsecured scripts and site hacking? <hjp-usenet2@hjp.at>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sun, 07 Jan 2007 17:36:50 +0100
From: Mark Clements <mark.clementsREMOVETHIS@wanadoo.fr>
Subject: Re: comparing large number of data
Message-Id: <45a1218d$0$25950$ba4acef3@news.orange.fr>
Perl Lover wrote:
> this whole exercise started in Perl because doing it in SQL
> just killed the RDBMS. Imagine doing it on a table with
> 30 million rows.
If you get the indexes right, then it shouldn't be a problem. Use your
RDBMS's analysis tools (EXPLAIN PLAN or whatever) to check that your
queries are, in fact, using the indexes that have been set.
Mark
------------------------------
Date: Sun, 7 Jan 2007 12:02:50 -0500
From: "Thrill5" <nospam@somewhere.com>
Subject: Re: comparing large number of data
Message-Id: <G6qdnSODnKkjujzYnZ2dnUVZ_qCmnZ2d@comcast.com>
"Mark Clements" <mark.clementsREMOVETHIS@wanadoo.fr> wrote in message
news:45a1218d$0$25950$ba4acef3@news.orange.fr...
> Perl Lover wrote:
>
>> this whole exercise started in Perl because doing it in SQL
>> just killed the RDBMS. Imagine doing it on a table with
>> 30 million rows.
>
> If you get the indexes right, then it shouldn't be a problem. Use your
> RDBMS's analysis tools (EXPLAIN PLAN or whatever) to check that your
> queries are, in fact, using the indexes that have been set.
>
> Mark
You could also add another field to the tables that has the date/time of the
last change, and use this one field to test for changes. You could update
this field in the basetable database using an ON UPDATE trigger. The
comparing each field in each row is probably what is killing the RDBMS. The
only thing you offload by doing this in another language is the comparison
operation, which means what you need to optimize is how to determine when a
record has been updated. Doing the compare on another system and another
language is not going to be much faster than doing it in SQL, and is
probably going to be slower.
Scott
------------------------------
Date: 7 Jan 2007 14:48:07 -0800
From: "DJ Stunks" <DJStunks@gmail.com>
Subject: Re: comparing large number of data
Message-Id: <1168210087.383957.44230@s34g2000cwa.googlegroups.com>
Mark Clements wrote:
> Perl Lover wrote:
>
> > this whole exercise started in Perl because doing it in SQL
> > just killed the RDBMS. Imagine doing it on a table with
> > 30 million rows.
>
> If you get the indexes right, then it shouldn't be a problem. Use your
> RDBMS's analysis tools (EXPLAIN PLAN or whatever) to check that your
> queries are, in fact, using the indexes that have been set.
exactly. or do portions at a time - id's between 0 - 399_999, then
400_000 to 799_999 etc or something similar.
there's absolutely no way Perl can out perform a database at this type
of task. I love Perl, but use the right tool for the job.
-jp
------------------------------
Date: 07 Jan 2007 11:51:39 -0500
From: Charlton Wilbur <cwilbur@chromatico.net>
Subject: Re: Elegant equivalent to this regex?
Message-Id: <87ejq6hjac.fsf@mithril.chromatico.net>
>>>>> "sff" == sherifffruitfly <sherifffruitfly@gmail.com> writes:
sff> If you (or anyone) wish to only answer questions *you* think
sff> I *should* be asking, rather than questions I am *actually*
sff> asking, that's fine, I suppose. I don't see how that requires
sff> any explanation from *me*, however.
The intent behind such questions is to help you come up with a better
solution. Regular expressions are a bad tool for many things, and
this may be one such. (I didn't look at your original question, so
don't have an opinion on that.) Imposing an arbitrary constraint
(such as "solutions MUST use regular expressions") without any
justification for that constraint means people will ignore the
constraint, on the assumptions that you have the full Perl toolkit
available and you're most interested in solving the problem clearly.
If there's a reason you *must* use a regular expression, explain what
it is. (But be prepared: "I'm doing this with PHP's preg_replace, so
I can't use HTML::Parser," for instance, such as I've seen before,
will get you redirected to a PHP forum.) If there's no reason to
restrict the solution to regular expressions, why not use the simplest
solution? And why get snarky and rude to people who are trying to
help, because you have constraints on the help you're willing to
accept that you can't or won't explain?
Charlton
--
Charlton Wilbur
cwilbur@chromatico.net
------------------------------
Date: Sun, 7 Jan 2007 19:40:25 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: File::Temp and external editor
Message-Id: <slrneq2fkp.9p3.hjp-usenet2@yoyo.hjp.at>
On 2007-01-02 21:48, xhoster@gmail.com <xhoster@gmail.com> wrote:
> Binand Sethumadhavan <binand@gmx.net> wrote:
>> $mtime1 = (stat($filename))[9];
>> system ("$cmd $filename");
>> $mtime2 = (stat($filename))[9];
>> if ($mtime2 - $mtime1 > 0) {
>> # Process the contents of $filename here
>> }
>
> I see no reason you need the $mtime stuff. What if the person edits
> and saves the file really quickly?
Some programs use that technique to decide whether the user saved the
file from the editor (thereby confirming that he wanted to proceed) or
exited the editor without saving (and thus canceled the whole
operation). As you notice, this can cause trouble, if the user manages
to save the file within the same second (rarely a problem with humans,
but quite possible if used within a script). An alternative is to check
whether the file really changed (either via a checksum or by saving the
original contents of the file), but that causes problems if saving the
file unchanged is a valid response. I don't see a general solution to
the problem.
hp
--
_ | Peter J. Holzer | > Wieso sollte man etwas erfinden was nicht
|_|_) | Sysadmin WSR | > ist?
| | | hjp@hjp.at | Was sonst wäre der Sinn des Erfindens?
__/ | http://www.hjp.at/ | -- P. Einstein u. V. Gringmuth in desd
------------------------------
Date: Sun, 07 Jan 2007 22:04:19 GMT
From: markhobley@hotpop.deletethisbit.com (Mark Hobley)
Subject: Iterator special variable $_
Message-Id: <506874-jua.ln1@neptune.markhobley.yi.org>
Keywords: perl,iterator,special,variable,$_,for,loop
The iterator special variable can be used in a foreach loop as follows:
@fruit = ( "apples", "bananas", "cherries" );
foreach (@fruit) {
print "I like $_\n";
}
Is it accepted to use the special variable in a for loop as follows?
use strict;
use warnings;
for ($_=0; $_ <=9; $_++) {
print "$_\n";
}
(The above works and I get no errors.)
Would you expect nested loops using the iterator variable to work?
use strict;
use warnings;
for ($_=0; $_<=3; $_++) {
print "Loop $_\n";
for ($_=0; $_<=6; $_++) {
print "Value $_\n";
}
}
(The above does not work, but I get no warnings)
Could I fix these loops to work by changing variable scope?
Is it valid to change the scope of special variables?
Is the observed behaviour above undefined, or is there a document clarifying
this somewhere?
Mark.
--
Mark Hobley
393 Quinton Road West
QUINTON
Birmingham
B32 1QE
Telephone: (0121) 247 1596
International: 0044 121 247 1596
Email: markhobley at hotpop dot donottypethisbit com
http://markhobley.yi.org/
------------------------------
Date: 7 Jan 2007 14:16:13 -0800
From: "Paul Lalli" <mritty@gmail.com>
Subject: Re: Iterator special variable $_
Message-Id: <1168208173.313488.90760@i15g2000cwa.googlegroups.com>
Mark Hobley wrote:
> Is it accepted to use the special variable in a for loop as follows?
>
> use strict;
> use warnings;
> for ($_=0; $_ <=9; $_++) {
> print "$_\n";
> }
>
> (The above works and I get no errors.)
If by "accepted" you mean "does it work", then as you discovered, yes.
If, however, you mean "is this a good idea", the answer is no. Why
would you do that? What reason would you have to use a default
variable in that manner? In general, $_ should never be assigned to.
It's an implicit variable, and should be used as such. Declare a
counter variable if you need it for some reason, or just use a foreach
loop like $_ was designed to be used:
for (0..9) {
print "$_\n";
}
>
> Would you expect nested loops using the iterator variable to work?
>
> use strict;
> use warnings;
> for ($_=0; $_<=3; $_++) {
> print "Loop $_\n";
> for ($_=0; $_<=6; $_++) {
> print "Value $_\n";
> }
> }
>
> (The above does not work, but I get no warnings)
Odd, the above code works perfectly for me. It first assigns $_ to 0
and prints out the first print statement. Then it assigns $_ to 0
(again), and prints the inner loop's statement. Then it goes through
the inner loop, ending when $_ is set to 7, which is not less than or
equal to 6. Then it goes back up to the condition of the first loop,
sees that $_ is now greater than 3, and the first loop exits.
What were you expecting it to do?
> Could I fix these loops to work by changing variable scope?
I have no idea what "fix" you're planning on making - the above code
does exactly what it should do.
> Is it valid to change the scope of special variables?\
You cannot make a lexical variable named $_, if that's what you mean.
You can, however, localize $_, at the start of each loop. That would
result in the 0..6 loop printing out four times. Is that what you
want?
> Is the observed behaviour above undefined, or is there a document clarifying
> this somewhere?
The above behavior is exactly what I expect it to be. If it's not what
you expect it to be, you'll have to explain what you expect. It's
"defined" in that it's completely non-ambiguous...
Paul Lalli
------------------------------
Date: Sun, 07 Jan 2007 23:23:44 +0100
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: Iterator special variable $_
Message-Id: <50da7hF1f6s8iU1@mid.individual.net>
Mark Hobley wrote:
> Would you expect nested loops using the iterator variable to work?
Yes, if you localize it.
> use strict;
> use warnings;
> for ($_=0; $_<=3; $_++) {
for (local $_ = 0; $_<=3; $_++) {
> print "Loop $_\n";
> for ($_=0; $_<=6; $_++) {
for (local $_=0; $_<=6; $_++) {
> print "Value $_\n";
> }
> }
But this would be a more readable and Perlish solution IMO:
foreach my $loop ( 0..3 ) {
print "Loop $loop\n";
foreach my $val ( 0..6 ) {
print "Value $val\n";
}
}
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: 7 Jan 2007 11:40:04 -0800
From: "Ishmael" <stahl.karl@gmail.com>
Subject: PERL5LIB variable does not work as expected
Message-Id: <1168198804.159760.72360@11g2000cwr.googlegroups.com>
I am trying to get Perl to recognize a module (Math::GMP) that is
installed in a local directory, i.e. not the same directory as the rest
of Perl. I've been trying to use the PERL5LIB environment variable to
get Perl to search on the correct path, but it doesn't work. Here's
what I mean:
First I set (what I believe to be) the search path:
setenv PERL5LIB
/home/kstahl/PERL/Digest-1.15:/home/kstahl/PERL/Digest-SHA1-2.11:
/home/kstahl/PERL/Scalar-List-Utils-1.19:/home/kstahl/PERL/gmp-4.2.1:
/home/kstahl/PERL/Math-GMP-2.04:/home/kstahl/PERL/Math-GMP-2.04/lib:
/home/kstahl/PERL/Math-GMP-2.04/blib:
/home/kstahl/PERL/Math-GMP-2.04/lib/site_perl/5.6.1/sol2.sun4/auto/Math/GMP:
/home/kstahl/PERL/Net-SSH-Perl-1.30
Now I check to see if it finds the module.
perl -MMath::GMP -e 'print "ok\n"'
Answer: Nope, it's still looking in the perl installation directory.
Can't load
'/apps/gnu/perl_local/lib/site_perl/5.6.1/sol2.sun4/auto/Math/GMP/GMP.so'
for
module Math::GMP: ld.so.1: perl: fatal: libgmp.so.3: open failed: No
such file
or directory at /apps/gnu/perl-5.6.1/lib/sol2.sun4/DynaLoader.pm line
206.
at -e line 0
Compilation failed in require.
BEGIN failed--compilation aborted.
Check to make sure correct directory is on the path:
perl -e 'while(<@INC>) { print "$_\n"; }'
/home/kstahl/PERL/Digest-1.15
/home/kstahl/PERL/Digest-SHA1-2.11
/home/kstahl/PERL/Scalar-List-Utils-1.19
/home/kstahl/PERL/gmp-4.2.1
/home/kstahl/PERL/Math-GMP-2.04
/home/kstahl/PERL/Math-GMP-2.04/lib/sol2.sun4
/home/kstahl/PERL/Math-GMP-2.04/lib
/home/kstahl/PERL/Math-GMP-2.04/blib
/home/kstahl/PERL/Math-GMP-2.04/lib/site_perl/5.6.1/sol2.sun4/auto/Math/GMP
/home/kstahl/PERL/Net-SSH-Perl-1.30
/apps/gnu/perl-5.6.1/lib/sol2.sun4
/apps/gnu/perl-5.6.1/lib
/apps/gnu/perl_local/lib/site_perl/5.6.1/sol2.sun4
/apps/gnu/perl_local/lib/site_perl/5.6.1
/apps/gnu/perl_local/lib/site_perl
.
Looks to me like the every conceivable subdirectory is included (as
well as the complete path to the actual library, GMP.so). What am I
missing here?
Thanks a priori!
------------------------------
Date: Sat, 6 Jan 2007 22:48:48 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: regex
Message-Id: <slrneq0utg.28o.tadmc@tadmc30.august.net>
lucas <lstouder@teksavvy.com> wrote:
> echo "http://fu.bar/test.jpg" | perl -ne 'print if((/\.jpg\n$/i)|
> (/\.bmp\n$/i)||(/\.png\n$/i));'
None of those dollar signs do anything, and so are not needed.
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Sun, 07 Jan 2007 22:29:47 GMT
From: "Arved Sandstrom" <asandstrom@accesswave.ca>
Subject: Re: regex
Message-Id: <vveoh.116055$rv4.12127@edtnps90>
"Abigail" <abigail@abigail.be> wrote in message
news:slrneq1jr9.fg.abigail@alexandra.abigail.be...
> lucas (lstouder@teksavvy.com) wrote on MMMMDCCCLXXVI September MCMXCIII
> in <URL:news:28e3e$45a01650$cef8bf30$16302@TEKSAVVY.COM-Free>:
> ^^ I have this perl oneliner that I'm using to recognize certain file
> ^^ extentions in a url. This works, but I was wondering if any of you
> know of
> ^^ a better way to write this.
> ^^
> ^^ echo "http://fu.bar/test.jpg" | perl -ne 'print if((/\.jpg\n$/i)|
> ^^ (/\.bmp\n$/i)||(/\.png\n$/i));'
>
> Yeah.
>
> echo "http://fu.bar/test.jpg"
>
> is a lot simpler.
I'm gathering - this is just me, mind you - that his actual input is not
manually typing in echo "http://fu.bar/test.jpg".
> Or did you want to ask how to grep? In that case, I'd use 'grep', not
> perl.
That works on Windows, does it?
Get over being so elitist.
AHS
------------------------------
Date: Sat, 6 Jan 2007 23:07:48 -0600
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: substitutions in a large file
Message-Id: <slrneq1014.28o.tadmc@tadmc30.august.net>
Nospam <nospam@home.com> wrote:
> Basically I have a local html file, called file1.html it has a series of
> links
> (with a particular domain name) in addition to the html code, I am trying to
> follow each of these links (based on the regular expression /on\.fe/) each
> of
> these links,
You should practice step-wise development.
First write a program to just print out the links you want to follow.
> in their content have a link to another page,
Then write a program that gets each linked page.
> (I would like to
> capture this particular page based on a regular expression /www\.arax/),
What does "capture a page" mean when you say it?
Then write a program that "captures a page" based on matching
a regular expression.
> and
> substitute for each link (with regular expression /on\.fe/)in file1.html
> with
> their corresponding link (with regular expression/www\.arax/)
Then write a program that does the substitution.
> So far this is
> what I have come up with, and am a little stuck
You are a lot stuck.
> #! perl\bin\perl
>
> use strict;
> use warnings;
> use WWW::Mechanize;
>
> my $mech = WWW::Mechanize->new();
>
> open(FILE, "< file1.html") || print "Unable to open the file file1 \n"
> +;
>
>
> while (<FILE>)
The line from the local file goes in $_.
Where in the code below do you operate on what is in $_?
> {
> for my $l ($mech->links) {
What web page have you told Mechanize to get before asking for
the links in the gotten page?
> my $url = $l->url;
> my $desc = $l->text;
> my $new = $url;
> $new =~ s/on\.fe/arax.com/;
>
> print "concerning $desc, $url should be $new\n";
>
> # $mech->get( $new );
> }
>
>
> }
>
>
> close(FILE);
>
>
> If it helps a sample of the html code from file1.html :
[snip HTML]
There is *nothing* there that matches /on\.fe/, so the
program should make no output at all.
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Sun, 7 Jan 2007 20:13:44 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: system command won't run with -T option on
Message-Id: <slrneq2hj8.9p3.hjp-usenet2@yoyo.hjp.at>
On 2007-01-03 20:44, xhoster@gmail.com <xhoster@gmail.com> wrote:
> "Paul Lalli" <mritty@gmail.com> wrote:
>> Correct. The "taintedness" of the PATH has nothing to do with
>> variables. It has to do with the fact that the PATH, by default, comes
>> from the environment - that is, it comes from outside your script. It
>> is tainted because it is theoretically possible for someone in the
>> environment to set a "bad" PATH before launching your script. And if
>> they do that, you're screwed.
>
> Well, most likely *they* are screwed, not you.
The -T switch is for situations where the person (or entity) controlling
the environment is not the same as the person (or entity) with whose
privileges the script is executing. So if the script is executing with
"your" privileges and "they" control the environment, "you" are screwed.
>> If your program calls 'mycmd.exe', and the malicious user has set
>> PATH to be "/my/really/nasty/viruses/" and puts a very bad program
>> named mycmd.exe in that PATH, you're in trouble. By defining the
>> PATH within the script itself, you eliminate that possibility.
>
> What are the ways for another party to change *my* PATH environment,
> without already having the system completely under their control anyway?
> I'm not really worried about someone else tricking Perl code (that I wrote,
> but which they, not I, am running) into running some nasty code within
> their own account, when they could have run the code in their own account
> directly anyway.
>
> The only one I can think of is setuid.
Right. That's the situation where the (potential) attacker has complete
control of the environment, and therefore also the PATH. In other
situations (CGI, sudo, etc.) the attacker may control only part of the
environment, which may or may not include PATH.
> In perhaps the most common situation in which -T is used, CGI, I think
> mistrusting PATH might be excessive, as anyone who can hack apache to
> change the PATH under which your script executes probably is already
> in the driver seat.
Probably, yes. However, perl doesn't know it is being run from Apache,
and it knew that, it would probably not be trivial to determine which
environment variables are always safe and which are potentially unsafe.
So while you as a sysadmin may determine that, yes, when this particular
script is run from this particular instance of apache, this environment
variable has a known value which cannot be influenced by an attacker,
perl in general cannot do that and has to treat the whole environment as
suspect.
hp
--
_ | Peter J. Holzer | > Wieso sollte man etwas erfinden was nicht
|_|_) | Sysadmin WSR | > ist?
| | | hjp@hjp.at | Was sonst wäre der Sinn des Erfindens?
__/ | http://www.hjp.at/ | -- P. Einstein u. V. Gringmuth in desd
------------------------------
Date: Sun, 7 Jan 2007 20:47:14 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: Unsecured scripts and site hacking?
Message-Id: <slrneq2ji2.9p3.hjp-usenet2@yoyo.hjp.at>
On 2007-01-03 20:43, Charlton Wilbur <cwilbur@chromatico.net> wrote:
>>>>>> "A" == Alison <invalid@NOSPAM.com> writes:
>
> A> Thanks for your reply. As far as root access goes, full access
> A> is only available locally to the server.
>
> As a general rule, anyone who has shell access to the box can acquire
> root access. Someone who can exploit flaws in a Perl script to get
> access to the machine can exploit flaws in other things to get root
> access.
That's a worst case assumption, not a general rule. If that was true in
general, multi-user systems would not be practical. We could just give
everybody uid 0 and stop bothering our users with those pesky file
permissions.
Sure, a knowledgable person with infinite amounts of time will sooner or
later be able to exploit a flaw before the sysadmin patches it, and it
is always a good idea to minimize privileges (i.e., shell access only
when necessary), but generally equating shell access with root access is
neither realistic nor psychologically useful (it leads to "we can't win
against the blackhats so we don't even have to bother").
(That said, for many tasks you don't need root access: If all you need
is a zombie for sending spam, an unprivileged account works just fine,
you may not even need a shell)
hp
--
_ | Peter J. Holzer | > Wieso sollte man etwas erfinden was nicht
|_|_) | Sysadmin WSR | > ist?
| | | hjp@hjp.at | Was sonst wäre der Sinn des Erfindens?
__/ | http://www.hjp.at/ | -- P. Einstein u. V. Gringmuth in desd
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 5
************************************
