[31964] in Perl-Users-Digest
Perl-Users Digest, Issue: 3228 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sun Dec 5 06:09:37 2010
Date: Sun, 5 Dec 2010 03:09:19 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sun, 5 Dec 2010 Volume: 11 Number: 3228
Today's topics:
Re: Optimisation for tight loops... <hjp-usenet2@hjp.at>
Re: Optimisation for tight loops... <kst-u@mib.org>
Re: variables inside a string sln@netherlands.com
Re: variables inside a string <tadmc@seesig.invalid>
Re: variables inside a string sln@netherlands.com
why doesn't my dbm file save my data? <jaialai.technology@gmail.com>
Re: why doesn't my dbm file save my data? <jaialai.technology@gmail.com>
Re: why doesn't my dbm file save my data? <marc.girod@gmail.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 4 Dec 2010 19:12:29 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: Optimisation for tight loops...
Message-Id: <slrnifl14d.sdt.hjp-usenet2@hrunkner.hjp.at>
On 2010-12-03 02:09, Uri Guttman <uri@StemSystems.com> wrote:
>>>>>> "KT" == Keith Thompson <kst-u@mib.org> writes:
>
> KT> "Uri Guttman" <uri@StemSystems.com> writes:
> >>>>>>> "KT" == Keith Thompson <kst-u@mib.org> writes:
> KT> [...]
> KT> Consider this complete program:
> >>
> KT> #!/usr/bin/perl
> >>
> KT> use strict;
> KT> use warnings;
> >>
> KT> my $i = 0;
> KT> for (1 .. 10) {
> KT> $i ++;
> KT> }
> KT> print "$i\n";
> >>
> KT> The Perl compiler/interpreter can see the entire program before it
> KT> produces any output. At least in principle, it *could* analyze it
> KT> and determine that $i isn't tied, that there are no sub calls in the
> KT> loop, that there are no calls to eval(), and so forth. Given that
> KT> information, it *could* compile the whole thing to the equivalent of
> >>
> >> and what if warnings was replaced by a trojan module that made declaring
> >> $i a tied variable?
>
> KT> Then the hypothetical loop-optimizing Perl implementation would
> KT> analyze the contents of warnings.pm, fail to prove that it doesn't
> KT> do anything disruptive, and refrain from performing the optimization.
>
> and if that module loads another module? ad infinitum.
If the module loads other modules ad infinitum, the compiler will spend
an infinite time compiling the program, so we don't need to worry about
what happens after it finishes ;-).
If the the number of loaded modules is finite, nothing changes. The
compiler knows all the code it has compiled so far - it makes no
difference whether all the code is in a single file or in multiple
modules. (This is BTW a major difference between Perl and languages
which are usually compiled to object files: In the latter the compiler
only sees the current compilation unit - only at link time the whole
code is merged together)
> welcome to the halting problem. do you even know what that is?
I think he does. Do you?
> KT> That doesn't follow. The halting problem is about the impossibility
> KT> of doing certain kinds of analysis on *all* programs. You are
> KT> claiming that certain kinds of analysis are impossible on *any*
> KT> programs, a much stronger claim.
>
> no, it is the same problem. you have to keep analyzing deeper and
> deeper. the issue isn't about small things being optimized, it is how
> much work you need to do to gain a worthy optimization. you have already
> exceeded that limit.
That's an engineering tradeoff. If the gain (shorter run-time) is
greater than the cost (longer compiler-time, implementation effort, code
complexity (harder to maintain, potential bugs) - note that these costs
are not directly comparable) it's worthy.
It may well be that loop unrolling and strength reduction are not worthy
optimizations for perl (in fact, for loop unrolling I'm quite sure it
isn't worth the effort - see my last posting in this thread), but you
can't argue with the halting problem here because these optimizations
are not equivalent to the halting problem. The halting problem is to
write a program which for *any* program and input can determine (in
finite time) whether that program will halt for that input. An optimizer
doesn't need to recognize all optimizable programs: It can err on the
safe side and leave some optimizable programs unoptimized.
> KT> (Note that I'm not arguing that it *is* worth the effort, merely that
> KT> your argument that it isn't is incorrect.)
>
> no, it is the same. we can easily write a solution for the halting
> problem for most programs. same as for analyzing code for
> optimization. that doesn't mean you solved the problem nor have you
> found a proper limit on when to stop. someone (like you! :) will
> always bitch that the analysis doesn't go deeply enough for your
> program. better to not even offer to do it.
"Somebody will always bitch that Perl doesn't offer a good enough
solution to their problems. Better to not even offer it."
If Larry had thought that way we wouldn't have Perl. In fact we would
have very little software at all since almost all software can be
improved and the authors know it.
hp
------------------------------
Date: Sun, 05 Dec 2010 00:26:54 -0800
From: Keith Thompson <kst-u@mib.org>
Subject: Re: Optimisation for tight loops...
Message-Id: <ln8w04oca9.fsf@nuthaus.mib.org>
"Uri Guttman" <uri@StemSystems.com> writes:
>>>>>> "KT" == Keith Thompson <kst-u@mib.org> writes:
>
> KT> "Uri Guttman" <uri@StemSystems.com> writes:
> >>>>>>> "KT" == Keith Thompson <kst-u@mib.org> writes:
> KT> [...]
> KT> Consider this complete program:
> >>
> KT> #!/usr/bin/perl
> >>
> KT> use strict;
> KT> use warnings;
> >>
> KT> my $i = 0;
> KT> for (1 .. 10) {
> KT> $i ++;
> KT> }
> KT> print "$i\n";
> >>
> KT> The Perl compiler/interpreter can see the entire program before it
> KT> produces any output. At least in principle, it *could* analyze it
> KT> and determine that $i isn't tied, that there are no sub calls in the
> KT> loop, that there are no calls to eval(), and so forth. Given that
> KT> information, it *could* compile the whole thing to the equivalent of
> >>
> >> and what if warnings was replaced by a trojan module that made declaring
> >> $i a tied variable?
>
> KT> Then the hypothetical loop-optimizing Perl implementation would
> KT> analyze the contents of warnings.pm, fail to prove that it doesn't
> KT> do anything disruptive, and refrain from performing the optimization.
>
> and if that module loads another module? ad infinitum. welcome to the
> halting problem. do you even know what that is?
You're missing the point. What if that module *doesn't* load
another module?
For *some* programs, the analysis is possible. For *some* programs,
it isn't. For *some* programs, determining whether the analysis
is possible or not requires more resources than is reasable.
> KT> That doesn't follow. The halting problem is about the impossibility
> KT> of doing certain kinds of analysis on *all* programs. You are
> KT> claiming that certain kinds of analysis are impossible on *any*
> KT> programs, a much stronger claim.
>
> no, it is the same problem. you have to keep analyzing deeper and
> deeper. the issue isn't about small things being optimized, it is how
> much work you need to do to gain a worthy optimization. you have already
> exceeded that limit.
Yes, it would require a lot of work, probably more than is worthwhile.
I do not dispute that. All I'm saying is that it's theoretically
possible in some cases.
> KT> (Note that I'm not arguing that it *is* worth the effort, merely that
> KT> your argument that it isn't is incorrect.)
>
> no, it is the same. we can easily write a solution for the halting
> problem for most programs. same as for analyzing code for
> optimization. that doesn't mean you solved the problem nor have you
> found a proper limit on when to stop. someone (like you! :) will always
> bitch that the analysis doesn't go deeply enough for your
> program. better to not even offer to do it.
"better" is irrevelant. I've repeatedly acknowledged that it's very
likely not worth the effort.
As for "when to stop", set any arbitrary limit you like. Some programs
will be simple enough not to exceed that limit.
--
Keith Thompson (The_Other_Keith) kst-u@mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
------------------------------
Date: Fri, 03 Dec 2010 11:18:25 -0800
From: sln@netherlands.com
Subject: Re: variables inside a string
Message-Id: <mtfif6dec8rj1040b9gbhc01p1a6ofio8g@4ax.com>
On 3 Dec 2010 18:31:34 GMT, Martin Keiter <Martin.Keiter_NOSPAM@gmx.de> wrote:
>On 2010-12-03, sln@netherlands.com <sln@netherlands.com> wrote:
>
>> If you going to use eval, you could always use the suggs posted
>> by others with something like this.
>>
>> -sln
>>
>> use strict;
>> use warnings;
>>
>> my @template = (
>> { regexp => q{(.*) comes in},
>> sentence => qq{"\$1 is here!\\n"}
>> },
>> { regexp => q{(.*) sometimes comes in},
>> sentence => qq{"\$1 is sometimes there!\\n"}
>> },
>> );
>
>I definitely have to learn about all this quoting! (you don't need to
>explain it - I'll read and learn!
>
>> my @instrings = (
>> 'foo comes in here',
>> 'bar sometimes comes in',
>> 'baz comes in here',
>> );
>>
>> for my $sample (@instrings)
>> {
>> for my $i (0..$#template)
>> {
>> if ( $sample =~ /$template[ $i ]{'regexp'}/ ) {
>> print eval $template[ $i ]{'sentence'};
>> }
>> }
>> }
>
>this also works. But is it not dangerous if the instrings are provided
>by potentially dangerous users?
Eval in itself is not dangerous, its what's in the eval after its code.
If, after its code, its result is a STRING scalar as opposed to bareword functions,
thats all it is. It depends on how you quote it.
Consider this:
print eval qq{"system(\\"dir aa.*\\");\\n"};
print eval qq{system("dir aa.*");};
I think the first cannot be executed, so if you set up
'sentence' by qq{ " <capture from user> "} you are quoting "" and it
( here ^ and here ^ )
is a scalar.
Anyway, thats the theory I think, but there could be oddities I guess.
-sln
------------------------------
Date: Fri, 03 Dec 2010 14:14:17 -0600
From: Tad McClellan <tadmc@seesig.invalid>
Subject: Re: variables inside a string
Message-Id: <slrnifiki2.8q3.tadmc@tadbox.sbcglobal.net>
Martin Keiter <Martin.Keiter_NOSPAM@gmx.de> wrote:
> On 2010-12-03, sln@netherlands.com <sln@netherlands.com> wrote:
> I definitely have to learn about all this quoting! (you don't need to
> explain it - I'll read and learn!
See the "Quote and Quote-like Operators", "Regexp Quote-Like Operators"
and "Quote-Like Operators" sections in:
perldoc perlop
>> print eval $template[ $i ]{'sentence'};
> But is it not dangerous if the instrings are provided
> by potentially dangerous users?
eval EXPR (which that is) is nearly always dangerous.
eval BLOCK is safe though.
--
Tad McClellan
email: perl -le "print scalar reverse qq/moc.liamg\100cm.j.dat/"
The above message is a Usenet post.
I don't recall having given anyone permission to use it on a Web site.
------------------------------
Date: Sat, 04 Dec 2010 13:12:40 -0800
From: sln@netherlands.com
Subject: Re: variables inside a string
Message-Id: <h0blf6dc06g8si8ejua6su9q344b4q6bku@4ax.com>
On Fri, 03 Dec 2010 11:18:25 -0800, sln@netherlands.com wrote:
>On 3 Dec 2010 18:31:34 GMT, Martin Keiter <Martin.Keiter_NOSPAM@gmx.de> wrote:
>
>>On 2010-12-03, sln@netherlands.com <sln@netherlands.com> wrote:
>>
>>> If you going to use eval, you could always use the suggs posted
>>> by others with something like this.
>>>
[...]
>>this also works. But is it not dangerous if the instrings are provided
>>by potentially dangerous users?
>
>Eval in itself is not dangerous, its what's in the eval after its code.
>If, after its code, its result is a STRING scalar as opposed to bareword functions,
>thats all it is. It depends on how you quote it.
>
>Consider this:
>
> print eval qq{"system(\\"dir aa.*\\");\\n"};
> print eval qq{system("dir aa.*");};
>
>I think the first cannot be executed, so if you set up
>'sentence' by qq{ " <capture from user> "} you are quoting "" and it
> ( here ^ and here ^ )
>is a scalar.
>Anyway, thats the theory I think, but there could be oddities I guess.
>
I'm going to have to backtrack on this whole thing.
So forget everything I said with the eval "" format.
It is not safe at all if using external, possibly malicious data.
The docs say that eval BLOCK is compiled only once. But that won't
do any good for dynamic variable substitution ala \$var.
The eval "" form will recompile every time.
The fact is that if a user knows (or guesses) your statement form,
he/she will be able to splice in code, in any construction of eval "".
I see no way around it, it is dangerous, so use another method.
Test case:
my $val = ").system('dir a*.*').qq(";
eval "print qq($val)";
-sln
------------------------------
Date: Sat, 4 Dec 2010 12:22:34 -0800 (PST)
From: jaialai technology <jaialai.technology@gmail.com>
Subject: why doesn't my dbm file save my data?
Message-Id: <5e9a0d08-977a-4d12-b760-e16f33c14f8c@z9g2000yqz.googlegroups.com>
The code below is saved to a file called, say, temp.pl.
I run $perl temp.pl a b
and then I immediately run
$perl temp.pl c d
On this second run I expect to see the key 'a' of my first run but I
do not.
Why?
I am not getting any errors and the
associated db files are in /tmp
-rw-r--r-- 2 jt jt 12306 2010-12-04 14:53 test_db.dir
-rw-r--r-- 2 jt jt 12306 2010-12-04 14:53 test_db.pag
----
temp.pl
----
my %TEST_DATA;
dbmopen(%TEST_DATA,"/tmp/test_db",0666);
foreach my $key (keys %TEST_DATA){
print "$key\n";
}
$TEST_DATA[$ARGV[0]]=$ARGV[1];
dbmclose(%TEST_DATA);
------------------------------
Date: Sat, 4 Dec 2010 12:52:04 -0800 (PST)
From: jaialai technology <jaialai.technology@gmail.com>
Subject: Re: why doesn't my dbm file save my data?
Message-Id: <f231afd8-c7ce-46bd-995d-bebfe7756a53@o4g2000yqd.googlegroups.com>
Nevermind. I see my error.
Silly mistake.
$TEST_DATA[$ARGV[0]]=$ARGV[1];
should read
$TEST_DATA{$ARGV[0]}=$ARGV[1];
------------------------------
Date: Sat, 4 Dec 2010 13:00:51 -0800 (PST)
From: Marc Girod <marc.girod@gmail.com>
Subject: Re: why doesn't my dbm file save my data?
Message-Id: <15415624-bddd-4e08-9f45-9ac7e9c52462@i18g2000yqn.googlegroups.com>
On Dec 4, 8:22=A0pm, jaialai technology <jaialai.technol...@gmail.com>
wrote:
> Why?
Because you do not write into your hash, but into a @TEST_DATA array:
> $TEST_DATA[$ARGV[0]]=3D$ARGV[1];
use strict; would have told you.
$TEST_DATA{$ARGV[0]}=3D$ARGV[1];
Marc
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 3228
***************************************
