[28545] in Perl-Users-Digest
Perl-Users Digest, Issue: 9909 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Tue Oct 31 14:10:26 2006
Date: Tue, 31 Oct 2006 11:10:18 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Tue, 31 Oct 2006 Volume: 10 Number: 9909
Today's topics:
Re: Interesting behaviour with lexical variable anno4000@radom.zrz.tu-berlin.de
Re: Interesting behaviour with lexical variable <bik.mido@tiscalinet.it>
Re: Interesting behaviour with lexical variable <jl_post@hotmail.com>
Re: Interesting behaviour with lexical variable xhoster@gmail.com
Re: Interesting behaviour with lexical variable <thepoet_nospam@arcor.de>
Re: Interesting behaviour with lexical variable <bol@adv.magwien.gv.at>
Re: Interesting behaviour with lexical variable <bik.mido@tiscalinet.it>
Re: Interesting behaviour with lexical variable <bik.mido@tiscalinet.it>
Re: Interesting behaviour with lexical variable anno4000@radom.zrz.tu-berlin.de
killing processes using perl (artsd) <andreas./@\.poipoi.de>
Re: Looking for a lightweight GUI editor on Windows j.m.sedgwick@gmail.com
Re: Looking for a lightweight GUI editor on Windows j.m.sedgwick@gmail.com
loop over a string to do search/replacement using regex <rick.peng.du@gmail.com>
Re: loop over a string to do search/replacement using r <noreply@gunnar.cc>
Re: Naive threading performance questions <brundlefly76@hotmail.com>
new CPAN modules on Tue Oct 31 2006 (Randal Schwartz)
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: 30 Oct 2006 19:57:59 GMT
From: anno4000@radom.zrz.tu-berlin.de
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <4qn3q7Fng4qaU1@news.dfncis.de>
jl_post@hotmail.com <jl_post@hotmail.com> wrote in comp.lang.perl.misc:
> Ferry Bolhar wrote:
> >
> > Hi to all,
> >
> > while playing with code like this:
> >
> > use warnings;
> > while (my $input = <STDIN>){
> > my $num = abs $input;
> > print add();
> >
> > sub add {
> > $num + $num;
> > }
> > }
>
> Unless you really know what you're doing, you should never define a
> function inside a loop or another function. In fact, many Perl
> programmers (me included) recommend that all your functions be defined
> at the top of your script, before you start your main code. Do that,
> and your problem will go away.
>
> > I found out that once the loop get executed for the second
> > time, the variable $sum gets "splitted", taking the new value
> > from $input in the loop, but leaving the old, previous value
> > in add().
>
> I believe that's because the add() function is never defined until
> the first time through the loop.
That isn't quite correct. The sub add() is defined at compile time, not
the first time through the loop. Proof: Add
die 'boo' unless defined &add;
before the loop. It won't die.
The confusing behavior is caused by sub dispatch, which is the
mechanism by which closures retain the lexical environment that
is present when the coderef is assigned. If sub add() is replaced
by an actual closure, the confusing behavior goes away:
while (my $input = <STDIN>){
my $num = abs $input;
my $add = sub {
$num + $num;
};
# sub add { $num + $num }
print $add->(), "\n";
}
prints a new value (the expected one) each time through, because
the sub is dispatched with the current value each time.
A named sub, as in the commented-out line is dispatched only once
and freezes that value of $num. I don't claim to know how this
happens exactly, but that's what it amounts to.
Note that $num in the loop is *not* the same variable each time,
as printing the ref shows. The named sub add() firmly believes
in the very first variable named $num (and its value) while the
loop advances and uses a new one every time. A closure knows
about the changes.
Anno
------------------------------
Date: Mon, 30 Oct 2006 21:01:29 +0100
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <51mck2hp3ff16vepn9qltkriv63649n5r4@4ax.com>
On Mon, 30 Oct 2006 17:37:43 +0100, "Ferry Bolhar"
<bol@adv.magwien.gv.at> wrote:
>while playing with code like this:
>
>use warnings;
>while (my $input = <STDIN>){
> my $num = abs $input;
> print add();
>
> sub add {
> $num + $num;
> }
>}
>
>I found out that once the loop get executed for the second
>time, the variable $sum gets "splitted", taking the new value
>from $input in the loop, but leaving the old, previous value
>in add().
This has been discussed before. Actually I have been claiming that
contrary to the common assumption, closures are an orthogonal concept
to that of anonymous subs. I was wrong: this turns out to be true in
general, but it has been shown to me that in Perl (5, that is, in 6 I
don't know (yet)) closures are only implemented as anonymous subs,
since as you discovered yourself, named ones only close around the
lexical scope surrounding them as of the first time they're found,
thus failing to take a 'snapshot' of those lexicals at any given time,
which is a fundamental closure charachteristic.
A much simpler example can be given than you code:
#!/usr/bin/perl -l
use strict;
use warnings;
print '[anon]';
print &$_ for
map { my $x=$_; sub () { $x } }
qw/foo bar baz/;
print '[named]';
print &$_ for
map { my $x=$_; sub foo () { $x }; \&foo }
qw/foo bar baz/;
__END__
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: 30 Oct 2006 12:38:05 -0800
From: "jl_post@hotmail.com" <jl_post@hotmail.com>
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <1162240685.150490.130020@b28g2000cwb.googlegroups.com>
anno4000@radom.zrz.tu-berlin.de wrote:
>
> while (my $input = <STDIN>){
> my $num = abs $input;
> my $add = sub {
> $num + $num;
> };
> # sub add { $num + $num }
> print $add->(), "\n";
>
> }
>
> Note that $num in the loop is *not* the same variable each time,
> as printing the ref shows. The named sub add() firmly believes
> in the very first variable named $num (and its value) while the
> loop advances and uses a new one every time. A closure knows
> about the changes.
I wouldn't think that a closure knows about the changes. That
closure only knows about the changes because a new closure gets created
each time through the loop (each time using a new value of $num).
But define the closure only once and it will always use the value of
$num that it had the first time through the loop:
my $add;
while (my $input = <STDIN>){
my $num = abs $input;
$add = sub {
$num + $num;
} unless defined $add;
print $add->(), "\n";
}
With this code, the $add only get assigned a closure once, and keeps
the first value of $num each time.
Because of this behavior, I was under the impression that the
original poster's code:
while (my $input = <STDIN>){
my $num = abs $input;
print add();
sub add {
$num + $num;
}
}
behaved the same way for similar reasons. (And if you replace "my"
with "our" all pieces of code will work the way the original poster
intended.)
So although &add is indeed defined at the beginning of the script,
for some reason the add() function it refuses to "re-define" its own
behavior after the first time through the loop. (As to why this
happens, I'm really not sure...)
-- Jean-Luc
------------------------------
Date: 30 Oct 2006 21:03:45 GMT
From: xhoster@gmail.com
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <20061030160412.996$CA@newsreader.com>
"jl_post@hotmail.com" <jl_post@hotmail.com> wrote:
> anno4000@radom.zrz.tu-berlin.de wrote:
> >
> > while (my $input = <STDIN>){
> > my $num = abs $input;
> > my $add = sub {
> > $num + $num;
> > };
> > # sub add { $num + $num }
> > print $add->(), "\n";
> >
> > }
> >
> > Note that $num in the loop is *not* the same variable each time,
> > as printing the ref shows. The named sub add() firmly believes
> > in the very first variable named $num (and its value) while the
> > loop advances and uses a new one every time. A closure knows
> > about the changes.
>
> I wouldn't think that a closure knows about the changes. That
> closure only knows about the changes because a new closure gets created
> each time through the loop (each time using a new value of $num).
>
> But define the closure only once and it will always use the value of
> $num that it had the first time through the loop:
>
> my $add;
> while (my $input = <STDIN>){
> my $num = abs $input;
> $add = sub {
> $num + $num;
> } unless defined $add;
> print $add->(), "\n";
> }
>
> With this code, the $add only get assigned a closure once, and keeps
> the first value of $num each time.
Furthermore, if you define the named sub in a string eval, so that it is
recompiled each time, it will behave like the non-stultified anonymous
closure does.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
------------------------------
Date: Tue, 31 Oct 2006 07:39:02 +0100
From: Christian Winter <thepoet_nospam@arcor.de>
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <4546ef79$0$18845$9b4e6d93@newsspool4.arcor-online.net>
Michele Dondi schrieb:
> On Mon, 30 Oct 2006 17:37:43 +0100, "Ferry Bolhar"
> <bol@adv.magwien.gv.at> wrote:
>
>> while playing with code like this:
>>
>> use warnings;
>> while (my $input = <STDIN>){
>> my $num = abs $input;
>> print add();
>>
>> sub add {
>> $num + $num;
>> }
>> }
>>
>> I found out that once the loop get executed for the second
>> time, the variable $sum gets "splitted", taking the new value
>> from $input in the loop, but leaving the old, previous value
>> in add().
>
> This has been discussed before. Actually I have been claiming that
> contrary to the common assumption, closures are an orthogonal concept
> to that of anonymous subs. I was wrong: this turns out to be true in
> general, but it has been shown to me that in Perl (5, that is, in 6 I
> don't know (yet)) closures are only implemented as anonymous subs,
> since as you discovered yourself, named ones only close around the
> lexical scope surrounding them as of the first time they're found,
> thus failing to take a 'snapshot' of those lexicals at any given time,
> which is a fundamental closure charachteristic.
I believe you're doing the word "closure" a bit of injustice
here, just because most examples and descriptions out there
mix up closures and function templates. The latter ones being
implement as anonymous subs in Perl, that much is true.
However, I agree very much with Randal L. Schwartz in that
any kind of subroutine that "freezes" it referenced lexicals
from the surrounding scope is a closure. To me, the definition
that a closure is created every time the sub referencing outside
lexicals is created, still holds true.
Perl is just a bit trickier to understand in this regard,
because named subs have a longer lifetime than lexicals, which
isn't true for most functional languages, where the term stems
from. The sub is created only once, so it captures only once.
What would you call the sub "increment" in the following
example, if not a closure?
{
my $startcount = 5;
sub increment
{
++startcount;
}
}
Whether the surrounding block is just some text between curly
brackets, whether it has a "do" keyword in front, or whether it
makes up a loop or sub body, is in the first step of no importance
to Perl. It just gets important for the programmer to be aware of
due to the side effects that come up becasue named subs exist over
the whole package lifetime.
If you take a look at the perlref.pod in bleadperl and at
the devel version of perlfaq7 at
https://svn.perl.org/modules/perlfaq/trunk
you'll see that this reasonings are reflected in the docs now.
-Chris
------------------------------
Date: Tue, 31 Oct 2006 10:51:36 +0100
From: "Ferry Bolhar" <bol@adv.magwien.gv.at>
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <1162288296.658899@proxy.dienste.wien.at>
Michele Dondi:
> print '[anon]';
> print &$_ for
> map { my $x=$_; sub () { $x } }
> qw/foo bar baz/;
In this code, "sub" acts as an _operator_ and therefore
is evaluated for every loop iteration at run-time, yielding
a new value for $x each time.
> print '[named]';
> print &$_ for
> map { my $x=$_; sub foo () { $x }; \&foo }
> qw/foo bar baz/;
In this code, "sub" acts as a _declarator_, getting
evaluated only once because the function declaration
is processed at compile time. Therefore, the first value
for $x will be retained. It's jst another kind of loop as
in my example.
Still leaving the question open, why, when declaring
your second example as function and call it in the main
code, Perl will issue the "will not stay shared" message,
while running the code as-is, it does not, as in these
examples:
{ sub f {
my $r; my $r;
sub x { $r } sub x { $r }
} }
-- No message -- Variable "$r" will not stay shared...
And still leaving the question open, why this code _works_
as expected:
{
my $r = 1;
sub f { print $r }
f(); # print 1
$r++;
f(); # print 2
}
although, as mentioned in previous postings, in this case,
function f is evaluated (compiled) only once?
BTW: again, when writing this as function, Perl still will
issue the "will not stay shared" message, although here,
$r will stay shared!
Greetings, Ferry
--
Ing Ferry Bolhar
Magistrat der Stadt Wien - MA 14
A-1010 Wien
E-Mail: bol@adv.magwien.gv.at
------------------------------
Date: Tue, 31 Oct 2006 11:18:40 +0100
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <498ek2pkde029ukr3uhmt1dldn6qvmmkqu@4ax.com>
On Tue, 31 Oct 2006 07:39:02 +0100, Christian Winter
<thepoet_nospam@arcor.de> wrote:
>> This has been discussed before. Actually I have been claiming that
>> contrary to the common assumption, closures are an orthogonal concept
>> to that of anonymous subs. I was wrong: this turns out to be true in
[snip]
>However, I agree very much with Randal L. Schwartz in that
>any kind of subroutine that "freezes" it referenced lexicals
>from the surrounding scope is a closure. To me, the definition
>that a closure is created every time the sub referencing outside
>lexicals is created, still holds true.
Indeed that was my point too, in the discussion I was more or less
implicitly referring to. However the subject matter is at best
controversial, in fact in *that* discussion good arguments were
brought to disproof that a named sub that freezes lexicals from the
surrounding scope is an actual closure. (FWIW) all in all *I* would
still say that it *closes* around them. But I don't feel like delving
into endless discussions about exact terms that eventually would boil
down to just nitpicking. What's important, I hope, is that one has
clear what's going on...
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: Tue, 31 Oct 2006 11:35:01 +0100
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <239ek2h2d7td01f80ngc2dsp0i3cvu8tbm@4ax.com>
On Tue, 31 Oct 2006 10:51:36 +0100, "Ferry Bolhar"
<bol@adv.magwien.gv.at> wrote:
>> map { my $x=$_; sub () { $x } }
>> qw/foo bar baz/;
>
>In this code, "sub" acts as an _operator_ and therefore
^^^^^^^^^^
^^^^^^^^^^
>is evaluated for every loop iteration at run-time, yielding
>a new value for $x each time.
Not exactly, at least as of the docs. Although I (we all, I suppose)
*do* understand what you mean.
: C:\temp>perldoc -f sub
: sub NAME BLOCK
: sub NAME (PROTO) BLOCK
: sub NAME : ATTRS BLOCK
: sub NAME (PROTO) : ATTRS BLOCK
: This is subroutine definition, not a real function *per se*.
: Without a BLOCK it's just a forward declaration. Without a NAME,
: it's an anonymous function declaration, and does actually return
: a value: the CODE ref of the closure you just created.
>And still leaving the question open, why this code _works_
>as expected:
>
>{
> my $r = 1;
> sub f { print $r }
> f(); # print 1
> $r++;
> f(); # print 2
>}
Now, I don't find that confusing. What would you have expected? Note
that it wouldn't be different with
{
my $r = 1;
my $f = sub { print $r };
$f->(); $r++; $f->();
}
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: 31 Oct 2006 13:19:31 GMT
From: anno4000@radom.zrz.tu-berlin.de
Subject: Re: Interesting behaviour with lexical variable
Message-Id: <4qp0r3Fnlii8U1@news.dfncis.de>
jl_post@hotmail.com <jl_post@hotmail.com> wrote in comp.lang.perl.misc:
> anno4000@radom.zrz.tu-berlin.de wrote:
> >
> > while (my $input = <STDIN>){
> > my $num = abs $input;
> > my $add = sub {
> > $num + $num;
> > };
> > # sub add { $num + $num }
> > print $add->(), "\n";
> >
> > }
> >
> > Note that $num in the loop is *not* the same variable each time,
> > as printing the ref shows. The named sub add() firmly believes
> > in the very first variable named $num (and its value) while the
> > loop advances and uses a new one every time. A closure knows
> > about the changes.
>
> I wouldn't think that a closure knows about the changes. That
> closure only knows about the changes because a new closure gets created
> each time through the loop (each time using a new value of $num).
Not necessarily. There are two steps involved in th creation of a
Perl sub. One is compilation, creating code from source text. That
step happens only once for every Perl sub, named or anonymous. Another
step, which has been called sub dispatch, establishes the connection
of the sub with its lexical environment. This step happens at least
once, but with anonymous subs it can happen any number of times.
> But define the closure only once and it will always use the value of
> $num that it had the first time through the loop:
In the interest of clarity I'd s/define/dispatch/ in that sentence.
> my $add;
> while (my $input = <STDIN>){
> my $num = abs $input;
> $add = sub {
> $num + $num;
> } unless defined $add;
> print $add->(), "\n";
> }
>
> With this code, the $add only get assigned a closure once, and keeps
> the first value of $num each time.
Right, because the association with a lexical environment only happens
once.
I'm not sure exactly what action triggers sub dispatch but obviously
assignment has to do with it. But the lexical scope of the variable
*assigned to* also matters. When the scope is identical to the one
the sub lives in we see one behavior: The "sub { ... }" expression
returns the exact same coderef every time. If the scope is larger,
a different coderef is returned each time. That can be seen from
this code:
sub gensub {
my $val = shift;
sub { $val };
}
for ( 1 .. 3 ) {
my $x = gensub( $_);
print "$x ", $x->(), "\n";
}
print "===========================\n";
my $x;
for ( 1 .. 3 ) {
$x = gensub( $_);
print "$x ", $x->(), "\n";
}
which prints
CODE(0x812ec00) 1
CODE(0x812ec00) 2
CODE(0x812ec00) 3
===========================
CODE(0x812e180) 1
CODE(0x82333a8) 2
CODE(0x823333c) 3
The difference disappears when the code returned by gensub() doesn't
actually access (close over) a lexical variable. After changing the
last line of gensub() to "sub {}", all six coderefs turn out identical.
In other words, the behavior appears to depend on a number of decisions
that are made behind the scene, which leads to different behavior in
apparently similar situations.
Anno
------------------------------
Date: Tue, 31 Oct 2006 19:37:48 +0100
From: "andreas" <andreas./@\.poipoi.de>
Subject: killing processes using perl (artsd)
Message-Id: <5f916$454797fc$5499ae3f$30163@news2.eu.disputo.net>
Hello all,
I'm using an unstable debian distribution (Debian 2.6.18-3) and at some
day I found several artsd processes running in the process list. Normally
there should be only one sound server.
Now I started thinking about how to reduce the artsd process count using a
perl script and wrote something which works and can possibly be used to
remove any processes (cron etc.)
#!/usr/bin/perl -w
# file: killartsd.pl
# kill additional artsd processes - only one is required!
# (c)2006 Andreas Mueller andreas%at%poipoi.de
#####################################
# get artsd process ids from ps piped into a sed script
$plist=qx'ps h -C artsd | sed "s/\(\([ 0-9]*\|[0-9]*\).*\)/\2/g"';
# remove space+newline to form a list of pids
$plist =~ s/ \n/,/g;
# remove the ugly last comma in that list $plength=length($plist);
$plist=substr($plist,0,$plength-1) unless $plength < 1 ;
# convert result to an array because kill() wants an array, isn't it?
@parray=split(/,/,$plist);
# how many process ids are there?
$numproc=@parray;
# kill all unless there is only one or none
kill ('KILL',@parray) unless $numproc < 2 ;
My question is whether one knows a better way to do the whole thing
or parts of the script?
best regards
-andreas
------------------------------
Date: 30 Oct 2006 16:03:52 -0800
From: j.m.sedgwick@gmail.com
Subject: Re: Looking for a lightweight GUI editor on Windows
Message-Id: <1162253032.612820.216110@k70g2000cwa.googlegroups.com>
I'm liking textpad so far. I'll report back. Thanks for all your advice
guys.
------------------------------
Date: 30 Oct 2006 20:11:03 -0800
From: j.m.sedgwick@gmail.com
Subject: Re: Looking for a lightweight GUI editor on Windows
Message-Id: <1162267863.288894.102490@e64g2000cwd.googlegroups.com>
Wow, I really like it. Just wondering, the default perl syntax
highlighting in textpad does not highlight scalars, lists, hashes, etc
or give you options to choose colors. I tried adding
$*
@*
%*
to a category in perl5.syn but that's not working. Anyone know of a
fix, or, perhaps, a place I could find a better syntax file?
------------------------------
Date: 30 Oct 2006 14:36:54 -0800
From: "Rick" <rick.peng.du@gmail.com>
Subject: loop over a string to do search/replacement using regexp?
Message-Id: <1162247814.071103.255550@k70g2000cwa.googlegroups.com>
Hi, guys
I need to loop over a string to find some pattern A, do some
computation according to A and replace A with B. I am trying to do this
within a while loop but I have no idea how to plug s///g into this
loop. Can anyone offer some clue?
Here is an example: In string 'a123ba456b', the pattern to find is
a(.*?)b and this string needs to be processed as 'a(1)ba(2)b', meaning
that the occurance of '123' is recorded and put back into the original
string.
Thanks a million!
Rick
------------------------------
Date: Tue, 31 Oct 2006 01:22:38 +0100
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: loop over a string to do search/replacement using regexp?
Message-Id: <4qnje9Fnnl0oU1@individual.net>
Rick wrote:
> I need to loop over a string to find some pattern A, do some
> computation according to A and replace A with B. I am trying to do this
> within a while loop but I have no idea how to plug s///g into this
> loop. Can anyone offer some clue?
>
> Here is an example: In string 'a123ba456b', the pattern to find is
> a(.*?)b and this string needs to be processed as 'a(1)ba(2)b', meaning
> that the occurance of '123' is recorded and put back into the original
> string.
my $string = 'a123ba456b';
my (%vars, $i);
( my $newstring = $string ) =~ s{a(.*?)b}{
$vars{ ++$i } = $1;
"a($i)b";
}eg;
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: 30 Oct 2006 20:04:56 -0800
From: "brundlefly76@hotmail.com" <brundlefly76@hotmail.com>
Subject: Re: Naive threading performance questions
Message-Id: <1162267496.471058.278030@b28g2000cwb.googlegroups.com>
95% of the time when you think you want multithreading, you actually
want multiprocessing.
Check out Parallel::ForkManager....it is ridiculously simple and
robust.
I use it liberally on an SMP media processing farm that keeps a 8 cores
busy for 12 hours a day.
Just tell it how many processes you can use at a time and it will keep
all of them busy.
Worky Workerson wrote:
> I'm doing ETL for a database, i.e. line-by-line transformation of
> fairly large data sets. From some basic profiling, I've determined
> that the transformation process is relatively slow and I am heavily CPU
> bound on (i.e. the DB can take data 10 times faster than I can
> transform it).
>
> Since I am on an 8-way box and each line is independent of the others,
> I decided to try my hand using perl threads. I came up with a naive
> implementation (below), that spawns a couple of "transformation
> threads", where each thread is fed via a Thread::Queue.
>
> Unfortunately, the threaded implementation performs about 3 times
> *slower* than the single threaded implementation. Am I doing something
> horribly wrong? Is there something I can be doing better? Is there
> some hidden synchronization bottleneck that I'm not seeing, or is a
> Thread::Queue not very efficient? Are there some common idioms for
> threading that I am missing?
>
> Thanks!
>
> # Sorry if this is incorrect ... its hand-copied from an isolated lab
>
> use threads;
> use threads::shared;
> use Thread::Queue;
>
> my $num_threads = 5;
> my $finished_processing : shared = 0;
> my $data_queue = Thread::Queue->new();
>
> threads->create("process_lines") for (1..$num_threads);
>
> while (<>) { chomp; $data_queue->enqueue($_); }
>
> $finished_processing = 1;
> $_->join() foreach (threads->list());
>
> # Transform thread
> sub process_lines {
> while (1) {
> my $line = $data_queue->dequeue_nb();
> last if $finish_processing && !$line;
> next unless $line;
>
> # Do a line transformation ....
>
> print $line;
> }
> }
------------------------------
Date: Tue, 31 Oct 2006 05:42:09 GMT
From: merlyn@stonehenge.com (Randal Schwartz)
Subject: new CPAN modules on Tue Oct 31 2006
Message-Id: <J7zJu9.HqJ@zorch.sf-bay.org>
The following modules have recently been added to or updated in the
Comprehensive Perl Archive Network (CPAN). You can install them using the
instructions in the 'perlmodinstall' page included with your Perl
distribution.
BatchSystem-SBS-0.07
http://search.cpan.org/~alexmass/BatchSystem-SBS-0.07/
a Simple Batch System
----
Bio-Trace-ABIF-0.01
http://search.cpan.org/~vita/Bio-Trace-ABIF-0.01/
Perl extension for reading and parsing ABIF (Applied Biosystems, Inc. Format) files
----
Catalyst-Model-ISBNDB-0.20
http://search.cpan.org/~rjray/Catalyst-Model-ISBNDB-0.20/
Provide Catalyst access to isbndb.com
----
Catalyst-Plugin-BuildURI-0.0.1
http://search.cpan.org/~zigorou/Catalyst-Plugin-BuildURI-0.0.1/
Build URI by action name, namespace, and args
----
Class-CompiledC2.21
http://search.cpan.org/~blade/Class-CompiledC2.21/
----
FabForce-DBDesigner4-0.04
http://search.cpan.org/~reneeb/FabForce-DBDesigner4-0.04/
Parse/Analyse XML-Files created by DBDesigner 4 (FabForce)
----
Geo-Forward-0.01
http://search.cpan.org/~mrdvt/Geo-Forward-0.01/
Calculate geographic location form lat, lon, distance, heading.
----
Geo-Forward-0.02
http://search.cpan.org/~mrdvt/Geo-Forward-0.02/
Calculate geographic location form lat, lon, distance, heading.
----
Geo-Forward-0.03
http://search.cpan.org/~mrdvt/Geo-Forward-0.03/
Calculate geographic location from lat, lon, distance, and heading.
----
IO-AIO-2.2
http://search.cpan.org/~mlehmann/IO-AIO-2.2/
Asynchronous Input/Output
----
IPC-System-Simple-0.04
http://search.cpan.org/~pjf/IPC-System-Simple-0.04/
Call system() commands with a minimum of fuss
----
InSilicoSpectro-Databanks-0.0.15
http://search.cpan.org/~alexmass/InSilicoSpectro-Databanks-0.0.15/
parsing protein/nucleotides sequence databanks (fasta, uniprot...)
----
Mail-Thread-2.55
http://search.cpan.org/~tmtm/Mail-Thread-2.55/
Perl implementation of JWZ's mail threading algorithm
----
Mozilla-Mechanize-GUITester-0.01
http://search.cpan.org/~bosu/Mozilla-Mechanize-GUITester-0.01/
enhances Mozilla::Mechanize with GUI testing.
----
Net-FTP-Simple-0.0005
http://search.cpan.org/~wilco/Net-FTP-Simple-0.0005/
Simplified interface to a few common FTP tasks with Net::FTP.
----
Net-Stomp-0.32
http://search.cpan.org/~lbrocard/Net-Stomp-0.32/
A Streaming Text Orientated Messaging Protocol Client
----
SRU-0.94
http://search.cpan.org/~bricas/SRU-0.94/
Search and Retrieval by URL
----
SVN-Notify-Config-0.09
http://search.cpan.org/~jpeacock/SVN-Notify-Config-0.09/
Config-driven Subversion notification
----
Sdict-2.8
http://search.cpan.org/~swaj/Sdict-2.8/
Module to work with Sdictionary .dct files
----
Sledge-Utils-0.01
http://search.cpan.org/~tokuhirom/Sledge-Utils-0.01/
utility functions for Sledge
----
Spreadsheet-SimpleExcel-1.3
http://search.cpan.org/~reneeb/Spreadsheet-SimpleExcel-1.3/
Create Excel files with Perl
----
Test-use-ok-0.02
http://search.cpan.org/~audreyt/Test-use-ok-0.02/
Alternative to Test::More::use_ok
----
Tk-JComboBox-1.14
http://search.cpan.org/~rcseege/Tk-JComboBox-1.14/
Create and manipulate JComboBox widgets
----
UML-Class-Simple-0.01
http://search.cpan.org/~agent/UML-Class-Simple-0.01/
Render simple UML class diagrams, often by loading the code.
----
WWW-Blog-Metadata-AccountAutoDiscovery-0.03
http://search.cpan.org/~kentaro/WWW-Blog-Metadata-AccountAutoDiscovery-0.03/
Extract online accounts from a page
----
WWW-Salesforce-0.081
http://search.cpan.org/~capoeirab/WWW-Salesforce-0.081/
v0.081 - this class provides a simple abstraction layer between SOAP::Lite and Salesforce.com.
----
XML-Simple-2.16
http://search.cpan.org/~grantm/XML-Simple-2.16/
Easy API to maintain XML (esp config files)
----
threads-1.46
http://search.cpan.org/~jdhedden/threads-1.46/
Perl interpreter-based threads
If you're an author of one of these modules, please submit a detailed
announcement to comp.lang.perl.announce, and we'll pass it along.
This message was generated by a Perl program described in my Linux
Magazine column, which can be found on-line (along with more than
200 other freely available past column articles) at
http://www.stonehenge.com/merlyn/LinuxMag/col82.html
print "Just another Perl hacker," # the original
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 9909
***************************************
