[32081] in Perl-Users-Digest
Perl-Users Digest, Issue: 3345 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Thu Apr 7 11:09:27 2011
Date: Thu, 7 Apr 2011 08:09:10 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Thu, 7 Apr 2011 Volume: 11 Number: 3345
Today's topics:
How to queue each loop iteration into a thread pool, wi <davidmichaelkarr@gmail.com>
Re: How to queue each loop iteration into a thread pool <cartercc@gmail.com>
Re: How to queue each loop iteration into a thread pool <willem@toad.stack.nl>
Re: Posting Guidelines for comp.lang.perl.misc ($Revisi sln@netherlands.com
Re: Posting Guidelines for comp.lang.perl.misc ($Revisi <ralph@happydays.com>
Re: split a special character stored in variable <tzz@lifelogs.com>
Re: Use of uninitialized value in open <hjp-usenet2@hjp.at>
Re: Use of uninitialized value in open <tadmc@seesig.invalid>
Why is array pushed in thread sub not set after join? <davidmichaelkarr@gmail.com>
Re: Why is array pushed in thread sub not set after joi <willem@toad.stack.nl>
Re: Why is array pushed in thread sub not set after joi <davidmichaelkarr@gmail.com>
Re: Why is array pushed in thread sub not set after joi <cartercc@gmail.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Wed, 6 Apr 2011 12:47:41 -0700 (PDT)
From: David Karr <davidmichaelkarr@gmail.com>
Subject: How to queue each loop iteration into a thread pool, with a max size?
Message-Id: <2d3579ad-c25d-479e-a9d1-297f29c5f9b9@glegroupsg2000goo.googlegroups.com>
I have a loop nested inside another loop. Each iteration of the inner loop=
is independent, so I could conceivably create a thread for each iteration.=
However, I would think that would create too many threads (couple hundred=
) and probably make my box thrash before it dies.
Is there a practical way to make it queue N loop iterations and then block =
on the N+1st until a slot is available? If I was doing this in Java I woul=
d use a ThreadPoolExecutor, but I'm not aware of a similar mechanism in Per=
l (not already written as a reusable library, that is).
------------------------------
Date: Thu, 7 Apr 2011 06:35:45 -0700 (PDT)
From: ccc31807 <cartercc@gmail.com>
Subject: Re: How to queue each loop iteration into a thread pool, with a max size?
Message-Id: <87928246-a4cd-4917-a34e-acce87c8d317@x1g2000yqb.googlegroups.com>
On Apr 6, 3:47=A0pm, David Karr <davidmichaelk...@gmail.com> wrote:
> I have a loop nested inside another loop. =A0Each iteration of the inner =
loop is independent, so I could conceivably create a thread for each iterat=
ion.
If the inner loop is independent of the outer loop, why loop them?
Obviously, you don't mean 'independent' in the usual sense, so what
I'm understanding you to say that there are no shared variables
between the two loops, but the environment of the inner loop depends
on the state of the outer loop.
One simple minded example that comes to me is averaging the grades of
students in a class -- you want to loop through the students one at a
time, but the students don't share state with each other. In such a
case, I could see what you might think of as an 'outer loop' given the
task of opening a file, reading the values line by line, packing each
line in an array (or a reference to an array), and calling a separate
Perl script as the 'inner loop', passing the array or reference to the
caller, or maybe writing independently to a file.
> Is there a practical way to make it queue N loop iterations and then bloc=
k on the N+1st until a slot is available? =A0If I was doing this in Java I =
would use a ThreadPoolExecutor, but I'm not aware of a similar mechanism in=
Perl (not already written as a reusable library, that is).
If the inner loop is independent of the outer loop, why would you want
the outer loop to block? Could you give a simple example of your
problem?
CC.
------------------------------
Date: Thu, 7 Apr 2011 15:00:15 +0000 (UTC)
From: Willem <willem@toad.stack.nl>
Subject: Re: How to queue each loop iteration into a thread pool, with a max size?
Message-Id: <slrniprkbv.1smr.willem@toad.stack.nl>
David Karr wrote:
) I have a loop nested inside another loop. Each iteration of the inner
) loop is independent, so I could conceivably create a thread for each
) iteration. However, I would think that would create too many threads
) (couple hundred) and probably make my box thrash before it dies.
)
) Is there a practical way to make it queue N loop iterations and then
) block on the N+1st until a slot is available? If I was doing this in
) Java I would use a ThreadPoolExecutor, but I'm not aware of a similar
) mechanism in Perl (not already written as a reusable library, that is).
You mean like Thread::Pool ?
SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
------------------------------
Date: Wed, 06 Apr 2011 17:02:31 -0700
From: sln@netherlands.com
Subject: Re: Posting Guidelines for comp.lang.perl.misc ($Revision: 1.9 $)
Message-Id: <pmvpp6pqe505h40hupku6chb31o6nahshu@4ax.com>
On Tue, 05 Apr 2011 02:14:53 -0500, tadmc@seesig.invalid wrote:
>Do not post binaries, HTML, or MIME
I'm just wondering if this means all the html posted on this ng is a violation of the
law?
-sln
------------------------------
Date: Thu, 07 Apr 2011 10:47:40 -0400
From: Ralph Malph <ralph@happydays.com>
Subject: Re: Posting Guidelines for comp.lang.perl.misc ($Revision: 1.9 $)
Message-Id: <e9988$4d9dce8d$ce534406$4946@news.eurofeeds.com>
On 4/6/2011 8:02 PM, sln@netherlands.com wrote:
> On Tue, 05 Apr 2011 02:14:53 -0500, tadmc@seesig.invalid wrote:
>
>> Do not post binaries, HTML, or MIME
>
> I'm just wondering if this means all the html posted on this ng is a violation of the
> law?
Nah, it just means that nobody reads, much less heeds, this autoposted
pile of shit.
usenet is a community of people, not a community of rules.
This clearly is not understood by some people.
------------------------------
Date: Wed, 06 Apr 2011 08:54:57 -0500
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: split a special character stored in variable
Message-Id: <871v1fqyby.fsf@lifelogs.com>
On Tue, 05 Apr 2011 11:40:02 -0700 Joe Smith <jms@sendmail.com> wrote:
JS> On 2011-04-01 7:08 AM, Ted Zlatanov wrote:
>> On Fri, ***1 Apr*** 2011 12:29:40 -0700 "ela"<ela@yantai.org> wrote:
>>
e> @subcells = /$delimiter/, $line;
>>
>> As Perl really, incredibly, lacks full out-of-ligature spacing,
JS> What does "out-of-ligature" spacing mean?
It has the first letters "ool" (take the first letters on that line ;)
Ted
------------------------------
Date: Wed, 6 Apr 2011 09:26:56 +0200
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: Use of uninitialized value in open
Message-Id: <slrnipo5e0.8ju.hjp-usenet2@hrunkner.hjp.at>
On 2011-04-05 22:27, Mark <google@markginsburg.com> wrote:
> I get "Use of uninitialized value in open" the second time through the
> loop. However, if I declare $buff before the loop (and not in the
> loop), I don't get the error.
>
> Can someone explain what is happening?
[...]
> foreach (1..2) {
> print "starting iteration $_\n";
> my $buff;
> open(my $TMPBUF,'>',\$buff) || die "open, $!\n";
[...]
> }
>
> Output::
> starting iteration 1
> starting iteration 2
> Use of uninitialized value in open at j.pl line 10.
I get this warning with perl 5.8.8 but not with 5.10.1, so I guess it's
a bug which was fixed in perl 5.10.
As a workaround you can just assign an empty string to $buff:
my $buff = "";
hp
------------------------------
Date: Wed, 06 Apr 2011 07:22:01 -0500
From: Tad McClellan <tadmc@seesig.invalid>
Subject: Re: Use of uninitialized value in open
Message-Id: <slrnipome7.qbo.tadmc@tadbox.sbcglobal.net>
Peter J. Holzer <hjp-usenet2@hjp.at> wrote:
> On 2011-04-05 22:27, Mark <google@markginsburg.com> wrote:
>> I get "Use of uninitialized value in open" the second time through the
>> loop. However, if I declare $buff before the loop (and not in the
>> loop), I don't get the error.
>>
>> Can someone explain what is happening?
> [...]
>> foreach (1..2) {
>> print "starting iteration $_\n";
>> my $buff;
>> open(my $TMPBUF,'>',\$buff) || die "open, $!\n";
> [...]
>> }
>>
>> Output::
>> starting iteration 1
>> starting iteration 2
>> Use of uninitialized value in open at j.pl line 10.
>
> I get this warning with perl 5.8.8 but not with 5.10.1, so I guess it's
> a bug which was fixed in perl 5.10.
I get it with 5.10.0, so it must have been 5.10.1 that fixed it...
--
Tad McClellan
email: perl -le "print scalar reverse qq/moc.liamg\100cm.j.dat/"
The above message is a Usenet post.
I don't recall having given anyone permission to use it on a Web site.
------------------------------
Date: Wed, 6 Apr 2011 12:15:00 -0700 (PDT)
From: David Karr <davidmichaelkarr@gmail.com>
Subject: Why is array pushed in thread sub not set after join?
Message-Id: <83315645-6a85-44ed-9acd-a510ef7047a3@glegroupsg2000goo.googlegroups.com>
Perl 5.10.0 on Cygwin 1.5.25.
I have a loop where each iteration makes 5 independent REST calls. After e=
ach of those 5 calls, some data from the result is pushed onto an array tha=
t is declared before the loop. After the loop, I iterate through the resul=
ting array. When this is implemented without threads, this works fine.
I decided that since each of those 5 calls are independent, it might be fas=
ter if I executed each of them in a thread, and then "join"ed each of the t=
hreads at the end of the iteration, using the "void" context, as I don't ne=
ed a return value from each of those (just the side effect of pushing onto =
the array).
the print statements seemed to show this was working fine, and it did appea=
r to go faster, but after the loop I checked the size of the array whose co=
ntents had been pushed from each of those threads, and it was size 0.
I also tried adding "use threads::shared;" after my "use threads;" declarat=
ion, but that had no effect.
Is there anything obviously or likely wrong with my approach?
The changes I made to implement threads were pretty simple. I just added a=
line like this before each of the 5 blocks:
my $xxthread =3D threads->create(sub {
with a matching "});" at the end of each of the 5 blocks, and then a set of=
5 "$xxthread->join();" calls at the end of the iteration.
There were 5 different "xx" values, as I stored each thread in a separate v=
ariable.
------------------------------
Date: Wed, 6 Apr 2011 19:28:03 +0000 (UTC)
From: Willem <willem@toad.stack.nl>
Subject: Re: Why is array pushed in thread sub not set after join?
Message-Id: <slrnippfm3.17he.willem@toad.stack.nl>
David Karr wrote:
) Perl 5.10.0 on Cygwin 1.5.25.
)
) I have a loop where each iteration makes 5 independent REST calls. After
) each of those 5 calls, some data from the result is pushed onto an array
) that is declared before the loop. After the loop, I iterate through the
) resulting array. When this is implemented without threads, this works
) fine.
)
) I decided that since each of those 5 calls are independent, it might be
) faster if I executed each of them in a thread, and then "join"ed each of
) the threads at the end of the iteration, using the "void" context, as I
) don't need a return value from each of those (just the side effect of
) pushing onto the array).
It would have been easier if you had the threads return the wanted value,
and then pushing that into the array when you 'join' them:
) the print statements seemed to show this was working fine, and it did
) appear to go faster, but after the loop I checked the size of the array
) whose contents had been pushed from each of those threads, and it was
) size 0.
)
) I also tried adding "use threads::shared;" after my "use threads;"
) declaration, but that had no effect.
)
) Is there anything obviously or likely wrong with my approach?
You probably need to specifically mark the array as a shared variable.
) The changes I made to implement threads were pretty simple. I just added
) a line like this before each of the 5 blocks:
)
) my $xxthread = threads->create(sub {
)
) with a matching "});" at the end of each of the 5 blocks, and then a set
) of 5 "$xxthread->join();" calls at the end of the iteration.
)
) There were 5 different "xx" values, as I stored each thread in a separate
) variable.
If you have 5 different 'xx' values, you're usually better off using an
array.
Returning the values and pushing them into an array when joining is a lot
cleaner, as it doesn't require sharing variables:
my @results = map { $_->join } map { threads->create(sub {
return get("http://www.example.invalid/$_.html");
}) } qw/some thing or other to get/;
And even if the five functions do something wildly different,
you can still do it like that, using an array of anon-subs:
my @results = map { $_->join } map { threads->create($_) } (
sub { return get('something_one'); },
sub { return do_something('two'); },
sub { $result = foo(); return $result; },
sub { return another_function() + 100; },
sub { return 1 + 2; }
);
HTH, HAND.
SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
------------------------------
Date: Wed, 6 Apr 2011 13:46:46 -0700 (PDT)
From: David Karr <davidmichaelkarr@gmail.com>
Subject: Re: Why is array pushed in thread sub not set after join?
Message-Id: <02cf2c63-df64-4e94-90f5-497eb8b8d40b@glegroupsg2000goo.googlegroups.com>
Having the thread return the sublist instead of relying on shared data was very useful. That resolved my problem.
------------------------------
Date: Thu, 7 Apr 2011 06:44:21 -0700 (PDT)
From: ccc31807 <cartercc@gmail.com>
Subject: Re: Why is array pushed in thread sub not set after join?
Message-Id: <02bfcc3c-b446-44c3-a684-82d4dd8decbe@w21g2000yqm.googlegroups.com>
On Apr 6, 4:46=A0pm, David Karr <davidmichaelk...@gmail.com> wrote:
> Having the thread return the sublist instead of relying on shared data wa=
s very useful. =A0That resolved my problem.
Languages seem to be moving toward a functional paradigm. Having
played with Erlang recently, I see that you could solve your problem
easily and quickly with a functional language (like Erlang) using
Erlang's message passing paradigm rather than shared state. This isn't
to say that one language is 'better' than another, just that different
languages are optimized for different jobs.
In any case, a functional kind of approach seems to solve some kinds
of problems that a shared data approach finds hard. This little essay
is interesting:
http://www.defmacro.org/ramblings/fp.html
CC.
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 3345
***************************************
