[31397] in Perl-Users-Digest
Perl-Users Digest, Issue: 2649 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sat Oct 24 18:09:47 2009
Date: Sat, 24 Oct 2009 15:09:09 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sat, 24 Oct 2009 Volume: 11 Number: 2649
Today's topics:
Alternatives to LWP::Parallel <pete.sundstrom@gmail.com>
Re: File locking using threads QoS@invalid.net
Re: File locking using threads QoS@invalid.net
Re: How to prevent hanging when writing lots of text to <jl_post@hotmail.com>
Re: How to prevent hanging when writing lots of text to <ben@morrow.me.uk>
Re: How to prevent hanging when writing lots of text to <tzz@lifelogs.com>
Re: How to prevent hanging when writing lots of text to <derykus@gmail.com>
Re: How would I do this in perl? <tadmc@seesig.invalid>
Re: Indirection in a hash <mvdwege@mail.com>
Re: Want to write a script to note specific IP addresse <martien.verbruggen@invalid.see.sig>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 24 Oct 2009 00:13:11 -0700 (PDT)
From: Peter <pete.sundstrom@gmail.com>
Subject: Alternatives to LWP::Parallel
Message-Id: <7fe9ea71-c001-458a-8eec-7351b122c173@x5g2000prf.googlegroups.com>
I'm in the process of writing a script that will take a whole bunch of
URL's and get their HTTP status. I was going to use
LWP::Parallel::UserAgent to handle parallel HTTP requests. However, I
found that this module does not work with later version of libwww and
it looks like this module is no longer maintained.
So I started looking around at other methods/modules I could use.
There are certainly quite a few to choose from and some are quite
complex for my needs.
Some of the modules I looked at were:
IO::Lambda::HTTP (this looked the most robust/well maintained, but has
a learning curve to it)
POE::Component::Client::HTTP (as above)
AnyEvent::HTTP (this looked promising, but I couldn't get any results
with it)
HTTP::Client::Parallel (doesn't appear to have any function to limit
number of connections)
HTTP::Async (a little buggy and didn't seem to produce expected
results).
One essential feature I need is the ability to throttle/limit the
number of simultaneous connections or number of connections/requests
per minute.
I'd appreciate any advice as to what method/module might be best for
me. The other governing factor is that I need it to run with
ActivePerl on Linux and Windows.
------------------------------
Date: Sat, 24 Oct 2009 15:24:12 GMT
From: QoS@invalid.net
Subject: Re: File locking using threads
Message-Id: <w_EEm.8817$U55.6574@newsfe02.iad>
Prince Al <timothy.hill@gmail.com> wrote in message-id: <48e6e1e3-bc67-4171-83a5-d39b4b5d38ac@r36g2000vbn.googlegroups.com>
>
> Hi,
>
> I am attempting to write a some code that spawns various threads. Each
> thread will need to access a common file and to ensure that this file
> is only written to by one thread at a time, it needs to be locked.
> However, I am having trouble even creating a test script to try out
> the logic, which is below. What happens is that both threads are able
> to acquire the lock, even though the first thread should have it and
> release it before the second thread is able to lock the file. Any
> assistance will be very gratefully received!
>
> Cheers
>
> Tim
>
> #!usr/bin/perl
>
> use strict;
> use threads;
>
> my $thr_1 = threads->new(\&flock_test,5, 20, "Thread 1");
> my $thr_2 = threads->new(\&flock_test,10, 5, "Thread 2");
>
> grep {$_->join;} ($thr_1,$thr_2);
> # Just In Case!
> while ( my(@list)=threads->list()) {
> print "$#list\n";
> grep { $_->join } @list;
> };
> exit;
>
> sub flock_test {
> open(my $fh, ">", "/tmp/flock_test.dat");
> my $sleep_1 = $_[0];
> my $sleep_2 = $_[1];
> my $thread_name = $_[2];
>
> sleep $sleep_1;
>
> flock($fh,2) || die "Could not acquire the lock\n";
> print "$thread_name: acquired lock!\n";
> sleep $sleep_2;
> close $fh;
> print "$thread_name: released lock!\n";
> }
You might like threads::shared for inter thread communication.
You set up your shared variables then launch your threads.
Have the threads monitor and update the shared variables
to determine the status of the file or other things.
All threads will have access to the entire %shash.
foreach my $tID (1..2) {
warn 'Launching thread: [' . $tID . "]\n";
foreach my $l qw (fileIsLocked, go, stop, quit,) {
share($shash{1}{$l});
$shash{$tID}{$l} = 0;
}
$threads{$tID} = threads->new(\&worker, 1);
warn 'Thread: [' . $tID . "] is active\n";
}
sub worker #------------------------------------------------------------
{
#called from main
my $TID = $_[0] || 0;
while(1) {
if ($shash{$TID}{quit} == 1) {
last;
}
elsif ($shash{$TID}{fileIsLocked} == 1) {
}
#etc...
sleep (1);
}
return (1);
}
Hth,
J
------------------------------
Date: Sat, 24 Oct 2009 15:29:31 GMT
From: QoS@invalid.net
Subject: Re: File locking using threads
Message-Id: <v3FEm.8881$U55.889@newsfe02.iad>
QoS@invalid.net wrote in message-id: <w_EEm.8817$U55.6574@newsfe02.iad>
>
>
> Prince Al <timothy.hill@gmail.com> wrote in message-id: <48e6e1e3-bc67-4171-83a5-d39b4b5d38ac@r36g2000vbn.googlegroups.com>
>
> >
> > Hi,
> >
> > I am attempting to write a some code that spawns various threads. Each
> > thread will need to access a common file and to ensure that this file
> > is only written to by one thread at a time, it needs to be locked.
> > However, I am having trouble even creating a test script to try out
> > the logic, which is below. What happens is that both threads are able
> > to acquire the lock, even though the first thread should have it and
> > release it before the second thread is able to lock the file. Any
> > assistance will be very gratefully received!
> >
> > Cheers
> >
> > Tim
> >
> > #!usr/bin/perl
> >
> > use strict;
> > use threads;
> >
> > my $thr_1 = threads->new(\&flock_test,5, 20, "Thread 1");
> > my $thr_2 = threads->new(\&flock_test,10, 5, "Thread 2");
> >
> > grep {$_->join;} ($thr_1,$thr_2);
> > # Just In Case!
> > while ( my(@list)=threads->list()) {
> > print "$#list\n";
> > grep { $_->join } @list;
> > };
> > exit;
> >
> > sub flock_test {
> > open(my $fh, ">", "/tmp/flock_test.dat");
> > my $sleep_1 = $_[0];
> > my $sleep_2 = $_[1];
> > my $thread_name = $_[2];
> >
> > sleep $sleep_1;
> >
> > flock($fh,2) || die "Could not acquire the lock\n";
> > print "$thread_name: acquired lock!\n";
> > sleep $sleep_2;
> > close $fh;
> > print "$thread_name: released lock!\n";
> > }
>
> You might like threads::shared for inter thread communication.
>
> You set up your shared variables then launch your threads.
> Have the threads monitor and update the shared variables
> to determine the status of the file or other things.
>
> All threads will have access to the entire %shash.
>
> foreach my $tID (1..2) {
> warn 'Launching thread: [' . $tID . "]\n";
> foreach my $l qw (fileIsLocked, go, stop, quit,) {
> share($shash{1}{$l});
> $shash{$tID}{$l} = 0;
> }
> $threads{$tID} = threads->new(\&worker, 1);
> warn 'Thread: [' . $tID . "] is active\n";
> }
>
> sub worker #------------------------------------------------------------
> {
> #called from main
> my $TID = $_[0] || 0;
>
> while(1) {
> if ($shash{$TID}{quit} == 1) {
> last;
> }
> elsif ($shash{$TID}{fileIsLocked} == 1) {
>
> }
> #etc...
> sleep (1);
> }
> return (1);
> }
>
> Hth,
> J
Hrmph, typo, just half way through my am coffee and still a bit tired.
For this to work right you would have to fix above thread launching code
replacing the 1's within the foreach my $tID (1..2) {} block with $tID
------------------------------
Date: Fri, 23 Oct 2009 13:00:51 -0700 (PDT)
From: "jl_post@hotmail.com" <jl_post@hotmail.com>
Subject: Re: How to prevent hanging when writing lots of text to a pipe?
Message-Id: <f1c397a8-39ec-4de8-a0f5-335e867cd19e@e4g2000prn.googlegroups.com>
Thank you all for your excellent (and rapid) responses. They were
all helpful and clear.
On Oct 23, 11:01 am, Ben Morrow <b...@morrow.me.uk> wrote:
>
> The solution is to add
>
> close($writeHandle);
>
> as the first statement executed in the parent after the child is forked;
That worked! Thanks! Although, I'm a little puzzled: Why should
the parent close the $writeHandle? I would think that it's enough for
the child to close it. After all, the child process (and not the
parent) is the one that's doing the writing.
> Note: I have no idea if this will work on Win32. Win32 perl's fork
> emulation can sometimes be a little peculiar.
I agree with your statement (as I've encountered Windows' fork()ing
peculiarities myself). However, I tested the code with your
modifications on Windows (on Vista running Strawberry Perl) and it
works just fine.
So here is my final script:
#!/usr/bin/perl
use strict;
use warnings;
print "Enter a number: ";
my $number =3D <STDIN>;
chomp($number);
my @lines =3D do
{
pipe(my $readHandle, my $writeHandle);
if (fork() =3D=3D 0)
{
# Only the child process gets here:
print $writeHandle "$_\n" foreach 1 .. $number;
exit(0);
}
close($writeHandle);
# Extract the output, line-by-line:
local $/ =3D "\n";
<$readHandle>
};
print "Extracted output lines:\n @lines";
__END__
Note that no fileHandles are explicitly closed except for
$writeHandle in the parent thread. Perhaps I should close
$writeHandle in the child thread and $readHandle in the parent thread,
but I figured that since I declared both fileHandles lexically, then
they'll automatically be closed at the end of their own scopes.
(Of course, that won't be true if the fileHandles aren't lexically
declared, so that's something to keep in mind when not using lexical
fileHandles.)
However, ilovelinux's and J=FCrgen Exner's responses got me thinking
about a non-pipe() way of doing what I wanted, so I looked at "perldoc
-f open" and read up on writing to a scalar. I was able to re-write
my code so that it still wrote output to a fileHandle, and that output
ended up in a scalar (and eventually into an array).
Here is that code:
#!/usr/bin/perl
use strict;
use warnings;
print "Enter a number: ";
my $number =3D <STDIN>;
chomp($number);
my @lines =3D do
{
use 5.008; # for writing-to-scalar support
my $output;
open(my $writeHandle, '>', \$output)
or die "Cannot write to \$output scalar: $!\n";
print $writeHandle "$_\n" foreach 1 .. $number;
close($writeHandle);
# Split after all newlines, except for the
# one at the very end of the $output string:
split m/(?<=3D\n)(?!\z)/, $output;
};
print "Extracted output lines:\n @lines";
__END__
This works great, and without the need to invoke a child thread.
However, according to the documentation this capability (writing to a
scalar) is only available in Perl 5.8 or later.
I've checked, and the target machine I'm writing code for has Perl
5.8.8 (even though I personally use Perl 5.10). However, I want to
write my code so that it runs on most machines. As to what "most
machines" means is rather tricky; most machines I've run "perl -v" on
(that might need to run my script) reveal that they are running Perl
5.8, and I haven't found one yet running an older version.
So I have a question for all of you: Which of the two above
approaches should I use? The fork()ing approach in the first script
(that runs on Unix and on at least some Windows platforms), or the open
() approach that only runs with Perl 5.8 or later?
I suppose I can combine the two. I can check if the Perl version
is 5.8 or later, and if it is, use the approach that uses open(). If
not, fall back to fork()ing and hope for the best. And unless someone
suggests a simpler way, I can do it like so:
eval { require 5.008 };
if ($@)
{
# use the fork() approach
}
else
{
# use the open() approach
}
So which should I go for? Should I go with the fork(), open(), or
both? Any thoughts are appreciated.
Thanks again for all the help already given.
-- Jean-Luc
------------------------------
Date: Fri, 23 Oct 2009 21:16:47 +0100
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: How to prevent hanging when writing lots of text to a pipe?
Message-Id: <fif9r6-bvk.ln1@osiris.mauzo.dyndns.org>
Quoth "jl_post@hotmail.com" <jl_post@hotmail.com>:
>
> Thank you all for your excellent (and rapid) responses. They were
> all helpful and clear.
>
> On Oct 23, 11:01 am, Ben Morrow <b...@morrow.me.uk> wrote:
> >
> > The solution is to add
> >
> > close($writeHandle);
> >
> > as the first statement executed in the parent after the child is forked;
>
>
> That worked! Thanks! Although, I'm a little puzzled: Why should
> the parent close the $writeHandle? I would think that it's enough for
> the child to close it. After all, the child process (and not the
> parent) is the one that's doing the writing.
A pipe doesn't report EOF until there are no handles on it opened for
writing. The parent still has its write handle open, and for all the OS
knows it might be wanting to write to the pipe too.
<snip>
> Note that no fileHandles are explicitly closed except for
> $writeHandle in the parent thread. Perhaps I should close
> $writeHandle in the child thread and $readHandle in the parent thread,
> but I figured that since I declared both fileHandles lexically, then
> they'll automatically be closed at the end of their own scopes.
The only problem with this approach is that an error on close will be
ignored. A handle open for writing should usually be explicitly closed
and the return value of close checked, to make sure everything wrote
successfully.
> This works great, and without the need to invoke a child thread.
> However, according to the documentation this capability (writing to a
> scalar) is only available in Perl 5.8 or later.
You can use IO::Scalar under 5.6. (Indeed, you could simply use
IO::Scalar under all versions of perl: it's a little less efficient and
a little less pretty than the PerlIO-based solution in 5.8, but it will
work just fine.)
> I've checked, and the target machine I'm writing code for has Perl
> 5.8.8 (even though I personally use Perl 5.10). However, I want to
> write my code so that it runs on most machines. As to what "most
> machines" means is rather tricky; most machines I've run "perl -v" on
> (that might need to run my script) reveal that they are running Perl
> 5.8, and I haven't found one yet running an older version.
5.6 is not dead yet, though it's getting there. OTOH, requiring 5.8 is
probably usual for Perl code nowadays.
Ben
------------------------------
Date: Fri, 23 Oct 2009 17:41:20 -0500
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: How to prevent hanging when writing lots of text to a pipe?
Message-Id: <87d44dg2cv.fsf@lifelogs.com>
On Fri, 23 Oct 2009 08:47:30 -0700 (PDT) "jl_post@hotmail.com" <jl_post@hotmail.com> wrote:
jpc> I'm trying to write text to a piped writeHandle (created with the
jpc> pipe() function) so that I can later read the text by extracting it
jpc> from a readHandle. However, I discovered that if I write a lot of
jpc> text to the pipe, my script just hangs. Here is an example program:
One way to do this is to write a file name to the pipe. Then the client
just looks at that file for the actual data when the I/O arrives.
Depending on the size and frequency of the data updates, you could also
use a SQLite database file. It supports access from multiple clients so
your client can just SELECT the new data if you have some way of
detecting it (maybe pass the row IDs over the pipe), and the server just
writes (INSERT/UPDATE) the data whenever it wants.
Ted
------------------------------
Date: Fri, 23 Oct 2009 19:44:49 -0700 (PDT)
From: "C.DeRykus" <derykus@gmail.com>
Subject: Re: How to prevent hanging when writing lots of text to a pipe?
Message-Id: <95ca3c6d-5047-4256-9b0e-85c78b1328db@u16g2000pru.googlegroups.com>
On Oct 23, 1:00=A0pm, "jl_p...@hotmail.com" <jl_p...@hotmail.com> wrote:
> ...
> =A0 =A0However, ilovelinux's and J=FCrgen Exner's responses got me thinki=
ng
> about a non-pipe() way of doing what I wanted, so I looked at "perldoc
> -f open" and read up on writing to a scalar. =A0I was able to re-write
> my code so that it still wrote output to a fileHandle, and that output
> ended up in a scalar (and eventually into an array).
>
> =A0 =A0Here is that code:
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
> print "Enter a number: ";
> my $number =3D <STDIN>;
> chomp($number);
>
> my @lines =3D do
> {
> use 5.008; # for writing-to-scalar support
> my $output;
> open(my $writeHandle, '>', \$output)
> or die "Cannot write to \$output scalar: $!\n";
>
> print $writeHandle "$_\n" foreach 1 .. $number;
>
> =A0 =A0close($writeHandle);
>
> =A0 =A0# Split after all newlines, except for the
> =A0 =A0# one at the very end of the $output string:
> =A0 =A0split m/(?<=3D\n)(?!\z)/, $output;
>
> };
>
> print "Extracted output lines:\n @lines";
>
> __END__
>
> =A0 =A0This works great, and without the need to invoke a child thread.
> However, according to the documentation this capability (writing to a
> scalar) is only available in Perl 5.8 or later.
>...
Hm, if there's no IPC involved, can't you simply populate
an array directly...eliminating filehandles, Perl version
worries, and the 'do' statement completely. Did I miss
something else?
my @lines;
push @lines, "$_\n" for 1 .. $number;
print "Extracted output lines:\n @lines";
--
Charles DeRykus
------------------------------
Date: Sat, 24 Oct 2009 15:34:16 -0500
From: Tad J McClellan <tadmc@seesig.invalid>
Subject: Re: How would I do this in perl?
Message-Id: <slrnhe6o8p.84j.tadmc@tadmc30.sbcglobal.net>
ccc31807 <cartercc@gmail.com> wrote:
> On Oct 20, 11:42 am, Ben Morrow <b...@morrow.me.uk> wrote:
>> AIUI reading (well-written) Lisp is supposed to be easy,
> My trouble with reading Lisp is that the style seems to promote the
> multiplication of small functions that are deeply nested,
And throw in a government bureaucracy like the Strategic Defense Initiative,
and you get one of the immortals from r.h.f:
http://newsgroups.derkeiler.com/Archive/Rec/rec.humor.funny.reruns/2006-04/msg00014.html
--
Tad McClellan
email: perl -le "print scalar reverse qq/moc.noitatibaher\100cmdat/"
------------------------------
Date: Sat, 24 Oct 2009 16:53:09 +0200
From: Mart van de Wege <mvdwege@mail.com>
Subject: Re: Indirection in a hash
Message-Id: <861vkslu7e.fsf@gareth.avalon.lan>
"Uri Guttman" <uri@StemSystems.com> writes:
>
> MvdW> I think we're talking past each other. That the compiler can't directly
> MvdW> access a value does not mean that the language syntax doesn't allow
> MvdW> it. It merely makes this an inefficient operation.
>
> no, i get both sides very well. i worked in a compiler company way back
> and know perl well too.
I have written code in low-level languages like Forth and 6502 assembly,
where I had to implement common data structures myself, so I know a bit
about this subject too. You obviously know a lot more, but I agree in
principle with your points.
> i was just emphasizing the difference between the abstraction of
> multidim arrays which perl supports and the implementation which is
> slower in perl. it does matter sometimes and you can't just say perl
> supports multidim arrays without that context.
>
I tend to take that context for granted, as I am fairly comfortable in
Perl. But you are right, it is not exactly 'walks like a duck, quacks
like a duck'. It is damn close though.
Mart
--
"We will need a longer wall when the revolution comes."
--- AJS, quoting an uncertain source.
------------------------------
Date: Sat, 24 Oct 2009 07:56:12 +1100
From: Martien Verbruggen <martien.verbruggen@invalid.see.sig>
Subject: Re: Want to write a script to note specific IP addresses.
Message-Id: <c95tbh.e65.ln@news.heliotrope.home>
On Thu, 22 Oct 2009 16:13:15 +0800,
Hongyi Zhao <hongyi.zhao@gmail.com> wrote:
> On Tue, 20 Oct 2009 13:20:18 -0500, "J. Gleixner"
><glex_no-spam@qwest-spam-no.invalid> wrote:
>
>>Quite a few modules that help with IPs, however
>>you could translate the IPs to integers and simply
>>check if one IP is between the start and end.
>>
>>Place to start:
>>
>>use Net::Netmask qw(quad2int);
>>
>>
>>To open/read/process files:
>>
>>perldoc -f open
>>perldoc perlopentut
>
> Currently, I use a complete binary IPdatabase named QQWry.Dat which
> include 373374 IP blocks oall over the world, you can download this
> IPdatabase from the following url:
>
> http://update.cz88.net/soft/qqwry.rar
> The above qqwry.pl has the following content:
>
> ------------
> sub ipwhere {
This is rather convoluted. Whoever wrote this code obviously had
documentation on the file format of QQWry.Dat.
> my $ipfile="./QQWry.Dat";
As another poster already suggested, create a SQL database once, from
this file (use SQLite or something like that) with a table with the
start and end ip in numerical form in two columns. Then use another
program to select from that database, as often as you want.
You could even create a second table in there, load in your other IP
addresses, and join, although that is unlikely to be faster than looping
explicitly.
I started trying to reverse engineer the file format from the code you
supplied, but got too tired after trying to read it for about 5 minutes.
I'm assuming the file format is described in one of the other files in
that archive, but, unfortunately, I don't read the language they're in.
Martien
--
|
Martien Verbruggen | For heaven's sake, don't TRY to be
first.last@heliotrope.com.au | cynical. It's perfectly easy to be
| cynical.
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 2649
***************************************
