[32438] in Perl-Users-Digest
Perl-Users Digest, Issue: 3705 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sun Jun 3 03:09:25 2012
Date: Sun, 3 Jun 2012 00:09:07 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sun, 3 Jun 2012 Volume: 11 Number: 3705
Today's topics:
Child processes don't get the close on pipe <lothar.braun@googlemail.com>
Re: Child processes don't get the close on pipe <rweikusat@mssgmbh.com>
Re: Help with install from CPAN - LONG <ben@morrow.me.uk>
Re: Help with install from CPAN <ben@morrow.me.uk>
Re: Help with install from CPAN <steve53@nomail.earthlink.net>
Re: Help with install from CPAN <dave@invalid.invalid>
Re: Help with install from CPAN <ben@morrow.me.uk>
Re: Help with install from CPAN (Seymour J.)
Re: Help with install from CPAN <steve53@nomail.earthlink.net>
How to properly recursively navigate into archives stor <davidmichaelkarr@gmail.com>
Re: How to properly recursively navigate into archives <ben@morrow.me.uk>
Re: How to properly recursively navigate into archives <davidmichaelkarr@gmail.com>
Re: How to properly recursively navigate into archives <glex_no-spam@qwest-spam-no.invalid>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 2 Jun 2012 04:53:03 -0700 (PDT)
From: Lothar Braun <lothar.braun@googlemail.com>
Subject: Child processes don't get the close on pipe
Message-Id: <2e29b3aa-debe-487c-b81d-3dcff3ef4972@a16g2000vby.googlegroups.com>
Hi,
I have a problem with pipes that I do not understand at all. My code
forks a number of child processes, and creates a pipe for every one of
them. The parent then writes data to the child's pipes, finishing the
whole process by closing the pipe.
The children consume to data and write results back to the parent
through the other side of the pipe. The problem is that the children
do not get an EOF on the pipes and expect more data, resulting in a
deadlock. I can't see what the problem with my code could be and
therefore extracted the relevant parts into the script pasted below
(you can also find it at pastebin: http://pastebin.com/KarksDXf )
Can someone point me to the problematic parts of the code?
#!/usr/bin/env perl -w
use strict;
use POSIX;
my $num_processes = 2;
my %process_data;
sub worker_thread {
my $id = shift;
my $reader = shift;
my $writer = shift;
print STDERR "Child $id: Starting to consume input ...\n";
while(<$reader>) {
chomp;
print STDERR "$id: Got line \"$_\"\"\n";
}
print STDERR "Finished consuming packets ...";
print $writer "$id: this is my result\n";
}
sub parent_reader {
for (0 .. $num_processes - 1) {
my $writer = ${$process_data{$_}}[1];
print $writer "Got some work\n";
}
}
################ main
# start the worker processes
for my $pnum (0 .. $num_processes - 1) {
print $pnum, "\n";
local *READER;
local *WRITER;
pipe(*READER, *WRITER);
my $pid = fork();
if (defined $pid && $pid == 0) {
worker_thread($pnum, *READER, *WRITER);
exit -1;
} elsif (defined $pid && $pid > 0) {
$process_data{$pnum} = [ *READER, *WRITER, $pnum];
}
}
# run the main reader loop, and pass data to the readers
parent_reader();
print "Parent: Finished reading input data from stdin...\n";
# close the pipes to indicate that there will be no more data
for my $i (keys %process_data) {
my $writer = $process_data{$i}[1];
print scalar $writer, "\n";
close($writer) or warn "Error closing pipe: $!";
}
# collect the results
for my $i (keys %process_data) {
my $reader = ${process_data{$i}}[0];
while (<$reader>) {
print "$_";
}
}
# wait for child processes to end
for my $i (keys %process_data) {
my $ret = waitpid(-1, 0);
if ($ret<0 || $ret>0) {
print "process $ret finished..\n";
delete $process_data{$ret};
}
}
------------------------------
Date: Sat, 02 Jun 2012 15:55:18 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: Child processes don't get the close on pipe
Message-Id: <87mx4lbw6h.fsf@sapphire.mobileactivedefense.com>
Lothar Braun <lothar.braun@googlemail.com> writes:
>
> I have a problem with pipes that I do not understand at all. My code
> forks a number of child processes, and creates a pipe for every one of
> them. The parent then writes data to the child's pipes, finishing the
> whole process by closing the pipe.
>
> The children consume to data and write results back to the parent
> through the other side of the pipe. The problem is that the children
> do not get an EOF on the pipes
[...]
> sub worker_thread {
> my $id = shift;
> my $reader = shift;
> my $writer = shift;
>
> print STDERR "Child $id: Starting to consume input ...\n";
> while(<$reader>) {
> chomp;
> print STDERR "$id: Got line \"$_\"\"\n";
> }
>
> print STDERR "Finished consuming packets ...";
> print $writer "$id: this is my result\n";
> }
[...]
> for my $pnum (0 .. $num_processes - 1) {
> print $pnum, "\n";
> local *READER;
> local *WRITER;
> pipe(*READER, *WRITER);
> my $pid = fork();
> if (defined $pid && $pid == 0) {
> worker_thread($pnum, *READER, *WRITER);
[...]
You'll get an EOF on a pipe as soon as one side of it has been
closed. Since the child inherits the 'writer' handle but doesn't close
it, this will never happen. You could use two pipes instead but it is
probably easier to use an AF_UNIX SOCK_STREAM socketpair (=> perldoc
-f socketpair), closing one of the handles in the parent process and
the other in the child.
------------------------------
Date: Fri, 1 Jun 2012 18:27:52 +0100
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: Help with install from CPAN - LONG
Message-Id: <olan99-62s.ln1@anubis.morrow.me.uk>
Quoth "Dave Saville" <dave@invalid.invalid>:
> On Tue, 29 May 2012 22:49:17 UTC, Ben Morrow <ben@morrow.me.uk> wrote:
>
> > Maybe the first thing to do is run the tests for TAP::* and see what
> > fails? Make sure HARNESS_SUBCLASS is *unset*, then
> >
> > cpan> look Test::Harness
> > $ perl Makefile.PL
> > $ make
> >
> > and then go through t/*.t and t/compat/*.t and run each one through
> >
> > perl -Mblib t/x.t | grep -E '^not ok'
> >
> > to see what fails.
>
> Lots and lots :-(
That's not actually a bad thing :). A failure that can't be reproduced
is a lot worse than one which can.
<snip>
> iterators.t
> 1..76
> not ok 12 - Process: ... and exit should now return 0
> (TAP::Parser::Iterator::Process)
> # Failed test 'Process: ... and exit should now return 0
> (TAP::Parser::Iterator::Process)'
> # at t/iterators.t line 123.
> # got: '-1'
> # expected: '0'
<snip>
> not ok 48 - Process (Perl -e): next() should return the first result
> # Failed test 'Process (Perl -e): next() should return the first
> result'
> # at t/iterators.t line 108.
> # got: undef
> # expected: 'one'
That's interesting. I was wondering if this was something to do with
your bad @INC remapping, but (AFAICS) this shouldn't be affected by
that.
If you 'make install' this version of Test::Harness for now, since it
doesn't seem to be any *worse* than your current version and at least
that way things are consistent, and then run this
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Parser::Iterator::Process;
my $I = TAP::Parser::Iterator::Process->new({
command => ["perl", "-le", "print for 123, 456"],
merge => 1,
});
while (my $l = $I->next) {
warn "LINE [$l]\n";
}
warn "WAIT [" . $I->wait . "]\n";
warn "EXIT [" . $I->exit . "]\n";
what do you get? Do you get different results if you run it several
times?
Ben
------------------------------
Date: Fri, 1 Jun 2012 02:18:46 +0100
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: Help with install from CPAN
Message-Id: <mshl99-92h.ln1@anubis.morrow.me.uk>
Quoth steve53@earthlink.net:
> On Thu, 31 May 2012 15:08:45 UTC, Ben Morrow <ben@morrow.me.uk> wrote:
>
> Since I am getting quoted here, I figure I might as well join in.
Thank you for pitching in... I'm doing my best, but I really know very
little about OS/2.
> I may have some bad news. Against my better nature, I installed a
> copy of the OS/2 5.14.2 in a location that avoids the @INC issues.
>
> Ben's try.pl script works perfectly here run with either perl 5.10 or
> 5.14.2 and compiling with either gcc 3.3.5 or gcc 4.4.6.
<snip>
>
> The generated .def file has a leading underscore that matches what the
> compiler generates
<snip>
>
> Given this, I have to suspect something specfic on Dave's setup.
<snip>
>
> Hmm. What about the code at line 120 of _write_os2()?
Line 120 of EU::Mksymlists, in both v6.62 and the version supplied with
perl 5.14.2, is
sub _write_win32 {
> my($data) = @_;
require Config;
I think what's happened here is that Paul has applied a patch (possibly
the one Shmuel posted to EUMM's RT) to ExtUtils::Mksymlists. Since
installing perl, Dave has (on my advice, attempting to fix something
else) upgraded EUMM, and therefore EU::Mksymlists, to the latest version
from CPAN. Since this does *not* contain the patch, linking started
failing at that point.
This is why I really hate it when distributors patch perl, *especially*
when they don't document that they have done so.
> > I don't believe either of those options will work. DynaLoader assumes it
> > can pass a function name, with no underscore, to DosQueryProcAddr, and
> > get a C-convention function address. If _System changes the calling
> > convention, that will obviously lead to segfaults; and unless
>
> _System and _cdecl are similar enough that segfaults are unlikely.
>
> > DosQueryProcAddr silently prepends an underscore, changing the export
>
> That does not happen.
OK. I think this issue is pretty-much cleared up now: Dave just needs to
apply the patch and reinstall EUMM. And, preferably, that patch or
something like it needs to get applied upstream, so this doesn't keep
happening again.
Ben
------------------------------
Date: Thu, 31 May 2012 22:37:40 -0500
From: "Steven Levine" <steve53@nomail.earthlink.net>
Subject: Re: Help with install from CPAN
Message-Id: <11p86vVJT4Oe-pn2-wdnilZldSuGk@slamain.slainc.com>
On Fri, 1 Jun 2012 01:18:46 UTC, Ben Morrow <ben@morrow.me.uk> wrote:
Hi Ben,
> I think what's happened here is that Paul has applied a patch (possibly
> the one Shmuel posted to EUMM's RT) to ExtUtils::Mksymlists.
Yes, he did. I missed it the first time I looked at the .diff.
> else) upgraded EUMM, and therefore EU::Mksymlists, to the latest version
> from CPAN. Since this does *not* contain the patch, linking started
> failing at that point.
OK.
> This is why I really hate it when distributors patch perl, *especially*
> when they don't document that they have done so.
I agree, but it takes organized effort to get patches pushed into the
repo. I suspect Ilya's patch has been around for a number of years.
> OK. I think this issue is pretty-much cleared up now: Dave just needs to
> apply the patch and reinstall EUMM. And, preferably, that patch or
> something like it needs to get applied upstream, so this doesn't keep
> happening again.
How do we make this happen?
I'd also like to see a fix for the @INC vs. PERLLIB_PREFIX issue too.
Best I can tell this is a regression caused by a rewrite to the logic
that builds @INC.
BTW, it was bothering me as to why 5.8 worked with CBuilder even
though the 5.8 Mksymlists was not patched to supply a leading
underscore. The reason is a behavioral change to the gcc toolchain.
In the past, the -Zomf generated externals without a leading
underscore. Actually, this is not quite true. The leading
underscores were generated by the compiler, but they were stripped by
emxomf during conversion from .o to .obj.
Steven
--
---------------------------------------------------------------------
Steven Levine <steve53@earthlink.bogus.net>
eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com
---------------------------------------------------------------------
------------------------------
Date: Fri, 1 Jun 2012 17:14:29 +0000 (UTC)
From: "Dave Saville" <dave@invalid.invalid>
Subject: Re: Help with install from CPAN
Message-Id: <fV45K0OBJxbE-pn2-BwQxsWtOmocI@localhost>
On Fri, 1 Jun 2012 03:37:40 UTC, "Steven Levine"
<steve53@nomail.earthlink.net> wrote:
> On Fri, 1 Jun 2012 01:18:46 UTC, Ben Morrow <ben@morrow.me.uk> wrote:
>
> Hi Ben,
>
> > I think what's happened here is that Paul has applied a patch (possibly
> > the one Shmuel posted to EUMM's RT) to ExtUtils::Mksymlists.
>
> Yes, he did. I missed it the first time I looked at the .diff.
>
> > else) upgraded EUMM, and therefore EU::Mksymlists, to the latest version
> > from CPAN. Since this does *not* contain the patch, linking started
> > failing at that point.
>
Actually I just diff'ed the updated install against Pauls MKsymlists
and the only difference apart from Paul's patch is the version number.
So this just gets us back to all tests failing or not even attempted
which is what seems to be happening and the missing os2ish.h &
os2thyread.h which are links in the distro that go nowhere.
I wonder how many other patrches Paul has put into the packaged perl
modules that wlii get dropped by an update from CPAN?
--
Regards
Dave Saville
------------------------------
Date: Fri, 1 Jun 2012 18:16:38 +0100
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: Help with install from CPAN
Message-Id: <m0an99-62s.ln1@anubis.morrow.me.uk>
Quoth steve53@earthlink.net:
> On Fri, 1 Jun 2012 01:18:46 UTC, Ben Morrow <ben@morrow.me.uk> wrote:
>
> > OK. I think this issue is pretty-much cleared up now: Dave just needs to
> > apply the patch and reinstall EUMM. And, preferably, that patch or
> > something like it needs to get applied upstream, so this doesn't keep
> > happening again.
>
> How do we make this happen?
If either you or Dave could
- download the latest dev version of ExtUtils-MakeMaker from CPAN,
- download the second of the two patches from
https://rt.cpan.org/Public/Bug/Display.html?id=75278 ,
- apply the patch, build and test,
- write to bug-ExtUtils-MakeMaker@rt.cpan.org, including the string
[rt.cpan.org #75278] in the subject line, stating that you have
done so and it was a success,
I would have thought that would help. I'm sure you can understand
Schwern may be a little nervous about applying a patch he can't test.
If you could include additional information about specific tests in the
MakeMaker distribution which fail without the patch and pass with it,
that would help; if there aren't any, then ideally some should be
written.
> I'd also like to see a fix for the @INC vs. PERLLIB_PREFIX issue too.
> Best I can tell this is a regression caused by a rewrite to the logic
> that builds @INC.
Is this logic useful now? Since 5.10 core perl has had support for
-Duserelocatableinc, which I believe does much the same thing in a more
portable manner. Since an OS/2-specific hack is not something most
module authors will be aware of, it seems worth migrating to a more
general solution if possible.
Ben
------------------------------
Date: Fri, 01 Jun 2012 09:12:21 -0400
From: Shmuel (Seymour J.) Metz <spamtrap@library.lspace.org.invalid>
Subject: Re: Help with install from CPAN
Message-Id: <4fc8bfb5$45$fuzhry+tra$mr2ice@news.patriot.net>
In <so3l99-3me.ln1@anubis.morrow.me.uk>, on 05/31/2012
at 10:17 PM, Ben Morrow <ben@morrow.me.uk> said:
>though I'm not entirely sure about using version.pm for
>anyth^Wparsing non-Perl version numbers.
I'm not sure either; it seemed reasonable, but there may be a better
way.
--
Shmuel (Seymour J.) Metz, SysProg and JOAT <http://patriot.net/~shmuel>
Unsolicited bulk E-mail subject to legal action. I reserve the
right to publicly post or ridicule any abusive E-mail. Reply to
domain Patriot dot net user shmuel+news to contact me. Do not
reply to spamtrap@library.lspace.org
------------------------------
Date: Sat, 02 Jun 2012 02:02:21 -0500
From: "Steven Levine" <steve53@nomail.earthlink.net>
Subject: Re: Help with install from CPAN
Message-Id: <11p86vVJT4Oe-pn2-tUwq7bs5g1u8@slamain.slainc.com>
On Fri, 1 Jun 2012 17:14:29 UTC, "Dave Saville" <dave@invalid.invalid>
wrote:
Hi Dave,
> I wonder how many other patrches Paul has put into the packaged perl
> modules that wlii get dropped by an update from CPAN?
There are 3 or 4 changes. Search the .diff for cpan and you will find
them.
Steven
--
---------------------------------------------------------------------
Steven Levine <steve53@earthlink.bogus.net>
eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com
---------------------------------------------------------------------
------------------------------
Date: Fri, 1 Jun 2012 09:57:01 -0700 (PDT)
From: David Karr <davidmichaelkarr@gmail.com>
Subject: How to properly recursively navigate into archives stored in archives
Message-Id: <2738738d-89ea-46f8-b261-13a8f5b44d7b@googlegroups.com>
I have a small perl script that I use to search archives for members matchi=
ng a name. I'd like to enhance this so that if it finds any members in the=
archive that are also archives (zip, jar, etc) it will then recursively sc=
an those, looking for the original desired pattern.
I've looked through the "Archive::Zip" documentation, and I thought I saw h=
ow to do this. I noticed the "fh()" and "readFromFileHandle()" methods. H=
owever, in my testing, it appears that the "fh()" call on an archive member=
returns the file handle for the containing archive, not the member. Perha=
ps I'm doing it wrong, but I appreciate an example of how to do this.
------------------------------
Date: Fri, 1 Jun 2012 18:34:55 +0100
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: How to properly recursively navigate into archives stored in archives
Message-Id: <v2bn99-62s.ln1@anubis.morrow.me.uk>
Quoth David Karr <davidmichaelkarr@gmail.com>:
> I have a small perl script that I use to search archives for members
> matching a name. I'd like to enhance this so that if it finds any
> members in the archive that are also archives (zip, jar, etc) it will
> then recursively scan those, looking for the original desired pattern.
>
> I've looked through the "Archive::Zip" documentation, and I thought I
> saw how to do this. I noticed the "fh()" and "readFromFileHandle()"
> methods. However, in my testing, it appears that the "fh()" call on an
> archive member returns the file handle for the containing archive, not
> the member. Perhaps I'm doing it wrong, but I appreciate an example of
> how to do this.
See Archive::Zip::MemberRead, and the readFileHandle method of member
objects.
Ben
------------------------------
Date: Fri, 1 Jun 2012 10:54:50 -0700 (PDT)
From: David Karr <davidmichaelkarr@gmail.com>
Subject: Re: How to properly recursively navigate into archives stored in archives
Message-Id: <d6bbba1d-2ff9-4a30-8dfa-e109ae6a3967@googlegroups.com>
On Friday, June 1, 2012 9:57:01 AM UTC-7, David Karr wrote:
> I have a small perl script that I use to search archives for members matc=
hing a name. I'd like to enhance this so that if it finds any members in t=
he archive that are also archives (zip, jar, etc) it will then recursively =
scan those, looking for the original desired pattern.
>=20
> I've looked through the "Archive::Zip" documentation, and I thought I saw=
how to do this. I noticed the "fh()" and "readFromFileHandle()" methods. =
However, in my testing, it appears that the "fh()" call on an archive memb=
er returns the file handle for the containing archive, not the member. Per=
haps I'm doing it wrong, but I appreciate an example of how to do this.
I got this answered, at: http://stackoverflow.com/questions/10854809/how-to=
-use-perl-archivezip-to-recursively-walk-archive-files
------------------------------
Date: Fri, 01 Jun 2012 12:54:57 -0500
From: "J. Gleixner" <glex_no-spam@qwest-spam-no.invalid>
Subject: Re: How to properly recursively navigate into archives stored in archives
Message-Id: <4fc901f2$0$63189$815e3792@news.qwest.net>
On 06/01/12 11:57, David Karr wrote:
> I have a small perl script that I use to search archives for members matching a name. I'd like to enhance this so that if it finds any members in the archive that are also archives (zip, jar, etc) it will then recursively scan those, looking for the original desired pattern.
>
> I've looked through the "Archive::Zip" documentation, and I thought I saw how to do this. I noticed the "fh()" and "readFromFileHandle()" methods. However, in my testing, it appears that the "fh()" call on an archive member returns the file handle for the containing archive, not the member. Perhaps I'm doing it wrong, but I appreciate an example of how to do this.
Keep looking at the documentation.. I haven't used it, but a quick
read and it looks like members(), and/or membersMatching() methods
seem to be what you're after.
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 3705
***************************************
