[29972] in Perl-Users-Digest
Perl-Users Digest, Issue: 1215 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sat Jan 19 21:09:42 2008
Date: Sat, 19 Jan 2008 18:09:09 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sat, 19 Jan 2008 Volume: 11 Number: 1215
Today's topics:
How do you use IPC::Open3.... <cdalten@gmail.com>
Re: How do you use IPC::Open3.... <ben@morrow.me.uk>
Re: How do you use IPC::Open3.... <cdalten@gmail.com>
Re: How do you use IPC::Open3.... xhoster@gmail.com
Re: Wait for background processes to complete <ced@blv-sam-01.ca.boeing.com>
Re: Wait for background processes to complete <pgodfrin@gmail.com>
Re: Wait for background processes to complete <m@rtij.nl.invlalid>
Re: Wait for background processes to complete <tadmc@seesig.invalid>
Re: Wait for background processes to complete <hjp-usenet2@hjp.at>
Re: Wait for background processes to complete xhoster@gmail.com
Re: Wait for background processes to complete xhoster@gmail.com
xml-to-word document <motic.mail@gmail.com>
Re: xml-to-word document <sensorflo@gmail.com>
Re: xml-to-word document <m@rtij.nl.invlalid>
Re: xml-to-word document <jurgenex@hotmail.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 19 Jan 2008 14:11:15 -0800 (PST)
From: grocery_stocker <cdalten@gmail.com>
Subject: How do you use IPC::Open3....
Message-Id: <92229539-0bfc-40a6-b9cd-20eb4787e20e@s8g2000prg.googlegroups.com>
to start a process in the background? I'm drawing a total blank.
------------------------------
Date: Sat, 19 Jan 2008 23:50:49 +0000
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: How do you use IPC::Open3....
Message-Id: <pvea65-kr2.ln1@osiris.mauzo.dyndns.org>
Quoth grocery_stocker <cdalten@gmail.com>:
[ How do you use IPC::Open3 ]
> to start a process in the background? I'm drawing a total blank.
Processes started with IPC::Open3 are 'in the background', insofar as
that means anything in the context of Perl. They are not connected to
your terminal, and they will run independantly of anything else as long
as they aren't blocking on IO.
What are you actually trying to acheive? Please post a short script that
demonstrates your problem, and explain how it's not doing what you want.
You may also want to look at IPC::Run, which is generally much easier to
drive correectly than IPC::Open3.
Ben
------------------------------
Date: Sat, 19 Jan 2008 16:21:45 -0800 (PST)
From: grocery_stocker <cdalten@gmail.com>
Subject: Re: How do you use IPC::Open3....
Message-Id: <f90c8c35-c861-43e5-a978-1b4cfa88c110@i7g2000prf.googlegroups.com>
On Jan 19, 3:50 pm, Ben Morrow <b...@morrow.me.uk> wrote:
> Quoth grocery_stocker <cdal...@gmail.com>:
> [ How do you use IPC::Open3 ]
>
> > to start a process in the background? I'm drawing a total blank.
>
> Processes started with IPC::Open3 are 'in the background', insofar as
> that means anything in the context of Perl. They are not connected to
> your terminal, and they will run independantly of anything else as long
> as they aren't blocking on IO.
>
> What are you actually trying to acheive? Please post a short script that
> demonstrates your problem, and explain how it's not doing what you want.
> You may also want to look at IPC::Run, which is generally much easier to
> drive correectly than IPC::Open3.
>
> Ben
Somewhere in the perl documentation I read that you could use
IPC::Open3 to start a process in the background. I was having a brain
fart at the time. This might be because I was thinking about sex. Who
knows.
------------------------------
Date: 20 Jan 2008 01:40:02 GMT
From: xhoster@gmail.com
Subject: Re: How do you use IPC::Open3....
Message-Id: <20080119204005.688$bV@newsreader.com>
grocery_stocker <cdalten@gmail.com> wrote:
> to start a process in the background? I'm drawing a total blank.
I'm not.
$ perldoc IPC::Open3 | wc
77 537 3860
Xho
--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
------------------------------
Date: Sat, 19 Jan 2008 15:24:59 -0800 (PST)
From: "comp.llang.perl.moderated" <ced@blv-sam-01.ca.boeing.com>
Subject: Re: Wait for background processes to complete
Message-Id: <7f7e0358-c6ad-4c71-9f67-2ca5555e4dfa@d21g2000prf.googlegroups.com>
On Jan 19, 3:12 am, "Peter J. Holzer" <hjp-usen...@hjp.at> wrote:
> On 2008-01-18 22:19, comp.llang.perl.moderated <c...@blv-sam-01.ca.boeing.com> wrote:
>
> ...
> > Again I disagree. A viable alternative solution could make use of a
> > SIGCHLD handler.
>
> It could. It's just completely useless.
>
> Assume that the loop which forks off the children registers them in
> %kids.
>
> Then the SIGCHLD handler could do something like this:
>
> sub REAPER {
> for(;;) {
> $kid = waitpid(-1, WNOHANG);
> last if $kid <= 0;
> delete $kids{$kid};
> }
>
> }
>
> So, at the end of the program we just need to loop until %kids is empty:
>
> while (keys %kids) {
>
> }
>
> But that's busy-waiting - it will consume 100 % CPU time. We could sleep
> inside the loop, but how long? Until the next child terminates. Well, we
> already have a function which does sleep until a child terminates - it's
> called wait. So the loop turns into:
>
> while (keys %kids) {
> my $kid = wait();
> delete $kids{$kid};
>
> }
>
> So now we have a loop at the end which waits for all children, and the
> REAPER function has no useful function anymore. So we can delete it.
>
That's true. The SIGCHLD handler has no apparent advantage here
unless
you have some critical reason to want immediate signal delivery. I'm
not sure what likely scenarios in a non-daemon setting might benefit
--
maybe someone decides to bump a reaped process count in the handler
in
order to sleep during the fork loop if some threshold is exceeded...
Dunno.
At any rate though, it's always seemed cleaner to me to reap child
processes
as soon as possible...particularly at the bargain basement price of a
POSIX
declaration and 1 or 2-liner handler. Not a big deal either way but I
think
it's worth mentioning as an altenative.
--
Charles DeRykus
------------------------------
Date: Sat, 19 Jan 2008 15:37:02 -0800 (PST)
From: pgodfrin <pgodfrin@gmail.com>
Subject: Re: Wait for background processes to complete
Message-Id: <04b0d8cd-f7db-49e5-93c5-a98561f25c11@i12g2000prf.googlegroups.com>
On Jan 18, 3:17 pm, "Peter J. Holzer" <hjp-usen...@hjp.at> wrote:
> On 2008-01-18 17:47, pgodfrin <pgodf...@gmail.com> wrote:
>
>
>
> > This was a lot of fun. I would like to respond to the various
> > observations, especially the ones about the Perl Documentation being
> > misleading.
> [...]
> > To restate the original task I wanted to solve:
>
> > To be able to execute commands in the background and wait for their
> > completion.
>
> > The documentation I am referring to ishttp://perldoc.perl.org/.
>
> > If you search on the concept of "background processes" this
> > documentation points you to the following in the Language reference >
> > perlipc > Background Process :
> ><begin quote>
> > You can run a command in the background with:
>
> > system("cmd &");
>
> > The command's STDOUT and STDERR (and possibly STDIN, depending on your
> > shell) will be the same as the parent's. You won't need
> > to catch SIGCHLD because of the double-fork taking place (see below
> > for more details).
> ><end quote>
> > There is no further reference to a "double fork" (except in the
> > perlfaq8 and only in the context of zombies, which it says are not an
> > issue using system("cmd &") ). This is confusing.
>
> I find it more confusing that this seems to be in the section with the
> title "Using open() for IPC". I fail to see what one has to do with the
> other.
>
> There is a general problem with perl documentation: Perl evolved in the
> Unix environment, and a lot of the documentation was written at a time
> when the "newbie perl programmer" could be reasonably expected to have
> already some programming experience on Unix (in C, most likely) and know
> basic Unix concepts like processes, fork(), filehandles, etc. So in a
> lot of places the documentation doesn't answer the question "how can I
> do X on Unix?" but the question "I already know how to do X in C, now
> tell me how I can do it in perl!". When you lern Perl without knowing
> Unix first, this can be confusing, because the Perl documentation
> generelly explains only Perl, but not Unix.
>
> I am not sure if that should be fixed at all: It's the perl
> documentation and not the unix documentation after all, and perl isn't
> unix specific, but has been ported to just about every OS.
>
> > The documentation for wait(), waitpid() and fork() do not explain that
> > the executing code should be placed in the "child" section of the if
> > construct.
>
> Of course not, that would be wrong. There can be code in both (if you
> wanted one process to do nothing, why fork?). What the parent should do
> and what the child should do depend on what you want them to do. It just
> happened that for your particular problem the parent had nothing to do
> between forking off children.
>
> > Some of the examples in perlipc show code occurring in both
> > the parent and the child section - so it is still not clear. If it
> > were, why was I insisting on trying to execute the "cp" command in the
> > parent section?
>
> I don't know. What did you expect that would do?
>
> Your problem was:
>
> I want a process to gather a list of files. Then, for each file, it
> should start another process which copies the file. These processes
> should run in parallel. Finally, it should wait for all these processes
> to terminate, and then terminate itself.
>
> Even this description makes it rather implicit, that the original
> process creates children and then the children do the copying. If you
> add the restriction that a process can only wait for its children, any
> other solution becomes extremely awkward.
>
> Besides it is exactly the same what your shell script did: The shell
> (the parent process) created child processes in a loop, each child
> process executed one cp command, and finally the parent waited for all
> the children.
>
> > Furthermore, while the perlipc section is quite verbose, nowhere is
> > the code snippet do {$x=wait; print "$x\n"} until $x==-1 or any
> > variation of that wait() call mentioned.
>
> Again, this is a solution to your specific problem. You said you wanted
> to wait for all children, so Xho wrote a loop which would do that. This
> is a rather rare requirement - I think I needed it maybe a handful of
> times in 20+ years of programming outside of a child reaper function.
>
> > There are references to a
> > while loop and the waitpid() function, but being in the context of a
> > signal handler and 'reaping' - it is not clear.
>
> This is the one situation where this construct is frequently used. It
> can happen that several children die before the parent can react to the
> signal - in this case the reaper function will be called only once, but
> it must wait for several children. This isn't obvious, so the docs
> mention it.
>
> However, in your case you *know* you started several children and you
> *want* to wait for all of them, so it's obvious that you need a loop. This
> hasn't anything to do with perl, it's just a requirement of your
> problem.
>
> > So, once again many thanks for the help. I would like to know, Peter,
> > are you in a position to amend the documentation? Also, the perlfaq8
> > does come closer to explaining, but it is simply not clear how the
> > fork process will emulate the Unix background operator (&). So how can
> > we make this better? How's about this:
>
> fork doesn't "emulate &". fork is one of the functions used by the shell
> to execute commands. Basically, the shell does this in a loop:
>
> display the prompt
> read the next command
> fork
> in the child:
> execute the command, then exit
> in the parent:
> if the command did NOT end with &:
> wait for the child to terminate
>
> So the Shell always forks before executing commands and it always
> executes them in a child process. But normally it waits for the child
> process to terminate before displaying the next prompt. If you add the
> "&" at the end of the line, it doesn't wait but displays the next prompt
> immediately.
>
> (ok, that's a bit too simplistic, but I don't want to go into the arcana
> of shell internals here - that would be quite off-topic).
>
> > In the perlipc, under background tasks. make the statement - "the call
> > system("cmd &") will run the cmd in the background, but will spawn a
> > shell to execute the command, dispatch the command, terminate the
> > shell and return to the calling program. The calling program will lose
> > the ability to wait for the process as is customary with the shell
> > script command 'wait'. In order to both execute one or more programs
> > in the background and have the calling program wait for the executions
> > to complete it will be necessary to use the fork() function."
>
> I think that would confuse just about anybody who doesn't have exactly
> your problem for exactly the same reason you were confused by the
> "double fork". It's very specific and should be clear to anyone who
> knows what system does and what a unix shell does when you use "&" - I
> suspect that sentence about the double fork was added after a discussion
> like this one. But I agree that fork should definitely be mentioned here.
> system("cmd &") almost never what you would want to do.
>
> > Then in the fork section. Show a simple example like the ones we have
> > been working with AND show a simple approach to using the wait
> > function.
> > Furthermore - add sample code to the wait() and fork()
> > functions that are simple and realistic,
>
> fork and wait are very simple building blocks which can be combined in a
> lot of different ways. Any sample code here can cover only basic
> constructs like
>
> my $pid = fork();
> if (!defined $pid) {
> die "fork failed: $!";
> } elsif ($pid == 0) {
> # child does something here
> # ...
> exit;
> } else {
> # parent does something here
> # ...
> # and then waits for child:
> waitpid($pid, 0);
> }
>
> More complex examples should go into perlipc (which needs serious
> cleanup, IMHO).
>
> > unlike the code same in the waitpid() function.
>
> That code is simple and realistic.
>
> > In closing, it is perhaps non-intuitive to me that a fork process
> > should have the child section actually executing the code,
>
> If you don't want the child process to do anything, why would you create
> it in the first place?
>
> > but I ask you how one can intuit that from the sample in the camel
> > book and the samples in thehttp://perldoc.perl.org/. To really drive
> > the point home, Xho's code:
>
> > fork or exec("cp $_ $_.old") ;
> > do {$x=wait;} until $x==-1 ;
>
> > Is STILL not intuitive that the child is executing the code!
>
> True. This code isn't meant to be intuitive. It's meant to be short.
> I wouldn't write that in production code, much less in an answer to a
> newbie question.
>
> hp
HI Peter,
Thanks - you've been quite fair. The only point I would argue is how
many times you needed to wait for background tasks. I guess the
salient point is - I'm not a systems programmer - so I use Perl like
shell scripting and because I think it's stupid to do if-fi and case-
esac pairs.
'nuff said. Good point about the Perl docs being about Perl and not
Unix...
cheers...
pg
p.s. what's an OP ?
------------------------------
Date: Sun, 20 Jan 2008 00:55:47 +0100
From: Martijn Lievaart <m@rtij.nl.invlalid>
Subject: Re: Wait for background processes to complete
Message-Id: <pan.2008.01.19.23.55.47@rtij.nl.invlalid>
On Sat, 19 Jan 2008 15:37:02 -0800, pgodfrin wrote:
> p.s. what's an OP ?
Original Poster
HTH,
M4
------------------------------
Date: Sat, 19 Jan 2008 18:08:54 -0600
From: Tad J McClellan <tadmc@seesig.invalid>
Subject: Re: Wait for background processes to complete
Message-Id: <slrnfp548m.kek.tadmc@tadmc30.sbcglobal.net>
pgodfrin <pgodfrin@gmail.com> wrote:
> p.s. what's an OP ?
http://catb.org/jargon/html/O/thread-OP.html
--
Tad McClellan
email: perl -le "print scalar reverse qq/moc.noitatibaher\100cmdat/"
------------------------------
Date: Sun, 20 Jan 2008 01:12:48 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: Wait for background processes to complete
Message-Id: <slrnfp54g4.1rm.hjp-usenet2@hrunkner.hjp.at>
On 2008-01-19 23:37, pgodfrin <pgodfrin@gmail.com> wrote:
> On Jan 18, 3:17 pm, "Peter J. Holzer" <hjp-usen...@hjp.at> wrote:
>> On 2008-01-18 17:47, pgodfrin <pgodf...@gmail.com> wrote:
>>
>>
>>
>> > This was a lot of fun. I would like to respond to the various
>> > observations, especially the ones about the Perl Documentation being
>> > misleading.
>> [...]
[76 lines snipped]
>> > Furthermore, while the perlipc section is quite verbose, nowhere is
>> > the code snippet do {$x=wait; print "$x\n"} until $x==-1 or any
>> > variation of that wait() call mentioned.
>>
>> Again, this is a solution to your specific problem. You said you wanted
>> to wait for all children, so Xho wrote a loop which would do that. This
>> is a rather rare requirement - I think I needed it maybe a handful of
>> times in 20+ years of programming outside of a child reaper function.
>>
[109 lines snipped - please quote only the relevant parts of the
articles you reply to]
> Thanks - you've been quite fair. The only point I would argue is how
> many times you needed to wait for background tasks.
I need to wait for background tasks quite often. But usually either I
need to wait for only one of them or some of them, but rarely all of
them. But I don't see how it matters. If you know how to wait for one
process, and you know how to write a loop, it is trivial to wait for all
processes.
> I guess the salient point is - I'm not a systems programmer - so I use
> Perl like shell scripting and because I think it's stupid to do if-fi
> and case- esac pairs.
That's ok. Perl just gives you a lot more flexibility than the shell,
at the price of higher complexity. For example, it would be quite easy
to change your script to do a configurable number of copies in parallel
- this is rather hard to do in the shell.
> p.s. what's an OP ?
"original poster": The person who started a thread, i.e. you in this case.
hp
------------------------------
Date: 20 Jan 2008 01:50:44 GMT
From: xhoster@gmail.com
Subject: Re: Wait for background processes to complete
Message-Id: <20080119205047.580$Xz@newsreader.com>
"Peter J. Holzer" <hjp-usenet2@hjp.at> wrote:
>
> Assume that the loop which forks off the children registers them in
> %kids.
>
> Then the SIGCHLD handler could do something like this:
>
> sub REAPER {
> for(;;) {
> $kid = waitpid(-1, WNOHANG);
> last if $kid <= 0;
> delete $kids{$kid};
> }
> }
>
> So, at the end of the program we just need to loop until %kids is empty:
>
> while (keys %kids) {
> }
>
> But that's busy-waiting - it will consume 100 % CPU time. We could sleep
> inside the loop, but how long? Until the next child terminates. Well, we
> already have a function which does sleep until a child terminates - it's
> called wait. So the loop turns into:
>
> while (keys %kids) {
> my $kid = wait();
> delete $kids{$kid};
> }
>
> So now we have a loop at the end which waits for all children, and the
> REAPER function has no useful function anymore. So we can delete it.
It might be useful in the highly unlikely event that you want to be
spawning enough jobs that you will run out of processes if you don't reap
any until all are spawned. Reaping early ones that exit even as the later
ones are still being spawning could help that (But I still wouldn't use a
sig handler, I'd just put a "waitpid(-1, WNOHANG);" inside the spawning
loop). But anyway, since the spawning is unthrottled, you are just racing
disaster anyway--you have no guarantee that enough will finish in time to
prevent you from running out of processes. So something throttling, like
ForkManager would be the way to go.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
------------------------------
Date: 20 Jan 2008 02:03:33 GMT
From: xhoster@gmail.com
Subject: Re: Wait for background processes to complete
Message-Id: <20080119210336.367$EG@newsreader.com>
"Peter J. Holzer" <hjp-usenet2@hjp.at> wrote:
> Xho's solution is safe (and was so in all perl
> versions and does what the OP wants. Well, almost - wait can return -1
> if it is interrupted so one should check $! in addition to the return
> value.
Are you sure that that is the case? I figured it would return undef on
error. Upon experimenting (with both >5.8 and <5.8) I found that (on my
system) wait returns neither -1 nor undef due to signals, because it
never returns due to signals. The Perl wait behind the scenes just keeps
recalling the system-level wait either until something is reaped, or until
the program dies, or until a die/eval pair causes it to jump out of the
execution path without actually returning.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.
------------------------------
Date: Sat, 19 Jan 2008 13:16:15 -0800 (PST)
From: Moti <motic.mail@gmail.com>
Subject: xml-to-word document
Message-Id: <0f7a42a7-3be7-4607-abdb-2a78ef4f0c8a@f10g2000hsf.googlegroups.com>
Hi all,
I would like to perform xml-to-word conversion. Is there a perl module
that can perform it?
If not, maybe there are other ways to do it ..
Thanks in advance, Moti.
------------------------------
Date: Sat, 19 Jan 2008 15:33:08 -0800 (PST)
From: Florian Kaufmann <sensorflo@gmail.com>
Subject: Re: xml-to-word document
Message-Id: <78232cec-1ca6-416f-ad84-1ab0f66e973a@y5g2000hsf.googlegroups.com>
I don't have an exact knowledge about XML, but I can say that XML is
much more than a text file format. You can describe/store things with
it that you cant describe/store with a word document. Thus it is not
possible to convert from XML in general to a word document.
------------------------------
Date: Sun, 20 Jan 2008 00:54:46 +0100
From: Martijn Lievaart <m@rtij.nl.invlalid>
Subject: Re: xml-to-word document
Message-Id: <pan.2008.01.19.23.54.46@rtij.nl.invlalid>
On Sat, 19 Jan 2008 13:16:15 -0800, Moti wrote:
> Hi all,
> I would like to perform xml-to-word conversion. Is there a perl module
> that can perform it?
> If not, maybe there are other ways to do it .. Thanks in advance, Moti.
Read Florian response first. That said, convert to html, word generally
reads that fine. Store in it a file with a .doc extension and it will
even be opened by Word by default on Windows.
M4
------------------------------
Date: Sun, 20 Jan 2008 01:03:22 GMT
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: xml-to-word document
Message-Id: <b175p3h8kgjiur8c5grbu96p3l7oq3542k@4ax.com>
Moti <motic.mail@gmail.com> wrote:
>I would like to perform xml-to-word conversion.
I don't understand what you are looking for. XML is just plain text with
specific syntax rules. Microsoft Office Word will open an XML file just
fine. There is no need for any conversion.
jue
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 1215
***************************************
