[32274] in Perl-Users-Digest
Perl-Users Digest, Issue: 3541 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sat Nov 12 06:09:43 2011
Date: Sat, 12 Nov 2011 03:09:06 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sat, 12 Nov 2011 Volume: 11 Number: 3541
Today's topics:
How to import only part of a large XML file? <dwightarmyofchampions@hotmail.com>
Re: How to import only part of a large XML file? <ben@morrow.me.uk>
Re: How to import only part of a large XML file? <dwightarmyofchampions@hotmail.com>
Re: How to import only part of a large XML file? <bjoern@hoehrmann.de>
Re: How to import only part of a large XML file? <dwightarmyofchampions@hotmail.com>
Re: How to import only part of a large XML file? <bjoern@hoehrmann.de>
Re: How to import only part of a large XML file? <klaus03@gmail.com>
Re: Parse using Text::CSV into Hash <rvtol+usenet@xs4all.nl>
Re: Parse using Text::CSV into Hash <poo@pee.com>
Re: Parse using Text::CSV into Hash <rweikusat@mssgmbh.com>
Re: Parse using Text::CSV into Hash <ben@morrow.me.uk>
Re: Parse using Text::CSV into Hash <rweikusat@mssgmbh.com>
Re: time-lag after DBI->commit <hjp-usenet2@hjp.at>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Fri, 11 Nov 2011 14:39:52 -0800 (PST)
From: Dwight Army of Champions <dwightarmyofchampions@hotmail.com>
Subject: How to import only part of a large XML file?
Message-Id: <a487fd35-5c93-40d1-8d18-f6bb3f6342e3@y12g2000vba.googlegroups.com>
I have a very large XML file that I want to load, but I don't want to
necessarily load the entire document; that takes too long. What I want
to do instead is only key/value pairs that meet certain criteria, like
only grab entries whose value fall within a certain date for a key
date_of_entry. Can I just use XML::Simple for this or do I need a
better module?
------------------------------
Date: Fri, 11 Nov 2011 16:59:51 -0600
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: How to import only part of a large XML file?
Message-Id: <Db6dnScxL5V6NSDTnZ2dnUVZ8mmdnZ2d@bt.com>
Quoth Dwight Army of Champions <dwightarmyofchampions@hotmail.com>:
> I have a very large XML file that I want to load, but I don't want to
> necessarily load the entire document; that takes too long. What I want
> to do instead is only key/value pairs that meet certain criteria, like
> only grab entries whose value fall within a certain date for a key
> date_of_entry. Can I just use XML::Simple for this or do I need a
> better module?
It sounds like you want either XML::Twig or one of the SAX modules.
XML::Simple, at least in non-SAX mode, will load the entire document
into a tree structure before letting you see any of it.
Ben
------------------------------
Date: Fri, 11 Nov 2011 15:10:01 -0800 (PST)
From: Dwight Army of Champions <dwightarmyofchampions@hotmail.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <30b318c5-00fd-40fc-9628-d2674a3c3d47@g7g2000vbd.googlegroups.com>
On Nov 11, 5:59=A0pm, Ben Morrow <b...@morrow.me.uk> wrote:
> Quoth Dwight Army of Champions <dwightarmyofchampi...@hotmail.com>:
>
> > I have a very large XML file that I want to load, but I don't want to
> > necessarily load the entire document; that takes too long. What I want
> > to do instead is only key/value pairs that meet certain criteria, like
> > only grab entries whose value fall within a certain date for a key
> > date_of_entry. Can I just use XML::Simple for this or do I need a
> > better module?
>
> It sounds like you want either XML::Twig or one of the SAX modules.
> XML::Simple, at least in non-SAX mode, will load the entire document
> into a tree structure before letting you see any of it.
>
> Ben
I'm glancing at XML::Twig on search.cpan.org, What methods can I use
to accomplish these tasks? I don't see any kind of "filter" method...
------------------------------
Date: Sat, 12 Nov 2011 00:26:39 +0100
From: Bjoern Hoehrmann <bjoern@hoehrmann.de>
Subject: Re: How to import only part of a large XML file?
Message-Id: <tjbrb7dmt0ec2eeoc1pff1im7n6valvctb@hive.bjoern.hoehrmann.de>
* Dwight Army of Champions wrote in comp.lang.perl.misc:
>I have a very large XML file that I want to load, but I don't want to
>necessarily load the entire document; that takes too long. What I want
>to do instead is only key/value pairs that meet certain criteria, like
>only grab entries whose value fall within a certain date for a key
>date_of_entry. Can I just use XML::Simple for this or do I need a
>better module?
It depends on what you mean by "key/value pairs". If you want to filter
elements based on attributes, and don't particularily need to look at
child elements, then the SAX modules are likely a good fit, they report
events like "start of element plus attributes" and "end of element" and
you have to manage state between the events. Generally, this should help
<http://perl-xml.sourceforge.net/faq/#parser_selection>, and if you have
special needs, Perl-XML@listserv.ActiveState.com is likely to give the
best advice.
--
Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
------------------------------
Date: Fri, 11 Nov 2011 16:11:21 -0800 (PST)
From: Dwight Army of Champions <dwightarmyofchampions@hotmail.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <68fb1612-f14c-4d17-81a3-f362bdc3eaaf@r28g2000yqj.googlegroups.com>
On Nov 11, 6:26=A0pm, Bjoern Hoehrmann <bjo...@hoehrmann.de> wrote:
> * Dwight Army of Champions wrote in comp.lang.perl.misc:
>
> >I have a very large XML file that I want to load, but I don't want to
> >necessarily load the entire document; that takes too long. What I want
> >to do instead is only key/value pairs that meet certain criteria, like
> >only grab entries whose value fall within a certain date for a key
> >date_of_entry. Can I just use XML::Simple for this or do I need a
> >better module?
>
> It depends on what you mean by "key/value pairs". If you want to filter
> elements based on attributes, and don't particularily need to look at
> child elements, then the SAX modules are likely a good fit, they report
> events like "start of element plus attributes" and "end of element" and
> you have to manage state between the events. Generally, this should help
> <http://perl-xml.sourceforge.net/faq/#parser_selection>, and if you have
> special needs, Perl-...@listserv.ActiveState.com is likely to give the
> best advice.
> --
> Bj=F6rn H=F6hrmann =B7 mailto:bjo...@hoehrmann.de =B7http://bjoern.hoehrm=
ann.de
> Am Badedeich 7 =B7 Telefon: +49(0)160/4415681 =B7http://www.bjoernsworld.=
de
> 25899 Dageb=FCll =B7 PGP Pub. KeyID: 0xA4357E78 =B7http://www.websitedev.=
de/
For example, suppose I have the following XML input file:
<?xml version=3D"1.0"?>
<library>
<book>
<title>Dreamcatcher</title>
<author>Stephen King</author>
<genre>Horror</genre>
<pages>899</pages>
<price>23.99</price>
<rating>5</rating>
<publication_date>11/27/2001</publication_date>
</book>
<book>
<title>Mystic River</title>
<author>Dennis Lehane</author>
<genre>Thriller</genre>
<pages>390</pages>
<price>17.49</price>
<rating>4</rating>
<publication_date>07/22/2003</publication_date>
</book>
<book>
<title>The Lord Of The Rings</title>
<author>J. R. R. Tolkien</author>
<genre>Fantasy</genre>
<pages>3489</pages>
<price>10.99</price>
<rating>5</rating>
<publication_date>10/12/2005</publication_date>
</book>
</library>
Suppose I only want to import books that were published after January
1, 2002. If I apply such a filter when I do my initial import, the
result should look like this:
$VAR1 =3D {
'book' =3D> [
{
'publication_date' =3D> '07/22/2003',
'price' =3D> '17.49',
'author' =3D> 'Dennis Lehane',
'title' =3D> 'Mystic River',
'rating' =3D> '4',
'pages' =3D> '390',
'genre' =3D> 'Thriller'
},
{
'publication_date' =3D> '10/12/2005',
'price' =3D> '10.99',
'author' =3D> 'J. R. R. Tolkien',
'title' =3D> 'The Lord Of The Rings',
'rating' =3D> '5',
'pages' =3D> '3489',
'genre' =3D> 'Fantasy'
}
]
};
The import will completely ignore entries that don't meet the
specified criteria (in this case, publication_date >=3D '1/1/2002').
------------------------------
Date: Sat, 12 Nov 2011 01:25:47 +0100
From: Bjoern Hoehrmann <bjoern@hoehrmann.de>
Subject: Re: How to import only part of a large XML file?
Message-Id: <fserb75v583pe99bbc2afaqvtschjjld37@hive.bjoern.hoehrmann.de>
* Dwight Army of Champions wrote in comp.lang.perl.misc:
>For example, suppose I have the following XML input file:
>Suppose I only want to import books that were published after January
>1, 2002. If I apply such a filter when I do my initial import, the
>result should look like this:
One way to do this would be with a SAX filter: you look for "book"
elements, store all events until you can decider whether you are
interested in this branch, and then re-emit or discard the events.
You can then use some module that turns the SAX stream into some
more Perl-ish data structure. There are some libraries that allow
you to filter in this fashion automatically ("xpath filtering"),
but I am not sure which, if any, modules for Perl do this for you.
Note that size is quite important here, with 100 MB you might just
suffer "too long" but with 5 GB you might suffer "impossible" for
some possible solutions. Some "reader"-style APIs allow you to go
to a "book" element, read everything up to the end of the element
into some DOM-style representation, and then make it easy to check
if you are interested in this branch as you have DOM-style access,
but only to the interesting part, so you save memory. Similar to
the SAX filter solution, except that you trade some memory and per-
haps speed for ease of programming.
--
Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
------------------------------
Date: Sat, 12 Nov 2011 02:28:54 -0800 (PST)
From: Klaus <klaus03@gmail.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <a39da306-0dd0-477a-89c4-3d43c282478d@m7g2000vbc.googlegroups.com>
On 12 nov, 01:11, Dwight Army of Champions
<dwightarmyofchampi...@hotmail.com> wrote:
> Suppose I only want to import books that were published after January
> 1, 2002. If I apply such a filter when I do my initial import, the
> result should look like this:
>
> $VAR1 =3D {
> =A0 =A0 =A0 =A0 =A0 'book' =3D> [
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'publication_date' =3D> '07/2=
2/2003',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'price' =3D> '17.49',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'author' =3D> 'Dennis Lehane'=
,
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'title' =3D> 'Mystic River',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'rating' =3D> '4',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'pages' =3D> '390',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'genre' =3D> 'Thriller'
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 },
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'publication_date' =3D> '10/1=
2/2005',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'price' =3D> '10.99',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'author' =3D> 'J. R. R. Tolki=
en',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'title' =3D> 'The Lord Of The=
Rings',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'rating' =3D> '5',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'pages' =3D> '3489',
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 'genre' =3D> 'Fantasy'
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ]
> =A0 =A0 =A0 =A0 };
That's a perfect Job for XML::Reader
use strict;
use warnings;
use XML::Reader;
use XML::Simple;
use Data::Dumper;
my $huge_xml =3D
q{<?xml version=3D"1.0"?>
<library>
<book>
<title>Dreamcatcher</title>
<author>Stephen King</author>
<genre>Horror</genre>
<pages>899</pages>
<price>23.99</price>
<rating>5</rating>
<publication_date>11/27/2001</publication_date>
</book>
<book>
<title>Mystic River</title>
<author>Dennis Lehane</author>
<genre>Thriller</genre>
<pages>390</pages>
<price>17.49</price>
<rating>4</rating>
<publication_date>07/22/2003</publication_date>
</book>
<book>
<title>The Lord Of The Rings</title>
<author>J. R. R. Tolkien</author>
<genre>Fantasy</genre>
<pages>3489</pages>
<price>10.99</price>
<rating>5</rating>
<publication_date>10/12/2005</publication_date>
</book>
</library>
};
my $selected =3D { book =3D> [] };
my $rdr =3D XML::Reader->new(\$huge_xml, {mode =3D> 'branches'},
{ root =3D> '/library/book', branch =3D> '*' });
while ($rdr->iterate) {
my $small_ref =3D XMLin($rdr->rvalue);
my ($day, $month, $year) =3D
$small_ref->{'publication_date'} =3D~
m{\A (\d+) / (\d+) / (\d+) \z}xms;
unless (defined $day) { $day =3D 0; }
unless (defined $month) { $month =3D 0; }
unless (defined $year) { $year =3D 0; }
my $date =3D sprintf('%04d-%02d-%02d', $year, $month, $day);
if ($date ge '2002-01-01') {
push @{$selected->{book}}, $small_ref;
}
}
print Dumper($selected);
> The import will completely ignore entries that don't meet the
> specified criteria (in this case, publication_date >=3D '1/1/2002').
Yes, the way it works is that XML::Reader reads from a huge XML only
small chunks (via $rdr->rvalue) (a small chunk being the '<book>...</
book> part). This small chunk is then fed into XML::Simple::XMLin() to
generate a small structure in memory which can then be used to extract
the date. if the date is >=3D 1/1/2002, then that small structure in
memory is pushed to a selected structure.
------------------------------
Date: Fri, 11 Nov 2011 20:33:58 +0100
From: "Dr.Ruud" <rvtol+usenet@xs4all.nl>
Subject: Re: Parse using Text::CSV into Hash
Message-Id: <4ebd78a6$0$6871$e4fe514c@news2.news.xs4all.nl>
On 2011-11-11 08:18, George Mpouras wrote:
> split ','
Hmm. What do you think that
split '[,]'
does?
--
Ruud
------------------------------
Date: Fri, 11 Nov 2011 15:22:08 -0500
From: batrams <poo@pee.com>
Subject: Re: Parse using Text::CSV into Hash
Message-Id: <j9k05g$dv2$1@dont-email.me>
On 11/10/2011 2:10 PM, J. Gleixner wrote:
> hash, see the getline_hr
Thanks! That looks like what I need :)
DB
------------------------------
Date: Fri, 11 Nov 2011 21:21:45 +0000
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: Parse using Text::CSV into Hash
Message-Id: <87pqgyfj86.fsf@sapphire.mobileactivedefense.com>
Ben Morrow <ben@morrow.me.uk> writes:
> Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
>> Ben Morrow <ben@morrow.me.uk> writes:
>>
>> > open (my $TEXTFILE, "<", $file)
>> > or die "Couldn't open '$file': $!\n";
>>
>> > - I put back the "\n" on the die message: file and line are useful
>> > for a programmer, and should be left in on messages you don't
>> > expect to ever see in production, but messages addressed to the
>> > end user don't want them.
>>
>> The end-user is going to report the problem as "It crashed" no matter
>> what actually occurred and any information said end-user can just
>> copy&paste into an e-mail is possibly helpful to the person who is
>> supposed to deal with the issue.
>
> You are certainly right for a large (and important) class of end-users.
> Programs aimed at such people should, as you say, simply give a message
> saying 'there was a problem' plus some way to send someone competent a
> logfile with as much detailed information as possible.
That wasn't what I was saying. I intended to argue in favor of
omitting the \n because an event which causes a program to die is
usually one some kind of developer will need to deal with and hence,
including information which is only relevant to a developer in the
diagnostic makes sense.
[...]
> Look at the error messages given by the standard Unix utilities. They
> (usually) do include filenames and strerror messages, but very rarely
> include __FILE__ and __LINE__ information.
In C-code, I always include the name of the function where the error
occurred. Granted, the default 'sudden death message suffix' is more
verbose than I consider necessary, OTOH, it contains enough
information to locate the operation that failed in the code and it is
already built in.
[...]
>> > Another advantage of lexical filehandles is they auto-close when they go
>> > out of scope. It's risky taking advantage of this if you have written to
>> > the handle: in that case you should be calling close explicitly and
>> > checking the return value.
>>
>> Or so the Linux close(2) manpage claims. The reason it does this is
>> because data written to a file residing on a NFS filesystem (and
>> possibly, other network file systems) isn't necessarily sent to the
>> server until the (NFS) filehandle is closed. Because of this, the
>> return value of a close of such a file may be some error which ocurred
>> during network communication.
>
> You don't need to go anywhere near that far. PerlIO (and stdio in C)
> buffers writes, so when you call close (that is, the equivalent of
> fclose(3), not close(2)) there is still a partial bufferful of data
> which hasn't been sent to the kernel yet. If that write(2) gives an
> error, that error will be returned from close.
So, more generally put: Using hidden buffering mechansisms is risky in
the sense that they decouple the operation which provides the data and
the operation which moves the data to its destination. As side effects
of that, the time window during which data loss could happen becomes
much larger and an application can no longer determine where in the
produced 'output data stream' the problem actually occurred (and try
to recover from that). The justification for hidden buffering is
'usually, it works and it usually improves the performance of the
system while it is working significantly'.
'Reliable I/O' or even just 'reliably reporting I/O errors' and
'hidden buffering' can't go together and ...
> Additionally, close will return an error if *any* of the previous
> writes failed (unless you call IO::Handle->clearerr), so checking
> the return value of close means you don't need to check the return
> value of print.
... checking the return value of close is just a fig leaf covering the
problem: If everything worked, it is redundant and when something
didn't work, reporting that "something didn't work" (but I know neither
what nor when and can't do anything to repair that) is not helpful.
>> For practical purpose, this is often irrelevant and - much more
>> importantly - checking the return value of the close doesn't help
>> except if reporting "This didn't work. Bad for you!" is considered
>> to be useful.
>
> It is. Always. It's better to *know* something failed, and have some
> idea as to why, than to think it succeeded when it didn't.
Let's assume that somebody runs a long-running data processing task
supposed to generate a lot of important output. On day three, the disk
is full but the program doesn't notice that. On day fifteen, after the
task has completed, it checks the return value of close and prints
"The disk became full! You lost!".
Do you really think a user will appreciate that?
>> Programs which care about the integrity of data written to some
>> persistent medium, at least to the degree this is possible, need to
>> use fsync before closing the file and need to implement some strategy
>> for dealing with errors returned by that (for Perl, this means
>> invoking IO::Handle::flush, followed by IO::Handle::Sync).
>
> Yes, if you have strict integrity requirements simply checking close
> isn't enough. It does guard against the two most common causes of error,
> though: disk full (or quota exceeded) and network failure.
Or so you hope: In case of a regular file residing on some local
persistent storage medium, a write system call encountering a 'too
little free space to write that' problem is supposed to write as many
bytes as can be written and return this count to the caller. This
means a final write on close will not return ENOSPC but will silently
truncate the file. Since close doesn't return a count, there's no way
to detect that. And there are - of course - people who think "Well, 0
is as good a number as any other number, so, in the case that write
really can't write anything, it is supposed to write as many bytes as
it can, namely, 0 bytes, and report this as successful write of size 0
to the caller" (I'm not making this up). Then, there are 'chamber of
horrors' storage media where 'sucessfully written' doesn't mean 'can
be read back again' and 'can be read back now' doesn't mean 'can still
be read back five minutes from now', aka 'flash ROM'. In case of
network communication, a remote system is perfectly capable of sending
an acknowledgment for something and die a sudden death before the
acknowledged data actually hit the persistent storage medium and so
on.
Eg, so far, I have written one program supposed to work reliably
despite it is running in Germany and writes to a NFS-mounted
filesystem residing in the UK and this program not only checks the
return value of each I/O operation but additionally checks that the
state of remote file system is what it was supposed to be if the
operation was actually performed sucessfully and retries everything
(with exponential backoff) until success or administrative
intervention --- network programming remains network programming, no
matter if the file system API is used for high-level convenience.
------------------------------
Date: Fri, 11 Nov 2011 16:55:22 -0600
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: Parse using Text::CSV into Hash
Message-Id: <Db6dnSQxL5VHOiDTnZ2dnUVZ8mmdnZ2d@bt.com>
Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
> Ben Morrow <ben@morrow.me.uk> writes:
> > Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
> >>
> >> The end-user is going to report the problem as "It crashed" no matter
> >> what actually occurred and any information said end-user can just
> >> copy&paste into an e-mail is possibly helpful to the person who is
> >> supposed to deal with the issue.
> >
> > You are certainly right for a large (and important) class of end-users.
> > Programs aimed at such people should, as you say, simply give a message
> > saying 'there was a problem' plus some way to send someone competent a
> > logfile with as much detailed information as possible.
>
> That wasn't what I was saying. I intended to argue in favor of
> omitting the \n because an event which causes a program to die is
> usually one some kind of developer will need to deal with
I disagree.
~% grep mauzo /etc/paswd
grep: /etc/paswd: No such file or directory
That message would *not* be improved by including the file and line in
grep's source where the error happened to be discovered.
> >> For practical purpose, this is often irrelevant and - much more
> >> importantly - checking the return value of the close doesn't help
> >> except if reporting "This didn't work. Bad for you!" is considered
> >> to be useful.
> >
> > It is. Always. It's better to *know* something failed, and have some
> > idea as to why, than to think it succeeded when it didn't.
>
> Let's assume that somebody runs a long-running data processing task
> supposed to generate a lot of important output. On day three, the disk
> is full but the program doesn't notice that. On day fifteen, after the
> task has completed, it checks the return value of close and prints
> "The disk became full! You lost!".
>
> Do you really think a user will appreciate that?
As opposed to the program falsely claiming it completed successfully?
Yes, absolutely. As opposed to a carefully-thought-out recovery
strategy? No, probably not.
Most programs don't run for fifteen days. Much of the time, dying with a
message like 'File not found' will allow the user to quickly and easily
fix the problem and try again. Obviously, there are circumstances where
this does not apply, and under such circumstances more thought needs to
be applied to the problem.
> >> Programs which care about the integrity of data written to some
> >> persistent medium, at least to the degree this is possible, need to
> >> use fsync before closing the file and need to implement some strategy
> >> for dealing with errors returned by that (for Perl, this means
> >> invoking IO::Handle::flush, followed by IO::Handle::Sync).
> >
> > Yes, if you have strict integrity requirements simply checking close
> > isn't enough. It does guard against the two most common causes of error,
> > though: disk full (or quota exceeded) and network failure.
>
> Or so you hope: In case of a regular file residing on some local
> persistent storage medium, a write system call encountering a 'too
> little free space to write that' problem is supposed to write as many
> bytes as can be written and return this count to the caller. This
> means a final write on close will not return ENOSPC but will silently
> truncate the file. Since close doesn't return a count, there's no way
> to detect that.
RTFS, of either perl or libc (or, you know, the manpages). fclose(3)
calls fflush(3), which will repeatedly call write(2) until the whole of
the remaining buffer has been successfully written, or an error occurs.
Perl's PerlIO functions do exactly the same thing. stdio certainly has
some design bugs, but it wasn't designed by idiots.
> And there are - of course - people who think "Well, 0
> is as good a number as any other number, so, in the case that write
> really can't write anything, it is supposed to write as many bytes as
> it can, namely, 0 bytes, and report this as successful write of size 0
> to the caller" (I'm not making this up).
SUSv4 says:
| Where this volume of POSIX.1-2008 requires -1 to be returned and errno
| set to [EAGAIN], most historical implementations return zero [...].
so, while this behaviour is not strictly POSIX-conforming, a decently-
written application should handle that case correctly. Unlike read(2),
there is no ambiguity in returning 0.
Ben
------------------------------
Date: Fri, 11 Nov 2011 23:20:21 +0000
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: Parse using Text::CSV into Hash
Message-Id: <87ty6az1oq.fsf@sapphire.mobileactivedefense.com>
Ben Morrow <ben@morrow.me.uk> writes:
> Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
>> Ben Morrow <ben@morrow.me.uk> writes:
>> > Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
>> >>
>> >> The end-user is going to report the problem as "It crashed" no matter
>> >> what actually occurred and any information said end-user can just
>> >> copy&paste into an e-mail is possibly helpful to the person who is
>> >> supposed to deal with the issue.
>> >
>> > You are certainly right for a large (and important) class of end-users.
>> > Programs aimed at such people should, as you say, simply give a message
>> > saying 'there was a problem' plus some way to send someone competent a
>> > logfile with as much detailed information as possible.
>>
>> That wasn't what I was saying. I intended to argue in favor of
>> omitting the \n because an event which causes a program to die is
>> usually one some kind of developer will need to deal with
>
> I disagree.
>
> ~% grep mauzo /etc/paswd
> grep: /etc/paswd: No such file or directory
>
> That message would *not* be improved by including the file and line in
> grep's source where the error happened to be discovered.
As I already wrote:
,----
|In C-code, I always include the name of the function where the error
|occurred. Granted, the default 'sudden death message suffix' is more
|verbose than I consider necessary, OTOH, it contains enough
|information to locate the operation that failed in the code and it is
|already built in.
`----
>> >> For practical purpose, this is often irrelevant and - much more
>> >> importantly - checking the return value of the close doesn't help
>> >> except if reporting "This didn't work. Bad for you!" is considered
>> >> to be useful.
>> >
>> > It is. Always. It's better to *know* something failed, and have some
>> > idea as to why, than to think it succeeded when it didn't.
>>
>> Let's assume that somebody runs a long-running data processing task
>> supposed to generate a lot of important output. On day three, the disk
>> is full but the program doesn't notice that. On day fifteen, after the
>> task has completed, it checks the return value of close and prints
>> "The disk became full! You lost!".
>>
>> Do you really think a user will appreciate that?
>
> As opposed to the program falsely claiming it completed successfully?
> Yes, absolutely.
I agree with that: Should the program print "I'm happy to inform you
that your task ran to completion" despite it didn't, the person whose
time got lost because of the lack of a sensible error handling
strategy will certainly be even more pissed then when the program just
prints "Sorry. Couldn't be bothered to check print return
values. Something failed".
To state this again: Either, the application doesn't care for the
result of an I/O operation (and there are valid reasons for that, eg,
short running, interactive tasks where any failure will be obvious to
the user). In this case, checking the return value of close is
pointless. Or the application does care for the results of I/O
operations. In this case, the hidden buffering facility whose
shortcomings are supposed to be plastered over by checking the return
value of close must not be used. I'll just say "Abracadabra" and firmly
believe everything's fine doesn't work.
>> >> Programs which care about the integrity of data written to some
>> >> persistent medium, at least to the degree this is possible, need to
>> >> use fsync before closing the file and need to implement some strategy
>> >> for dealing with errors returned by that (for Perl, this means
>> >> invoking IO::Handle::flush, followed by IO::Handle::Sync).
>> >
>> > Yes, if you have strict integrity requirements simply checking close
>> > isn't enough. It does guard against the two most common causes of error,
>> > though: disk full (or quota exceeded) and network failure.
>>
>> Or so you hope: In case of a regular file residing on some local
>> persistent storage medium, a write system call encountering a 'too
>> little free space to write that' problem is supposed to write as many
>> bytes as can be written and return this count to the caller. This
>> means a final write on close will not return ENOSPC but will silently
>> truncate the file. Since close doesn't return a count, there's no way
>> to detect that.
>
> RTFS, of either perl or libc (or, you know, the manpages). fclose(3)
> calls fflush(3), which will repeatedly call write(2) until the whole of
> the remaining buffer has been successfully written,
Or so you hope. You cannot possibly have checked the source code of
each past, present and future hidden buffering facility and I
seriously doubt that you did check even a large minority of them, not
the least because the source code of many of them isn't available. And
- of course - since the kernel will usually provide another layer of
hidden buffering, this is still technically equivalent to "La la la. I
didn't hear a thing!".
> Perl's PerlIO functions do exactly the same thing. stdio certainly has
> some design bugs, but it wasn't designed by idiots.
This is not a matter of the design but of the implementation.
>> And there are - of course - people who think "Well, 0
>> is as good a number as any other number, so, in the case that write
>> really can't write anything, it is supposed to write as many bytes as
>> it can, namely, 0 bytes, and report this as successful write of size 0
>> to the caller" (I'm not making this up).
>
> SUSv4 says:
>
> | Where this volume of POSIX.1-2008 requires -1 to be returned and errno
> | set to [EAGAIN], most historical implementations return zero [...].
>
> so, while this behaviour is not strictly POSIX-conforming, a decently-
> written application should handle that case correctly.
And that is
These functions shall fail if:
[EAGAIN]
The O_NONBLOCK flag is set for the file descriptor and the thread
would be delayed in the write() operation.
Not applicable. In case of the 'write returning zero instead of
ENOSPC' issue (and this is a real implementation), the program will
loop forever in the flushing routine.
------------------------------
Date: Sat, 12 Nov 2011 10:33:34 +0100
From: "Peter J. Holzer" <hjp-usenet2@hjp.at>
Subject: Re: time-lag after DBI->commit
Message-Id: <slrnjbsfbe.hj8.hjp-usenet2@hrunkner.hjp.at>
On 2011-11-09 01:45, xhoster@gmail.com <xhoster@gmail.com> wrote:
> "dn.perl@gmail.com" <dn.perl@gmail.com> wrote:
>> One correction: The database in question is MySQL, not Oracle.
>
> Well, there is a big difference. Which storage engine are you using with
> MySQL? Not one of the transactional ones, I bet.
And is there replication or a cluster involved? I don't see how that
behaviour could arise even with MyISAM on a single machine: The insert
returns only when it is complete, so when the script exits all 50'000
inserts are complete and a select count(*) should show them.
But I would expect that behaviour in a replicated environment where
inserts go to a master and selects to a slave. MySQL replication is
asynchronous, so the inserts show up on the slave some time later than
on the master (and since MyISAM doesn't have transactions, each row
shows up individually; with Innodb all rows would become visible on the
slave only once the commit was replicated).
hp
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 3541
***************************************
