[32278] in Perl-Users-Digest
Perl-Users Digest, Issue: 3545 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Wed Nov 16 14:09:22 2011
Date: Wed, 16 Nov 2011 11:09:05 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Wed, 16 Nov 2011 Volume: 11 Number: 3545
Today's topics:
Re: How to import only part of a large XML file? <cartercc@gmail.com>
Re: How to import only part of a large XML file? <rweikusat@mssgmbh.com>
Re: How to import only part of a large XML file? <rweikusat@mssgmbh.com>
Re: How to import only part of a large XML file? <willem@toad.stack.nl>
Re: How to import only part of a large XML file? <cartercc@gmail.com>
Re: How to import only part of a large XML file? <cartercc@gmail.com>
Re: How to import only part of a large XML file? <klaus03@gmail.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Wed, 16 Nov 2011 08:32:24 -0800 (PST)
From: ccc31807 <cartercc@gmail.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <b6deb458-c27f-4735-ab02-0ef18814b3c8@o17g2000yqa.googlegroups.com>
On Nov 11, 5:39=A0pm, Dwight Army of Champions
<dwightarmyofchampi...@hotmail.com> wrote:
> I have a very large XML file that I want to load, but I don't want to
> necessarily load the entire document; that takes too long. What I want
> to do instead is only key/value pairs that meet certain criteria, like
> only grab entries whose value fall within a certain date for a key
> date_of_entry. Can I just use XML::Simple for this or do I need a
> better module?
This depends on the nature of your input. I do this kind of thing
every day, and use a simple regular expression to filter the file. Of
course, you still have to read every line of the file to make sure
that you catch all of your intended targets, but you would have to do
that anyway. This is the kind of task for which it's a lot easier to
hand roll your own parser than it is to look for, evaluate, learn,
install, and use some third party module. In my opinion anyway. For
example:
SCRIPT
#! perl
use warnings;
use strict;
my %filter;
while (<DATA>)
{
next unless /\w/;
chomp;
if ($_ =3D~ m!<order>(\d+)</order>!)
{
my $key =3D $1;
while (<DATA>)
{
last if $_ =3D~ m!</pres>!;
next unless $_ =3D~ m!<last>(\w+)</last>!;
$filter{$key} =3D $1;
}
}
}
print "Finished processing file\n";
foreach my $key (sort keys %filter) { print "$key =3D> $filter{$key}
\n"; }
exit(0);
__DATA__
<pres>
<order>1</order>
<first>George</first>
<last>Washington</last>
<year>1788</year>
</pres>
<pres>
<order>2</order>
<first>John</first>
<last>Adams</last>
<year>1796</year>
</pres>
<pres>
<order>3</order>
<first>Thomas</first>
<last>Jefferson</last>
<year>1800</year>
</pres>
OUTPUT
$perl filter_test.plx
Finished processing file
1 =3D> Washington
2 =3D> Adams
3 =3D> Jefferson
4 =3D> Madison
5 =3D> Monroe
6 =3D> Adams
------------------------------
Date: Wed, 16 Nov 2011 16:50:37 +0000
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <87hb24f1uq.fsf@sapphire.mobileactivedefense.com>
ccc31807 <cartercc@gmail.com> writes:
> On Nov 11, 5:39 pm, Dwight Army of Champions
> <dwightarmyofchampi...@hotmail.com> wrote:
>> I have a very large XML file that I want to load, but I don't want to
>> necessarily load the entire document; that takes too long. What I want
>> to do instead is only key/value pairs that meet certain criteria, like
>> only grab entries whose value fall within a certain date for a key
>> date_of_entry. Can I just use XML::Simple for this or do I need a
>> better module?
>
> This depends on the nature of your input.
[...]
> while (<DATA>)
> {
> next unless /\w/;
> chomp;
> if ($_ =~ m!<order>(\d+)</order>!)
> {
> my $key = $1;
> while (<DATA>)
> {
> last if $_ =~ m!</pres>!;
> next unless $_ =~ m!<last>(\w+)</last>!;
> $filter{$key} = $1;
> }
> }
> }
AFAIK, a well-formed XML file could have an order description looking like
this:
<order
>1</order
>
meaning, it is not really possible to parse XML without doing a
character-by-character lexical analysis of the input data stream
first.
------------------------------
Date: Wed, 16 Nov 2011 16:54:06 +0000
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <87d3csf1ox.fsf@sapphire.mobileactivedefense.com>
Dwight Army of Champions <dwightarmyofchampions@hotmail.com> writes:
> I have a very large XML file that I want to load, but I don't want to
> necessarily load the entire document; that takes too long. What I want
> to do instead is only key/value pairs that meet certain criteria, like
> only grab entries whose value fall within a certain date for a key
> date_of_entry.
This is impossible. Technically, XML is a sequential character stream
and any structured data encoded as XML can only be recovered by
aggregating characters into tokens based on the rules for XML tokens
and parsing the resulting token stream.
------------------------------
Date: Wed, 16 Nov 2011 17:04:14 +0000 (UTC)
From: Willem <willem@toad.stack.nl>
Subject: Re: How to import only part of a large XML file?
Message-Id: <slrnjc7r8e.2ksv.willem@toad.stack.nl>
Rainer Weikusat wrote:
) AFAIK, a well-formed XML file could have an order description looking like
) this:
)
)<order
)
)
)>1</order
)
)
)
)
)>
)
) meaning, it is not really possible to parse XML without doing a
) character-by-character lexical analysis of the input data stream
) first.
Indeed. To me, this is an argument that XML is usually a bad choice,
especially when you use it to store, transmit and retrieve data.
It's a _markup_ language, people! Not a data storage language.
SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
------------------------------
Date: Wed, 16 Nov 2011 09:06:31 -0800 (PST)
From: ccc31807 <cartercc@gmail.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <470f4a99-a3b2-4844-bc10-848a24dbbefa@q9g2000yqe.googlegroups.com>
On Nov 16, 11:50=A0am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
> AFAIK, a well-formed XML file could have an order description looking lik=
e
> this:
>
> <order
>
> >1</order
>
> meaning, it is not really possible to parse XML without doing a
> character-by-character lexical analysis of the input data stream
> first.
As I said, it depends on the nature of your input. XML handles
'ragged' data as well as the kind of normalized data we would expect
to use for an RDBMS. If you aren't sure of the format of your data,
you obviously have to validate it somehow. Part of this might be
removing whitespace at the beginning and ends of lines. Sometimes it
might be removing newlines from several lines until you match some
kind of closing tag.
I don't advocate reinventing wheels. I also don't advocate searching
for a CPAN module as the first step in solving a particular
programming problem. If you need to run a script continually
processing the same kind of input, it might pay to cobble together
some code that does EXACTLY what you need, no more and no less, that
to use someone else's code.
I say this as a promiscuous user of CPAN modules -- hardly a week goes
by that I don't install a new module for one reason or another -- and
frequently I just look at the source, modify it to do what I need, and
don't use or require the module.
TIMTOWTDI, CC.
------------------------------
Date: Wed, 16 Nov 2011 09:17:05 -0800 (PST)
From: ccc31807 <cartercc@gmail.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <aad91d5c-d42e-4287-837d-61db6efacff8@g21g2000yqc.googlegroups.com>
On Nov 11, 7:11=A0pm, Dwight Army of Champions
<dwightarmyofchampi...@hotmail.com>
> <?xml version=3D"1.0"?>
> <library>
> <book>
> =A0 =A0 =A0 =A0 <title>Dreamcatcher</title>
> =A0 =A0 =A0 =A0 <author>Stephen King</author>
> =A0 =A0 =A0 =A0 <genre>Horror</genre>
> =A0 =A0 =A0 =A0 <pages>899</pages>
> =A0 =A0 =A0 =A0 <price>23.99</price>
> =A0 =A0 =A0 =A0 <rating>5</rating>
> =A0 =A0 =A0 =A0 <publication_date>11/27/2001</publication_date>
> </book>
...
> </library>
If I had this kind of file, and it was a static file, I would read it
into some kind of database. If you used something like SQLite, you
could read it into a table <book> element by <book> element, and then
use normal SQL to munge your data.
Alternative, you could convert the file into CSV format, which in many
ways is a lot easier to handle than XML.
It strikes me that using XML for this kind of work is overkill, unless
you had a specific requirement to use XML. If you had to use XML it
might pay to learn a little XSLT and use that instead of Perl. Perl is
a great language for string processing, but in some cases XSLT works
better.
CC.
------------------------------
Date: Wed, 16 Nov 2011 10:31:37 -0800 (PST)
From: Klaus <klaus03@gmail.com>
Subject: Re: How to import only part of a large XML file?
Message-Id: <ee03d757-c120-4925-8ccc-3e85bfd351f8@v5g2000yqn.googlegroups.com>
On 16 nov, 18:17, ccc31807 <carte...@gmail.com> wrote:
> On Nov 11, 7:11=A0pm, Dwight Army of Champions
> <dwightarmyofchampi...@hotmail.com>
>
> > <?xml version=3D"1.0"?>
> > <library>
> > <book>
> > =A0 =A0 =A0 =A0 <title>Dreamcatcher</title>
> > =A0 =A0 =A0 =A0 <author>Stephen King</author>
> > =A0 =A0 =A0 =A0 <genre>Horror</genre>
> > =A0 =A0 =A0 =A0 <pages>899</pages>
> > =A0 =A0 =A0 =A0 <price>23.99</price>
> > =A0 =A0 =A0 =A0 <rating>5</rating>
> > =A0 =A0 =A0 =A0 <publication_date>11/27/2001</publication_date>
> > </book>
> ...
> > </library>
>
> If I had this kind of file, and it was a static file, I would read it
> into some kind of database. If you used something like SQLite, you
> could read it into a table <book> element by <book> element, and then
> use normal SQL to munge your data.
>
> Alternative, you could convert the file into CSV format, which in many
> ways is a lot easier to handle than XML.
Converting to CSV is as easy as:
use strict;
use warnings;
use XML::Reader;
use Text::CSV_XS;
my $rdr =3D XML::Reader->new('huge.xml', {mode =3D> 'branches'},
{ root =3D> '/library/book', branch =3D> [
'/title',
'/author',
'/genre',
'/pages',
'/price',
'/rating',
'/publication_date',
]},
{ root =3D> '/library/music', branch =3D> [
'/title',
'/artist',
'/release_date',
'/label',
]});
my $csv =3D Text::CSV_XS->new({ sep_char =3D> ',', binary =3D> 1, eol =3D>
$/ });
open my $ofh, '>', 'out.csv' or die $!;
while ($rdr->iterate) {
$csv->print($ofh, [ ($rdr->rx =3D=3D 0 ? 'book' : 'music'), $rdr-
>value ]);
}
close $ofh;
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 3545
***************************************
