[26544] in Perl-Users-Digest
Perl-Users Digest, Issue: 8688 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sat Nov 19 18:07:05 2005
Date: Sat, 19 Nov 2005 15:05:04 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sat, 19 Nov 2005 Volume: 10 Number: 8688
Today's topics:
Lunge crowd, kill file robic0
Re: XML Parsing too slow with XML::Simple <toddrw69@excite.com>
Re: XML Parsing too slow with XML::Simple <daveandniki@ntlworld.com>
Re: XML Parsing too slow with XML::Simple <1usa@llenroc.ude.invalid>
Re: XML Parsing too slow with XML::Simple <matthew.garrish@sympatico.ca>
Re: XML Parsing too slow with XML::Simple <matthew.garrish@sympatico.ca>
Re: XML Parsing too slow with XML::Simple <1usa@llenroc.ude.invalid>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 19 Nov 2005 13:25:14 -0800
From: robic0
Subject: Lunge crowd, kill file
Message-Id: <2j4vn113qkrud4bqkcjnrvavggsus4amb2@4ax.com>
Thinking about newsgroup kill files I had some thoughts.
In neophyte groups (like this) where those that live by
the creed that they are in the expert group (debateable)
prey upon the new either poster and/or to Perl neo's,
will lunge. The stupider the question, the more the "lunge".
Bad English? Hey, more lunge. The reply is not for the
OP's benifit, the propensity to reply is based on the
aggregate "lunge" and proximity. There is usually a
small minority of enthusiast's who read through the
OP's fopa's to answer their question. Funny how that is.
The majority of the "lungers" oh just passs that by.
Then theres me... A hard assed motherfucker, drinkin,
swearin, fuckin, seen all this bullshit before who says
it like it is to the OP who will post code and read
through the fopa's (that I don't give a rats ass about).
I'm writing Perl 60 hrs/week and I crank on it. My
interests are technique and some style.
I put these "lungers" in thier place and I end up on their
kill files. Thats the way I want it! I crawle up
thier asses and put them in thier place. The OP's
"lunge victim" can now see them in context, and
get good solutions to thier problem.
The only good solution to OP's questions are actual
code snip's and recommendations to other code.
Outside of those kinds of recommendations in this group
give a orgasm to the "lunge" crowd.
The names of the "LUNGE" crowd are:
(if you don't fill this in, I will!!!)
------------------------------
Date: Sat, 19 Nov 2005 14:11:42 GMT
From: "Todd W" <toddrw69@excite.com>
Subject: Re: XML Parsing too slow with XML::Simple
Message-Id: <yoGff.243$fO5.123@newssvr33.news.prodigy.com>
"jabby" <krising@gmail-dot-com.no-spam.invalid> wrote in message
news:SdadnTQ6M66PteLeRVn-hg@giganews.com...
> I have some files ranging from 6 meg to 15 meg that I need to process
> to input into a database. I'm inheriting some old code using
> XML::Simple and this process is taking forever (an hour or more). I
> have eliminated the database as the time hog in this equation.
>
> is there an alternative XML module I should be using?
>
There are several API's to XML in perl. Most of them are going to be faster
than XML::Simple.
For high performance XML parsing I use SAX. SAX is a language independent
light weight XML interface. As with all XML interfaces, it has somewhat of a
learning curve.
If you would like, please post an example program that demonstrates your
problem space and a set of test data and we will be happy to help.
Todd W.
------------------------------
Date: Sat, 19 Nov 2005 19:37:17 +0100
From: "Dave" <daveandniki@ntlworld.com>
Subject: Re: XML Parsing too slow with XML::Simple
Message-Id: <437f70e3$0$18345$8fcfb975@news.wanadoo.fr>
"jabby" <krising@gmail-dot-com.no-spam.invalid> wrote in message
news:SdadnTQ6M66PteLeRVn-hg@giganews.com...
>I have some files ranging from 6 meg to 15 meg that I need to process
> to input into a database. I'm inheriting some old code using
> XML::Simple and this process is taking forever (an hour or more). I
> have eliminated the database as the time hog in this equation.
>
> is there an alternative XML module I should be using?
>
>
> Sent via Archivaty.com
Even with a 2Mb file I was finding XML::Simple far too slow, (I think it
depends on the complexity of your XML markup). Switching to XML::Parser
solved the problem (10 seconds for a 2Mb file on a 1Ghz laptop).
------------------------------
Date: Sat, 19 Nov 2005 19:26:49 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: XML Parsing too slow with XML::Simple
Message-Id: <Xns971392F803BB7asu1cornelledu@127.0.0.1>
"Dave" <daveandniki@ntlworld.com> wrote in
news:437f70e3$0$18345$8fcfb975@news.wanadoo.fr:
>
> "jabby" <krising@gmail-dot-com.no-spam.invalid> wrote in message
> news:SdadnTQ6M66PteLeRVn-hg@giganews.com...
>>I have some files ranging from 6 meg to 15 meg that I need to process
>> to input into a database. I'm inheriting some old code using
>> XML::Simple and this process is taking forever (an hour or more). I
>> have eliminated the database as the time hog in this equation.
>>
>> is there an alternative XML module I should be using?
>>
>>
>> Sent via Archivaty.com
>
> Even with a 2Mb file I was finding XML::Simple far too slow, (I think
> it depends on the complexity of your XML markup). Switching to
> XML::Parser solved the problem (10 seconds for a 2Mb file on a 1Ghz
> laptop).
I normally would prefer XML::Simple for, guess what, simple files.
That said, I got curious, so I decided to check. I wrote the following
quick script:
#!/usr/bin/perl
use strict;
use warnings;
use XML::Simple;
my $conf;
if (@ARGV) {
$conf = XMLin();
} else {
for my $key1 ('a' .. 'z') {
for my $key2 ('a' .. 'z') {
for my $key3 ('a' .. 'z') {
$conf->{$key1}->{$key2}->{$key3} = ($key1) x 1024;
$conf->{$key1}->{$key3}->{$key2} = ($key1) x 1024;
$conf->{$key3}->{$key1}->{$key2} = ($key2) x 1024;
$conf->{$key3}->{$key2}->{$key1} = ($key2) x 1024;
$conf->{$key2}->{$key3}->{$key1} = ($key3) x 1024;
$conf->{$key2}->{$key1}->{$key3} = ($key3) x 1024;
}
}
}
XMLout($conf, OutputFile => 'x.xml');
}
__END__
If called with an argument, the script reads the xml file it generated
in a previous run without an argument.
D:\Home\asu1\UseNet\clpmisc\xml> dir x.xml
...
11/19/2005 02:20 PM 18,097,887 x.xml
D:\Home\asu1\UseNet\clpmisc\xml> timethis x 1
TimeThis : Command Line : x 1
TimeThis : Start Time : Sat Nov 19 14:23:26 2005
TimeThis : End Time : Sat Nov 19 14:23:28 2005
TimeThis : Elapsed Time : 00:00:01.562
This is on a AMD64 laptop running at 800 Mhz (I am on batteries right
now) with at least 60 other windows programs running, including
PowerPoint, Firefox with about 30 tabs, and Apache).
I would not call that slow.
Sinan
--
A. Sinan Unur <1usa@llenroc.ude.invalid>
(reverse each component and remove .invalid for email address)
comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: Sat, 19 Nov 2005 15:57:10 -0500
From: "Matt Garrish" <matthew.garrish@sympatico.ca>
Subject: Re: XML Parsing too slow with XML::Simple
Message-Id: <JlMff.3790$KP5.489934@news20.bellglobal.com>
"A. Sinan Unur" <1usa@llenroc.ude.invalid> wrote in message
news:Xns971392F803BB7asu1cornelledu@127.0.0.1...
> "Dave" <daveandniki@ntlworld.com> wrote in
> news:437f70e3$0$18345$8fcfb975@news.wanadoo.fr:
>
>>
>> "jabby" <krising@gmail-dot-com.no-spam.invalid> wrote in message
>> news:SdadnTQ6M66PteLeRVn-hg@giganews.com...
>>>I have some files ranging from 6 meg to 15 meg that I need to process
>>> to input into a database. I'm inheriting some old code using
>>> XML::Simple and this process is taking forever (an hour or more). I
>>> have eliminated the database as the time hog in this equation.
>>>
>>> is there an alternative XML module I should be using?
>>>
>>>
>>> Sent via Archivaty.com
>>
>> Even with a 2Mb file I was finding XML::Simple far too slow, (I think
>> it depends on the complexity of your XML markup). Switching to
>> XML::Parser solved the problem (10 seconds for a 2Mb file on a 1Ghz
>> laptop).
>
> I normally would prefer XML::Simple for, guess what, simple files.
>
> That said, I got curious, so I decided to check. I wrote the following
> quick script:
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
> use XML::Simple;
>
> my $conf;
>
> if (@ARGV) {
> $conf = XMLin();
> } else {
> for my $key1 ('a' .. 'z') {
> for my $key2 ('a' .. 'z') {
> for my $key3 ('a' .. 'z') {
> $conf->{$key1}->{$key2}->{$key3} = ($key1) x 1024;
> $conf->{$key1}->{$key3}->{$key2} = ($key1) x 1024;
> $conf->{$key3}->{$key1}->{$key2} = ($key2) x 1024;
> $conf->{$key3}->{$key2}->{$key1} = ($key2) x 1024;
> $conf->{$key2}->{$key3}->{$key1} = ($key3) x 1024;
> $conf->{$key2}->{$key1}->{$key3} = ($key3) x 1024;
> }
> }
> }
> XMLout($conf, OutputFile => 'x.xml');
> }
>
> __END__
>
> If called with an argument, the script reads the xml file it generated
> in a previous run without an argument.
>
> D:\Home\asu1\UseNet\clpmisc\xml> dir x.xml
> ...
> 11/19/2005 02:20 PM 18,097,887 x.xml
>
> D:\Home\asu1\UseNet\clpmisc\xml> timethis x 1
> TimeThis : Command Line : x 1
> TimeThis : Start Time : Sat Nov 19 14:23:26 2005
> TimeThis : End Time : Sat Nov 19 14:23:28 2005
> TimeThis : Elapsed Time : 00:00:01.562
>
> This is on a AMD64 laptop running at 800 Mhz (I am on batteries right
> now) with at least 60 other windows programs running, including
> PowerPoint, Firefox with about 30 tabs, and Apache).
>
> I would not call that slow.
>
That's not a fair test. Your xml file may be 18Mb, but it only has one level
of nesting, ~600 tags and the bulk of the size is data in attributes. I
would suggest making an 18Mb file with records like the following:
<record>
<test>
<data1>blech</data1>
<data2>
<data3>more blech</data3>
<data3>other blech</data3>
<data4 attr="value">doh</data4>
</data2>
</test>
</record>
And you'll find a considerable difference in parsing time (I've waited eight
minutes now and it's still chugging away draining resources). Even when it
completes, my system is going so slow right now that any processing on the
data in the hash after it is built is going to take a protracted amount of
time. If the OP is doing a lot of work on the data I can see how it could
require an inordinate amount of time to finish. XML-Simple is best for small
configuration files you want to get quickly into a hash. As another poster
noted, XML-SAX is a better choice for a file of this size.
Matt
------------------------------
Date: Sat, 19 Nov 2005 16:20:00 -0500
From: "Matt Garrish" <matthew.garrish@sympatico.ca>
Subject: Re: XML Parsing too slow with XML::Simple
Message-Id: <2GMff.3817$KP5.492965@news20.bellglobal.com>
"Matt Garrish" <matthew.garrish@sympatico.ca> wrote in message
news:JlMff.3790$KP5.489934@news20.bellglobal.com...
>
> "A. Sinan Unur" <1usa@llenroc.ude.invalid> wrote in message
> news:Xns971392F803BB7asu1cornelledu@127.0.0.1...
>>
>> D:\Home\asu1\UseNet\clpmisc\xml> timethis x 1
>> TimeThis : Command Line : x 1
>> TimeThis : Start Time : Sat Nov 19 14:23:26 2005
>> TimeThis : End Time : Sat Nov 19 14:23:28 2005
>> TimeThis : Elapsed Time : 00:00:01.562
>>
>> This is on a AMD64 laptop running at 800 Mhz (I am on batteries right
>> now) with at least 60 other windows programs running, including
>> PowerPoint, Firefox with about 30 tabs, and Apache).
>>
>> I would not call that slow.
>>
>
> That's not a fair test. Your xml file may be 18Mb, but it only has one
> level of nesting, ~600 tags and the bulk of the size is data in
> attributes. I would suggest making an 18Mb file with records like the
> following:
>
It finally completed after 695s, and promptly took another two minutes after
printing the benchmark line before it finally was able to exit (which
considering that it was doing nothing after reading the file probably gives
some indication of the OP's problem). The memory usage was also about
360,000K by the end and had my computer in gridlock.
Matt
------------------------------
Date: Sat, 19 Nov 2005 21:24:58 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: XML Parsing too slow with XML::Simple
Message-Id: <Xns9713A70044236asu1cornelledu@127.0.0.1>
"Matt Garrish" <matthew.garrish@sympatico.ca> wrote in
news:JlMff.3790$KP5.489934@news20.bellglobal.com:
>
> "A. Sinan Unur" <1usa@llenroc.ude.invalid> wrote in message
> news:Xns971392F803BB7asu1cornelledu@127.0.0.1...
>> "Dave" <daveandniki@ntlworld.com> wrote in
>> news:437f70e3$0$18345$8fcfb975@news.wanadoo.fr:
>>
>>>
>>> "jabby" <krising@gmail-dot-com.no-spam.invalid> wrote in message
>>> news:SdadnTQ6M66PteLeRVn-hg@giganews.com...
>>>>I have some files ranging from 6 meg to 15 meg that I need to
>>>>process
>>>> to input into a database. I'm inheriting some old code using
>>>> XML::Simple and this process is taking forever (an hour or more). I
>>>> have eliminated the database as the time hog in this equation.
>>>>
>>>> is there an alternative XML module I should be using?
>>>>
>>>>
>>>> Sent via Archivaty.com
>>>
>>> Even with a 2Mb file I was finding XML::Simple far too slow, (I
>>> think it depends on the complexity of your XML markup). Switching to
>>> XML::Parser solved the problem (10 seconds for a 2Mb file on a 1Ghz
>>> laptop).
>>
>> I normally would prefer XML::Simple for, guess what, simple files.
[ snip silly code ]
> That's not a fair test. Your xml file may be 18Mb, but it only has one
> level of nesting, ~600 tags and the bulk of the size is data in
> attributes. I would suggest making an 18Mb file with records like the
> following:
>
> <record>
> <test>
> <data1>blech</data1>
> <data2>
> <data3>more blech</data3>
> <data3>other blech</data3>
> <data4 attr="value">doh</data4>
> </data2>
> </test>
> </record>
OK, so I tried with 100,000 copies of the snippet above:
D:\Home\asu1\UseNet\clpmisc\xml> dir x.xml
...
11/19/2005 04:16 PM 21,800,015 x.xml
D:\Home\asu1\UseNet\clpmisc\xml> timethis x 1
TimeThis : Command Line : x 1
TimeThis : Start Time : Sat Nov 19 16:16:44 2005
TimeThis : End Time : Sat Nov 19 16:18:14 2005
TimeThis : Elapsed Time : 00:01:30.703
> XML-Simple is best for small configuration files you want to
> get quickly into a hash.
Of course. I am *not* saying that XML::Simple should be used for
anything that is not, ehm, simple.
> As another poster noted, XML-SAX is a better choice for a file of this
> size.
Ditto.
Sinan
--
A. Sinan Unur <1usa@llenroc.ude.invalid>
(reverse each component and remove .invalid for email address)
comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 8688
***************************************
