[26603] in Perl-Users-Digest
Perl-Users Digest, Issue: 8727 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Thu Dec 1 06:05:28 2005
Date: Thu, 1 Dec 2005 03:05:04 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Thu, 1 Dec 2005 Volume: 10 Number: 8727
Today's topics:
Re: My experience with XML::DOM VS XML::LibXML <jvs@india.ti.com>
Whats the reason for this?? <recker_us@yahoo.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Thu, 1 Dec 2005 15:58:17 +0530
From: Jahagirdar Vijayvithal S <jvs@india.ti.com>
Subject: Re: My experience with XML::DOM VS XML::LibXML
Message-Id: <slrndotk21.q1s.jvs@bstclindt008.india.ti.com>
* robic0 <robic0> wrote:
> On Thu, 1 Dec 2005 10:49:38 +0530, Jahagirdar Vijayvithal S
> <jvs@india.ti.com> wrote:
>
>>I have a script where I am
>>1> Opening a pipe to a program which reads in a binary file(400MB) and dumps out XML data(XXX GB's) (tethereal)
>>2> Grabing chunk's of data within tags <packet>.....</packet> (approx 4
>>to 20 K)
> So the data within tags contains the xml you want to parse?
Right.
>>3> parsing the XML
> within the packet tags
right again
>>4> post processing based on fields and attributes in the XML document.
> which document?
Oops sorry for the confusion I meant based on elements and attributes
within the <packet> </packet> tags.
>>
>>Initially I used XML::DOM and found that my memory consumption
>>constantly increased filling up the entire RAM and SWAP space before
>>crashing(approx 32GB RAM and 160+ GB Swap consumed).
>>Switching to XML:LibXML and replacing the XML::DOM constructs with their
>>equivalent I find that my worst case Memory consumption remains below
>>2GB each (RAM and swap) and average is around 50 MB each.
one additional observation is that the XML::DOM would crash after
processing around 6K such packets where as XML::LibXML was able to
handle more than .6G packets
since the only thing that changed was the line
use XML::DOM;
my $parser = XML::DOM::Parser->new();
and the function getval();
and the changes were that of replacing these lines by their equivalent
in XML::LibXML I was wondering what could be the problem. while my
initial guess is that of some sort of memory leak......
>>
> use SAX, nodes will fall off the ends of the earth
>>While my problem is solved I am curious to know wether there is any
>>known Issues which caused the above problems?
>>
>>code fragment used by me is as below
>>-----------------------------Code-----------------
>>use XML::DOM;
>>my $parser = XML::DOM::Parser->new();
>>open XML ,"$tethereal -r $pcapfile -T pdml|" or die "Cant open a simple pipe? go smoke one!";
>> while(<XML>){
>> #print;
>> if(my $range=/<packet>/.../<\/packet>/){
> ^ ^^^
> bad Perl? (...) capture?
bad perl: why?
... => range operator with evaluation of RHS pattern in the next cycle.
and $range capturing the sequence number.
> what would happen if:
> <packet>adsfbasdfbabf</packet><packet>adsfbasdfbabf
> </packet><packet>adsfbasdfbabf</packet>
> <packet>adsfbasdfbabf
> </packet>
While theoritically possible the tool converting the binary data to XML
always dumps out one element per line e.g.
<packet>
<proto name="geninfo" pos="0" showname="General information" size="114">
<field name="num" pos="0" show="1" showname="Number" value="1" size="114"/>
<field name="len" pos="0" show="114" showname="Packet Length" value="72" size="114"/>
<field name="caplen" pos="0" show="114" showname="Captured Length" value="72" size="114"/>
<field name="timestamp" pos="0" show="Sep 30, 2005 11:34:22.158787000" showname="Captured Time" value="1128060262.158787000" size="114"/>
</proto>
....
</packet>
> Thats the funny thing about xml, you can edit it in notepad.
>> if($range==1){
> will it even get here if $range == 0?
no, refer perldoc perlop for range operator.
>> $data="<JVS_PARSER>";
>> }
>> $data="$data $_";
> what are you grabbing here? i thought you just needed packet data?
Thats right. I am reading each line from XML, as long as it falls
between the <packet></packet> tags appending it to $data to form one
single scalar with the required data.
>> if ($range=~/E0/){
>> $data="$data</JVS_PARSER>";
>>........ various calls to function getval and Other processingstuff
>> }
>> }
>>}
>>
>>sub getval(){
> sub getval(???){ ## ?
prototype declared at top of file (not included in code snippet)
>> my ($data,$name,$attribute)=@_;
>> return unless defined($data);
>> my $doc = $parser->parse($data);#Should be parsing this just once outside the function call
>> foreach my $element ($doc->getElementsByTagName('field')){
>> if ($element->getAttribute('name') eq $name){
>> return $element->getAttribute($attribute);
>> }
>> }
>> return -1; #Error
>>}
>>
>>--------------------------End Code-------------------------
>>
>>Regards
>>Jahagirdar Vijayvithal S
>
> Don't know how you got anything to work on this.
I hope the comments above would clarify some of your doubts.
> You would be better off if you use SAX to start with.
while I have been following the SAX vs ... debate in other thread. I was
already familiar with XML::DOM and had a code base using DOM to build
on. which facilitated a quick workable code which could be used to
perform the actual processing I am interested in.
> I know the whole XXX gigabyte attracts attention to you
> question but you should give more details.
Regards
Jahagirdar Vijayvithal S
--
------------------------------
Date: Thu, 01 Dec 2005 16:33:29 +0530
From: JP <recker_us@yahoo.com>
Subject: Whats the reason for this??
Message-Id: <z0Bjf.8$MC6.52@news.oracle.com>
perl -e '1+1+1'
The answer is 3.
but
perl -e 'print (1+1)+1'
thje answer in this case is 2.
Any explination why I get geting a different value in the second case??
Thanks,
JP
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 8727
***************************************
