[26943] in Perl-Users-Digest
Perl-Users Digest, Issue: 8908 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Tue Jan 31 21:05:16 2006
Date: Tue, 31 Jan 2006 18:05:06 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Tue, 31 Jan 2006 Volume: 10 Number: 8908
Today's topics:
Re: FAQ 1.3 Which version of Perl should I use? <brian.d.foy@gmail.com>
Memory usage <rugowski1spm@nhw.pl>
Re: Memory usage <matthew.garrish@sympatico.ca>
Re: Memory usage <1usa@llenroc.ude.invalid>
Re: Memory usage <rugowski1spm@nhw.pl>
Re: Memory usage <rvtol+news@isolution.nl>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Tue, 31 Jan 2006 19:03:45 -0600
From: brian d foy <brian.d.foy@gmail.com>
Subject: Re: FAQ 1.3 Which version of Perl should I use?
Message-Id: <310120061903452997%brian.d.foy@gmail.com>
In article <Xns975CA4BFB1A8ebohlmanomsdevcom@130.133.1.4>, Eric Bohlman
<ebohlman@omsdev.com> wrote:
> PerlFAQ Server <comdog@pair.com> wrote in
> news:drk6m4$c2u$1@reader2.panix.com:
>
> > * There is no perl6.x for the next couple of years. Stay tuned,
> > but
> > don't worry that you'll have to change major versions of Perl
> > soon (i.e. before 2006).
>
> Ahem!
>
Maybe I should make that "(i.e. two years from when you read this") !
Thanks,
------------------------------
Date: Wed, 01 Feb 2006 00:39:22 +0100
From: Witold Rugowski <rugowski1spm@nhw.pl>
Subject: Memory usage
Message-Id: <drosfe$n8i$1@hplms2.hpl.hp.com>
Hi!
I have following problem. My perl script is parsing large log files (to be exact many medium sized) up to 3 or 4 GB of data. But all what is done, it is extracting some data (IP address and volume of traffic).
Data is read line by line, from file by file, with something like (not real code):
while (@files) {
open FILE, files[0] or die;
shift;
while (<FILE>) { if ( $_ =~ /(MY_PATTERN)/ ) call_sum_function($1); }
}
print_results();
All hashes, storing data are very compact (after processing 90% of 1.6 GB data all my defined hashes have less than 30 entries, every having two integers).
And after those 90% shit happens:
Out of memory during request for 156 bytes, total sbrk() is 536711168 bytes!
And I don't have idea why?? Of course I can read, that perl allocated 512MB and that is enough for him, but I have no idea:
a) what consumed this memory
b) how to avoid this error
Any help?
--
Witold Rugowski
rugowski_at_nhw_pl
http://nhw.pl/blg/
------------------------------
Date: Tue, 31 Jan 2006 18:58:05 -0500
From: "Matt Garrish" <matthew.garrish@sympatico.ca>
Subject: Re: Memory usage
Message-Id: <eQSDf.7390$Iw6.396030@news20.bellglobal.com>
"Witold Rugowski" <rugowski1spm@nhw.pl> wrote in message
news:drosfe$n8i$1@hplms2.hpl.hp.com...
> Hi!
> I have following problem. My perl script is parsing large log files (to be
> exact many medium sized) up to 3 or 4 GB of data. But all what is done, it
> is extracting some data (IP address and volume of traffic).
>
> Data is read line by line, from file by file, with something like (not
> real code):
>
Then why ask for help? If you don't post real code how is anyone supposed to
guess what you've done wrong?
Matt
------------------------------
Date: Wed, 01 Feb 2006 00:19:57 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: Memory usage
Message-Id: <Xns975CC4BD9B8C9asu1cornelledu@127.0.0.1>
Witold Rugowski <rugowski1spm@nhw.pl> wrote in
news:drosfe$n8i$1@hplms2.hpl.hp.com:
> Data is read line by line, from file by file, with something like (not
> real code):
Please post real code.
> while (@files) {
> open FILE, files[0] or die;
> shift;
What do you think this shift is shifting. Oh, this is not real code.
> And after those 90% shit happens:
Well, as they say, shit does happen occasionally. It's just a fact of
life.
> a) what consumed this memory
Who knows. You are in the best position to know because you have access
to the real code, and the real data.
Do you think we are mindreaders (or we can telepathically access the
hard drive on your computer, or the biometric USB drive on which this
script is stored)?
> b) how to avoid this error
I guess you could use:
http://search.cpan.org/~podmaster/Acme-MJD-0.02/lib/Acme/MJD.pm
Sinan
--
A. Sinan Unur <1usa@llenroc.ude.invalid>
(reverse each component and remove .invalid for email address)
comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: Wed, 01 Feb 2006 01:20:42 +0100
From: Witold Rugowski <rugowski1spm@nhw.pl>
Subject: Re: Memory usage
Message-Id: <5078f$43dffedb$d4ba5a95$15162@news.chello.pl>
John Bokma wrote:
> Bring back your problem to the smallest possible program that has this
> problem and post the *real* code.
OK. I post it in a few hours...
--
Witold Rugowski
rugowski_at_nhw_pl
http://nhw.pl/blg/
------------------------------
Date: Wed, 1 Feb 2006 01:55:24 +0100
From: "Dr.Ruud" <rvtol+news@isolution.nl>
Subject: Re: Memory usage
Message-Id: <drp4ev.tc.1@news.isolution.nl>
Witold Rugowski schreef:
> (not real code):
use strict;
use warnings;
> while (@files) {
> open FILE, files[0] or die;
See: perldoc -f open
Use the three-arg form.
If your Perl is 5.8+, use a Perl scalar filehandle.
Use $! in the die.
What is "files[0]"?
> shift;
Why the "shift"?
Did you maybe start with something like the following code,
and changed all underscores to "files", s/_/files/g?
while (@_) {
open FILE, $_[0] or die;
> while (<FILE>) {
> if ( $_ =~ /(MY_PATTERN)/ ) call_sum_function($1);
> }
> }
> print_results();
--
Affijn, Ruud
"Gewoon is een tijger."
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 8908
***************************************
