[27435] in Perl-Users-Digest
Perl-Users Digest, Issue: 9074 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Tue Mar 21 21:05:35 2006
Date: Tue, 21 Mar 2006 18:05:07 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Tue, 21 Mar 2006 Volume: 10 Number: 9074
Today's topics:
Re: How to release memory ? robic0
Re: How to release memory ? robic0
Re: How to release memory ? <nospam-abuse@ilyaz.org>
Safe Pipe Open question <paolucci@dunsel.cuug.ab.ca>
Re: Safe Pipe Open question robic0
Re: Safe Pipe Open question <john@castleamber.com>
Re: Safe Pipe Open question <jgibson@mail.arc.nasa.gov>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Tue, 21 Mar 2006 15:54:22 -0800
From: robic0
Subject: Re: How to release memory ?
Message-Id: <f64122psh6729333icvgc8n5m7qprumf49@4ax.com>
On Tue, 21 Mar 2006 10:47:23 +0100, zdbrg <zdbrg@mail.con> wrote:
>Ilya Zakharevich a dit le Mon, 20 Mar 2006 21:28:26 +0000 (UTC):
>>Good rule is: do not expect your memory going back to
>>system in any real-life situation.
>>
>
>Thanks for your answer.
>
>So, how do perl developers handle quite commons tasks where one program needs
>to use/allocate really big variables periodicaly which then should be released
>to let others tasks run
>?
>
>Starting a new process for those jobs which are known to use a lot of memory is
>a solution, is there others
>?
Consider the price for loose form and structure provided by Pseudo languages
such as Perl, Java, ur's here... to be paid! Actually you should program in
'C/C++' with a good os api. In that environment you can do your own memory
management, including low level routines that do block allocation as well
as defragmentation, without leaving the program..
Perl is a garbage collector that provides no usefull control of the os
primatives.
IMO Perl hasn't got long to live. If you put all ur eggs in this basket
you will be in trouble careerwise as well...
robic0
------------------------------
Date: Tue, 21 Mar 2006 16:19:00 -0800
From: robic0
Subject: Re: How to release memory ?
Message-Id: <8q51221uv3ktsst41a7aamqd1rf3najgh1@4ax.com>
On 21 Mar 2006 20:34:45 GMT, xhoster@gmail.com wrote:
>zdbrg <zdbrg@mail.con> wrote:
>> xhoster@gmail.com a dit le 21 Mar 2006 14:50:07 GMT:
>>
>> >Mostly I just let the VM system do it's job. Are you sure that won't
>> >work for you?
>> >
>> >Xho
>>
>> Yes it will, at the cost of swapping hundred megs of memory to disk.
>
>A hundred meg here, an hundred meg there, pretty soon you may be talking
>about real IO :).
>
>Swapping lots of memory to disk is often a small price to pay for the
>benefit of not having to micromanage your program's memory use. And when
>that price is too high, then your optimization choices are so intimately
>tied in with the nitty-gritty details of your code that it is hard to
>discuss them meaningfully in generalities. At various times, I've:
So ahh, when do you have *ANY* programitic control of when and how much
gets swapped to *disk* ?????? Huh...?
>
>Opened "+<" then used seek, tell, read, etc.
>Used Acme::Abuse to isolate certain parts to separate process.
>Used various tie modules to tie the data to disk.
>Used mmap through XS code (although doing it again I'd probably look into
>Sys::Mmap first) Made my code readily restartable, so I could exit after a
>particularly memory heavy usage and then pick up where it left off in a
>fresh interpreter.
>
>But the vast majority of the time I don't do any of these rather painful
>and time consuming (my time, that is) things, I just let the VM system swap
>out hundreds of megabytes.
>
>>
>> As zentara pointed out, using some kind of persistant database (which was
>> not wanted at the first time), will surely drop down memory use, adding a
>> bit of complexity, and probably slowlyness.
>
>This is often a good idea, especially if your large data structure is a
>hash. But if it is a scalar, like your example seems to suggest, it is less
>clear how useful this will be.
>
>> Forking or using a crontab
>> (on *nix systems) for that is surely another good solution.
>
>I don't see how crontab would help here, but forking certainly could if you
>have a lot of mostly independent tasks where some are very big and some
>are not. But if in that situation, I'd tend to use forking anyway, even
>if memory was not a concern.
>
>Xho
Your such a blow hard. Excuse me but you don't know your ass from your
elbow. You never did, you never will..
robic0
------------------------------
Date: Wed, 22 Mar 2006 02:04:24 +0000 (UTC)
From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
Subject: Re: How to release memory ?
Message-Id: <dvqbb8$oc4$1@agate.berkeley.edu>
[A complimentary Cc of this posting was sent to
zdbrg
<zdbrg@mail.con>], who wrote in article <441fcbab$0$26160$626a54ce@news.free.fr>:
> >Good rule is: do not expect your memory going back to
> >system in any real-life situation.
> So, how do perl developers handle quite commons tasks where one
> program needs to use/allocate really big variables periodicaly which
> then should be released to let others tasks run ?
If this is A FEW of REALLY BIG allocations, then indeed, an
implementation of malloc() which has a possibility of releasing some
regions of memory back to system may indeed have a chance to ACTUALLY
USE this possibility. So disabling `usemymalloc' in such situations
will be beneficial.
However, in my experience, typical memory hog Perl applications would
hog memory by a lot of relatively small chunks. Then `usemymalloc'
will have a lot of advantages over a typical "system" malloc(): it
should be quickier, use less memory, and, most important of all, it
gives feedback to Perl; in addition to additional speedups, this
feedback simplifies memory debugging a lot.
With feedback, the developer has a real chance to improve the
algorithm. Think about it this way: it is the algorithm, not Perl,
which hogs the memory. 1/3 ;-)
Hope this helps,
Ilya
------------------------------
Date: Wed, 22 Mar 2006 01:00:29 GMT
From: Mike Paolucci <paolucci@dunsel.cuug.ab.ca>
Subject: Safe Pipe Open question
Message-Id: <Nk1Uf.11$B_1.10@edtnps89>
I'm getting unexplained errors when I read from a log file. I'm using a
"Safe Pipe Open", and my reference for this is:
http://www.litespeed.net/perldocs/lib/Pod/perlipc.html#Using_open_for_IPC
Here's the code:
my $logFile = 'full_path_to_logfile'; (not specified for confidentially)
my $ipAddress = '12.34.56.78'; (not specified for confidentially)
my $fh;
open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
die "Cannot open logfile!!\n";
# get the last lines
my $logData = '';
while ( <$fh> )
{
$logData .= $_;
}
close( $fh ) or die "Cannot close logfile!\n";
# display these lines
print $logData;
The code works perfectly and displays no errors. The probem is that
when I look in the system error file I see lines and lines of:
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
grep: writing output: Broken pipe
Why are these errors there?
What do they mean?
How do I prevent them?
------------------------------
Date: Tue, 21 Mar 2006 17:20:32 -0800
From: robic0
Subject: Re: Safe Pipe Open question
Message-Id: <ng91225j4eei4nlktbmeolncv4e3fdntnh@4ax.com>
On Wed, 22 Mar 2006 01:00:29 GMT, Mike Paolucci <paolucci@dunsel.cuug.ab.ca> wrote:
>I'm getting unexplained errors when I read from a log file. I'm using a
>"Safe Pipe Open", and my reference for this is:
>
>http://www.litespeed.net/perldocs/lib/Pod/perlipc.html#Using_open_for_IPC
>
>
>Here's the code:
>
>my $logFile = 'full_path_to_logfile'; (not specified for confidentially)
>my $ipAddress = '12.34.56.78'; (not specified for confidentially)
>
>my $fh;
>open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
> die "Cannot open logfile!!\n";
>
># get the last lines
>my $logData = '';
>while ( <$fh> )
>{
> $logData .= $_;
>}
>close( $fh ) or die "Cannot close logfile!\n";
>
># display these lines
>print $logData;
>
>
>The code works perfectly and displays no errors. The probem is that
>when I look in the system error file I see lines and lines of:
>
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>grep: writing output: Broken pipe
>
>
>Why are these errors there?
>
>What do they mean?
>
>How do I prevent them?
Since the code works perfectly, I would say
"grep" has written a problem message out.
I could be wrong though...
robic0
------------------------------
Date: 22 Mar 2006 01:21:19 GMT
From: John Bokma <john@castleamber.com>
Subject: Re: Safe Pipe Open question
Message-Id: <Xns978DC4E20CE2Dcastleamber@130.133.1.4>
Mike Paolucci <paolucci@dunsel.cuug.ab.ca> wrote:
> my $fh;
> open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
> die "Cannot open logfile!!\n";
It's $! not !!.
--
John Bokma Freelance software developer
&
Experienced Perl programmer: http://castleamber.com/
------------------------------
Date: Tue, 21 Mar 2006 17:46:04 -0800
From: Jim Gibson <jgibson@mail.arc.nasa.gov>
Subject: Re: Safe Pipe Open question
Message-Id: <210320061746042666%jgibson@mail.arc.nasa.gov>
In article <Nk1Uf.11$B_1.10@edtnps89>, Mike Paolucci
<paolucci@dunsel.cuug.ab.ca> wrote:
> I'm getting unexplained errors when I read from a log file. I'm using a
> "Safe Pipe Open", and my reference for this is:
>
> http://www.litespeed.net/perldocs/lib/Pod/perlipc.html#Using_open_for_IPC
>
>
> Here's the code:
>
> my $logFile = 'full_path_to_logfile'; (not specified for confidentially)
> my $ipAddress = '12.34.56.78'; (not specified for confidentially)
>
> my $fh;
> open( $fh, "grep $ipAddress $logFile | tail 2>&1 |" ) or
> die "Cannot open logfile!!\n";
>
> # get the last lines
> my $logData = '';
> while ( <$fh> )
> {
> $logData .= $_;
> }
> close( $fh ) or die "Cannot close logfile!\n";
>
> # display these lines
> print $logData;
>
>
> The code works perfectly and displays no errors. The probem is that
> when I look in the system error file I see lines and lines of:
>
> grep: writing output: Broken pipe
Your code works perfectly for me (Mac OS 10.4, perl 5.8.6), although I
did not try it on any really big files. I suggest if you just want to
get your program working, you read and process the log file within your
Perl program and save lines that contain a substring:
#!/usr/local/bin/perl
use warnings;
use strict;
my $logFile = 'full_path_to_logfile';
my $ipAddress = '12.34.56.78';
my $fh;
open( $fh, '<', $logFile ) or
die "Cannot open logfile!!\n";
my $logData = '';
my $matches = 0;
while ( <$fh> )
{
if( index($_,$ipAddress) >= 0 ) {
$logData .= $_ ;
last if ++$matches >= 10;
}
}
close( $fh ) or die "Cannot close logfile!\n";
# display these lines
print $logData;
Posted Via Usenet.com Premium Usenet Newsgroup Services
----------------------------------------------------------
** SPEED ** RETENTION ** COMPLETION ** ANONYMITY **
----------------------------------------------------------
http://www.usenet.com
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 9074
***************************************
