[19082] in Perl-Users-Digest
Perl-Users Digest, Issue: 1277 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Wed Jul 11 03:37:01 2001
Date: Wed, 11 Jul 2001 00:36:38 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Message-Id: <994836998-v10-i1277@ruby.oce.orst.edu>
Content-Type: text
Perl-Users Digest Wed, 11 Jul 2001 Volume: 10 Number: 1277
Today's topics:
Re: Net::Telnet logging on problem..... <manuel.hoelss@uni-giessen.de>
Re: Net::Telnet logging on problem..... (Logan Shaw)
Re: Net::Telnet logging on problem..... <ayamanita.nospam@bigfoot.com>
Re: Net::Telnet logging on problem..... <manuel.hoelss@uni-giessen.de>
Re: NNTP servers <goldbb2@earthlink.net>
no pragma in loop <intmktg@gloria.cam.org>
Re: no pragma in loop <tony_curtis32@yahoo.com>
Re: no pragma in loop (Jay Tilton)
open file for update (hao tran)
Re: open file for update (Eric Bohlman)
Re: open file for update <gnarinn@hotmail.com>
Open Process with a space <hwendt@premitech.com>
Re: Open Process with a space (Eric Bohlman)
Pattern <jocke30_gbg@hotmail.com>
Performance problem--memory bloat or crawl <cjmackie@princeton.edu>
Re: Performance problem--memory bloat or crawl (Logan Shaw)
Re: Performance problem--memory bloat or crawl (Logan Shaw)
Re: Performance problem--memory bloat or crawl (Logan Shaw)
Re: Performance problem--memory bloat or crawl <krahnj@acm.org>
Re: Perl and the NT Event Log. <rothd@roth.net>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Tue, 10 Jul 2001 19:47:13 +0200
From: Manuel =?ISO-8859-1?Q?H=F6l=DF?= <manuel.hoelss@uni-giessen.de>
Subject: Re: Net::Telnet logging on problem.....
Message-Id: <3B4B3FA1.4040701@uni-giessen.de>
>>>
>>>print @test."\n";
>>>
>>>It should print loads of help commands... but instead prints 0... ive
>>>tried other commadns and instead prints just numbers....
>>>
You are using an array in scalar context. According to the perl-reference
this gives the number of Array-Elements. To print the Array Elements you
can use <join> :
print (join "\n",@test)."\n";
This will print your Array-Elements separated by Newlin Charakters.
------------------------------
Date: 10 Jul 2001 15:29:05 -0500
From: logan@cs.utexas.edu (Logan Shaw)
Subject: Re: Net::Telnet logging on problem.....
Message-Id: <9ifoih$46s$1@charity.cs.utexas.edu>
In article <3B4B3FA1.4040701@uni-giessen.de>,
Manuel =?ISO-8859-1?Q?H=F6l=DF?= <manuel.hoelss@uni-giessen.de> wrote:
>To print the Array Elements you
>can use <join> :
>
> print (join "\n",@test)."\n";
I find it cleaner to do this instead:
print map ("$_\n", @test);
Hope that helps.
- Logan
--
my your his her our their _its_
I'm you're he's she's we're they're _it's_
------------------------------
Date: Tue, 10 Jul 2001 21:31:27 GMT
From: Akira Yamanita <ayamanita.nospam@bigfoot.com>
Subject: Re: Net::Telnet logging on problem.....
Message-Id: <3B4B7429.369AC580@bigfoot.com>
Manuel Hölß wrote:
>
> >>>print @test."\n";
> >>>
> >>>It should print loads of help commands... but instead prints 0... ive
> >>>tried other commadns and instead prints just numbers....
> >>>
>
> You are using an array in scalar context. According to the perl-reference
> this gives the number of Array-Elements. To print the Array Elements you
> can use <join> :
>
> print (join "\n",@test)."\n";
>
> This will print your Array-Elements separated by Newlin Charakters.
You could just do:
print @test;
I think that what you propose would double-space the lines. Of
course, there's a bigger problem if the number of array elements
is 0.
------------------------------
Date: Tue, 10 Jul 2001 22:59:26 +0200
From: Manuel =?ISO-8859-1?Q?H=F6l=DF?= <manuel.hoelss@uni-giessen.de>
Subject: Re: Net::Telnet logging on problem.....
Message-Id: <3B4B6CAE.9050206@uni-giessen.de>
Akira Yamanita wrote:
> Manuel H=F6l=DF wrote:
>=20
>>>>>print @test."\n";
>>>>>
>>>>>It should print loads of help commands... but instead prints 0... iv=
e
>>>>>tried other commadns and instead prints just numbers....
>>>>>
>>>>>
>>You are using an array in scalar context. According to the perl-referen=
ce
>>this gives the number of Array-Elements. To print the Array Elements yo=
u
>>can use <join> :
>>
>> print (join "\n",@test)."\n";
>>
>>This will print your Array-Elements separated by Newlin Charakters.
>>
>=20
> You could just do:
>=20
> print @test;
I see. I thought the print-command has to be followed by a scalar.
But the problem was that the Array was concatenated with "\n" and here
it appeared in scalar context.
>=20
> I think that what you propose would double-space the lines.
If the array-elements already have their newline-charakters
------------------------------
Date: Mon, 09 Jul 2001 11:46:10 -0400
From: Benjamin Goldberg <goldbb2@earthlink.net>
Subject: Re: NNTP servers
Message-Id: <3B49D1C2.53FD326F@earthlink.net>
Aman wrote:
>
> hey guys what NNTP public servers you are using? I am badly wanting to
> know any servers that i can use without supplying password(public). I
> want to be able to:
>
> read the news groups (esp. comp.lang.perl.*)
> post messages also esp. comp.lang.perl.*
>
> i am sorry, but i am tired of using the google/deja style web groups
> that take hours to update or post. Please help me!!!!
For a list of links to pages with lists of free news servers, see:
http://www.free2seek.com/free/newserver.shtml
One of those pages with a list of free news servers is:
http://www.serverseekers.com/server.html
--
The longer a man is wrong, the surer he is that he's right.
------------------------------
Date: Mon, 9 Jul 2001 17:31:14 -0400
From: Marc Tardif <intmktg@gloria.cam.org>
Subject: no pragma in loop
Message-Id: <Pine.LNX.4.10.10107091726490.8575-100000@gloria.cam.org>
Is there a difference in using the no pragma in a loop as opposed to
setting it outside a loop? For instance, consider the following code:
use warnings;
use strict;
foreach my $var (@vars) {
my($key, $value) = split(/=/, $pair);
$value =~ s/foo//;
...
}
In the above, there will be a warning because $value might be undefined.
So can I set "no warnings qw(uninitialized);" inside the loop so that it
can be in the tightest scope while not suffering any performance loss from
having the code being run for every iteration? Or should it be set just
before the loop at the risk of having the pragma applied to code after the
loop (unless I reset the warning on uninitialized variables after, which
adds that much more clutter)?
Marc
------------------------------
Date: 09 Jul 2001 17:54:00 -0500
From: Tony Curtis <tony_curtis32@yahoo.com>
Subject: Re: no pragma in loop
Message-Id: <87ofqt4stz.fsf@limey.hpcc.uh.edu>
>> On Mon, 9 Jul 2001 17:31:14 -0400,
>> Marc Tardif <intmktg@gloria.cam.org> said:
> Is there a difference in using the no pragma in a loop
> as opposed to setting it outside a loop? For instance,
> consider the following code:
> use warnings; use strict;
> foreach my $var (@vars) { my($key, $value) = split(/=/,
> $pair); $value =~ s/foo//; ... }
> In the above, there will be a warning because $value
> might be undefined.
So test for it; don't try to tweak the compiler/run-time
system with hints, code it right in the first place:
if (defined $value) {
$value =~ .......
...
} else {
# handle not defined case (or ignore or whatever)
}
(This isn't a roll-your-own CGI question in disguise is it?
Go with CGI.pm instead.)
hth
t
--
Beep beep! Out of my way, I'm a motorist!
------------------------------
Date: Mon, 09 Jul 2001 23:33:29 GMT
From: tiltonj@erols.com (Jay Tilton)
Subject: Re: no pragma in loop
Message-Id: <3b4a3cc1.24968312@news.erols.com>
On Mon, 9 Jul 2001 17:31:14 -0400, Marc Tardif <intmktg@gloria.cam.org>
wrote:
>So can I set "no warnings qw(uninitialized);" inside the loop so that it
>can be in the tightest scope while not suffering any performance loss from
>having the code being run for every iteration? Or should it be set just
>before the loop at the risk of having the pragma applied to code after the
>loop (unless I reset the warning on uninitialized variables after, which
>adds that much more clutter)?
Restrict the pragma to an enclosing block. Minimal clutter with zero risk.
{ no warnings 'uninitialized';
foreach my $var (@vars) {
...#some operation that may detonate the warning
}
}
------------------------------
Date: 9 Jul 2001 18:39:26 -0700
From: hao7@yahoo.com (hao tran)
Subject: open file for update
Message-Id: <3913d5b4.0107091739.665ca237@posting.google.com>
Hi all,
I try to write script to open a file for update but somehow got stuck
when execute the script. I use perl ver 4.0 under Solaris platform.
Here is my script
#!/usr/local/bin/perl
open(FH, "+< test.txt");
$line = <FH>;
while($line ne "")
{
$line =~ s/hithere/goodbye/;
print FH $line;
$line = <FH>;
}
close(FH);
And test.txt has the line "hithere"
Thanks.
------------------------------
Date: 10 Jul 2001 08:34:22 GMT
From: ebohlman@omsdev.com (Eric Bohlman)
Subject: Re: open file for update
Message-Id: <9ieeme$rdg$2@bob.news.rcn.net>
hao tran <hao7@yahoo.com> wrote:
> I try to write script to open a file for update but somehow got stuck
> when execute the script. I use perl ver 4.0 under Solaris platform.
> Here is my script
> #!/usr/local/bin/perl
> open(FH, "+< test.txt");
> $line = <FH>;
> while($line ne "")
> {
> $line =~ s/hithere/goodbye/;
> print FH $line;
> $line = <FH>;
> }
> close(FH);
You can't do it that way; no commonly used OS supports an "update in
place" filesystem like the one you're assuming. See the section about
editing, updating, etc. in perlfaq5 for information about how to
accomplish your task.
------------------------------
Date: Tue, 10 Jul 2001 09:55:23 +0000
From: gnari <gnarinn@hotmail.com>
Subject: Re: open file for update
Message-Id: <994758923.272823812905699.gnarinn@hotmail.com>
In article <3913d5b4.0107091739.665ca237@posting.google.com>,
hao tran <hao7@yahoo.com> wrote:
>I try to write script to open a file for update but somehow got stuck
>when execute the script. I use perl ver 4.0 under Solaris platform.
>Here is my script
>
>#!/usr/local/bin/perl
>
>open(FH, "+< test.txt");
>$line = <FH>;
>while($line ne "")
>{
> $line =~ s/hithere/goodbye/;
> print FH $line;
> $line = <FH>;
>}
>close(FH);
>
In what way does this not do what you want?
Do you know about
perldoc -q "change one line"
If really want to do a random access update, then you forgot
to do a seek back to the position of the line between read and write.
gnari
------------------------------
Date: Tue, 10 Jul 2001 16:12:49 +0200
From: "Henrik Wendt" <hwendt@premitech.com>
Subject: Open Process with a space
Message-Id: <S3E27.1807$6x5.222430@news000.worldonline.dk>
Hi,
I'm trying to run the below program using ActiveState Perl on Windows NT
$prog="D:/temp/New Folder/cvs";
open CVSIN, "$prog status $ARGV[0]|" or die "Can't open $prog ";
while (<CVSIN>) {
my ($myrev) = /Working revision:\s(\d+\.\d+)/ ;
$rev = $myrev if $myrev ;
}
But gets the error:
The name specified is not recognized as an internal or external command,
operable program or batch file.
However if i move the programfile to
$prog="D:/temp/cvs";
There is no problem
Can anyone help me out here? I think it has to do with the space in the
filename.
BR
Henrik
------------------------------
Date: 10 Jul 2001 15:25:51 GMT
From: ebohlman@omsdev.com (Eric Bohlman)
Subject: Re: Open Process with a space
Message-Id: <9if6pv$302$1@bob.news.rcn.net>
Henrik Wendt <hwendt@premitech.com> wrote:
> I'm trying to run the below program using ActiveState Perl on Windows NT
> $prog="D:/temp/New Folder/cvs";
> open CVSIN, "$prog status $ARGV[0]|" or die "Can't open $prog ";
> while (<CVSIN>) {
> my ($myrev) = /Working revision:\s(\d+\.\d+)/ ;
> $rev = $myrev if $myrev ;
> }
> But gets the error:
> The name specified is not recognized as an internal or external command,
> operable program or batch file.
> However if i move the programfile to
> $prog="D:/temp/cvs";
> There is no problem
> Can anyone help me out here? I think it has to do with the space in the
> filename.
Yep. You have to work literal double quotes into the directory name in
order for the command processor to recognize it:
$prog='"D:/temp/New Folder/cvs"';
------------------------------
Date: Tue, 10 Jul 2001 09:51:39 +0200
From: "JJ" <jocke30_gbg@hotmail.com>
Subject: Pattern
Message-Id: <9iec51$i9d$1@vg170.it.volvo.se>
Hi,
I'm not that good at Perl and need some help.
I have the following code that converts datatypes from Oracle to mySQL but
the conversion for RAW doesn't work.
Can you see what's wrong?
############################################
# Rules are..:
# Datatype (ORA) Length Returns (MySQL)
# NUMBER Any NUMERIC
# DEC Any NUMERIC
# DECIMAL Any NUMERIC
# NUMERIC Any NUMERIC
# DOUBLE PRECISION Any NUMERIC
# FLOAT Any NUMERIC
# REAL Any NUMERIC
# SMALLINT Any SMALLINT
# VARCHAR <256 VARCHAR
# VARCHAR2 <256 VARCHAR
# CHAR <256 CHAR
# VARCHAR2 >255 TEXT
# VARCHAR >255 TEXT
# CHAR >255 TEXT
# LONG <256 VARCHAR
# LONGRAW <256 VARCHAR
# RAW <256 VARCHAR
# LONG >255 TEXT
# LONGRAW >255 TEXT
# RAW >255 TEXT
# DATE - DATETIME (Since DATE in oracle can include time
# information!)
#
# List return consists of:
# 1. Datatype name
# 2. Datalength
# 3. Complete datatype spec. to be used within CREATE TABLE statement
#############################################
return ('NUMERIC', $datalength, 'NUMERIC('.$datalength.')') if $datatype =~
/^DEC(IMAL)?|NUMERIC|NUMBER|DOUBLE PRECISION|FLOAT|REAL$/;
return ('INTEGER', 38, 'INTEGER(38)') if $datatype =~
/^(SMALL)?INT(EGER)?$/;
return ('VARCHAR', $datalength, 'VARCHAR('.$datalength.')') if $datatype =~
/^N?VARCHAR2?|LONG( RAW)?$/ && $datalength <= 255;
return ('TEXT', '', 'TEXT') if $datatype =~ /^N?(VAR)?CHAR2?|LONG( RAW)?$/
&& $datalength > 255;
return ('CHAR', $datalength, 'CHAR('.$datalength.')') if $datatype =~
/^N?CHAR$/ && $datalength <= 255;
return ('DATETIME', '', 'DATETIME') if $datatype eq 'DATE';
Best regards
Joacim Jarkeborn
------------------------------
Date: Tue, 10 Jul 2001 19:31:12 -0400
From: "Christopher J. Mackie" <cjmackie@princeton.edu>
Subject: Performance problem--memory bloat or crawl
Message-Id: <9ig380$abd$1@cnn.Princeton.EDU>
Apologies for the length of this--I want to be as clear as I can. I'm
needing to read a large binary file and do a simple manipulation of the
data. It's not working--either it crashes with an 'out of memory!' message,
or, if I choke down the memory limit enough, it works so slowly that the
complete run will take more than a year. I don't know whether the problem
is in my code, in the nature of the memory overhead for perl data
structures, or what.
I'm pretty stuck, and could use all the help anyone can offer.
TIA, --Chris
---------------------------------------------------
I'm running 64-bit enabled perl 5.6.1 on Irix 6.2 (SGI Origin 4000). The
input file is a matrix with known row and column lengths. The layout is:
0 1 2 3 4 5
6 7 8 9 10 11
12 13 14 15 16 17
...
stored in the file as: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 ...
(The foregoing is just for illustration--the actual dimensions are 297 rows
by 6.4M+ columns. Cell values are double-precision.) I need to change it
to:
12 6 0
13 7 1
14 8 2
15 9 3
16 10 4
17 11 5
(I also need to write it out as single-precision)
Problem is, the file is too large (14Gb) to read in all at once. I wrote a
subroutine to perform the manipulation in pieces, using the following logic:
1. calculate a subset size, based on available memory
2. read first row in one chunk (i.e., '0 1 2 3 4 5')
3. subset out a piece of that row (e.g., '0 1 2')
4. append the subset to a temporary buffer
5. repeat steps 1-4 for all rows in the file, to fill the buffer.
0 1 2 6 7 8 12 13 14
this is a 'vertical slice' of the original file:
0 1 2
6 7 8
12 13 14
6. use a nested loop to read the data out to another temp buffer in the
output format
12 6 0 13 7 1 14 8 2
or
12 6 0
12 7 1
14 8 2
7. repack the buffer as a float, and print it to the outfile
8. increment the substring offset to capture the next vertical slice
9. repeat steps 1-8 until the file is processed.
I started out using multi-d arrays, since that made the reformatting logic
the clearest. The script worked perfectly on smaller files (< 500Mb), but
it consumed as much as 8x more core than was strictly required for the two
buffers. On the bigger file, the memory overhead was even worse--in fact,
it seems to increase exponentially with file size in a way that I don't
fully understand. For the 14Gb file and two 100Mb buffers (a buffer size
that works fine on a 100Mb input file), the script kept steadily consuming
core until it crashed at 1.7Gb, without even completing the first loop
through the file--and did so at a pace that would have taken 388 days to
complete the entire subroutine.
I don't know why the script consumes nearly ten times the memory that it
needs just to hold the values in the temp buffers. I've tried several
alternatives, including taking bigger and smaller reads and writes,
pre-filling the buffers with zeros (in case it was the constant buffer
resizing that was causing speed problems), using strings for subsetting
rather than arrays, and so on. Nothing seems to work.
Below is the actual subroutine, in its most recent form (in a small
testframe). I would appreciate anything that anyone can suggest about why
it's leaking memory so severely on big files. Alternative solutions would
also be very welcome. (I've looked at things like PDL and Matrix::Real, but
I'm not sure I understand them well enough to add that complication to the
situation.)
=============================
#!/usr/bin/perl
# the testframe--a 100_000 value dummy matrix, with a 20_000 byte buffer
limit
open (OUTFILE, ">test.in") or die "Which?\n";
binmode(OUTFILE); # not necessary on my machine, but someday I will need to
use this on Windoze
for (0..99999) {
$string = pack("d", $_);
print OUTFILE $string;
}
close OUTFILE;
rotateMatrix('test.in','test.out', 500, 200, 100000, 20000, 8);
sub rotateBuffer {
my($infile, $outfile, $nVectors, $nFactors, $nItems, $MEM_LIMIT,
$bytesize, $type)=@_;
my ($NumLoops, $Residuum, $BufferVectors, $BufferMem, $loopnum, $offset,
@inbuffer, @cells, $mem, $i, $j, $BUFFER, $OUTBUFFER);
my $bytesize = 8; # sizeof(double)
{ # a little integer math, to generate buffer sizes
use integer;
$BufferVectors=$MEM_LIMIT/($nFactors*$bytesize);
die "MemLimit too small for buffering\n" if ($BufferVectors < 1 );
$NumLoops=$nVectors/$BufferVectors;
$Residuum=$nVectors-($BufferVectors*$NumLoops);
$BufferMem = $BufferVectors * $nFactors*$bytesize;
$NumLoops++ if ($Residuum);
print "\tBV: $BufferVectors nV: $nVectors nL: $NumLoops nR:
$Residuum MemLimit: $BufferMem\n";
}
$offset=0;
for($loopnum=0; $loopnum < $NumLoops; $loopnum++) {
# Unless $nVectors % $BufferVectors==0, last loop will have fewer
BufferVectors (i.e., Residuum)
if ($loopnum==($NumLoops-1) && $Residuum >0)
{
$BufferVectors=$Residuum;
print "Last loop: BufferVectors=$BufferVectors\n";
}
print "Loop: $loopnum beg: $offset end: ",
($offset+$BufferVectors), "...";
# Read the entire file, 'line' by 'line', then slice out only the
portion of the line we want in this pass.
# a 'line' is all vectors for one factor
open(INFILE, "<$infile") or die "Can't open $infile\n";
binmode(INFILE);
undef (@inbuffer);
while (($nVectors*$bytesize)==sysread(INFILE, $BUFFER, ($nVectors
* $bytesize),0))
{
@cells=unpack("d*", $BUFFER);
# here's the slice
push (@inbuffer,
@cells[$offset..($offset+$BufferVectors-1)]);
}# end WHILE read
# Once it's all read, we write
print "Outputting...";
open (OUTFILE, ">>$outfile") or die "Can't open $outfile: $!";
binmode(OUTFILE);
undef $OUTBUFFER;
my $startpoint = (($nFactors-1) * $nVectors);
for ($i=0; $i<$nFactors-1; $i++)
{
for ($j=$startpoint; $j >= 0; $j -= $nVectors)
{
$OUTBUFFER .= pack("f", $inbuffer[$i+$j]);
}
}
print OUTFILE $OUTBUFFER;
print "Done.\n";
# Update slice location for next loop
$offset += $BufferVectors ;
} # end FOR loopnum
close(INFILE);
close(OUTFILE);
} # end SUB rotateBuffer
------------------------------
Date: 10 Jul 2001 19:53:38 -0500
From: logan@cs.utexas.edu (Logan Shaw)
Subject: Re: Performance problem--memory bloat or crawl
Message-Id: <9ig82i$489$1@charity.cs.utexas.edu>
In article <9ig380$abd$1@cnn.Princeton.EDU>,
Christopher J. Mackie <cjmackie@princeton.edu> wrote:
>needing to read a large binary file and do a simple manipulation of the
>data. It's not working--either it crashes with an 'out of memory!' message,
:
:
>---------------------------------------------------
>I'm running 64-bit enabled perl 5.6.1 on Irix 6.2 (SGI Origin 4000). The
>input file is a matrix with known row and column lengths. The layout is:
>
>0 1 2 3 4 5
>6 7 8 9 10 11
>12 13 14 15 16 17
>...
>
>stored in the file as: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 ...
>
>(The foregoing is just for illustration--the actual dimensions are 297 rows
>by 6.4M+ columns. Cell values are double-precision.) I need to change it
>to:
>
>12 6 0
>13 7 1
>14 8 2
>15 9 3
>16 10 4
>17 11 5
>(I also need to write it out as single-precision)
>
>Problem is, the file is too large (14Gb) to read in all at once. I wrote a
>subroutine to perform the manipulation in pieces, using the following logic:
It seems to me that you can accomplish the task with simpler logic, and
I see no reason why this wouldn't be efficient with memory and with
disk seeks.
Step 1: Pick a number, say 10000. Call it $n. Make sure it doesn't
exceed the number of columns you have left to process.
Step 2: For each row, read $n columns of data. Store it in an array
of arrays. The inner arrays will be (a slice of) a given
row's data; the outer one will be an array of row arrays.
This will involve seeking to the appropriate place within a
row once per row, and then one big sequential read.
Step 3: Take your array of vertical row slices and write everything
in it to your output file. If @v_slice is the array of arrays
of row data, the Perl code might look like this:
@v_slice = reverse @v_slice;
# as long as we have data left in the first row...
while (@{ $v_slice[0] } > 0)
{
foreach my $row (@v_slice)
{
print OUTPUT shift $row; # (output style is simplified)
}
}
This prints one column (from the point of view of the input)
every time the whole inner loop runs, and it eats up the data
as it goes since it uses "shift". And don't worry, shift is a
constant time operation -- it doesn't actually shift every
element internally. The outer loop exits when the first row
doesn't have any data left, which should mean the other rows
don't either.
Step 4: If not finished, go back to step #1.
This method takes advantage of the fact that you have a relatively
small number of rows.
- Logan
--
my your his her our their _its_
I'm you're he's she's we're they're _it's_
------------------------------
Date: 10 Jul 2001 20:00:22 -0500
From: logan@cs.utexas.edu (Logan Shaw)
Subject: Re: Performance problem--memory bloat or crawl
Message-Id: <9ig8f6$529$1@charity.cs.utexas.edu>
In article <9ig82i$489$1@charity.cs.utexas.edu>,
Logan Shaw <logan@cs.utexas.edu> wrote:
>It seems to me that you can accomplish the task with simpler logic, and
>I see no reason why this wouldn't be efficient with memory and with
>disk seeks.
Heh, I just realized my method is very, very similar to yours. I
promise I read the description. I was just confused by it somehow.
I'll post another follow-up to your original post in a minute with
comments about your specific code.
- Logan
--
my your his her our their _its_
I'm you're he's she's we're they're _it's_
------------------------------
Date: 10 Jul 2001 20:21:42 -0500
From: logan@cs.utexas.edu (Logan Shaw)
Subject: Re: Performance problem--memory bloat or crawl
Message-Id: <9ig9n6$6dg$1@charity.cs.utexas.edu>
In article <9ig380$abd$1@cnn.Princeton.EDU>,
Christopher J. Mackie <cjmackie@princeton.edu> wrote:
> open(INFILE, "<$infile") or die "Can't open $infile\n";
You don't need to open/close the input file every time through the
loop. That's not your biggest problem, but it's still inefficient.
> binmode(INFILE);
> undef (@inbuffer);
> while (($nVectors*$bytesize)==sysread(INFILE, $BUFFER, ($nVectors
>* $bytesize),0))
> {
> @cells=unpack("d*", $BUFFER);
> # here's the slice
> push (@inbuffer,
>@cells[$offset..($offset+$BufferVectors-1)]);
Three areas of significant inefficiency here:
1. You're reading the whole row when you only want a small part of
it. It'd be much better to use seek() to go to the exact location
you want, then sysread() exactly the amount you want.
2. Your ".." operator probably creates a temporary list of indices of
a size of about $BufferVectors. Another reason why slicing isn't
the most efficient.
3. You are appending onto the array @inbuffer. Perl allocates a
certain amount of space for an array, and if you stuff more elments
into the array than it can presently hold, behind the scenes Perl
asks the memory manager for more storage than what it previously
had. Then it copies the existing contents, changes a pointer (and
presumably garbage collects the old area), and proceeds where it
left off.
So what's wrong with this? If it first allocates 64 bytes and then
has to up that to (say) 128 bytes, it probably can't use any of the
original 64 bytes because the next 64 bytes right after it have
probably been allocated to something else already. So it picks 128
bytes from somewhere else.
Then it needs 256 bytes, and can't use the 64 or the 128, so it
grabs 256 bytes from somewhere else. And on and on. Eventually,
it's using twice the memory it should be using. This isn't
terrible, but it's not a good thing either.
> for ($j=$startpoint; $j >= 0; $j -= $nVectors)
> {
> $OUTBUFFER .= pack("f", $inbuffer[$i+$j]);
> }
Again, appending is probably not lending itself to good memory
management. I'm not sure whether scalars get reallocated every time
you append even a few bytes to them like that or not, but if they do,
the amount of memory you'd be using could be about equal to the square
of the largest size of the output buffer.
Anyway, you don't even need to buffer your output; when you use Perl's
"print", it does some output buffering for you.
Hope that helps.
- Logan
--
my your his her our their _its_
I'm you're he's she's we're they're _it's_
------------------------------
Date: Wed, 11 Jul 2001 04:01:52 GMT
From: "John W. Krahn" <krahnj@acm.org>
Subject: Re: Performance problem--memory bloat or crawl
Message-Id: <3B4BCFAB.7477E33B@acm.org>
"Christopher J. Mackie" wrote:
>
> Apologies for the length of this--I want to be as clear as I can. I'm
> needing to read a large binary file and do a simple manipulation of the
> data. It's not working--either it crashes with an 'out of memory!' message,
> or, if I choke down the memory limit enough, it works so slowly that the
> complete run will take more than a year. I don't know whether the problem
> is in my code, in the nature of the memory overhead for perl data
> structures, or what.
>
> I'm pretty stuck, and could use all the help anyone can offer.
> TIA, --Chris
I tried to run your example but was unsuccessful:
$ Mackie.pl
BV: 12 nV: 500 nL: 42 nR: 8 MemLimit: 19200
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 61.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
Use of uninitialized value in pack at /home/john/bin/Mackie.pl line 62.
...
> undef $OUTBUFFER;
> my $startpoint = (($nFactors-1) * $nVectors);
> for ($i=0; $i<$nFactors-1; $i++) { # line 61
> for ($j=$startpoint; $j >= 0; $j -= $nVectors) { # line 62
> $OUTBUFFER .= pack("f", $inbuffer[$i+$j]);
> }
> }
> print OUTFILE $OUTBUFFER;
> print "Done.\n";
John
--
use Perl;
program
fulfillment
------------------------------
Date: Tue, 10 Jul 2001 00:05:48 -0700
From: "Dave Roth" <rothd@roth.net>
Subject: Re: Perl and the NT Event Log.
Message-Id: <FOx27.26681$rh.513884@news6.giganews.com>
It should already be installed if you have ActivePerl. Otherwise try auto
installing it:
ppm install win32-eventlog
For comprehensive explanations on how to use the extension (along with
sample scripts) take a look at my first book
(http://www.roth.net/books/extensions). The second edition is due out in a
month or two.
My second book also has a chapter on monitoring including scripts that use
the event log: http://www.roth.net/books/handbook/
dave
<Nick Porter> wrote in message
news:mhaekt8tvd77h1jci6e790uspd4ucjk1d7@4ax.com...
> On 06 Jul 2001 12:54:21 +0100, nobull@mail.com wrote:
>
> >Nick Porter <nickp@flyingpig.com> writes:
> >
> >> Can anyone explain to this newbie how to interrogate the NT event log.
> >
> >This isn't really a Perl question. Win32 provides an API to
> >interrogate the event log. The Win32::EventLog module provides a Perl
> >mapping to this API. The semantics of the API itself are largely
> >independant of language and need to be understood (or guessed) before
> >the Perl mapping of the API makes a lot of sense.
>
> [NP] Thanks for that. I'll check out the Win32 module.
>
> >> I've had a search at CPAN and other places for a module but have been
> >> unsuccessfull.
> >
> >Do you mean you could not find Win32::EventLog or do mean something
> >else by "unsuccessfull"?
>
> [NP] I searched cpan with the command CPAN> search "event log" and
> CPAN> search "windows nt". I also went to;
> http://search.cpan.org/Development_Support and;
> http://search.cpan.org/Catalog/Operating_System_Interfaces/ but
> couldn't find any module descriptions that referred to the NT event
> log. So by 'unsuccessful' I meant that I was unable to find what I was
> looking for. Was it really that unclear? I am unaware of any other
> methods for searching CPAN than the ones I used above. I am a new at
> this so perhaps you could enlighten me on methods which are more
> discriminating, other than posting to a newsgroup?
>
>
>
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 1277
***************************************
