[19117] in Perl-Users-Digest
Perl-Users Digest, Issue: 1312 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Mon Jul 16 14:06:08 2001
Date: Mon, 16 Jul 2001 11:05:11 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Message-Id: <995306711-v10-i1312@ruby.oce.orst.edu>
Content-Type: text
Perl-Users Digest Mon, 16 Jul 2001 Volume: 10 Number: 1312
Today's topics:
Re: Advice REQ for newbie (Tad McClellan)
Re: Advice REQ for newbie nobull@mail.com
arrays, textfiles & creating directories (Casey Chambliss)
Re: arrays, textfiles & creating directories <mbudash@sonic.net>
ARRAYS: find common values <whaidri@ford.com>
Re: ARRAYS: find common values <jmccrea@guideguide.com>
Re: ARRAYS: find common values <jmccrea@guideguide.com>
Re: ARRAYS: find common values <jeff@vpservices.com>
Re: ARRAYS: find common values <mbudash@sonic.net>
Re: ARRAYS: find common values <mjcarman@home.com>
changing password in password file <ub98aa@brocku.ca>
Re: Check for Dups! (Eric Bohlman)
Re: Checking for changed or missing links in perl (dave)
Re: CPAN NNTP Module recommendations? <gerard@NOSPAMlanois.com>
Graph.pm example <james@alphyra.ie>
Re: Hard DBI question nobull@mail.com
Re: Hard DBI question <cpryce@pryce.net>
Re: Performance problem--memory bloat or crawl <cjmackie@princeton.edu>
Re: Performance problem--memory bloat or crawl (Michel Dalle)
Re: problem with @INC (Kenny)
Re: problem with @INC <mjcarman@home.com>
Re: Problems moving from an Apache server to a Windows <lmoran@wtsg.com>
Question about indenting in emacs cperl-mode (Don Eisele)
Re: sortlen -- filter to sort text by line length <ren@tivoli.com>
Re: sortlen -- filter to sort text by line length <uri@sysarch.com>
The brackets and arrows of outrageous fortune... (compl <cjmackie@princeton.edu>
versions (Matt)
Re: versions <mbudash@sonic.net>
Re: versions <mjcarman@home.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Mon, 16 Jul 2001 10:33:50 -0400
From: tadmc@augustmail.com (Tad McClellan)
Subject: Re: Advice REQ for newbie
Message-Id: <slrn9l5uqe.5ld.tadmc@tadmc26.august.net>
[ Please do not send stealth Cc's. It makes people angry. ]
Pete <bloke6789@hotmail.com> wrote:
>> Yes. I advise that you post a short and complete program that we
>> can run that illustrates your problem when asking questions.
>
>Tad,
If you are going to comment on one of my postings, then your
References: header should have the message ID of my message.
It does not. Please don't break stuff like that.
Please followup correctly or I may not see your responses.
Your article hit 3 of my scorefile rules, so your posts are
mostly invisible to me.
Followups with a correct References header are scored high
enough to override the normal negative score that your
articles get.
>Maybe I should have stuck at "hello world". That was my complete script!
>
>> How do you know that the substitution does not take place?
>
>Because my file remains unchanged.
Then you do _not_ know that the substitution failed, since your
program does not make any output at all.
>> I do not see your second piece of code making any output at all.
>
>It didn't, I kinda thought it would just change the existing file.
^^^^^^
You kinda thought incorrectly then.
>> Here is a short and complete program that you can run:
[snip code]
>Thanks, I'll try it and I'll learn from it.
You could learn even more by checking the Perl FAQ *before*
posting to the Perl newsgroup:
perldoc -q change
"How do I change one line in a file/delete a line in a
file/insert a line in the middle of a file/append to the
beginning of a file?"
See also the -i command line switch if you want to do in-place edits:
perldoc perlrun
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: 16 Jul 2001 18:10:21 +0100
From: nobull@mail.com
Subject: Re: Advice REQ for newbie
Message-Id: <u9u20cn6ki.fsf@wcl-l.bham.ac.uk>
"Pete" <bloke6789@hotmail.com> writes:
> Subject: Advice REQ for newbie
> Please can you advise me?
http://www.perl.com/CPAN/authors/Dean_Roehrich/subjects.post
HTH
--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\
------------------------------
Date: 16 Jul 2001 10:33:02 -0700
From: cchambli@lycos.com (Casey Chambliss)
Subject: arrays, textfiles & creating directories
Message-Id: <a15e3f18.0107160933.1fe87f73@posting.google.com>
Ok, I have a text file (projlist.txt) with each line that has the name
of a project (proj1, proj2, proj3, ect.)
I have put the text file into an array. How can I then use that array
to create a directory for each project? Let me show you what I've
tried... but I don't know why it isn't working...
##############
Begin Code ###
##############
open(LISTTXT, 'C:\projlist.txt');
@list = <LISTTXT>;
for( $i=0; $i<=5; $i +=1)
{
mkdir (join("C:\\PROJECTS\\", $list[$i]));
}
close(LISTTXT);
##############
End Code #####
##############
Why dosen't this work? The syntax for the directories is
acceptable... no problems there. If I do this with a manually created
array by:
@list = ('proj1', 'proj2', 'proj3');
it will work just fine... but it won't when reading the array from a
txt file.
What do I need to do?
------------------------------
Date: Mon, 16 Jul 2001 17:44:17 GMT
From: Michael Budash <mbudash@sonic.net>
Subject: Re: arrays, textfiles & creating directories
Message-Id: <mbudash-239EA9.10441716072001@news.sonic.net>
In article <a15e3f18.0107160933.1fe87f73@posting.google.com>,
cchambli@lycos.com (Casey Chambliss) wrote:
> Ok, I have a text file (projlist.txt) with each line that has the name
> of a project (proj1, proj2, proj3, ect.)
>
> I have put the text file into an array. How can I then use that array
> to create a directory for each project? Let me show you what I've
> tried... but I don't know why it isn't working...
>
> ##############
> Begin Code ###
> ##############
>
> open(LISTTXT, 'C:\projlist.txt');
> @list = <LISTTXT>;
>
> for( $i=0; $i<=5; $i +=1)
> {
> mkdir (join("C:\\PROJECTS\\", $list[$i]));
> }
>
> close(LISTTXT);
> ##############
> End Code #####
> ##############
>
> Why dosen't this work? The syntax for the directories is
> acceptable... no problems there. If I do this with a manually created
> array by:
>
> @list = ('proj1', 'proj2', 'proj3');
>
> it will work just fine... but it won't when reading the array from a
> txt file.
>
> What do I need to do?
what does "doesn't work" mean? are there errors? or do the directories
simply not created?
1) @list as taken from the file will contain line terminators.
2) the system will tell you why a mkdir failed.
3) it looks like you're expecting join to do something it won't.
try this before the loop:
chomp (@list);
and this in the loop:
$newdir = 'C:\PROJECTS\' . $list[$i];
mkdir ($newdir) or die ("Cannot create $newdir: $!");
hth-
--
Michael Budash ~~~~~~~~~~ mbudash@sonic.net
------------------------------
Date: Mon, 16 Jul 2001 10:18:04 -0400
From: Wali Haidri <whaidri@ford.com>
Subject: ARRAYS: find common values
Message-Id: <3B52F79C.360C2B2B@ford.com>
I want to out put the common values in the two arrays below. Following
code does work but it is very slow. I hope somebody can suggest a
better way.
sub get_usersByToken3{
# my (@users) defined somewhere else and has a list of users.
# my (users2) " " " " "
my (@finalusers);
my ($tval, $val1, $val2);
# FIRST SORT BOTH ARRAYS
@users = sort { uc($a) cmp uc($b) } @users;
@users2 = sort { uc($a) cmp uc($b) } @users2;
# NOW FOR EACH ELEMENT IN LIST 1, TRY TO FIND IT IN LIST 2. IF FOUND
# PUSH IT TO AN OUTPUT ARRAY AND EXIT OUT OF THIS LOOK BACK TO THE FIRST
# foreach LOOP.
LINE:
foreach $val1 (@users){
foreach $val2 (@users2){
if ( $val2 eq uc($val1) ) {
push(@finalusers, uc($val1));
next LINE;
}
}
} # end of foreach
return (@finalusers);
}
Please cc to my email address if possible. Thanks.
Wali Haidri
whaidri@ford.com
------------------------------
Date: Mon, 16 Jul 2001 17:27:38 +0100
From: Jasper McCrea <jmccrea@guideguide.com>
To: Wali Haidri <whaidri@ford.com>
Subject: Re: ARRAYS: find common values
Message-Id: <3B5315FA.44B1AC12@guideguide.com>
Wali Haidri wrote:
>
> I want to out put the common values in the two arrays below. Following
> code does work but it is very slow. I hope somebody can suggest a
> better way.
isn't this an FAQ. ie use a hash.
# haven't tested this, and am assuming (probably correctly :) )
# that uc works on $_
my %users2 = map { uc => 1 } @users2;
foreach my $user (sort { $a cmp $b} map { uc } @users) {
push @finalusers, $user if $users2{$user};
}
return @finalusers;
probably you'll find that this isn't much faster.
But it isn't what perl was designed to do, I guess.
I also not that you're using next LINE when the code doesn't do anything
afterward, anyway. There's only a necessity to put a next in if there's
more code you want to avoid.
I have also uc'd all values straight away. You waste time by testing
against a uced $val1 (this ucing being done every time), and then if you
actually want to use it, you uc it anyway.
You're also declaring variables that only need a small scope with a much
larger scope ($val1, $val2).
Jasper
>
> sub get_usersByToken3{
>
> # my (@users) defined somewhere else and has a list of users.
> # my (users2) " " " " "
> my (@finalusers);
> my ($tval, $val1, $val2);
>
> # FIRST SORT BOTH ARRAYS
>
> @users = sort { uc($a) cmp uc($b) } @users;
> @users2 = sort { uc($a) cmp uc($b) } @users2;
>
> # NOW FOR EACH ELEMENT IN LIST 1, TRY TO FIND IT IN LIST 2. IF FOUND
> # PUSH IT TO AN OUTPUT ARRAY AND EXIT OUT OF THIS LOOK BACK TO THE FIRST
> # foreach LOOP.
>
> LINE:
> foreach $val1 (@users){
> foreach $val2 (@users2){
> if ( $val2 eq uc($val1) ) {
> push(@finalusers, uc($val1));
> next LINE;
> }
> }
> } # end of foreach
> return (@finalusers);
> }
>
> Please cc to my email address if possible. Thanks.
>
> Wali Haidri
> whaidri@ford.com
--
split//,'019617511192'.
'17011111610114101114'.
'21011141011840799901'.
'17101174';
foreach(0..
$#_){$_[$_
++]^=$_[$_
--]^=$_[$_
]^=$_[++ $_]if!($_%
2)}$g.=$_ ,chr($g)=~
/(\w)/&&($o.=$1and
$g='')foreach@_;
print"$o\n"
------------------------------
Date: Mon, 16 Jul 2001 17:33:24 +0100
From: Jasper McCrea <jmccrea@guideguide.com>
Subject: Re: ARRAYS: find common values
Message-Id: <3B531754.7943006@guideguide.com>
Jasper McCrea wrote:
>
> Wali Haidri wrote:
> >
> > I want to out put the common values in the two arrays below. Following
> > code does work but it is very slow. I hope somebody can suggest a
> > better way.
>
> isn't this an FAQ. ie use a hash.
>
> # haven't tested this, and am assuming (probably correctly :) )
> # that uc works on $_
> my %users2 = map { uc => 1 } @users2;
>
> foreach my $user (sort { $a cmp $b} map { uc } @users) {
> push @finalusers, $user if $users2{$user};
> }
>
> return @finalusers;
>
Don't know what I was thinking.
my %users2 = map { uc => 1 } @users2;
my @final = grep { $users2{$_} } sort { $a cmp $b} map { uc } @users;
return @final;
:)
Jasp
not sure if you'll want to look up map and grep in camel, Wali, but I'm
guessing so.
--
split//,'019617511192'.
'17011111610114101114'.
'21011141011840799901'.
'17101174';
foreach(0..
$#_){$_[$_
++]^=$_[$_
--]^=$_[$_
]^=$_[++ $_]if!($_%
2)}$g.=$_ ,chr($g)=~
/(\w)/&&($o.=$1and
$g='')foreach@_;
print"$o\n"
------------------------------
Date: Mon, 16 Jul 2001 09:46:15 -0700
From: Jeff Zucker <jeff@vpservices.com>
Subject: Re: ARRAYS: find common values
Message-Id: <3B531A57.2F3ADEBF@vpservices.com>
Wali Haidri wrote:
>
> I want to out put the common values in the two arrays below.
If each of the two arrays is itself unique (i.e. there are no duplicates
within the array itself), this will work:
my %is_member;
my @array1 = ('bird','DOG','cat','worm');
my @array2 = ('bird','dog','mouse','fish');
my @common = grep($is_member{lc $_}++, @array1,@array2);
print "@common"; # prints "bird dog"
If there are duplicates within one or both of the arrays, this will
work:
my(%is_member_1,%is_member_2);
my @array1 =('bird','DOG','cat','worm','worm');
my @array2 =('bird','dog','mouse','fish','fish');
@array1 = grep !$is_member_1{lc $_}++, @array1;
my @common = grep( ($is_member_1{lc $_} and !$is_member_2{lc $_}++),
@array2 );
print "@common"; # prints "bird dog"
If you want 'DOG' and 'dog' treated as separate entries, change {lc $_}
to just {$_}.
-- Jeff
------------------------------
Date: Mon, 16 Jul 2001 16:50:00 GMT
From: Michael Budash <mbudash@sonic.net>
Subject: Re: ARRAYS: find common values
Message-Id: <mbudash-012B6B.09500016072001@news.sonic.net>
In article <3B52F79C.360C2B2B@ford.com>, Wali Haidri <whaidri@ford.com>
wrote:
> I want to out put the common values in the two arrays below. Following
> code does work but it is very slow. I hope somebody can suggest a
> better way.
>
> sub get_usersByToken3{
>
> # my (@users) defined somewhere else and has a list of users.
> # my (users2) " " " " "
> my (@finalusers);
> my ($tval, $val1, $val2);
>
> # FIRST SORT BOTH ARRAYS
>
> @users = sort { uc($a) cmp uc($b) } @users;
> @users2 = sort { uc($a) cmp uc($b) } @users2;
>
> # NOW FOR EACH ELEMENT IN LIST 1, TRY TO FIND IT IN LIST 2. IF FOUND
> # PUSH IT TO AN OUTPUT ARRAY AND EXIT OUT OF THIS LOOK BACK TO THE FIRST
> # foreach LOOP.
>
> LINE:
> foreach $val1 (@users){
> foreach $val2 (@users2){
> if ( $val2 eq uc($val1) ) {
> push(@finalusers, uc($val1));
> next LINE;
> }
> }
> } # end of foreach
> return (@finalusers);
> }
here's the basic idea:
@hash1{@array1} = (1) x scalar(@array1);
@hash2{@array2} = (1) x scalar(@array2);
foreach (keys %hash1) {
push (@finalusers, $_) if ($hash2{$_});
}
the '@hash{@array}' idiom is called a hash slice. go to google.com and
type in 'hash slice' to see some articles on it... very useful...
hth-
--
Michael Budash ~~~~~~~~~~ mbudash@sonic.net
------------------------------
Date: Mon, 16 Jul 2001 11:28:07 -0500
From: Michael Carman <mjcarman@home.com>
Subject: Re: ARRAYS: find common values
Message-Id: <3B531617.10C3BCE1@home.com>
[Posted and courtesy copy mailed.]
Wali Haidri wrote:
>
> I want to out put the common values in the two arrays below.
The normal way of doing this is to use a hash, as hashes are better
suited to set-type operations. See perlfaq4: "How do I compute the
difference of two arrays? How do I compute the intersection of two
arrays?"
> Following code does work but it is very slow.
Yep -- you're doing all your searching in a nested loop.
> I hope somebody can suggest a better way.
Here's a nice generic sub for computing intersections and differences:
#!/usr/local/bin/perl5 -w
use strict;
my @A = qw(a e i o u q r s t v w);
my @B = qw(a b c d e f g h i j k l m n o p);
my ($ao, $bo, $ab) = Compute_Sets(\@A, \@B);
print "BOTH: @$ab\n";
print " A: @$ao\n";
print " B: @$bo\n";
#-----------------------------------------------------------------------
# Subroutine: Compute_Sets(\@,\@)
# Purpose : Determine the difference and intersection of two arrays.
# Notes : Takes two array refs as arguments, returns three
# anonymous array refs.
#-----------------------------------------------------------------------
sub Compute_Sets {
my $a_ref = shift;
my $b_ref = shift;
my (%a_sigs, %b_sigs, %uniq_sigs);
my (@a_only, @b_only, @both);
# Use a hash slice to autovivify keys. Values are undefined, so we
# will check for existance (rather than definedness or truth) later.
@a_sigs{@$a_ref} = ();
@b_sigs{@$b_ref} = ();
foreach (keys %a_sigs, keys %b_sigs) {
$uniq_sigs{$_}++;
}
foreach (sort keys %uniq_sigs) {
if (exists $a_sigs{$_} && exists $b_sigs{$_}) {
push(@both, $_);
}
elsif (exists $a_sigs{$_}) {
push(@a_only, $_);
}
else { # (exists $b_sigs{$_})
push(@b_only, $_);
}
}
return(\@a_only, \@b_only, \@both);
}
__END__
BOTH: a e i o
A: q r s t u v w
B: b c d f g h j k l m n p
If you're worried about case differences (as your code indicated) change
these lines:
@a_sigs{@$a_ref} = ();
@b_sigs{@$b_ref} = ();
to this:
@a_sigs{map {uc($_)} @$a_ref} = ();
@b_sigs{map {uc($_)} @$b_ref} = ();
and your results will converted to uppercase as in your code.
-mjc
------------------------------
Date: Mon, 16 Jul 2001 12:37:07 -0400
From: Umair Tariq Bajwa <ub98aa@brocku.ca>
Subject: changing password in password file
Message-Id: <3B531833.9DE26C5B@brocku.ca>
Hi,
I am writing a script so users can change their passwords in password
file using web GUI. In my post section i am executing command like that
...............
system "htpasswd -m filename newuser";
What should I do after that so it just simply update the file. Any idea?
Thanks in advance.
Umair
------------------------------
Date: 16 Jul 2001 17:10:36 GMT
From: ebohlman@omsdev.com (Eric Bohlman)
Subject: Re: Check for Dups!
Message-Id: <9iv76c$bvr$3@bob.news.rcn.net>
Anno Siegel <anno4000@lublin.zrz.tu-berlin.de> wrote:
> According to millside <millettNOSPAM@lblueyonder.co.uk>:
>> > > [ snip }
>> >
>> > use a tied hash instead of a flat file for your e-mail addresses.
>> >
>> > perldoc -f tie
>> > perldoc DB_File
>> > perldoc AnyDBM_File
>> >
>> >
>> I need to use a flat file as it's shared with other programs.
> That's not a compelling reason, you can lock a database as well.
I assume he meant that the other programs were written to read a flat file
and that he didn't want to have to, or couldn't, rewrite them to use a
tied hash instead (maybe they're not even his programs).
------------------------------
Date: 16 Jul 2001 10:27:07 -0700
From: usted@cyberspace.org (dave)
Subject: Re: Checking for changed or missing links in perl
Message-Id: <e2c00ae.0107160927.2354c378@posting.google.com>
Tore Aursand <tore@extend.no> wrote in message news:<MPG.15bd1c4f845ea807989697@news.online.no>...
> In article <ffd662ea.0107160424.3ea21ae8@posting.google.com>,
> ggrothendieck@volcanomail.com says...
> > I would like to check all the links on a web site
> > to see if they point to missing or *changed* web
> > pages.
Go to:
http://examples.oreilly.com/cookbook/
and download the perl code examples. Recipe 20.3 shows how to get the
links out of the webpage, and then 20.7 and 20.8 show how to tell if
they have been changed very recently or if they are bad (stale,
because server may be down temporarily when script runs)
Your question almost seems to imply that you want to know if the
webpage has *changed* in some way so that you no longer want to link
to it. This may require human intelligence. The example scripts just
use LWP -> HTTP -> HEAD to check when the page was changed.
dave
------------------------------
Date: 16 Jul 2001 10:02:23 -0700
From: Gerard Lanois <gerard@NOSPAMlanois.com>
Subject: Re: CPAN NNTP Module recommendations?
Message-Id: <uae247qow.fsf@NOSPAMlanois.com>
hkmp5@spray.se (Alex SC) writes:
> I'm in the process of writing a perl program
> that accesses our news server. Having taken
> a look at CPAN I see there are a lot of different
> modules available for NNTP access. Has anybody
> tried them or does somebody have something
> good to say about any particular choise?
>
> TIA,
> asc
I use Net::NNTP from the libnet distribution.
http://search.cpan.org/search?dist=libnet
It works well, and is nicely RFC-compliant.
If you do end up using Net::NNTP, watch out for this
little gotcha, lurking down in Net::Cmd:
http://www.geocities.com/gerardlanois/perl/20010126_000000.html
libnet is now part of the core perl distribution, so
if you have a fairly recent perl, you might already
have libnet.
-Gerard
http://www.geocities.com/gerardlanois/perl/
------------------------------
Date: Mon, 16 Jul 2001 17:09:22 +0100
From: James Heneghan <james@alphyra.ie>
Subject: Graph.pm example
Message-Id: <3B5311B2.2130057A@alphyra.ie>
Hi All,
>
> I downloaded Graph.pm and tried to run the example.
>
> Here was my output ...
>
> ./example.pl
> Setting title to Daily Web Site Hits
> Setting subtitle to Content pages only
> Setting keys_label to Day of the Week
> Setting values_label to Hits
> Setting value_min to 0
> Setting value_max to 35000
> Setting value_labels to 10000,20000,30000
> Setting background_image to bg.gif
> Minimum Data value: 10712
> Maximum Data value: 29976
> Minimum Y value set to 0
> Maximum Y value set to 35000
> Image Created: 928 x 367
> Colors Allocated
> Can't locate object method "newFromGif" via package "GD::Image" at
> /usr/lib/perl5/site_perl/5.6.0/i386-linux/Graph.pm line 761.
>
> I've tried to locate this method too but its not there!
>
> Anyone know the solution?
>
> Thanks,
>
> James
------------------------------
Date: 16 Jul 2001 18:11:26 +0100
From: nobull@mail.com
Subject: Re: Hard DBI question
Message-Id: <u9snfwn6ip.fsf@wcl-l.bham.ac.uk>
isterin@hotmail.com (isterin) writes:
> Why not post on the dbi-user group list.
I can suggest one possible "why". Because it's even more full of
off-topic noise than the newsgroups!
I tried subscribing for a week but I found the s:n ratio (c. 1:10)
unbarable.
Is there any chance that dbi-user could become comp.lang.perl.dbi? At
least then it would be easy to redirect off-topic threads to a more
appropriate forum.
--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\
------------------------------
Date: Mon, 16 Jul 2001 12:41:48 -0500
From: cp <cpryce@pryce.net>
Subject: Re: Hard DBI question
Message-Id: <B778918C.8A4D%cpryce@pryce.net>
in article u9snfwn6ip.fsf@wcl-l.bham.ac.uk, nobull@mail.com at
nobull@mail.com wrote on 07/16/2001 12:11 PM:
> I can suggest one possible "why". Because it's even more full of
> off-topic noise than the newsgroups!
>
> I tried subscribing for a week but I found the s:n ratio (c. 1:10)
> unbarable.
>
> Is there any chance that dbi-user could become comp.lang.perl.dbi? At
> least then it would be easy to redirect off-topic threads to a more
> appropriate forum.
Here! Here!
cp
------------------------------
Date: Mon, 16 Jul 2001 11:04:47 -0400
From: "Christopher J. Mackie" <cjmackie@princeton.edu>
Subject: Re: Performance problem--memory bloat or crawl
Message-Id: <9iuvqf$cgl$1@cnn.Princeton.EDU>
Thanks to all who offered suggestions for this. Unfortunately, nothing
worked: no matter how I tweaked, it kept bloating to the point of failure.
I ended up splitting the input file into sections and processing each
individually. That worked fine, albeit slowly. Think I'll do this in C,
next time... --Chris
"Christopher J. Mackie" <cjmackie@princeton.edu> wrote in message
news:9ig380$abd$1@cnn.Princeton.EDU...
> Apologies for the length of this--I want to be as clear as I can. I'm
> needing to read a large binary file and do a simple manipulation of the
> data. It's not working--either it crashes with an 'out of memory!'
message,
> or, if I choke down the memory limit enough, it works so slowly that the
> complete run will take more than a year. I don't know whether the problem
> is in my code, in the nature of the memory overhead for perl data
> structures, or what.
>
> I'm pretty stuck, and could use all the help anyone can offer.
> TIA, --Chris
> ---------------------------------------------------
> I'm running 64-bit enabled perl 5.6.1 on Irix 6.2 (SGI Origin 4000). The
> input file is a matrix with known row and column lengths. The layout is:
>
> 0 1 2 3 4 5
> 6 7 8 9 10 11
> 12 13 14 15 16 17
> ...
>
> stored in the file as: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 ...
>
> (The foregoing is just for illustration--the actual dimensions are 297
rows
> by 6.4M+ columns. Cell values are double-precision.) I need to change it
> to:
>
> 12 6 0
> 13 7 1
> 14 8 2
> 15 9 3
> 16 10 4
> 17 11 5
> (I also need to write it out as single-precision)
>
> Problem is, the file is too large (14Gb) to read in all at once. I wrote
a
> subroutine to perform the manipulation in pieces, using the following
logic:
>
> 1. calculate a subset size, based on available memory
> 2. read first row in one chunk (i.e., '0 1 2 3 4 5')
> 3. subset out a piece of that row (e.g., '0 1 2')
> 4. append the subset to a temporary buffer
> 5. repeat steps 1-4 for all rows in the file, to fill the buffer.
> 0 1 2 6 7 8 12 13 14
> this is a 'vertical slice' of the original file:
> 0 1 2
> 6 7 8
> 12 13 14
> 6. use a nested loop to read the data out to another temp buffer in the
> output format
> 12 6 0 13 7 1 14 8 2
> or
> 12 6 0
> 12 7 1
> 14 8 2
> 7. repack the buffer as a float, and print it to the outfile
> 8. increment the substring offset to capture the next vertical slice
> 9. repeat steps 1-8 until the file is processed.
>
> I started out using multi-d arrays, since that made the reformatting logic
> the clearest. The script worked perfectly on smaller files (< 500Mb), but
> it consumed as much as 8x more core than was strictly required for the two
> buffers. On the bigger file, the memory overhead was even worse--in fact,
> it seems to increase exponentially with file size in a way that I don't
> fully understand. For the 14Gb file and two 100Mb buffers (a buffer size
> that works fine on a 100Mb input file), the script kept steadily consuming
> core until it crashed at 1.7Gb, without even completing the first loop
> through the file--and did so at a pace that would have taken 388 days to
> complete the entire subroutine.
>
> I don't know why the script consumes nearly ten times the memory that it
> needs just to hold the values in the temp buffers. I've tried several
> alternatives, including taking bigger and smaller reads and writes,
> pre-filling the buffers with zeros (in case it was the constant buffer
> resizing that was causing speed problems), using strings for subsetting
> rather than arrays, and so on. Nothing seems to work.
>
> Below is the actual subroutine, in its most recent form (in a small
> testframe). I would appreciate anything that anyone can suggest about why
> it's leaking memory so severely on big files. Alternative solutions would
> also be very welcome. (I've looked at things like PDL and Matrix::Real,
but
> I'm not sure I understand them well enough to add that complication to the
> situation.)
> =============================
> #!/usr/bin/perl
>
> # the testframe--a 100_000 value dummy matrix, with a 20_000 byte buffer
> limit
> open (OUTFILE, ">test.in") or die "Which?\n";
> binmode(OUTFILE); # not necessary on my machine, but someday I will need
to
> use this on Windoze
> for (0..99999) {
> $string = pack("d", $_);
> print OUTFILE $string;
> }
> close OUTFILE;
>
> rotateMatrix('test.in','test.out', 500, 200, 100000, 20000, 8);
>
> sub rotateBuffer {
> my($infile, $outfile, $nVectors, $nFactors, $nItems, $MEM_LIMIT,
> $bytesize, $type)=@_;
> my ($NumLoops, $Residuum, $BufferVectors, $BufferMem, $loopnum,
$offset,
> @inbuffer, @cells, $mem, $i, $j, $BUFFER, $OUTBUFFER);
> my $bytesize = 8; # sizeof(double)
>
> { # a little integer math, to generate buffer sizes
> use integer;
> $BufferVectors=$MEM_LIMIT/($nFactors*$bytesize);
> die "MemLimit too small for buffering\n" if ($BufferVectors <
1 );
> $NumLoops=$nVectors/$BufferVectors;
> $Residuum=$nVectors-($BufferVectors*$NumLoops);
> $BufferMem = $BufferVectors * $nFactors*$bytesize;
> $NumLoops++ if ($Residuum);
> print "\tBV: $BufferVectors nV: $nVectors nL: $NumLoops nR:
> $Residuum MemLimit: $BufferMem\n";
> }
>
> $offset=0;
> for($loopnum=0; $loopnum < $NumLoops; $loopnum++) {
> # Unless $nVectors % $BufferVectors==0, last loop will have fewer
> BufferVectors (i.e., Residuum)
> if ($loopnum==($NumLoops-1) && $Residuum >0)
> {
> $BufferVectors=$Residuum;
> print "Last loop: BufferVectors=$BufferVectors\n";
> }
> print "Loop: $loopnum beg: $offset end: ",
> ($offset+$BufferVectors), "...";
> # Read the entire file, 'line' by 'line', then slice out only
the
> portion of the line we want in this pass.
> # a 'line' is all vectors for one factor
> open(INFILE, "<$infile") or die "Can't open $infile\n";
> binmode(INFILE);
> undef (@inbuffer);
> while (($nVectors*$bytesize)==sysread(INFILE, $BUFFER,
($nVectors
> * $bytesize),0))
> {
> @cells=unpack("d*", $BUFFER);
> # here's the slice
> push (@inbuffer,
> @cells[$offset..($offset+$BufferVectors-1)]);
> }# end WHILE read
>
> # Once it's all read, we write
> print "Outputting...";
> open (OUTFILE, ">>$outfile") or die "Can't open $outfile: $!";
> binmode(OUTFILE);
>
> undef $OUTBUFFER;
> my $startpoint = (($nFactors-1) * $nVectors);
> for ($i=0; $i<$nFactors-1; $i++)
> {
> for ($j=$startpoint; $j >= 0; $j -= $nVectors)
> {
> $OUTBUFFER .= pack("f", $inbuffer[$i+$j]);
> }
> }
> print OUTFILE $OUTBUFFER;
> print "Done.\n";
>
> # Update slice location for next loop
> $offset += $BufferVectors ;
>
> } # end FOR loopnum
>
> close(INFILE);
> close(OUTFILE);
>
> } # end SUB rotateBuffer
>
>
------------------------------
Date: Mon, 16 Jul 2001 16:44:48 GMT
From: news@mikespub.net (Michel Dalle)
Subject: Re: Performance problem--memory bloat or crawl
Message-Id: <9iv5iv$7jc$1@news.mch.sbs.de>
In article <9iuvqf$cgl$1@cnn.Princeton.EDU>, "Christopher J. Mackie" <cjmackie@princeton.edu> wrote:
>Thanks to all who offered suggestions for this. Unfortunately, nothing
>worked: no matter how I tweaked, it kept bloating to the point of failure.
>I ended up splitting the input file into sections and processing each
>individually. That worked fine, albeit slowly. Think I'll do this in C,
>next time... --Chris
The vec() approach proposed by ... (someone whose post didn't make
it into Google) seems rather interesting.
When you combine that with your idea of "column slices", you only
need the following memory storage at any one point :
1) a bit vector containing N rows x M column slices x 4 bytes (single-
precision float on this box) - adjust M according to the available memory
2) an array containing the M column slice for 1 row
Of course, you need to do N * (column size) / M sysseek()s and place
every single float at the right place in the bit vector, so it's not really
fast, but you *can* do the whole operation with limited memory.
You're right that for a project like this, the development time in C would
probably be comparable to that in Perl, so exceptionnally, I'd go for C
as well here :)
Michel.
------------------------------
Date: 16 Jul 2001 09:03:01 -0700
From: kenny@barmeister.com (Kenny)
Subject: Re: problem with @INC
Message-Id: <589a1080.0107160803.126651de@posting.google.com>
At the top of the script you can also add
push(@INC,'/usr/lib/perl5/site_perl/5.6.0');
Les Ander <citykid@nospam.com> wrote in message news:<Pine.LNX.4.33.0107151214370.3170-100000@schewanella.stanford.edu>...
> Hi,
> I installed Net:ssh module and unfotunately it decided
> to upgrade my perl from 5.6 to 5.6.1. Now when ever i try to use
> a module that i had previously installed does'nt work anymore.
> For example, if i type perl -e 'use DBI;'
> it says...
> ---[~>> perl -e 'use DBI;'
> Can't locate DBI.pm in @INC (@INC contains:
> /usr/local/lib/perl5/5.6.1/i686-linux /usr/local/lib/perl5/5.6.1
> /usr/local/lib/perl5/site_perl/5.6.1/i686-linux
> /usr/local/lib/perl5/site_perl/5.6.1 /usr/local/lib/perl5/site_perl .) at
> -e line 1.
> BEGIN failed--compilation aborted at -e line 1.
>
> but
> ---[~>> locate DBI.pm
> /usr/lib/perl5/site_perl/5.6.0/i386-linux/Bundle/DBI.pm
> /usr/lib/perl5/site_perl/5.6.0/i386-linux/DBI.pm
> /usr/lib/perl5/site_perl/DBI.pm
>
> How can i add the /usr/lib/perl5/site_perl/5.6.0
> into my path? Is there another way to fix this?
> thanks
------------------------------
Date: Mon, 16 Jul 2001 11:32:32 -0500
From: Michael Carman <mjcarman@home.com>
Subject: Re: problem with @INC
Message-Id: <3B531720.F80B4D74@home.com>
[Jeapordectomy performed. Please don't top-post.]
Kenny wrote:
>
> Les Ander <citykid@nospam.com> wrote:
> > Can't locate DBI.pm in @INC (@INC contains:
[...]
> >
> > How can i add the /usr/lib/perl5/site_perl/5.6.0
> > into my path? Is there another way to fix this?
> > thanks
> At the top of the script you can also add
> push(@INC,'/usr/lib/perl5/site_perl/5.6.0');
That won't happen until runtime. You need to put it in a BEGIN block to
make it happen at compile time.
-mjc
------------------------------
Date: Mon, 16 Jul 2001 14:02:28 -0400
From: Lou Moran <lmoran@wtsg.com>
Subject: Re: Problems moving from an Apache server to a Windows server
Message-Id: <rpa6ltotiq13vs6b5u49jjg5tgjl0erhpu@4ax.com>
On Sat, 07 Jul 2001 21:40:27 GMT, Bart Lateur <bart.lateur@skynet.be>
wrote wonderful things about sparkplugs:
>Kerry Shetline wrote:
>
>>What kinds of things can crash Perl and not even give you an error message?
>
>Memory problems.
>
SNIP
>
>Rewrite them in (plain C),
SNIP
Bart has the most correct answers. We were forced recently to use IIs
and Memory Problems were rampant (who would have ever though 1GB was
every going to be a memory amount much less not enough!)
C helped a lot, we used CygWin which has a really nice Bash shell too.
------------------------------
Date: 16 Jul 2001 12:41:13 -0500
From: eisele@ksu.edu (Don Eisele)
Subject: Question about indenting in emacs cperl-mode
Message-Id: <9iv8vp$9ie@insanity.ksu.ksu.edu>
I am having a problem in which some code inside blocks
aren't getting indented in the same way as some
code outside of blocks.
following is a sample of the type of code I am having problems
with (I want it to look like the second one):
#1
{
print "dog",
"cat",
"frog",
"nee";
}
#2
print "dog",
"cat",
"frog",
"nee";
--
Don Eisele, KSU CNS Unix Systems Administrator -- eisele@ksu.edu
"A common mistake that people make when trying to design something completely
foolproof was to underestimate the ingenuity of complete fools."
-- Douglas Adams, Mostly Harmless
------------------------------
Date: 16 Jul 2001 11:18:17 -0500
From: Ren Maddox <ren@tivoli.com>
Subject: Re: sortlen -- filter to sort text by line length
Message-Id: <m3u20cu9ti.fsf@dhcp9-161.support.tivoli.com>
On Fri, 13 Jul 2001, gbacon@HiWAAY.net wrote:
[snipped sorting by length with ST or GRT]
> A scalar already knows its length, so all the time and space used
> in constructing, accessing, and stripping the augmented list is
> wasted.
Ack! I thought that perhaps the GRT advantage of the built-in
comparison would overcome this, but some quick benchmarking has not
revealed that to be true.
--
Ren Maddox
ren@tivoli.com
------------------------------
Date: Mon, 16 Jul 2001 17:26:57 GMT
From: Uri Guttman <uri@sysarch.com>
Subject: Re: sortlen -- filter to sort text by line length
Message-Id: <x73d7wwzrx.fsf@home.sysarch.com>
>>>>> "RM" == Ren Maddox <ren@tivoli.com> writes:
RM> On Fri, 13 Jul 2001, gbacon@HiWAAY.net wrote:
RM> [snipped sorting by length with ST or GRT]
>> A scalar already knows its length, so all the time and space used
>> in constructing, accessing, and stripping the augmented list is
>> wasted.
RM> Ack! I thought that perhaps the GRT advantage of the built-in
RM> comparison would overcome this, but some quick benchmarking has not
RM> revealed that to be true.
i haven't looked at this problem in detail but you have to note that
both the ST and GRT will be slower in real terms for short lists where
the setup overhead they have is less than the comparison overhead. they
will win when you have enough comparisons that the setup overhead is
swamped by the compare calls. this is standard O() algorithm
theory. even a O(N*N) bubble sort can be faster than a O(N * Log N) sort
for short lists.
uri
--
Uri Guttman --------- uri@sysarch.com ---------- http://www.sysarch.com
SYStems ARCHitecture and Stem Development ------ http://www.stemsystems.com
Learn Advanced Object Oriented Perl from Damian Conway - Boston, July 10-11
Class and Registration info: http://www.sysarch.com/perl/OOP_class.html
------------------------------
Date: Mon, 16 Jul 2001 11:02:15 -0400
From: "Christopher J. Mackie" <cjmackie@princeton.edu>
Subject: The brackets and arrows of outrageous fortune... (complex object/reference syntax)
Message-Id: <9iuvln$cg5$1@cnn.Princeton.EDU>
This is a bit embarrassing--I'm having trouble using the following data
structure (it's a parse tree produced by Parse::RecDescent using the
<autotree> option, and displayed using Data::Dumper).
$VAR1 = bless( {
'story' => [
bless( {
'head' => bless( {
'headtag' => [
bless( {
'Section' => bless( {
'__VALUE__' => 'Section (SE) HOUSTON'
}, 'Section' ),
'__RULE__' => 'headtag'
}, 'headtag' ), # and so on...
I want to retrieve the text 'Section (SE) HOUSTON'. Can someone help me
with the syntax?
TIA, --Chris
------------------------------
Date: 16 Jul 2001 09:05:07 -0700
From: kanem@ic.edu (Matt)
Subject: versions
Message-Id: <459b8ca2.0107160805.4850e7b7@posting.google.com>
how can I find out what version of perl is on a linux server? I'm not
familiar with a command.
------------------------------
Date: Mon, 16 Jul 2001 16:25:06 GMT
From: Michael Budash <mbudash@sonic.net>
Subject: Re: versions
Message-Id: <mbudash-FC793B.09250716072001@news.sonic.net>
In article <459b8ca2.0107160805.4850e7b7@posting.google.com>,
kanem@ic.edu (Matt) wrote:
> how can I find out what version of perl is on a linux server? I'm not
> familiar with a command
type:
perl -v
at the command prompt.
hth-
--
Michael Budash ~~~~~~~~~~ mbudash@sonic.net
------------------------------
Date: Mon, 16 Jul 2001 11:33:00 -0500
From: Michael Carman <mjcarman@home.com>
Subject: Re: versions
Message-Id: <3B53173C.DC58E93B@home.com>
Matt wrote:
>
> how can I find out what version of perl is on a linux server? I'm not
> familiar with a command.
perl -v
perl -V
-mjc
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 1312
***************************************
