[23250] in Perl-Users-Digest
Perl-Users Digest, Issue: 5471 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Tue Sep 9 14:10:59 2003
Date: Tue, 9 Sep 2003 11:10:13 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Tue, 9 Sep 2003 Volume: 10 Number: 5471
Today's topics:
Re: My perl script is "Killed" - Ran out of memory ctcgag@hotmail.com
Re: Perl Arguements <michael.p.broida@boeing.com>
Re: Perl Arguements <michael.p.broida@boeing.com>
Re: perl simple cms <tzz@lifelogs.com>
Re: Perl vs TCL <abigail@abigail.nl>
Re: Perl vs TCL <mpapec@yahoo.com>
Re: Perl vs TCL <jwillmore@cyberia.com>
Problem installing libwww-perl-5.69 on Solaris.Failing (Daniel Roberts)
Re: qr// question <mpapec@yahoo.com>
Re: qr// question (Anno Siegel)
Re: qr// question (Anno Siegel)
Re: Quotes and circular references <bharnish@technologist.com>
Reading Data File Records (Graham)
Re: Reading Data File Records <no@email.com>
Re: Reading Data File Records <jwillmore@cyberia.com>
Re: Reading Data File Records <twhu@lucent.com>
tar gone wild under perl <mdudley@execonn.com>
Re: <bwalton@rochester.rr.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: 09 Sep 2003 17:37:06 GMT
From: ctcgag@hotmail.com
Subject: Re: My perl script is "Killed" - Ran out of memory
Message-Id: <20030909133706.666$aa@newsreader.com>
dna@888.nu (Marcus Brody) wrote:
> >Don't store as much! (If you told us what you were doing, I might be
> >able to tell you how to not store as much.
>
> It's (somebody elses) biological data. This isn't even the raw data,
> which would be too huuuuge to even imagine.... Therefore it isnt my
> fault I am storing so much. Biological data is notorious ;-)
You think that's notorious, chemical data is at least bad. Then
try to do the cartesian join between the two! :)
> I am parsing it to extract/aggregate the small percentage which is
> useful to me.
OK, I just want to verify that you are not accidentally also storing
the data that isn't useful to you.
> >Aggregating how? sum, count, min, max by group? by how many groups?
>
> Essentially, there are a couple of useful things in the "excel" (tab
> delimetid) files I want to grab hold of. Each file is a table with
> columns denoting ID number, gene name, lots of associated info,
> various results, normalisations and statistical calculations etc.
> Each row is the results of a single "experiment", giving values for
> the above fields.
I think you are already doing this, but just in case...
Don't store the gene name, just the (much smaller) ascension number.
The name can then be looked up later.
> The same gene appears in many experiments. I want to aggregate the
> data for genes togethor for one of the results. E.g. So for each
> gene, I have every result for one type of experiment ordered togethor.
> To give an example of my data structure, this is what I spit out at
> the end:
So essentially you have data of this form:
exp_id, gene_id, results
where the data is effectively sorted (i.e. grouped) by exp_id (because they
are all in the same file), and you want to instead have it grouped by
gene_id.
My first effort would be to parse each file, and for every line in the
file print STDOUT "$gene-id\t$exp_id\t$relevant_result\n";
and invoke this script as
./my_script | gzip > out.txt.gz
And see if this will fit in available disk space.
Then I would use system utilities (are you using a real operating system,
or Windows? If Windows, good look!) to sort out.txt so it would be
grouped by gene_id rather then exp_id, and then have another Perl script
parse that already grouped data.
If you don't have enough disk scratch space to do the sort as one
command, there are various work-arounds.
> foreach my $key (keys %genes) {
> #This is the gene name, followed by how many experiments were for
> that gene
> print "$key $genes{$key}{occurences}\t";
>
> #These are all the values for the experimental results for that
> gene. #The number of results varies from gene to gene.
> foreach my $value ( @{$genes{$key}{result_value}} ) {
> print "$value\t";
> }
> print "\n";
> }
I don't see any key by experiment here! Surely you can't just
load all the different experiment results into one amorphous array,
with no way to distinguish them.
>
> #please excuse poor code, I am a newbie to *all of this*
> Furthermore, there is some other associated info in that hash, wich i
> am not printing out at this stage, but may wish to when the thing is
> working.
I would start by not even storing that associated info. Once you get
the bare bones working, then go back and see if you can add it.
>
> >> I guestimate that my data structure contains 250 million
> >> variables (all very simple - either decimal points (to about 6
> places)
> >> or short names).
> >
> >This is what the input data structure is, or this is what the working
> >memory structure is? If stored in standar Perl variables, that's
> going to
> >be at least 5 gig, even if you they are all floats and never used in
> a
> >stringish way.
>
> The input data is lots more than that. I figure I would have around
> 50,000 genes * 2500 results stored in my hash at the end.
Even if each result is a single float value and nothing more, that
is going to take at least 2.5 gig of memory. If you pack the
floats it would take ~500 meg of memory. This doesn't include
whatever labels would be necessary to identify which experiment
a given value is for.
I think this means you are going to have abandon in-memory hashes, and
go to hashes tied to disk, or parsing the data into a RDBMS (MySQL,
Oracle), both of which will likely take more disk space than you seem to
have, or using system tools to manipulate flat files, or resorting to
multi-pass methods.
>
> >> Any ideas how I could stop running out of memory? Declaring data
> >> types ala C?
> >>
> >You could use "pack" to pack the data into C-type variables that are
> >held (in bulk) in strings. But that would still take at least 1 gig,
> >and I assume you actually want to do something with these numbers,
> which
> >would be difficult if they are all packed.
>
> Yes I do want to do something else with the numbers, but printing them
> out would be fine for now. Once I have everything sorted in a logical
> way, I can easily do what I want to.
Yes, so you can pack the numbers as you aggregate them, then unpack
one-by-one at the end when you want to do something with them. But even
this will probably use too much memory.
>
> >Does your output consists of all 250 million items? If not, then
> >perhaps you don't need to store them all afterall.
>
> Yes my output does, and I do need them all.
Damn! Then there is no easy solution. I just wanted to make sure
that was the case before delving into the harder ones.
> Is there a cleverer way I
> could print them out as I was going along? E.g. I could have each
> gene name that i come across inserted at the beggining of the next
> free row, and then put the associated results value in the appropriate
> place in the file.
I don't think it's feasible with ordinary files in perl with a single pass
approach. I'd suggest the flat files and system sort utility. If not,
then maybe something something like this:
write a func() which, given a gene_id, returns a number from 0 to 49,
with each number having about the same number of genes fall under it.
Open 50 file handles to compressing pipes into files:
foreach ('00'..'49') {
my $fh; open $fh, "|gzip > $_.gz" or die $!
push @handle, $fh;
};
Now, parse the "excel" files, and for every row, print to the
appropriate file:
print {$handle[func($gene_id)]} $data;
Now each file
1) contains all the information for any gene for which it conains any
information. 2) is a managable size.
So these can re-processed to aggregate the by-gene info.
If you don't have enough disk space to hold all 50 output files
simultaneously, then you could simply have the program output
to a single file, but only if func($gene_id) == $ARGV[0].
Then, you'd have to run the parser program fifty times with each
argument from 00 to 49, processing the intermediate file
(and then removing it) between each parse run.
In summary, buy more disk space!
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service New Rate! $9.95/Month 50GB
------------------------------
Date: Tue, 9 Sep 2003 17:21:49 GMT
From: "Michael P. Broida" <michael.p.broida@boeing.com>
Subject: Re: Perl Arguements
Message-Id: <3F5E0C2D.9335DAA3@boeing.com>
Keith Keller wrote:
>
> On 2003-09-08, Michael P. Broida <michael.p.broida@boeing.com> wrote:
> >
> > Do you have the "shebang" line at the beginning?
> > #!/usr/bin/perl
> >
> > I'm thinking that if you DON'T have that line and you
> > run it without specifying "perl", then it's running
> > as commands to a shell (not executing "perl") which
> > -might- output what you see.
>
> My shell, bash, says "print: command not found" or some such.
>
> > (Or maybe not; it's just a thought. I'm on Win32, so
> > can't test this theory.)
>
> Why not? Doesn't Win32 have a command shell?
Not one that could pay attention to the "shebang" line
INSIDE the file.
Here on Win2K, there is a "print" command; I could have
tested that, BUT our network printer queue management
system doesn't work well from the DOS (CommandPrompt)
side.
Mike
------------------------------
Date: Tue, 9 Sep 2003 17:24:44 GMT
From: "Michael P. Broida" <michael.p.broida@boeing.com>
Subject: Re: Perl Arguements
Message-Id: <3F5E0CDC.ADA830F2@boeing.com>
John Bokma wrote:
>
> And for those who really want the same experience on Windows:
> http://www.cygwin.com/
I am interested in Cygwin, but we need to keep all of
our systems here in pretty nearly the same configuration
so we don't introduce problems from dependencies on a
particular tool.
We had problems on an older project under Solaris when
one person decided to use "tcsh" instead of our "standard"
csh. Some of his scripts didn't work correctly for the
rest of us or for our customers. :) So we enforce a
standard set of tools.
Mike
------------------------------
Date: Tue, 09 Sep 2003 13:05:48 -0400
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: perl simple cms
Message-Id: <4nn0dd51ub.fsf@lockgroove.bwh.harvard.edu>
On Mon, 8 Sep 2003, andrew@nospam_andicrook.demon.co.uk wrote:
> can any of you recommend how I should go about storing the
> information in a mysql database (database structure). Just need a
> few ideas to work from
Use the Class::DBI module to get and store data in a MySQL database
(it handles other RDBMSs too).
http://search.cpan.org/author/TMTM/Class-DBI/lib/Class/DBI.pm
Ted
------------------------------
Date: 09 Sep 2003 15:32:10 GMT
From: Abigail <abigail@abigail.nl>
Subject: Re: Perl vs TCL
Message-Id: <slrnblrsjp.djm.abigail@alexandra.abigail.nl>
Selwyn Leeke (selwyn.leeke@camcon.co.uk) wrote on MMMDCLXI September
MCMXCIII in <URL:news:f6ba12d3.0309090456.3a61f60e@posting.google.com>:
][ Hi,
][
][ I'm thinking about writing a script in either Perl or TCL, but before
][ I start, I thought it would be a good idea to find out how they
][ compare to each other, so I can decide which is best for me.
][
][ Does anyone have any experience/opinions of TCL or Perl's Maths
][ functionality? Do you need to install extra modules to make it useful?
][
][ What about text manipulation? Is Perl better than TCL? Are there any
][ modules that improve on the core distribution?
TCL is far, far better in al aspects. That's why every Unix vendor
supplies TCL, and have you let install Perl yourself.
Furthermore, TCL programmers are in high demand. Noone ever hires a
Perl programmer.
][ Thanks in advance -
][ Selwyn
][
][ PS. Just out of interest... A lot of modules for perl seem to be
][ ported from TCL (tk, for example) - are there any that have gone the
][ other way?
No. Almost anything found on CPAN or the Perl standard library was
borrowed, copied or stolen from TCL. The Perl communitie doesn't
create anything itself, so no porting to TCL was ever done.
Abigail
--
perl -wle'print"Κυστ αξοτθες Πεςμ Θαγλες"^"\x80"x24'
------------------------------
Date: Tue, 09 Sep 2003 18:30:15 +0200
From: Matija Papec <mpapec@yahoo.com>
Subject: Re: Perl vs TCL
Message-Id: <tuvrlvko8t27184a6rmcru5csb5226pgfh@4ax.com>
X-Ftn-To: Abigail
Abigail <abigail@abigail.nl> wrote:
>][ modules that improve on the core distribution?
>
>TCL is far, far better in al aspects. That's why every Unix vendor
>supplies TCL, and have you let install Perl yourself.
>
>Furthermore, TCL programmers are in high demand. Noone ever hires a
>Perl programmer.
>
>][ Thanks in advance -
>][ Selwyn
>][
>][ PS. Just out of interest... A lot of modules for perl seem to be
>][ ported from TCL (tk, for example) - are there any that have gone the
>][ other way?
>
>
>No. Almost anything found on CPAN or the Perl standard library was
>borrowed, copied or stolen from TCL. The Perl communitie doesn't
>create anything itself, so no porting to TCL was ever done.
Very, very cruel, I must say. :)
--
Matija
------------------------------
Date: Tue, 09 Sep 2003 16:37:46 GMT
From: James Willmore <jwillmore@cyberia.com>
Subject: Re: Perl vs TCL
Message-Id: <20030909123749.0cfa013b.jwillmore@cyberia.com>
On 9 Sep 2003 05:56:24 -0700
selwyn.leeke@camcon.co.uk (Selwyn Leeke) wrote:
> I'm thinking about writing a script in either Perl or TCL, but
> before I start, I thought it would be a good idea to find out how
> they compare to each other, so I can decide which is best for me.
>
> Does anyone have any experience/opinions of TCL or Perl's Maths
> functionality? Do you need to install extra modules to make it
> useful?
>
> What about text manipulation? Is Perl better than TCL? Are there any
> modules that improve on the core distribution?
>
> Thanks in advance -
> Selwyn
>
> PS. Just out of interest... A lot of modules for perl seem to be
> ported from TCL (tk, for example) - are there any that have gone the
> other way?
<sigh>
I could compare COBOL with Perl, BASIC with Perl, C++ with Perl, etc.,
etc
Each language has its strengths and its weaknesses.
My advise is ... learn both, then create an application in both.
Which ever you feel most comfortable with, use.
This type of question has been asked and answered many times. Try
Google. The result is always the same - "my language is better than
your language" threads that take up bandwidth.
--
Jim
Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.
a fortune quote ...
Mark's Dental-Chair Discovery: Dentists are incapable of asking
questions that require a simple yes or no answer.
------------------------------
Date: 9 Sep 2003 10:31:18 -0700
From: daniel.roberts@aventis.com (Daniel Roberts)
Subject: Problem installing libwww-perl-5.69 on Solaris.Failing Tests
Message-Id: <7eef4575.0309090931.3f15d3e7@posting.google.com>
Hello All
I am at a loss on how to get around the meke test failures shown below
when trying to install libwww-perl-5.69 on solaris.
How do I go about solving the errors listed below?
If I ignore the errors listed below and run a make install I don't get
any readliy apparent errors.
Thanks for any help!
Dan
A snip from make test shows>
robot/rules-dbm.......ok
robot/rules...........ok
robot/ua-get..........HTTP Server terminated
FAILED tests 1-3, 5, 7
Failed 5/7 tests, 28.57% okay
robot/ua..............HTTP Server terminated
FAILED tests 1-3, 5, 7
Failed 5/7 tests, 28.57% okay
local/autoload-get....ok
local/autoload........ok
local/get.............ok
local/http-get........Can't call method "is_redirect" on an undefined
value at local/http-get.t line
214, <DAEMON> line 1.
HTTP Server terminated
dubious
Test returned status 29 (wstat 7424, 0x1d00)
DIED. FAILED tests 1-19
Failed 19/19 tests, 0.00% okay
local/http............Can't call method "is_redirect" on an undefined
value at local/http.t line 188
, <DAEMON> line 1.
HTTP Server terminated
dubious
Test returned status 29 (wstat 7424, 0x1d00)
DIED. FAILED tests 1-18
Failed 18/18 tests, 0.00% okay
local/protosub........ok
Failed 4/26 test scripts, 84.62% okay. 47/343 subtests failed, 86.30%
okay.
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
local/http-get.t 29 7424 19 19 100.00% 1-19
local/http.t 29 7424 18 18 100.00% 1-18
robot/ua-get.t 7 5 71.43% 1-3 5 7
robot/ua.t 7 5 71.43% 1-3 5 7
make: *** [test] Error 29
------------------------------
Date: Tue, 09 Sep 2003 18:27:14 +0200
From: Matija Papec <mpapec@yahoo.com>
Subject: Re: qr// question
Message-Id: <88trlv4pt651sl5hhq1ipmc467qdeb0dbh@4ax.com>
X-Ftn-To: Anno Siegel
anno4000@lublin.zrz.tu-berlin.de (Anno Siegel) wrote:
>> and that "some particular" may be used more then once for matching,
>> while some regexes may not be used at all. If regex compiling is
>> significantly time consuming then this matters.
>
>Access the regexes through this routine:
>
> sub to_regex {
> ref $_[ 0] ? $_[ 0] : $_[ 0] = qr/$_[0]/;
> }
>
>Strings will successively be turned into regexes on first use.
Tnx, so ref knows all about them! :) Unfortunately perldoc isn't so
informative. :!
--
Matija
------------------------------
Date: 9 Sep 2003 16:35:24 GMT
From: anno4000@lublin.zrz.tu-berlin.de (Anno Siegel)
Subject: Re: qr// question
Message-Id: <bjkvgc$hfm$2@mamenchi.zrz.TU-Berlin.DE>
Jeff 'japhy' Pinyan <pinyaj@rpi.edu> wrote in comp.lang.perl.misc:
> On 9 Sep 2003, Anno Siegel wrote:
>
> >Access the regexes through this routine:
> >
> > sub to_regex {
> > ref $_[ 0] ? $_[ 0] : $_[ 0] = qr/$_[0]/;
> > }
> >
> >Strings will successively be turned into regexes on first use.
>
> ... and every time after that. = binds more loosely than ?:
Oh, right. Thanks for catching that. Parentheses around the assignment
would fix it, but, as noted in another branch of the thread, it doesn't
save on regex compilation either way: qr// (well, the regex compiler itself)
already does that.
Anno
------------------------------
Date: 9 Sep 2003 17:40:25 GMT
From: anno4000@lublin.zrz.tu-berlin.de (Anno Siegel)
Subject: Re: qr// question
Message-Id: <bjl3a9$jlo$2@mamenchi.zrz.TU-Berlin.DE>
Matija Papec <mpapec@yahoo.com> wrote in comp.lang.perl.misc:
> X-Ftn-To: Anno Siegel
>
> anno4000@lublin.zrz.tu-berlin.de (Anno Siegel) wrote:
> >> and that "some particular" may be used more then once for matching,
> >> while some regexes may not be used at all. If regex compiling is
> >> significantly time consuming then this matters.
> >
> >Access the regexes through this routine:
> >
> > sub to_regex {
> > ref $_[ 0] ? $_[ 0] : $_[ 0] = qr/$_[0]/;
> > }
> >
> >Strings will successively be turned into regexes on first use.
>
> Tnx, so ref knows all about them! :) Unfortunately perldoc isn't so
> informative. :!
You can think of qr/.../ values as objects of class Regexp, which has
stringification overloaded. That describes most of their behavior, though
things aren't really that simple.
perl -Moverload -le'print "yup" if overload::Overloaded( qr//)'
prints nothing.
Anno
------------------------------
Date: Tue, 09 Sep 2003 16:26:07 GMT
From: Brian Harnish <bharnish@technologist.com>
Subject: Re: Quotes and circular references
Message-Id: <pan.2003.09.09.16.26.26.507319@technologist.com>
-----BEGIN xxx SIGNED MESSAGE-----
Hash: SHA1
On Tue, 09 Sep 2003 06:11:50 -0700, JR wrote:
[snip...]
> [I suppose the it may not matter whether I'm dealing with a circular
> reference or not-I just first noticed this situation when I was
> dealing with them.]
>
[snip...]
> I guess the moral of the story is simply not to force stringification
> of unblessed references in any complex data structure, by
> double-quoting the reference.
What you're seeing doesn't have to do with circular references. What you
are trying to do is use a string as a hash reference. If you get rid of
the bulk of your code, it's really trying to do this:
#!/usr/bin/perl -w
use strict;
my $a={this=>1,that=>2};
print $_ for %{$a};
print $_ for %{"$a"}; # <- Error here, can't use a string as a hashref!
__END__
After interopelation (sp?), it ceases being a reference to a hash, and is
now a string.
- Brian
-----BEGIN xxx SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
iD8DBQE/Xf8siK/rA3tCpFYRAjGMAJ0YswFDiX60T5B5eHQcNZuEXXOV1wCcDz2j
QtdEwK3vxtgn31g5mN3oxXs=
=hHwg
-----END PGP SIGNATURE-----
------------------------------
Date: 9 Sep 2003 08:14:57 -0700
From: GrahamWilsonCA@yahoo.ca (Graham)
Subject: Reading Data File Records
Message-Id: <eda30d78.0309090714.2a6f6431@posting.google.com>
I'm a little frustrated with Perl's line-by-line file reading and I am
hoping that someone can help me.
I have a data file that looks like:
--
! Comment 1
! Comment 2
! Comment ...
5 ! number of levels
*aaa [aaa units] ! space deliminated is common
1.0 2.0 3.0 4.0 5.0
*bbb [bbb units] ! csv is possible
1.0, 2.0, 3.0,
4.0 5.0
*ccc [ccc units] ! the file is written from fortran and the number of
columns is not fixed
10.0
20.0
30.0
40.0
50.0
...
--
Essentially, there is a header block that always begins with '!' in
the first column. This is followed by the number of elements in each
data block and an unknown number of data blocks having a set number of
elements.
The file is generated using about five lines of FORTRAN so it seems
somehwat surprising that I am up to 30 lines of perl with almost no
end in sight... Does anyone have an example showing how to process a
file in blocks using Perl?
Thanks,
Graham
------------------------------
Date: Tue, 9 Sep 2003 17:01:38 +0100
From: "Brian Wakem" <no@email.com>
Subject: Re: Reading Data File Records
Message-Id: <bjkth4$jsvsd$1@ID-112158.news.uni-berlin.de>
"Graham" <GrahamWilsonCA@yahoo.ca> wrote in message
news:eda30d78.0309090714.2a6f6431@posting.google.com...
> I'm a little frustrated with Perl's line-by-line file reading and I am
> hoping that someone can help me.
>
> I have a data file that looks like:
>
> --
> ! Comment 1
> ! Comment 2
> ! Comment ...
> 5 ! number of levels
> *aaa [aaa units] ! space deliminated is common
> 1.0 2.0 3.0 4.0 5.0
> *bbb [bbb units] ! csv is possible
> 1.0, 2.0, 3.0,
> 4.0 5.0
> *ccc [ccc units] ! the file is written from fortran and the number of
> columns is not fixed
> 10.0
> 20.0
> 30.0
> 40.0
> 50.0
> ...
> --
>
> Essentially, there is a header block that always begins with '!' in
> the first column. This is followed by the number of elements in each
> data block and an unknown number of data blocks having a set number of
> elements.
>
> The file is generated using about five lines of FORTRAN so it seems
> somehwat surprising that I am up to 30 lines of perl with almost no
> end in sight... Does anyone have an example showing how to process a
> file in blocks using Perl?
What do you want to do with it?
--
Brian Wakem
------------------------------
Date: Tue, 09 Sep 2003 16:44:19 GMT
From: James Willmore <jwillmore@cyberia.com>
Subject: Re: Reading Data File Records
Message-Id: <20030909124422.43486b64.jwillmore@cyberia.com>
On 9 Sep 2003 08:14:57 -0700
GrahamWilsonCA@yahoo.ca (Graham) wrote:
<snip>
> The file is generated using about five lines of FORTRAN so it seems
> somehwat surprising that I am up to 30 lines of perl with almost no
> end in sight... Does anyone have an example showing how to process
> a file in blocks using Perl?
Post your code - I have no idea what you are trying to do. Maybe it's
just me ;)
--
Jim
Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.
a fortune quote ...
You cannot kill time without injuring eternity.
------------------------------
Date: Tue, 9 Sep 2003 12:50:53 -0400
From: "Tulan W. Hu" <twhu@lucent.com>
Subject: Re: Reading Data File Records
Message-Id: <bjl0dk$t8o@netnews.proxy.lucent.com>
"Graham" <GrahamWilsonCA@yahoo.ca> wrote in message ...
[snip..]
> The file is generated using about five lines of FORTRAN so it seems
> somehwat surprising that I am up to 30 lines of perl with almost no
> end in sight... Does anyone have an example showing how to process a
> file in blocks using Perl?
I would download the File::Slurp module from cpan and installed it.
http://search.cpan.org/author/MUIR/File-Slurp-2004.0904/
====
#!/usr/bin/perl
use File::Slurp;
@allLines = read_file("data_file_name");
foreach my $line (@allLine) {
# in case you need process each line
if ($line =~ /^!/) { # comment lines }
else { # datalines}
}
------------------------------
Date: Tue, 09 Sep 2003 13:02:47 -0400
From: Marshall Dudley <mdudley@execonn.com>
Subject: tar gone wild under perl
Message-Id: <3F5E07B7.66A03359@execonn.com>
I am having a problem with tar suddenly taking 100% of the cpu and never
completing using:
FreeBSD 4.8-RELEASE (NEWKERNEL) #1: Fri Aug 8 16:04:58 EDT 2003
Server Version: Apache/1.3.28 (Unix) PHP/4.1.2 mod_perl/1.28
mod_ssl/2.8.15 OpenSSL/0.9.7b
This is perl, version 5.005_03 built for i386-freebsd
The perl code line I am using is:
exec("/usr/bin/tar cflz - $store_path
/home/kingcart/html/store/$username");
What it does is tar and zip up two directories and send them to the
client via stdout.
If the transfer is not terminated, everything works fine and the
download is performed properly.
If the transfer is aborted before waiting too long (It appears the time
is 5 minutes which is the Timeout in apache), then the tar is aborted
immediately, and all is well.
If the transfer is aborted after 5 minutes but before it finishes, then
the following stays unchanged in memory for several minutes:
nobody 48147 0.0 0.0 512 324 ?? I 10:22AM 0:00.17
/usr/bin/tar cflz - /usr/store/kingcart /home/k.....
nobody 48148 0.0 0.0 484 280 ?? I 10:22AM 0:00.02
/usr/bin/tar cflz - /usr/store/kingcart /home/k.....
nobody 48149 0.0 0.0 576 420 ?? I 10:22AM 0:00.87 gzip
Then after a few minutes one of these disappears, and I get the
following:
nobody 48147 91.8 0.0 512 324 ?? R 10:22AM 1:02.53
/usr/bin/tar cflz - /usr/store/kingcart /home/k
From this point until I kill it manually, it will run and take up 100%
of free cpu time. Lat night one used 18 cpu hours and was was still
hung.
Is this a bug in FreeBSD, Apache, or Perl? Is it a bug at all? Any
ideas on how I can get around it?
Also a few other questions.
Why is the transfer continuing even after the 5 minute timeout?
Why do two instances of tar occur every time I try this, the first one
accumulating about 10 times the cpu time of the second one.
Thanks,
Marshall
--
PLEASE NOTE: comp.infosystems.www.authoring.cgi is a
SELF-MODERATED newsgroup. aa.net and boutell.com are
NOT the originators of the articles and are NOT responsible
for their content.
HOW TO POST to comp.infosystems.www.authoring.cgi:
http://www.thinkspot.net/ciwac/howtopost.html
------------------------------
Date: Sat, 19 Jul 2003 01:59:56 GMT
From: Bob Walton <bwalton@rochester.rr.com>
Subject: Re:
Message-Id: <3F18A600.3040306@rochester.rr.com>
Ron wrote:
> Tried this code get a server 500 error.
>
> Anyone know what's wrong with it?
>
> if $DayName eq "Select a Day" or $RouteName eq "Select A Route") {
(---^
> dienice("Please use the back button on your browser to fill out the Day
> & Route fields.");
> }
...
> Ron
...
--
Bob Walton
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 5471
***************************************
