[24681] in Perl-Users-Digest
Perl-Users Digest, Issue: 6843 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Fri Aug 6 21:06:11 2004
Date: Fri, 6 Aug 2004 18:05:08 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Fri, 6 Aug 2004 Volume: 10 Number: 6843
Today's topics:
Re: A-Z listing <Joe.Smith@inwap.com>
Re: any resources to assist with why perl is better? <3432513@yahoo.de>
Re: Breakpoint on a builtin? <nospam-abuse@ilyaz.org>
Re: Breakpoint on a builtin? <nospam-abuse@ilyaz.org>
Re: Failing File Tests? (Jay Sun Ex)
Re: Failing File Tests? <1usa@llenroc.ude.invalid>
Re: Failing File Tests? <gnari@simnet.is>
Re: Failing File Tests? (Kevin Collins)
Re: How do I detect if I have incoming data in <STDIN> <gnari@simnet.is>
Re: In search of grammars for parsing <joe_y_perroREMOVETHIS@yahooREMOVETHISALSOcom>
Re: Inline C <tassilo.von.parseval@rwth-aachen.de>
NET::TELNET and displaying output on-screen (ZoloftGuy)
Re: partially matching a regexp (Charles DeRykus)
Perl fails with out of memory (sunil)
Re: Perl fails with out of memory (Kevin Collins)
Re: Perl fails with out of memory ctcgag@hotmail.com
Re: Perl fails with out of memory <jgibson@mail.arc.nasa.gov>
Program slows as I run it <dgmiller@u.washington.edu>
Re: Program slows as I run it <noreply@gunnar.cc>
Re: Program slows as I run it <gifford@umich.edu>
Re: Program slows as I run it <gnari@simnet.is>
Re: Program slows as I run it ctcgag@hotmail.com
Re: transforming german characters <someone@example.com>
Re: transforming german characters <noreply@gunnar.cc>
Re: transforming german characters <me@example.com>
Re: transforming german characters <me@example.com>
Unbuffered piped command output? <bryan@ayesha.phys.Virginia.EDU>
Re: Unbuffered piped command output? <gifford@umich.edu>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Fri, 06 Aug 2004 19:51:21 GMT
From: Joe Smith <Joe.Smith@inwap.com>
Subject: Re: A-Z listing
Message-Id: <ZiRQc.256051$XM6.215027@attbi_s53>
BernieH wrote:
> Joe, I've been having a closer look at this, and I'm not too sure how to get
> the program reading my input file (i'm a total perl novice)!
>
> I've been starting the program with: "perl program.pl my_file.txt" but this
> is reading only the filename, not the contents of the file. Obviously I'm
> doing something not right!
You're doing it right. When starting the perl program, provide the names of
the input file(s) on the command line. The perl construct
while(<>){ ... $_ ... }
will automatically open the specified input file and read it line by line.
-Joe
------------------------------
Date: Fri, 06 Aug 2004 14:14:48 -0400
From: L D Jones <3432513@yahoo.de>
Subject: Re: any resources to assist with why perl is better?
Message-Id: <2ni04aF14l8tU1@uni-berlin.de>
davout wrote:
> Can anybody help point me at some web resources that explain why Perl is
> better than say Python, TCL etc?
>
>
What does "better" mean? Search the archives for this group. The
question has (unfortunately) come up thousands of times
------------------------------
Date: Fri, 6 Aug 2004 18:14:59 +0000 (UTC)
From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
Subject: Re: Breakpoint on a builtin?
Message-Id: <cf0hr3$2p40$1@agate.berkeley.edu>
[A complimentary Cc of this posting was sent to
Brian McCauley
<nobull@mail.com>], who wrote in article <cf0cfc$b0$1@sun3.bham.ac.uk>:
> > Not enough. One needs to special-case the 0-arguments case. [See
> > Fatal for how to auto-implement this.]
> I've had a look at Fatal version 1.03 in Perl 5.8.5 and I can't see how
> Fatal destinguishes no arguments ( e.g. unlink() ) from an empty list (
> unlink( my @empty_list = ()))
> What an I missing?
That I did not realize there is a difference. THE VERDICT: functions
which take a list, AND default to $_ are not overridable (via
importing, or via CORE::GLOBAL)... Do not see how to get this info
via caller() either...
Bad luck for Perl programmers, they may need to switch to a different
language;
Ilya
------------------------------
Date: Fri, 6 Aug 2004 18:22:21 +0000 (UTC)
From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
Subject: Re: Breakpoint on a builtin?
Message-Id: <cf0i8t$2p9n$1@agate.berkeley.edu>
[A complimentary Cc of this posting was sent to
Keith Arner nospam
<spam.keith.me.arner.not@marconi.com>], who wrote in article <Pine.GSO.4.42.0408060850270.15519-100000@zimbra>:
> > You could override (documented in perlsub) the unlink function and set
> > a breakpoint on that. This should do (untested, I'm stalwartly
> > ignoring the debugger):
>
> Hhmm... no dice. I can correctly override the builtin (that is, it will
> execute the function that I override with), but when I set a breakpoint on
> my function, the debugger does not hit it:
I can't set a breakpoint in CORE::GLOBAL::* here. But setting it in
foo works with this:
perl -dwle 'sub foo {warn qq((@_))} BEGIN {*CORE::GLOBAL::unlink = \&foo} $_= "bar"; unlink or die'
Hope this helps,
Ilya
------------------------------
Date: 6 Aug 2004 11:11:28 -0700
From: nefarious_6x3@yahoo.co.uk (Jay Sun Ex)
Subject: Re: Failing File Tests?
Message-Id: <135fa1e0.0408061011.1733a787@posting.google.com>
I figured out the problem. Below is the working code. I also figured
out the answer to the module question. Thanks for all the help!
&recurse($ARGV[0]);
sub recurse {
opendir DIR, $_[0];
foreach $file (readdir DIR) {
if (($file eq ".") or ($file eq "..")) {
#Do nothing
} elsif (-d ($_[0] . "\\" . $file)) {
print "\n$file";
$folder_path = $_[0] . "\\" . $file;
push @directories, $folder_path;
} else {
print "\n$file is not a directory";
}
}
close DIR;
foreach $newpath (@directories) {
shift @directories;
&recurse($newpath);
}
}
Thanks again everyone!
------------------------------
Date: 6 Aug 2004 19:38:13 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: Failing File Tests?
Message-Id: <Xns953D9F120A9Aasu1cornelledu@132.236.56.8>
nefarious_6x3@yahoo.co.uk (Jay Sun Ex) wrote in
news:135fa1e0.0408061011.1733a787@posting.google.com:
use strict;
use warnings;
> &recurse($ARGV[0]);
You should only call subroutines with & if you need the specific behavior.
Read perldoc perlsub for more info.
> sub recurse {
> opendir DIR, $_[0];
opendir might fail.
> foreach $file (readdir DIR) {
> if (($file eq ".") or ($file eq "..")) {
> #Do nothing
next if ($file eq '.' or $file eq '..');
> } elsif (-d ($_[0] . "\\" . $file)) {
I would really recommend using File::Spec->catfile for this purpose. It
helps with one aspect of portability and enhances readibility as well as
helps one avoid typos.
> print "\n$file";
> $folder_path = $_[0] . "\\" . $file;
Why do the same thing twice?
> push @directories, $folder_path;
> } else {
> print "\n$file is not a directory";
> }
> }
> close DIR;
> foreach $newpath (@directories) {
> shift @directories;
> &recurse($newpath);
> }
> }
May I recommend the following:
#! perl
use strict;
use warnings;
use File::Spec::Functions qw(catfile);
$| = 1;
my $top = $ARGV[0];
$top = '.' unless defined $top;
recurse($top);
sub recurse {
my $dir = shift;
my @list;
{
opendir my $DIR, $dir or die "Cannot open $dir: $!";
@list = readdir $DIR;
closedir $DIR or die "Error closing $dir: $!";
}
@list = grep { $_ ne '.' and $_ ne '..' } @list;
@list = map { catfile $dir, $_ } @list;
my @todo;
for (@list) {
unless( -d ) {
print "$_ is not a directory\n";
next;
}
print "$_\n";
push @todo, $_;
}
recurse($_) for @todo;
}
--
A. Sinan Unur
1usa@llenroc.ude.invalid
(remove '.invalid' and reverse each component for email address)
------------------------------
Date: Fri, 6 Aug 2004 19:45:12 -0000
From: "gnari" <gnari@simnet.is>
Subject: Re: Failing File Tests?
Message-Id: <cf0n0g$h2$1@news.simnet.is>
"Jay Sun Ex" <nefarious_6x3@yahoo.co.uk> wrote in message
news:135fa1e0.0408061011.1733a787@posting.google.com...
> I figured out the problem. Below is the working code. I also figured
> out the answer to the module question. Thanks for all the help!
>
> &recurse($ARGV[0]);
as other have mentioned, the '&' is not necessary.
recurse($ARGV[0]);
is considered better form (by many)
i assume your newsposter destroyed your indent.
> sub recurse {
> opendir DIR, $_[0];
> foreach $file (readdir DIR) {
> if (($file eq ".") or ($file eq "..")) {
> #Do nothing
> } elsif (-d ($_[0] . "\\" . $file)) {
alternatives:
} elsif (-d ($_[0] . '\' . $file)) {
or:
} elsif (-d "$_[0]\\$file") {
> print "\n$file";
strange to see you print "\n" *before* each string
> ...
> foreach $newpath (@directories) {
> shift @directories;
> &recurse($newpath);
really strange way of doing this. I suggest you declare
my @directories;
at the start of the sub, and change this loop to:
recurse($_) for @directories;
> }
> }
gnari
------------------------------
Date: Fri, 06 Aug 2004 20:23:25 GMT
From: spamtotrash@toomuchfiction.com (Kevin Collins)
Subject: Re: Failing File Tests?
Message-Id: <slrnch7q95.oe3.spamtotrash@doom.unix-guy.com>
In article <cf0n0g$h2$1@news.simnet.is>, gnari wrote:
> "Jay Sun Ex" <nefarious_6x3@yahoo.co.uk> wrote in message
> news:135fa1e0.0408061011.1733a787@posting.google.com...
>> I figured out the problem. Below is the working code. I also figured
>> out the answer to the module question. Thanks for all the help!
>>
>> &recurse($ARGV[0]);
>
> as other have mentioned, the '&' is not necessary.
> recurse($ARGV[0]);
> is considered better form (by many)
>
> i assume your newsposter destroyed your indent.
>
>> sub recurse {
>> opendir DIR, $_[0];
>> foreach $file (readdir DIR) {
>> if (($file eq ".") or ($file eq "..")) {
>> #Do nothing
>> } elsif (-d ($_[0] . "\\" . $file)) {
>
> alternatives:
> } elsif (-d ($_[0] . '\' . $file)) {
> or:
> } elsif (-d "$_[0]\\$file") {
It should also be mentioned that Perl does not care about '\' versus '/' in
file paths. So, unless you specifically need to use '\' (for example, a
system() call might need them), you can just use '/' and avoid the escaping:
} elsif (-d "$_[0]/$file") {
>> print "\n$file";
>
> strange to see you print "\n" *before* each string
>
>> ...
>> foreach $newpath (@directories) {
>> shift @directories;
>> &recurse($newpath);
>
> really strange way of doing this. I suggest you declare
> my @directories;
> at the start of the sub, and change this loop to:
> recurse($_) for @directories;
>
>
>> }
>> }
>
Kevin
------------------------------
Date: Fri, 6 Aug 2004 19:21:18 -0000
From: "gnari" <gnari@simnet.is>
Subject: Re: How do I detect if I have incoming data in <STDIN> when I pipe something to my perl script ?
Message-Id: <cf0ljl$dk$1@news.simnet.is>
"Richard Gration" <richard@zync.co.uk> wrote in message
news:cevs6s$jqo$1@news.freedom2surf.net...
> In article <cevblc$2ds$1@newstree.wise.edt.ericsson.se>, "Andreas Berg"
> <andreas.berg@ericsson.com> wrote:
> > Actually, if the first command line parameter is a file, you can read it
> > using <STDIN>, or maybe its just <> you can read it with, but if I
> > recall correctly, they are the same thing.
>
> Right second time. The <> is some perl magic which treats the command
> line arguments as filenames and allows you to read their content via <>.
> This has nothing to do with stdin though, <> and <STDIN> are different
> beasts.
different beasts, but related.
do not forget that in the abscence of filename arguments, <> reads
from STDIN
gnari
------------------------------
Date: Fri, 06 Aug 2004 14:51:23 -0700
From: Joe Y Perro <joe_y_perroREMOVETHIS@yahooREMOVETHISALSOcom>
Subject: Re: In search of grammars for parsing
Message-Id: <qhu7h0li4adhvph43cmeema969lfqdr2p6@4ax.com>
Hi,
>
>- Parse::Yapp includes YappParse.yp, which partially parses most of a YACC
> grammar. I'd have to modify it to parse the tokens and union
> declaration.
In the test directory for Yapp, the file stress.t includes a full C++
grammar.
>
>== Parsing C/C++ ==
For C, check the source code for gcc at http://www.gnu.org/
>
>- I could try Inline::C::ParseRegExp or Inline::C::ParseRecDescent for C,
> and Inline::CPP::grammar for C++.
>- PERCEPS (http://starship.python.net/crew/tbryan/PERCEPS/) is a Perl
> header file parser. Not sure how well it would work for the head section
> of a YACC or LEX input file.
>
>In the worst case, both the bison and flex distributions have grammars,
>which I could adapt to suitable input for Parse::RecDescent or
>Parse::Yapp. But I can't help but think that someone has already done all
>this work.
I would stick with Yapp. Its more compatible than RecDescent,
especially if your re-using existing YACC/LEX files. For a project
last winter I tried to move a YACC grammar to RecDescent, but the
left/right recursions issues made it too time consuming to fix. Yapp
worked. Note you will have to write your own lexer routine --
converting from a LEX file shouldn't be too difficult.
Keep us posted.
Good luck.
Joe
------------------------------
Date: Fri, 6 Aug 2004 20:22:43 +0200
From: "Tassilo v. Parseval" <tassilo.von.parseval@rwth-aachen.de>
Subject: Re: Inline C
Message-Id: <2ni0jmF1658fU1@uni-berlin.de>
Also sprach PerlNovice:
> Below is the message from Tassilo that shows how to work the XS
> module. I did exactly what he said and it is not working for me.
> When I run: perl Makefile.PL && make; I get an error:
> Writing Makefile for Hello::World
> make: Error -- rem: The system cannot find the file specified.
> make: Error code -1
> I have no idea what this means and why it occurs.
Neither have I. To me this looks as though your environment is somewhat
borked. What operating system are you using?
Have you been able to install Perl modules containing XS portions in the
past at all? The above already failed before any compilation was
attempted. Maybe something wrong with your make program?
> I also tried Inline C and I get the following error:
> Undefined subroutine &main::hello called at ./test_inline.pl line 4.
You haven't shown us how test_inline.pl looks, therefore making a proper
diagnosis is tricky.
Tassilo
--
$_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus})!JAPH!qq(rehtona{tsuJbus#;
$_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexiixesixeseg;y~\n~~dddd;eval
------------------------------
Date: 6 Aug 2004 12:40:01 -0700
From: zoloftguy@myrealbox.com (ZoloftGuy)
Subject: NET::TELNET and displaying output on-screen
Message-Id: <d4376e99.0408061140.1978c947@posting.google.com>
I'm a newbie to Perl and I am writing a script to telnet in an shut
off one of our Nortel Contivity VPNs. It works but, I'd like some of
the activity to display on the screen rather then just in the logs.
Is there anyway to do this? Here is the script. Please tell me if
there is anything back-asswords about it. Thanks:
$t = new Net::Telnet (Timeout => 30, Prompt => '/[\$]/');
# Contivity
system "cls"; #This clears the screen.
print "Ok, First, lets shut down the Contivities\n";
$t->open("contivity");
$t->waitfor('/Login:/');
$t->print("admin");
$t->waitfor('/Password:/');
$t->print("pass");
$t->waitfor('/CES\>/');
$t->print("en");
$t->waitfor('/Password:/');
$t->print("pass");
$t->print("shutdown -h");
$t->errmode(sub {die @_ unless $_[0] =~ /eof/});
@output1 = $t->cmd('y');
print @output1;
$t->close;
------------------------------
Date: Fri, 6 Aug 2004 23:48:47 GMT
From: ced@bcstec.ca.boeing.com (Charles DeRykus)
Subject: Re: partially matching a regexp
Message-Id: <I21u5B.Jxq@news.boeing.com>
In article <cettfn$4o9$1@meiner.onlinehome.de>,
Thomas Koenig <Thomas.Koenig@online.de> wrote:
>Assume I have a regexp, /^(hello)|(goodbye)$/ for example.
>
>I want to see wether a particular string matches part of that
>particular regexp, so "", "h", "he", "hel", "hell", "hello", "g",
>"go", "goo" "good", "goodb", "goodby" and "goodbye" would be ok,
>and anything else wouldn't.
>
>I could hand-craft this example easily enough, but it grows
>tedious and error-prone for more general regular expressions,
>and automation would be much preferred.
>
>Ideas?
Sounds as if 'index' (perldoc -f index) might be easier and
in some cases faster than a regex:
my $substring = ...
for ( qw/hello goodbye/ ) {
print "$substring matched $_\n" if index($_, $substring) != -1;
}
--
Charles DeRykus
------------------------------
Date: 6 Aug 2004 15:42:23 -0700
From: sunilsreenivas2001@yahoo.com (sunil)
Subject: Perl fails with out of memory
Message-Id: <d924fa71.0408061442.43a15252@posting.google.com>
Hi,
I run nm command in perl on some 15000 files to collect symbol
information as:
@$oplines = qx "nm -A @$ipfiles";
This works fine on SUNs but on IBMs fails with:
Out of memory!
nm: rmcuall.o: 0654-207 The sort process was stopped prematurely."
This is not nm error since when I do nm on object files at shell
prompt, it is ok. Problem seems to be assigning o/p from nm to array
in perl. But I don't understand why it says nm stopped prematurely.
Ideas?
Thanks,
Sunil.
------------------------------
Date: Fri, 06 Aug 2004 22:57:52 GMT
From: spamtotrash@toomuchfiction.com (Kevin Collins)
Subject: Re: Perl fails with out of memory
Message-Id: <slrnch837g.otp.spamtotrash@doom.unix-guy.com>
In article <d924fa71.0408061442.43a15252@posting.google.com>, sunil wrote:
> Hi,
> I run nm command in perl on some 15000 files to collect symbol
> information as:
> @$oplines = qx "nm -A @$ipfiles";
> This works fine on SUNs but on IBMs fails with:
> Out of memory!
> nm: rmcuall.o: 0654-207 The sort process was stopped prematurely."
> This is not nm error since when I do nm on object files at shell
> prompt, it is ok. Problem seems to be assigning o/p from nm to array
> in perl. But I don't understand why it says nm stopped prematurely.
> Ideas?
> Thanks,
> Sunil.
This is not a Perl problem - this is a shell/system problem. Either the user or
system memory limit is being hit. See the man page for whatever shell you run
under and search for ulimit. Also, check the output of 'ulimit' on both systems
and see what the differences are.
Kevin
------------------------------
Date: 06 Aug 2004 23:17:58 GMT
From: ctcgag@hotmail.com
Subject: Re: Perl fails with out of memory
Message-Id: <20040806191758.546$q4@newsreader.com>
sunilsreenivas2001@yahoo.com (sunil) wrote:
> Hi,
> I run nm command in perl on some 15000 files to collect symbol
> information as:
> @$oplines = qx "nm -A @$ipfiles";
> This works fine on SUNs but on IBMs fails with:
> Out of memory!
> nm: rmcuall.o: 0654-207 The sort process was stopped prematurely."
> This is not nm error
I bet it is an nm or shell error. I base this on the fact that the error
starts with "nm:".
> since when I do nm on object files at shell
> prompt, it is ok.
On the exact same list of files as you do it on in perl?
> Problem seems to be assigning o/p from nm to array
> in perl. But I don't understand why it says nm stopped prematurely.
> Ideas?
Fix the nm problem, or do:
push @$oplines, qx"nm -A $_" foreach @$ipfiles;
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
------------------------------
Date: Fri, 06 Aug 2004 16:24:40 -0700
From: Jim Gibson <jgibson@mail.arc.nasa.gov>
Subject: Re: Perl fails with out of memory
Message-Id: <060820041624406796%jgibson@mail.arc.nasa.gov>
In article <d924fa71.0408061442.43a15252@posting.google.com>, sunil
<sunilsreenivas2001@yahoo.com> wrote:
> Hi,
> I run nm command in perl on some 15000 files to collect symbol
> information as:
> @$oplines = qx "nm -A @$ipfiles";
> This works fine on SUNs but on IBMs fails with:
> Out of memory!
> nm: rmcuall.o: 0654-207 The sort process was stopped prematurely."
> This is not nm error since when I do nm on object files at shell
> prompt, it is ok. Problem seems to be assigning o/p from nm to array
> in perl. But I don't understand why it says nm stopped prematurely.
> Ideas?
Are you trying to run nm on all 15000 files at once? The @$ipfiles will
interpolate to its members separated by a space between double quotes.
How many elements does @$ipfiles contain? Try running nm on each file
separately and capture and process the output from each execution.
------------------------------
Date: Fri, 6 Aug 2004 14:31:31 -0700
From: "Daniel Miller" <dgmiller@u.washington.edu>
Subject: Program slows as I run it
Message-Id: <cf0t8g$cft$1@gnus01.u.washington.edu>
Hi, I wrote a program that basically takes one element at a time from a
large file (>one million lines) and searches some arrays etc. and spits out
data. Problem is that after I start it, it slowly grinds to a crawl as it
runs. I've re-written it several ways so as not to chew up my RAM (windows
says over 700MB free!) Any thoughts on how to approach this problem? Is this
a CPU cache problem?
Thanks
Dan
------------------------------
Date: Fri, 06 Aug 2004 23:39:32 +0200
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: Program slows as I run it
Message-Id: <2nic42F1bt5dU1@uni-berlin.de>
Daniel Miller wrote:
> Hi, I wrote a program that basically takes one element at a time
> from a large file (>one million lines) and searches some arrays
> etc. and spits out data.
Then we know *exactly* what the reason for your problem is...
> Problem is that after I start it, it slowly grinds to a crawl as it
> runs. I've re-written it several ways so as not to chew up my RAM
> (windows says over 700MB free!) Any thoughts on how to approach
> this problem?
Of course: Re-write it again.
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: 06 Aug 2004 17:39:20 -0400
From: Scott W Gifford <gifford@umich.edu>
Subject: Re: Program slows as I run it
Message-Id: <qszy8ks0wvr.fsf@rygar.gpcc.itd.umich.edu>
"Daniel Miller" <dgmiller@u.washington.edu> writes:
> Hi, I wrote a program that basically takes one element at a time from a
> large file (>one million lines) and searches some arrays etc. and spits out
> data. Problem is that after I start it, it slowly grinds to a crawl as it
> runs. I've re-written it several ways so as not to chew up my RAM (windows
> says over 700MB free!) Any thoughts on how to approach this problem? Is this
> a CPU cache problem?
It sounds to me like a problem with your algorithm. Can you describe
in more detail what your program is doing?
-----ScottG.
------------------------------
Date: Fri, 6 Aug 2004 22:03:53 -0000
From: "gnari" <gnari@simnet.is>
Subject: Re: Program slows as I run it
Message-Id: <cf0v4g$1gq$1@news.simnet.is>
"Daniel Miller" <dgmiller@u.washington.edu> wrote in message
news:cf0t8g$cft$1@gnus01.u.washington.edu...
> Hi, I wrote a program that basically takes one element at a time from a
> large file (>one million lines) and searches some arrays etc. and spits
out
> data. Problem is that after I start it, it slowly grinds to a crawl as it
> runs. I've re-written it several ways so as not to chew up my RAM (windows
> says over 700MB free!) Any thoughts on how to approach this problem? Is
this
> a CPU cache problem?
use hashes instead of the arrays
gnari
------------------------------
Date: 06 Aug 2004 23:25:22 GMT
From: ctcgag@hotmail.com
Subject: Re: Program slows as I run it
Message-Id: <20040806192522.162$10@newsreader.com>
"Daniel Miller" <dgmiller@u.washington.edu> wrote:
> Hi, I wrote a program that basically takes one element at a time from a
> large file (>one million lines) and searches some arrays etc.
Are the arrays static once initially set, or do they change throughout?
> and spits
> out data. Problem is that after I start it, it slowly grinds to a crawl
> as it runs. I've re-written it several ways so as not to chew up my RAM
> (windows says over 700MB free!) Any thoughts on how to approach this
> problem? Is this a CPU cache problem?
It is kind of hard to say when you don't how us any of the progam.
95.54% chance the problem is you are using arrays when you should use
hashes.
2.87% chance the you are making some other poor algorithmic choice.
1.58% chance that the problem is simply too big for you machine's
britches. 0.01% chance that the problem lies solely in the CPU cache.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
------------------------------
Date: Fri, 06 Aug 2004 19:20:13 GMT
From: "John W. Krahn" <someone@example.com>
Subject: Re: transforming german characters
Message-Id: <NRQQc.34144$yT2.23545@clgrps13>
steve_f wrote:
> I want to transform special German characters to obtain the following
> variations:
>
> groß bräu
> gross bräu
> gross braeu
>
> there are two sets -
>
> set one:
> ß = ss = \xDF
>
> set two:
> Ä = Ae = \xC4
> Ö = Oe = \xD6
> Ü = Ue = \xDC
> ä = ae = \xE4
> ö = oe = \xF6
> ü = ue = \xFC
>
> basically, the rules are transform ß independently
> and with set two, they are either all on or off together.
>
> I wrote the follow which works well, but looks
> pretty bad I think.
It doesn't look too bad, I've seen worse. :-)
> so again this is a style question...
> can anyone suggest a cleaner approach? TIA
The usual idiom is to use a hash for the search and replace tables.
> sub transform_characters {
> my @input = @_;
> my @output;
> for my $string (@input) {
> push @output, $string;
> if ($string =~ /\xDF/) {
> $string =~ s/\xDF/ss/g;
Using a match followed by a substitution is a usual beginner mistake.
You only need the substitution.
if ( $string =~ s/\xDF/ss/g ) {
> push @output, $string;
> if (test_for_character($string)) {
> $string = swap_all($string);
> push @output, $string;
> }
> next;
> }
> if (test_for_character($string)) {
> $string = swap_all($string);
> push @output, $string;
> }
> }
> return @output;
> }
>
> [snip code]
Using a hash you could write that as:
my %set1 = (
"\xDF" => 'ss',
);
# Use a character class because all keys are single characters
# If keys are multiple characters use alternation instead
my $key1 = '[' . join( '', keys %set1 ) . ']';
my %set2 = (
"\xC4" => 'Ae',
"\xD6" => 'Oe',
"\xDC" => 'Ue',
"\xE4" => 'ae',
"\xF6" => 'oe',
"\xFC" => 'ue',
);
my $key2 = '[' . join( '', keys %set2 ) . ']';
sub transform_characters {
my @input = @_;
my @output;
for my $string ( @input ) {
push @output, $string;
if ( $string =~ s/($key1)/$set1{$1}/og ) {
push @output, $string;
if ( $string =~ s/($key2)/$set2{$1}/og ) {
push @output, $string;
}
next;
}
if ( $string =~ s/($key2)/$set2{$1}/og ) {
push @output, $string;
}
}
return @output;
}
John
--
use Perl;
program
fulfillment
------------------------------
Date: Fri, 06 Aug 2004 21:35:18 +0200
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: transforming german characters
Message-Id: <2ni4r3F18ot3U1@uni-berlin.de>
steve_f wrote:
> I want to transform special German characters to obtain the
> following variations:
>
> groß bräu
> gross bräu
> gross braeu
>
> there are two sets -
>
> set one:
> ß = ss = \xDF
>
> set two:
> Ä = Ae = \xC4
> Ö = Oe = \xD6
> Ü = Ue = \xDC
> ä = ae = \xE4
> ö = oe = \xF6
> ü = ue = \xFC
>
> basically, the rules are transform ß independently
> and with set two, they are either all on or off together.
As John said, there is no reason to look for the characters with
separate regexes, and accordingly there is no reason to distinguish
between two sets.
> for my $string (@input) {
> push @output, $string;
Here you copy the whole original text to @output ...
> if ($string =~ /\xDF/) {
> $string =~ s/\xDF/ss/g;
> push @output, $string;
... and here you *add* the converted string. In the suggestion below,
I'm assuming that was a mistake.
sub transform_characters {
my @text = @_;
my %replace = (
"\xDF" => 'ss',
"\xC4" => 'Ae',
"\xD6" => 'Oe',
"\xDC" => 'Ue',
"\xE4" => 'ae',
"\xF6" => 'oe',
"\xFC" => 'ue',
);
for (@text) {
s/(\xDF|\xC4|\xD6|\xDC|\xE4|\xF6|\xFC)/$replace{$1}/g;
}
@text
}
my @output = transform_characters(@input);
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: Fri, 06 Aug 2004 20:13:29 -0400
From: steve_f <me@example.com>
Subject: Re: transforming german characters
Message-Id: <a778h05a3n2fc5h98tiephsijt21s31bl6@4ax.com>
Thanks Gunnar, some great stuff here....I can use simple
statements to just brute force things, but I know there is
a more elegent way.
On Fri, 06 Aug 2004 21:35:18 +0200, Gunnar Hjalmarsson <noreply@gunnar.cc> wrote:
>steve_f wrote:
>> I want to transform special German characters to obtain the
>> following variations:
>>
>> groß bräu
>> gross bräu
>> gross braeu
>>
>> there are two sets -
>>
>> set one:
>> ß = ss = \xDF
>>
>> set two:
>> Ä = Ae = \xC4
>> Ö = Oe = \xD6
>> Ü = Ue = \xDC
>> ä = ae = \xE4
>> ö = oe = \xF6
>> ü = ue = \xFC
>>
>> basically, the rules are transform ß independently
>> and with set two, they are either all on or off together.
>
>As John said, there is no reason to look for the characters with
>separate regexes, and accordingly there is no reason to distinguish
>between two sets.
The ß can either be on or off independent of the others so
you can get:
groß bräu
gross bräu
gross braeu
I should of stated the problem more directly:
if set one - set one on & set two on
set one off & set two on
set one off & set two off
if only set two - set two all on
- set two all off
>> for my $string (@input) {
>> push @output, $string;
>
>Here you copy the whole original text to @output ...
>
>> if ($string =~ /\xDF/) {
>> $string =~ s/\xDF/ss/g;
>> push @output, $string;
>
>... and here you *add* the converted string. In the suggestion below,
>I'm assuming that was a mistake.
>
> sub transform_characters {
> my @text = @_;
>
> my %replace = (
> "\xDF" => 'ss',
> "\xC4" => 'Ae',
> "\xD6" => 'Oe',
> "\xDC" => 'Ue',
> "\xE4" => 'ae',
> "\xF6" => 'oe',
> "\xFC" => 'ue',
> );
>
I really like the idea of the hash. Yes, I have heard you are not
thinking in Perl if you are not using hashes.
> for (@text) {
> s/(\xDF|\xC4|\xD6|\xDC|\xE4|\xF6|\xFC)/$replace{$1}/g;
> }
this is super! thanks
>
> @text
> }
>
> my @output = transform_characters(@input);
------------------------------
Date: Fri, 06 Aug 2004 20:46:30 -0400
From: steve_f <me@example.com>
Subject: Re: transforming german characters
Message-Id: <q988h01bb50q3ts2icpuq156o20clk6pmn@4ax.com>
Thank you John, this is really useful. Just to start, I must always remind
myself if I am doing something too many times to generalize.
>John W. Krahn wrote:
[ snip - my statement of problem ]
>>
>> I wrote the follow which works well, but looks
>> pretty bad I think.
>
>It doesn't look too bad, I've seen worse. :-)
>
I was able to brute force my way through it ;-)
>
>> so again this is a style question...
>> can anyone suggest a cleaner approach? TIA
>
>The usual idiom is to use a hash for the search and replace tables.
>
yes, I see and it is very good...changes the whole approach
>
>> sub transform_characters {
>> my @input = @_;
>> my @output;
>> for my $string (@input) {
>> push @output, $string;
>> if ($string =~ /\xDF/) {
>> $string =~ s/\xDF/ss/g;
>
>Using a match followed by a substitution is a usual beginner mistake.
>You only need the substitution.
>
> if ( $string =~ s/\xDF/ss/g ) {
>
ahh...ok...that's good to learn
[ snip code ]
>
>Using a hash you could write that as:
>
>my %set1 = (
> "\xDF" => 'ss',
> );
># Use a character class because all keys are single characters
># If keys are multiple characters use alternation instead
can you explain this a bit further? I'm not quite sure what you mean
by alternation, but I really only looked up the escaped values for
this particular problem.
>my $key1 = '[' . join( '', keys %set1 ) . ']';
also here I start to get really lost....ok, you are loading into a scalar
the keys as one long string...joining them with no space between...
with two brackets so
$key1 = [\xDF]
$key2 = [\xC4\xD6\xDC\xE4\xF6\xFC]
correct?
I see you use it down below in this substitution but it is a bit hard
for me to understand:
if ( $string =~ s/($key1)/$set1{$1}/og )
well, if you have the time please give me a bit more clarrification
on this because I haven't seen it before.
>
>my %set2 = (
> "\xC4" => 'Ae',
> "\xD6" => 'Oe',
> "\xDC" => 'Ue',
> "\xE4" => 'ae',
> "\xF6" => 'oe',
> "\xFC" => 'ue',
> );
>my $key2 = '[' . join( '', keys %set2 ) . ']';
>
>sub transform_characters {
> my @input = @_;
> my @output;
> for my $string ( @input ) {
> push @output, $string;
> if ( $string =~ s/($key1)/$set1{$1}/og ) {
> push @output, $string;
> if ( $string =~ s/($key2)/$set2{$1}/og ) {
> push @output, $string;
> }
> next;
> }
> if ( $string =~ s/($key2)/$set2{$1}/og ) {
> push @output, $string;
> }
> }
> return @output;
> }
>
>
>
>John
Thanks again John.
Steve
------------------------------
Date: Fri, 6 Aug 2004 18:11:16 +0000 (UTC)
From: "Bryan K. Wright" <bryan@ayesha.phys.Virginia.EDU>
Subject: Unbuffered piped command output?
Message-Id: <cf0hk4$rd6$1@murdoch.acc.Virginia.EDU>
Keywords: unbuffered autoflush flush pipe
Hi folks,
The following seems simple, but it's driving me crazy.
I'd like to create a command pipe with open, like this:
open ( CMD, "mycommand |");
then print out each line of the command's output AS IT HAPPENS.
Here's what I've tried:
#!/usr/bin/perl
use strict;
use FileHandle;
STDOUT->autoflush(1);
open ( COMMAND, "mycommand |" ) or die "Can't run command: $!\n";
COMMAND->autoflush(1);
while (<COMMAND>) {
print;
}
close ( COMMAND );
But the command's output is still buffered until the command
finishes. What am I doing wrong here?
Thanks,
Bryan
--
===============================================================================
Bryan Wright |"If you take cranberries and stew them like
Physics Department | applesauce, they taste much more like prunes
University of Virginia | than rhubarb does." -- Groucho
Charlottesville, VA 22901 |
(434) 924-7218 | bryan@virginia.edu
===============================================================================
------------------------------
Date: 06 Aug 2004 14:27:44 -0400
From: Scott W Gifford <gifford@umich.edu>
Subject: Re: Unbuffered piped command output?
Message-Id: <qsz4qng3yvz.fsf@rygar.gpcc.itd.umich.edu>
"Bryan K. Wright" <bryan@ayesha.phys.Virginia.EDU> writes:
> Hi folks,
>
> The following seems simple, but it's driving me crazy.
> I'd like to create a command pipe with open, like this:
>
> open ( CMD, "mycommand |");
>
> then print out each line of the command's output AS IT HAPPENS.
A discussion of this exact problem is going on right now in the thread
with subject:
Input from subprocess called using open() buffered?
The answers there may be helpful.
-----ScottG.
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 6843
***************************************
