[18531] in Perl-Users-Digest
Perl-Users Digest, Issue: 699 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sat Apr 14 18:05:44 2001
Date: Sat, 14 Apr 2001 15:05:11 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Message-Id: <987285911-v10-i699@ruby.oce.orst.edu>
Content-Type: text
Perl-Users Digest Sat, 14 Apr 2001 Volume: 10 Number: 699
Today's topics:
Re: A CGI question <flavell@mail.cern.ch>
Re: A CGI question <gellyfish@gellyfish.com>
Easy Perl Question... <robbie@totaln.com>
Re: Easy Perl Question... <jboes@qtm.net>
FAQ .: Where to get this document <faq@denver.pm.org>
File upload basics trouble <pric3596@cs.uidaho.edu>
Re: help needed with sorting Hash (Alan Barclay)
How can u group together fields and sum them up (sdfsd)
Re: How can u group together fields and sum them up (Mark Jason Dominus)
Re: How can u group together fields and sum them up (Eric Bohlman)
Re: How can u group together fields and sum them up <keesh@users.pleaseremovethisbit.sourceforge.net>
Re: How can u group together fields and sum them up (Logan Shaw)
Re: How can u group together fields and sum them up (Tad McClellan)
Re: How can u group together fields and sum them up <dan@tuatha.sidhe.org>
Re: How can u group together fields and sum them up (Logan Shaw)
Re: How can u group together fields and sum them up (sdfsd)
Re: How can u group together fields and sum them up <uri@sysarch.com>
Re: How can u group together fields and sum them up (Abigail)
Re: How can u group together fields and sum them up (Mark Jason Dominus)
Re: Including common code as text in many scripts <postmaster@hollandnumerics.co.uk>
need help cgi/perl/form variables <sigi@mail2me.de>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 14 Apr 2001 20:22:38 +0200
From: "Alan J. Flavell" <flavell@mail.cern.ch>
Subject: Re: A CGI question
Message-Id: <Pine.LNX.4.30.0104142021450.7696-100000@lxplus003.cern.ch>
On Sat, 14 Apr 2001, Mario Rizzuti wrote in response to:
> brian d foy <comdog@panix.com>
[...]
> [ignored]
[plonked]
------------------------------
Date: 14 Apr 2001 20:44:49 GMT
From: Jonathan Stowe <gellyfish@gellyfish.com>
Subject: Re: A CGI question
Message-Id: <9bacs1$2d2$1@plutonium.btinternet.com>
Mario Rizzuti <diab.lito@usa.net> wrote:
>
> brian d foy <comdog@panix.com> wrote in message
> comdog-D53754.23112511042001@news.panix.com...
>
>> which is why i said you shouldn't post it. if you can't include the
>> necessary items, don't post. don't allow your bad answers to pollute
>> the newsgroup archives. don't make other people respond to you so
>> that the next poor soul doesn't think that you know what you are
>> doing and wastes his time with your answer. don't pretend to know
>> more than you do. don't think that the number of posts you make has
>> any affect on your cluefulness. don't post code without testing it.
>> don't post code without understanding it. don't post.
>
> [ignored]
>
[killfiled]
/J\
--
Jonathan Stowe |
<http://www.gellyfish.com> | This space for rent
|
------------------------------
Date: Sat, 14 Apr 2001 21:17:23 GMT
From: "RobbieB" <robbie@totaln.com>
Subject: Easy Perl Question...
Message-Id: <D73C6.26102$PF4.42458@news.iol.ie>
Hi,
I am a newbie to perl. Basically, I wanted to know, if I can create a Link,
in a script, to another part of that script? Use a form in a script, have it
parsed by that script, and then the content of the form used by that script?
Thanks for any help you guys give me.
--Robbie Burke
------------------------------
Date: Sat, 14 Apr 2001 17:57:49 -0400
From: Jeff Boes <jboes@qtm.net>
Subject: Re: Easy Perl Question...
Message-Id: <6hhhdtc8q1i27c3gmih4ns8ml2to9q1j6l@4ax.com>
At some point in time, "RobbieB" <robbie@totaln.com> wrote:
>I am a newbie to perl. Basically, I wanted to know, if I can create a Link,
>in a script, to another part of that script? Use a form in a script, have it
>parsed by that script, and then the content of the form used by that script?
Your question isn't very clear, but I'll give it a shot.
By "link" and "form", it appears you are talking about a CGI script (as opposed
to some other kind of Perl program). Ergo, you probably want to know about:
CGI Programming FAQ <http://htmlhelp.com/faq/cgifaq.html>
Perl CGI Tutorial <http://agora.leeds.ac.uk/Perl/Cgi/start.html>
The Idiot's Guide to Solving Perl CGI Problems
<ftp://ftp.epix.net/pub/languages/perl/doc/FAQs/cgi/idiots-guide.html>
(I haven't checked these recently to make sure they are all active links.)
It's a very common thing to have a single CGI script present a form, and parse
the results. Usually it looks like this:
#!/usr/local/bin/perl -wT
use strict;
use CGI qw(:standard);
my $cgi = new CGI;
if ($cgi-param()) {
# do something with form contents here
} else {
# print the form here
}
In the printing of the form you will include something like
print start_form(-action=>'/cgi-bin/path/to/this/script.cgi');
--
~~~~~~~~~~~~~~~~|It is by caffeine alone I set my mind in motion,
Jeffery Boes |It is by the beans of Java that thoughts acquire speed,
jboes@qtm.net |The hands acquire shaking, the shaking becomes a warning,
UIN 3394914 |It is by caffeine alone I set my mind in motion.
------------------------------
Date: Sat, 14 Apr 2001 18:17:02 GMT
From: PerlFAQ Server <faq@denver.pm.org>
Subject: FAQ .: Where to get this document
Message-Id: <yu0C6.972$T3.193142784@news.frii.net>
This message is one of several periodic postings to comp.lang.perl.misc
intended to make it easier for perl programmers to find answers to
common questions. The core of this message represents an excerpt
from the documentation provided with every Standard Distribution of
Perl.
+
Where to get this document
This document is posted regularly to comp.lang.perl.announce and several
other related newsgroups. It is available in a variety of formats from
CPAN in the /CPAN/doc/FAQs/FAQ/ directory, or on the web at
http://www.perl.com/perl/faq/ .
-
Documents such as this have been called "Answers to Frequently
Asked Questions" or FAQ for short. They represent an important
part of the Usenet tradition. They serve to reduce the volume of
redundant traffic on a news group by providing quality answers to
questions that keep comming up. If you are some how irritated by
seeing these postings you are free to ignore them or add the sender
to your killfile. If you find errors or other problems with these
postings please send corrections or comments to the posting email
address.
If you are not able to find this or other Perl documentation from
your installation you may access it via the web by following the
appropriate links from one of the addresses listed below.
http://theoryx5.uwinnipeg.ca/mod_perl/cpan-search
http://www.perldoc.com
http://www.cpan.org
http://www.perl.com
Answers to questions about LOTS of other stuff, mostly not related to
Perl, can be found at
news:news.answers
and in the many thousands of other useful Usenet news groups.
Please note that the FAQ text posted by this server has been modified
from that distributed in the stable Perl release. It has been
edited to reflect the additions, changes and corrections provided
by respondents, reviewers, and critics to previous postings of
these FAQ. Complete text of these FAQ is available on request.
The perlfaq manual pages contain the following copyright notice.
AUTHOR AND COPYRIGHT
Copyright (c) 1997-1999 Tom Christiansen and Nathan
Torkington. All rights reserved.
When included as an integrated part of the Standard
Distribution of Perl or of its documentation (printed or
otherwise), this work is covered under Perl's Artistic
License. For separate distributions of all or part of
this FAQ outside of that, see the perlfaq manpage.
Irrespective of its distribution, all code examples here
are public domain. You are permitted and encouraged to
use this code and any derivatives thereof in your own
programs for fun or for profit as you see fit. A simple
comment in the code giving credit to the FAQ would be
courteous but is not required.
This work is provided in the hope that it will be useful but does
not represent a commitment of any kind on the part of the contributers,
authors or their agents.
--
This space intentionally left blank
------------------------------
Date: Sat, 14 Apr 2001 13:39:00 -0700
From: "Preston Price" <pric3596@cs.uidaho.edu>
Subject: File upload basics trouble
Message-Id: <9bacas$3qj$1@kestrel.csrv.uidaho.edu>
I got this code out of a book and most of the stuff in it works, however I
can seem to figure out this one.
1>#!usr/local/bin/perl
2>#upload1.cgi
3>use strict;
4>use CGI qw(:standard);
5>use CGI::Carp qw(fatalsToBrowser);
6>my $File_Name = param('fileName');
7>my $Mime = uploadInfo($File_Name)->{'Content-Type'};
8>$CGI::POST_MAX = 1024 * 250;
9>Print_Results();
10>sub Print_Results{
11> print header;
12> print start_html('File Upload Example 1');
13> print qq(<PRE><B>File Name: </B> $File_Name\n);
14> print qq(<B>Mime:</B> $Mime\n);
15> print qq(<B>File Contents:</B><XMP>\n\n);
16> while(<$File_Name>) {print; }
17> print qq(</XMP></PRE>);
18> print end_html;
19>}
The error that I get when I try to compile it is: "Can't use an undefined
value as a HASH reference at upload1.cgi line 10."
I would be greatful if anyone knows what is happening here. Thanks in
advance!
------------------------------
Date: 14 Apr 2001 20:38:20 GMT
From: gorilla@elaine.furryape.com (Alan Barclay)
Subject: Re: help needed with sorting Hash
Message-Id: <987280700.227911@elaine.furryape.com>
In article <3AD5F51D.1A5C53B5@netscape.net>,
kelli norman <kellikellin@netscape.net> wrote:
>such friendly people out there!
>
>there is no way to actually make a hash have it's keys in sorted order,
>but if you just want to print the values in order:
Actually there is. Use the Tie::IxHash module.
------------------------------
Date: 14 Apr 2001 13:51:01 -0500
From: sdfes@dsf.com (sdfsd)
Subject: How can u group together fields and sum them up
Message-Id: <3ad89b62$0$66056$45beb828@newscene.com>
We have a unix file with 100 million lines (broken into 12 files of 8 million
each)
fields are as follows:
acct# type1 amt1 type2 amt2 type3 amt3 c_type4 amt4
125 x2 34 x41 43 x14 76 x54 54
each line has unique types with a corresponding amout. however sometimes
(around 1-2 %) we get duplicate types with their amouts as follows
125 x2 34 x41 43 x2 76 x41 54 x14 65
in that case we need to combine the duplicate types, summing up their
corresponding amounts.
our output should be normalized that is a line for each type:
125 x2 110
125 x41 97
125 x14 65
with any dupes having been combined and summed
how can we (or can we?) do this in Perl?
Speed IS important. Currently a vendor sorting program is doing it but it
takes 15 hours. We need to this very fast. I asked in the AWK group but they
passed due to speed issue
p.s. there are no spaces between fields and its one rec per line.
hardware is Sun 10k, 8 gigs ram 900 gigs HD, 10 CPUS. Sun 5.6 is OS.
------------------------------
Date: Sat, 14 Apr 2001 19:07:27 GMT
From: mjd@plover.com (Mark Jason Dominus)
Subject: Re: How can u group together fields and sum them up
Message-Id: <3ad89ff5.62af$1d8@news.op.net>
In article <3ad89b62$0$66056$45beb828@newscene.com>,
sdfsd <sdfes@dsf.com> wrote:
>fields are as follows:
>125 x2 34 x41 43 x14 76 x54 54
>
>how can we (or can we?) do this in Perl?
I wrote a prgoram to do it, but I don't think it will work for you,
because my program assumes that the fields have spaces in between, as
you showed in your example. But then you said:
>p.s. there are no spaces between fields and its one rec per line.
So I have no idea what the data format is.
>Speed IS important. Currently a vendor sorting program is doing it but it
>takes 15 hours.
Your vendor is incompetent. My Perl program (seven lines long)
processed 20,000 records in 10.725 seconds. This is 100 million
records in 15 hours, the same as your vendor---but my program was
running on a Pentium 233. Using your modern hardware should cut this
by an order of magnitude.
Too bad you didn't supply your email address.
--
@P=split//,".URRUU\c8R";@d=split//,"\nrekcah xinU / lreP rehtona tsuJ";sub p{
@p{"r$p","u$p"}=(P,P);pipe"r$p","u$p";++$p;($q*=2)+=$f=!fork;map{$P=$P[$f^ord
($p{$_})&6];$p{$_}=/ ^$P/ix?$P:close$_}keys%p}p;p;p;p;p;map{$p{$_}=~/^[P.]/&&
close$_}%p;wait until$?;map{/^r/&&<$_>}%p;$_=$d[$q];sleep rand(2)if/\S/;print
------------------------------
Date: 14 Apr 2001 19:07:08 GMT
From: ebohlman@omsdev.com (Eric Bohlman)
Subject: Re: How can u group together fields and sum them up
Message-Id: <9ba74s$4g$1@bob.news.rcn.net>
sdfsd <sdfes@dsf.com> wrote:
> We have a unix file with 100 million lines (broken into 12 files of 8 million
> each)
> fields are as follows:
> acct# type1 amt1 type2 amt2 type3 amt3 c_type4 amt4
> 125 x2 34 x41 43 x14 76 x54 54
> each line has unique types with a corresponding amout. however sometimes
> (around 1-2 %) we get duplicate types with their amouts as follows
> 125 x2 34 x41 43 x2 76 x41 54 x14 65
>
> in that case we need to combine the duplicate types, summing up their
> corresponding amounts.
> our output should be normalized that is a line for each type:
> 125 x2 110
> 125 x41 97
> 125 x14 65
> with any dupes having been combined and summed
> how can we (or can we?) do this in Perl?
Very easily, because Perl has a wonderful data structure called a hash
(read up on hashes in the perldata manpage) which is exactly what you need
for this type of problem (actually, you'll need a "hash of hashes" which
is implemented using references so you'll want to look at perlreftut and
perlref as well).
> Speed IS important. Currently a vendor sorting program is doing it but it
> takes 15 hours. We need to this very fast. I asked in the AWK group but they
> passed due to speed issue
A hash based solution should be *much* faster than a sort-based solution
because it needs to look at each line only once, whereas sorting requires
each line to be looked at approximately 26 times (assuming a good sort
algorithm). It's likely that better than 95% of the time taken by a
hash-based solution will be file I/O time.
------------------------------
Date: Sat, 14 Apr 2001 20:21:49 +0100
From: "Ciaran McCreesh" <keesh@users.pleaseremovethisbit.sourceforge.net>
Subject: Re: How can u group together fields and sum them up
Message-Id: <9ba3m1$asa$1@news7.svr.pol.co.uk>
In article <3ad89ff5.62af$1d8@news.op.net>, mjd@plover.com wrote:
> Your vendor is incompetent. My Perl program (seven lines long)
> processed 20,000 records in 10.725 seconds. This is 100 million records
> in 15 hours, the same as your vendor---but my program was running on a
> Pentium 233. Using your modern hardware should cut this by an order of
> magnitude.
Is it a linear relationship? I would have thought that the more items
there are in a hash, the longer any operation on an item would take. More
items, more search time, isn't it?
I don't know enough about the internals of Perl to be sure, but I doubt
that it's that straightforward. Then again, 15 hours is still very poor.
Ciaran
--
Ciaran McCreesh
mail: keesh@users.sourceforge.net
web: http://www.opensourcepan.com/
------------------------------
Date: 14 Apr 2001 14:38:03 -0500
From: logan@cs.utexas.edu (Logan Shaw)
Subject: Re: How can u group together fields and sum them up
Message-Id: <9ba8ur$a7q$1@charity.cs.utexas.edu>
In article <3ad89b62$0$66056$45beb828@newscene.com>,
sdfsd <sdfes@dsf.com> wrote:
>We have a unix file with 100 million lines (broken into 12 files of 8 million
>each)
>
>fields are as follows:
>
>acct# type1 amt1 type2 amt2 type3 amt3 c_type4 amt4
>125 x2 34 x41 43 x14 76 x54 54
>
>each line has unique types with a corresponding amout. however sometimes
>(around 1-2 %) we get duplicate types with their amouts as follows
>
>125 x2 34 x41 43 x2 76 x41 54 x14 65
>
>in that case we need to combine the duplicate types, summing up their
>corresponding amounts.
>
>our output should be normalized that is a line for each type:
>
>125 x2 110
>125 x41 97
>125 x14 65
>
>with any dupes having been combined and summed
>
>how can we (or can we?) do this in Perl?
That's easy enough. Here's a first crack at it:
while (<>)
{
chomp; # remove newlines
@fields = split; # split on whitespace
$acct = shift @fields; # get the acct#
# build an associative array of types and amounts
%fields = ();
while ($key = shift @fields)
{
$fields{$key} += shift @fields;
}
# print the array in a random order
while (($key, $value) = each %fields)
{
print join ("\t", $acct, $key, $value), "\n";
}
}
This solution assumes that the output lines can occur in any order,
i.e. that the order the key/value pairs appear on the input line isn't
significant. It also doesn't have much error checking. It also
uses global variables, so don't look at it as an example of good style.
>Speed IS important. Currently a vendor sorting program is doing it but it
>takes 15 hours. We need to this very fast.
Sorting program? What kind of sorting is there? You just need to
change the way each line looks, right?
I don't see why it should be so slow. I created a sample input
file with a million lines of this:
125 x2 34 x41 43 x2 76 x41 54 x14 65
and ran the above Perl code on it. I got a million repetitions of
this:
125 x2 110
125 x14 65
125 x41 97
And it took 91 seconds on my 600 MHz Athlon (running Solaris 8) using
perl 5.6.0. That works out to about 11,000 lines a second, which means
on my home machine, I could do the whole thing in about 2.5 hours,
assuming I had the disk space and assuming that the sample data I
used was a reasonable representation.
Of course, if you really want speed, write a C++ program. It's not
that hard, especially since you can use STL types like "list" and "map"
to store everything. My guess is that even a somewhat sloppy C++
implementation would be fast enough that I/O would become the
bottleneck.
>p.s. there are no spaces between fields and its one rec per line.
There aren't? It sure looks like there are in the examples you gave.
Or do you mean blank lines?
>hardware is Sun 10k, 8 gigs ram 900 gigs HD, 10 CPUS. Sun 5.6 is OS.
Great, you can process all 12 files at once, so you should be able to
do the whole thing a lot quicker. Unfortunately, since you have a
small number of jobs, some might complete earlier than others, but
there's no easy solution to that. I'd run all 12 jobs at once since
they're going to partly be I/O bound, and you don't want those
processors sitting idle.
- Logan
--
my your his her our their _its_
I'm you're he's she's we're they're _it's_
------------------------------
Date: Sat, 14 Apr 2001 14:46:32 -0400
From: tadmc@augustmail.com (Tad McClellan)
Subject: Re: How can u group together fields and sum them up
Message-Id: <slrn9dh6o8.jma.tadmc@tadmc26.august.net>
Ciaran McCreesh <keesh@users.pleaseremovethisbit.sourceforge.net> wrote:
>In article <3ad89ff5.62af$1d8@news.op.net>, mjd@plover.com wrote:
>> Your vendor is incompetent. My Perl program (seven lines long)
>> processed 20,000 records in 10.725 seconds. This is 100 million records
>> in 15 hours, the same as your vendor---but my program was running on a
>> Pentium 233. Using your modern hardware should cut this by an order of
>> magnitude.
>
>Is it a linear relationship?
Yes.
>I would have thought that the more items
>there are in a hash, the longer any operation on an item would take. More
>items, more search time, isn't it?
No, not with a hash data structure (sans lots of collisions).
>I don't know enough about the internals of Perl to be sure, but I doubt
>that it's that straightforward.
The "hash data structure" is standard Computer Science fare.
The reason it is covered in CS curriculums is because it is *designed*
to be O(1) (that's fancy college talk for "same access time no matter
how many items").
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Sat, 14 Apr 2001 19:44:46 GMT
From: Dan Sugalski <dan@tuatha.sidhe.org>
Subject: Re: How can u group together fields and sum them up
Message-Id: <OM1C6.2192$Ce4.215108@news1.rdc1.ct.home.com>
Ciaran McCreesh <keesh@users.pleaseremovethisbit.sourceforge.net> wrote:
> In article <3ad89ff5.62af$1d8@news.op.net>, mjd@plover.com wrote:
>> Your vendor is incompetent. My Perl program (seven lines long)
>> processed 20,000 records in 10.725 seconds. This is 100 million records
>> in 15 hours, the same as your vendor---but my program was running on a
>> Pentium 233. Using your modern hardware should cut this by an order of
>> magnitude.
> Is it a linear relationship? I would have thought that the more items
> there are in a hash, the longer any operation on an item would take. More
> items, more search time, isn't it?
Not really, no. There's not a whole lot of searching that's done within
hashes--that's one of the main points of hashes in general.
What perl does is take the key and apply a hashing function to it, which
returns an integer that's used as the offset into an array. As long as
there are no collisions (you use multiple keys that get the same hash value),
access time is constant. If there are collisions then access time is linear
relative to the number of other keys that hash to the value. Perl's hashing
function's generally pretty good so you won't normally see many collisions,
even on a largish hash, and when you do see collisions they're generally
spread evenly across the hash. (Unless you're unlucky enough to have some
degenerate data that hashes really badly)
Dan
------------------------------
Date: 14 Apr 2001 14:45:39 -0500
From: logan@cs.utexas.edu (Logan Shaw)
Subject: Re: How can u group together fields and sum them up
Message-Id: <9ba9d3$a9d$1@charity.cs.utexas.edu>
In article <9ba3m1$asa$1@news7.svr.pol.co.uk>,
Ciaran McCreesh <keesh@users.pleaseremovethisbit.sourceforge.net> wrote:
>In article <3ad89ff5.62af$1d8@news.op.net>, mjd@plover.com wrote:
>> Your vendor is incompetent. My Perl program (seven lines long)
>> processed 20,000 records in 10.725 seconds. This is 100 million records
>> in 15 hours, the same as your vendor---but my program was running on a
>> Pentium 233. Using your modern hardware should cut this by an order of
>> magnitude.
>
>Is it a linear relationship? I would have thought that the more items
>there are in a hash, the longer any operation on an item would take. More
>items, more search time, isn't it?
Granted, the OP's description wasn't totally clear, but it seemed like
to me that this was a task with 100,000,000 separate computational
problems, i.e. each line of input is completely unrelated to all the
others. In that case, all you have to worry about is whether a hash
will perform well for the number of items on one line, which presumably
it would.
By the way, you're right that there is a greater search time for more
items, but the difference should not be that large. The whole point of
a hash is to make lookup time nearly constant, so that adding extra
elements will not cause the hash to get much slower. In practice,
this isn't really possible (bigger hashes are more likely to cause
cache misses, for example), but a hash should come close.
In general, the big problem with hashes is if you don't know in advance
how many items you need to put in them, because if they start to get
too full, they slow down and sooner or later you need to copy your hash
to a new, larger one to avoid this problem. I don't know the specific
mechanism perl uses to deal with this, but since it's such an important
core language feature, I'd imagine it's pretty good. (Still, it can
only be *so* good, because there are fundamental limitations.)
- Logan
--
my your his her our their _its_
I'm you're he's she's we're they're _it's_
------------------------------
Date: 14 Apr 2001 14:50:08 -0500
From: sdfes@dsf.com (sdfsd)
Subject: Re: How can u group together fields and sum them up
Message-Id: <3ad8a938$0$66056$45beb828@newscene.com>
In article <3ad89ff5.62af$1d8@news.op.net>, mjd@plover.com (Mark Jason Dominus) wrote:
>In article <3ad89b62$0$66056$45beb828@newscene.com>,
>
>I wrote a prgoram to do it, but I don't think it will work for you,
>because my program assumes that the fields have spaces in between, as
>you showed in your example. But then you said:
>
>>p.s. there are no spaces between fields and its one rec per line.
>
>So I have no idea what the data format is.
it is fixed lenght each line is 256 characters long with each field taking up
a fixed amount of space. the types are all there 9 of them, but some have 0
for amount. It may be possible to add a delimiter.
one other req is for the program to grab one of the fields and use it to do a
lookup to a table (its a file that can be load into mem).
>
>>Speed IS important. Currently a vendor sorting program is doing it but it
>>takes 15 hours.
>
>Your vendor is incompetent. My Perl program (seven lines long)
>processed 20,000 records in 10.725 seconds. This is 100 million
>records in 15 hours, the same as your vendor---but my program was
>running on a Pentium 233. Using your modern hardware should cut this
>by an order of magnitude.
>
>Too bad you didn't supply your email address.
>
------------------------------
Date: Sat, 14 Apr 2001 20:54:53 GMT
From: Uri Guttman <uri@sysarch.com>
Subject: Re: How can u group together fields and sum them up
Message-Id: <x7y9t3kxr6.fsf@home.sysarch.com>
>>>>> "s" == sdfsd <sdfes@dsf.com> writes:
>> So I have no idea what the data format is.
s> it is fixed lenght each line is 256 characters long with each field
s> taking up a fixed amount of space. the types are all there 9 of
s> them, but some have 0 for amount. It may be possible to add a
s> delimiter.
use unpack for fixed width fields.
s> one other req is for the program to grab one of the fields and use
s> it to do a lookup to a table (its a file that can be load into
s> mem).
repost with an example of that and clarify the operations you need on
each record.
uri
--
Uri Guttman --------- uri@sysarch.com ---------- http://www.sysarch.com
SYStems ARCHitecture and Stem Development ------ http://www.stemsystems.com
Learn Advanced Object Oriented Perl from Damian Conway - Boston, July 10-11
Class and Registration info: http://www.sysarch.com/perl/OOP_class.html
------------------------------
Date: Sat, 14 Apr 2001 21:57:28 +0000 (UTC)
From: abigail@foad.org (Abigail)
Subject: Re: How can u group together fields and sum them up
Message-Id: <slrn9dhhu8.j76.abigail@tsathoggua.rlyeh.net>
Tad McClellan (tadmc@augustmail.com) wrote on MMDCCLXXXIII September
MCMXCIII in <URL:news:slrn9dh6o8.jma.tadmc@tadmc26.august.net>:
][ Ciaran McCreesh <keesh@users.pleaseremovethisbit.sourceforge.net> wrote:
][ >In article <3ad89ff5.62af$1d8@news.op.net>, mjd@plover.com wrote:
][ >> Your vendor is incompetent. My Perl program (seven lines long)
][ >> processed 20,000 records in 10.725 seconds. This is 100 million records
][ >> in 15 hours, the same as your vendor---but my program was running on a
][ >> Pentium 233. Using your modern hardware should cut this by an order of
][ >> magnitude.
][ >
][ >Is it a linear relationship?
][
][ Yes.
][
Yes, but it's not so simple.
20,000 records might easily fit in RAM, and the program wouldn't need
to do much swapping. With 100 million records (assuming a relative low
number of duplicates), it's very likely the program starts swapping like
a mad man, and will even die part of the way if you don't have enough
swap space.
If you would make a graph of size vs running time, you're likely to
see something that resembles two line segments, sharing a common point.
And right of the common point, the line segment is much steeper than on
the left. The common point of course indicates where the program starts
to swap.
Abigail
------------------------------
Date: Sat, 14 Apr 2001 22:03:13 GMT
From: mjd@plover.com (Mark Jason Dominus)
Subject: Re: How can u group together fields and sum them up
Message-Id: <3ad8c927.6684$352@news.op.net>
In article <slrn9dhhu8.j76.abigail@tsathoggua.rlyeh.net>,
Abigail <abigail@foad.org> wrote:
>][ >Is it a linear relationship?
>][
>][ Yes.
>][
>
>Yes, but it's not so simple.
Perhaps I misunderstood the task, but the program I wrote has a fixed
memory requirement.
The question turns on whether, given the following two records of input:
125 x2 34 x41 43 x14 76 x54 54
125 x2 34 x41 43 x2 76 x41 54 x14 65
should the output contain seven records, or only four?
My understanding of the original question was that the output should
contain seven records: Four from the first input record, and
three from the second.
--
@P=split//,".URRUU\c8R";@d=split//,"\nrekcah xinU / lreP rehtona tsuJ";sub p{
@p{"r$p","u$p"}=(P,P);pipe"r$p","u$p";++$p;($q*=2)+=$f=!fork;map{$P=$P[$f^ord
($p{$_})&6];$p{$_}=/ ^$P/ix?$P:close$_}keys%p}p;p;p;p;p;map{$p{$_}=~/^[P.]/&&
close$_}%p;wait until$?;map{/^r/&&<$_>}%p;$_=$d[$q];sleep rand(2)if/\S/;print
------------------------------
Date: Sat, 14 Apr 2001 20:39:33 +0100
From: Phil Holland <postmaster@hollandnumerics.co.uk>
Subject: Re: Including common code as text in many scripts
Message-Id: <5iEZwBA1dK26Ewlo@hollandnumerics.demon.co.uk>
In article <comdog-9F3E45.17582913042001@news.panix.com>, brian d foy
<comdog@panix.com> writes
>In article <GEgdtCA8N216Ew5C@hollandnumerics.demon.co.uk>, Phil Holland
><postmaster@hollandnumerics.co.uk> wrote:
>
>> I have a number of similar Perl5 scripts where about 80% of the code is
>> common to all of them. The code includes a large number of variables
>> that are globally scoped to cover all of each script, so my attempts to
>> use 'require' have failed with undefined variable error in the included
>> code (which, by chance, is made up of a collection of subroutines that
>> use some of these global variables).
>
>can you show us a small example of what you are doing? you are on
>the right track but maybe missed a detail.
>
>
>> What I would like to be able to do is put all the common code in a
>> separate text file, but include the text in all of my scripts prior to
>> compilation. This would reduce my maintenance effort by at least 80%!!
>
>absolutely. when you talk to people about this, heavy use of
>the word "refactoring" will make you sound like a programming
>guru :)
>
Programming guru has a nice ring to it, but, thanks to everyone who
offered suggestions, I seem to have solved the problem.
The original code looked something like this:
[in program.cgi]
...
my $g_var = 1;
...
require "subprogram.pl";
...
sub sub1 {
...
$temp1 = $g_var;
$temp2 = &sub2($temp1);
...
}
[in subprogram.pl]
sub sub2 {
my ($temp3) = @_;
...
$test2 = $g_var;
return 5;
}
1
This code gave numerous errors due to $g_var not being initialized in
subprogram.pl.
The fixed code has:
- the subroutines in subprogram.pm
- all the 'global' variables declared as $main::xxx
and it seems to work...
The fixed code looks like this:
[in program.cgi]
...
$main::g_var = 1;
...
require subprogram;
...
sub sub1 {
...
$temp1 = $main::g_var;
$temp2 = &sub2($temp1);
...
}
[in subprogram.pm]
sub sub2 {
my ($temp3) = @_;
...
$test2 = $main::g_var;
return 5;
}
1
Thanks to everyone for their assistance in saving loads of maintenance
effort.
..............Phil
--
--------------------------------------------------------
Holland Numerics Ltd
94 Green Drift, Royston, Herts SG8 5BT, UK
mobile : +44-(0)7714-279085
email : <NEWS@hollandnumerics.com>
URL : http://www.hollandnumerics.com/
--------------------------------------------------------
------------------------------
Date: Sat, 14 Apr 2001 22:12:25 +0200
From: Sigi <sigi@mail2me.de>
Subject: need help cgi/perl/form variables
Message-Id: <Pine.BSF.4.05.10104142202570.89680-100000@manfred.isk.de>
Hello
I have a problem with an cgi perl script.
normally:
form.htm:
<form method=post action=/cgi-bin/check.pl>
<input name=NAME>
<input type=submit>
</form>
check.pl:
get all form variables (I use CGI.pm)
check and work on the variables
like NAME =~ tr/a-z/A-Z/; (silly sample)
redirect with
print "Location: other.htm?NAME=tested&NEXT=new ..."
other.htm:
show checked and new variables
--------------------
I want to do:
form.htm:
<form method=post action=/cgi-bin/other.htm>
<input name=NAME>
<input type=submit>
</form>
other.htm:
<!--#exec=/cgi-bin/check.pl-->
or on roxen/caudium: <cgi script=/cgi-bin/check.pl>
show checked and new variables
check.pl:
get all form variables (I use CGI.pm)
check and work on the variables
like NAME =~ tr/a-z/A-Z/; (silly sample)
give all form variables back to form environment
---------------------
Is there any way to do that?
Why I want that? ;-)
In the normally way I get URLs like
http://..../?NAME=tested&NEXT=new... .
Using the second way doesn't show
such ugly URLs.
Many Thanks for help
and for understanding my bad english
Sigi
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 699
**************************************
