[11342] in Perl-Users-Digest
Perl-Users Digest, Issue: 4941 Volume: 8
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Fri Feb 19 15:07:49 1999
Date: Fri, 19 Feb 99 12:00:30 -0800
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Fri, 19 Feb 1999 Volume: 8 Number: 4941
Today's topics:
Re: -T warning explanation (Greg Ward)
Re: Apache module to do Authentication/Authorization ag jsheffie@my-dejanews.com
Re: Apache module to do Authentication/Authorization ag jsheffie@tivoli.com
Breaking the 1K DBM barrier <breger@nwu.edu>
can I run perl on Win 98?????? t_alter@hotmail.com
exchanging info between two programs <jabbott@dingo.smig.net>
Re: exchanging info between two programs <tchrist@mox.perl.com>
FAQ 8.15: How do I set the time and date? <perlfaq-suggestions@perl.com>
FAQ 8.16: How can I sleep() or alarm() for under a seco <perlfaq-suggestions@perl.com>
FAQ 8.20: How can I call my system's unique C functions <perlfaq-suggestions@perl.com>
FAQ 8.21: Where do I get the include files to do ioctl( <perlfaq-suggestions@perl.com>
FMTEYEWTK: Sorting <tchrist@mox.perl.com>
Special: Digest Administrivia (Last modified: 12 Dec 98 (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: 19 Feb 1999 18:22:33 GMT
From: gward@cnri.reston.va.us (Greg Ward)
Subject: Re: -T warning explanation
Message-Id: <7aka59$2ve$3@news0-alterdial.uu.net>
Sanjay Malunjkar <smalunjk@cisco.com> wrote:
> I am getting following error message at the open (MAIL....) statement:
>
> Insecure $ENV{PATH} while running with -T switch at
> /home/cmrs/mosaic/cgi-bin//c
> po/lib/Email.pm line 30, <> chunk 1.
>
> Since I am specifying explicit path I dont understand why I get this
> error message.
It doesn't matter what path you specify -- any command that might invoke
a shell will howl if you have an insecure PATH (or IFS, or ENV, or a few
other environment variables that affect the shell). So, you either have
to set $ENV{'PATH'} to something secure, or (I think) use system/exec in
a secure way (pass a list, not a string).
> sub send_email($$$$)
> {
> my ($sender,$reciepent, $subject, $message)=@_;
>
> #untaint the variable that goes on command line.
> $sender =~ s/[\?\[\]\;\<\>\*\|\'\&\$\!\#\(\)\{\}\:\`\"\%\/\\\s]//g;
>
> #Redirect the stdout to trash as it clobbers up the HTML page
> #and cuases server error.
> open (MAIL, "|/usr/lib/sendmail -t -oi -f '$sender' >/dev/null")||
> die "Can't open sendmail";
I really wouldn't do it this way in a program meant to be secure. This
is much safer [disclaimer: untested, off-the-top-of-my-head code!]:
$pid = open (MAIL, "|-");
die "couldn't fork: $!\n" unless defined $pid;
unless ($pid) # in the child
{
open (STDOUT, ">/dev/null") ||
die "couldn't redirect stdout to /dev/null: $!\n";
exec $sendmail, "-t", "-oi", "-f", $sender;
die "couldn't exec $sendmail: $!\n";
}
else # in the parent
{
print MAIL ...
close (MAIL);
die "$sendmail failed\n" unless $? == 0;
}
This way, you are guaranteed that no shell will touch your precious
command, because you specify the argument list explicitly as a Perl
list. "perldoc -f exec" for an explanation. Also, "perldoc -f open"
for an explanation of that funky "|-" argument to open -- handy trick,
that.
Greg
--
Greg Ward - software developer gward@cnri.reston.va.us
Corporation for National Research Initiatives
1895 Preston White Drive voice: +1-703-620-8990 x287
Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
------------------------------
Date: Fri, 19 Feb 1999 18:22:05 GMT
From: jsheffie@my-dejanews.com
Subject: Re: Apache module to do Authentication/Authorization against the NAB (Name and Address Book) of Lotus Notes/Domino.
Message-Id: <7aka42$1il$1@nnrp1.dejanews.com>
Hello Kai,
Very good, I need to implement a very similar solution.
Although your post brings up a couple of questions.
Based on your statement
--> sends the login as CN to domino and get's the scrambled password
I am assuming that you are running a Notes LDAP server,
which Notes server version are you using to generate your LDAP server? (4.5,
4.6)
--> We took the mod_ldap and modified it a bit to do md5 password-encryption.
Ok no problem.
but why MD5?
Notes does not store the http passwd (Short Name :: http passwd)
in MD5.
--> Then we wrote an agent for Domino to use the same algorithm.
What is this agent doing?
--
Jeff Sheffield
jsheffie@dev.tivoli.com
Development "Widow" Webmaster
Good artist copy, Great artist steal
--Pablo Picaso
In article <36B845C6.A725FC90@syseca.de>,
Kai.Krebber@syseca.de wrote:
> Good News: It's possible (we're doing it)!
> Bad news: As far as I know, there's no out-of-the-box-module available for
> it. We took the mod_ldap and modified it a bit to do md5
> password-encryption.
> Then we wrote an agent for Domino to use the same algorithm. So if now
> somebody accesses our apache, the apache prompts for the login-name and
> password, sends the login as CN to domino and get's the scrambled password
> back, if domino finds this user and he / she has the intranet /
> internetpassword set.
> Then it scrambles the password typed in by the user and compares the
> cyphers. If the are the same, access is granted.
>
> Kai Krebber
>
> Stefaan Onderbeke schrieb:
>
> > Hi,
> >
> > I searched the WEB but I could not fined an answer on the following
> > question :
> >
> > Does somebody know if following module is available :
> >
> > Apache module to do Authentication/Authorization against the NAB (Name
> > and Address Book) of Lotus Notes/Domino.
> >
> > Or if this is possible and how this can be done ?
> >
> > Thanks in advance,
> >
> > Stefaan Onderbeke
>
>
-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/ Search, Read, Discuss, or Start Your Own
------------------------------
Date: Fri, 19 Feb 1999 18:22:11 GMT
From: jsheffie@tivoli.com
Subject: Re: Apache module to do Authentication/Authorization against the NAB (Name and Address Book) of Lotus Notes/Domino.
Message-Id: <7aka48$1io$1@nnrp1.dejanews.com>
Hello Kai,
Very good, I need to implement a very similar solution.
Although your post brings up a couple of questions.
Based on your statement
--> sends the login as CN to domino and get's the scrambled password
I am assuming that you are running a Notes LDAP server,
which Notes server version are you using to generate your LDAP server? (4.5,
4.6)
--> We took the mod_ldap and modified it a bit to do md5 password-encryption.
Ok no problem.
but why MD5?
Notes does not store the http passwd (Short Name :: http passwd)
in MD5.
--> Then we wrote an agent for Domino to use the same algorithm.
What is this agent doing?
--
Jeff Sheffield
jsheffie@dev.tivoli.com
Development "Widow" Webmaster
Good artist copy, Great artist steal
--Pablo Picaso
In article <36B845C6.A725FC90@syseca.de>,
Kai.Krebber@syseca.de wrote:
> Good News: It's possible (we're doing it)!
> Bad news: As far as I know, there's no out-of-the-box-module available for
> it. We took the mod_ldap and modified it a bit to do md5
> password-encryption.
> Then we wrote an agent for Domino to use the same algorithm. So if now
> somebody accesses our apache, the apache prompts for the login-name and
> password, sends the login as CN to domino and get's the scrambled password
> back, if domino finds this user and he / she has the intranet /
> internetpassword set.
> Then it scrambles the password typed in by the user and compares the
> cyphers. If the are the same, access is granted.
>
> Kai Krebber
>
> Stefaan Onderbeke schrieb:
>
> > Hi,
> >
> > I searched the WEB but I could not fined an answer on the following
> > question :
> >
> > Does somebody know if following module is available :
> >
> > Apache module to do Authentication/Authorization against the NAB (Name
> > and Address Book) of Lotus Notes/Domino.
> >
> > Or if this is possible and how this can be done ?
> >
> > Thanks in advance,
> >
> > Stefaan Onderbeke
>
>
-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/ Search, Read, Discuss, or Start Your Own
------------------------------
Date: Fri, 19 Feb 1999 11:58:18 -0600
From: Bernie Reger <breger@nwu.edu>
Subject: Breaking the 1K DBM barrier
Message-Id: <36CDA63A.CA3FBF5E@nwu.edu>
I am wrting a database program in Perl using DBM files. Unknowingly I
just ran into a small problem. Apparently the DBM file type used by
default has a limit to the size of keys and values that I use, and this
limit appears to be around 1K.
Does anyone know how to break this barrier? I've looked in the
AnyDBM_File manpage and read about the odbm, ndbm, sdbm, gdbm, and
bsd-db file types. From what I can tell, forcing my program to use one
of these would be a matter of having a 'use *DBM_File;' at the the
beginning of my program. I tried using the GDBM file and appaently my
version of Perl (5.004_04) doesn't have that module.
Does anyone have any experience with this? Thanks. Any input would be
appreciated.
-Bernie Reger
***********************************************************
Bernie Reger
Northwestern University breger@nwu.edu
(312) 503-5712 http://lims.mech.nwu.edu/~breger
------------------------------
Date: Fri, 19 Feb 1999 19:19:17 GMT
From: t_alter@hotmail.com
Subject: can I run perl on Win 98??????
Message-Id: <7akdfb$4p8$1@nnrp1.dejanews.com>
Hi everybody;
I am starting to learn perl. Can I do it with my Win98 PC?
I have been informed that I will have to go for Either UNIX or WinNT for that.
Please drop an email at t_alter@hotmail.com
Thanks a million.
Tom
-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/ Search, Read, Discuss, or Start Your Own
------------------------------
Date: Fri, 19 Feb 1999 18:24:59 +0000
From: abbott in Northfield <jabbott@dingo.smig.net>
Subject: exchanging info between two programs
Message-Id: <36CDAC7B.37620EE6@dingo.smig.net>
I have a perl program that does the following:
Cat a text file into it and it looks at the information contained in it
and runs one of three possible other perl programs depended on what is
in the text file.
I am doing this via
system ("/usr/home/data/secondfile.pl") or die "$status but it failed on
system call";
and this works. The second program runs just fine as long as everything
is perfect. However, what I would like to do instead is get back the
information that the second program generates. Something *like* this,
(however this does not work)
if ($sub =~ m/$namelist[1]/i){
#yes to the file name
$status="It might have worked. Running $namelist[2] ";
$test=system ("$namelist[2]") or die "$status but it failed
on system call";
$status="$status and test is:$test\n";
&sendreport ($status);
}
Where test would equal a bunch of output about records that were
imported to a database.
Any suggestions? Is a system call not the best way to do this?
--john abbott
------------------------------
Date: 19 Feb 1999 11:59:19 -0700
From: Tom Christiansen <tchrist@mox.perl.com>
Subject: Re: exchanging info between two programs
Message-Id: <36cdb487@csnews>
[courtesy cc of this posting sent to cited author via email]
In comp.lang.perl.misc, abbott in Northfield <jabbott@dingo.smig.net> writes:
:I am doing this via
:system ("/usr/home/data/secondfile.pl") or die "$status but it failed on
:system call";
:
:and this works. The second program runs just fine as long as everything
:is perfect. However, what I would like to do instead is get back the
:information that the second program generates.
Somehow you appear to have missed the proper answer when you studiously
searched the standard Perl documentation on your very own system.
It was there waiting for you.
% man perlfaq8
...
Why can't I get the output of a command with system()?
You're confusing the purpose of system() and backticks (``).
system() runs a command and returns exit status information (as
a 16 bit value: the low 7 bits are the signal the process died
from, if any, and the high 8 bits are the actual exit value).
Backticks (``) run a command and return what it sent to STDOUT.
$exit_status = system("mail-users");
$output_string = `ls`;
:Any suggestions? Is a system call not the best way to do this?
You haven't made a system call. A system call is a transition from user
space to kernel space, such as you find in section two of the standard
manual, or in Perl when you run something like this:
syscall(&SYS_gettimeofday, $start, 0)
You are talking about running external commands through the shell and
ignoring their output. This is not a system call. (Well, actually,
it's many, but you know what I mean -- I hope.)
--tom
--
This is a serious lapse of taste and judgement but does not imply that they
are stupid, lazy, or incompetent. Indeed, their intelligence, diligence,
and competence in service to the x86 are all too depressingly obvious. --Henry Spencer (henry@zoo.toronto.edu)
------------------------------
Date: 19 Feb 1999 08:11:39 -0700
From: Tom Christiansen <perlfaq-suggestions@perl.com>
Subject: FAQ 8.15: How do I set the time and date?
Message-Id: <36cd7f2b@csnews>
(This excerpt from perlfaq8 - System Interaction
($Revision: 1.36 $, $Date: 1999/01/08 05:36:34 $)
part of the standard set of documentation included with every
valid Perl distribution, like the one on your system.
See also http://language.perl.com/newdocs/pod/perlfaq8.html
if your negligent system adminstrator has been remiss in his duties.)
How do I set the time and date?
Assuming you're running under sufficient permissions, you should be able
to set the system-wide date and time by running the date(1) program.
(There is no way to set the time and date on a per-process basis.) This
mechanism will work for Unix, MS-DOS, Windows, and NT; the VMS
equivalent is `set time'.
However, if all you want to do is change your timezone, you can probably
get away with setting an environment variable:
$ENV{TZ} = "MST7MDT"; # unixish
$ENV{'SYS$TIMEZONE_DIFFERENTIAL'}="-5" # vms
system "trn comp.lang.perl.misc";
--
When the dinosaurs are mating, climb a tree. --Steve Johnson
------------------------------
Date: 19 Feb 1999 08:32:13 -0700
From: Tom Christiansen <perlfaq-suggestions@perl.com>
Subject: FAQ 8.16: How can I sleep() or alarm() for under a second?
Message-Id: <36cd83fd@csnews>
(This excerpt from perlfaq8 - System Interaction
($Revision: 1.36 $, $Date: 1999/01/08 05:36:34 $)
part of the standard set of documentation included with every
valid Perl distribution, like the one on your system.
See also http://language.perl.com/newdocs/pod/perlfaq8.html
if your negligent system adminstrator has been remiss in his duties.)
How can I sleep() or alarm() for under a second?
If you want finer granularity than the 1 second that the sleep()
function provides, the easiest way is to use the select() function as
documented in the section on "select" in the perlfunc manpage. If your
system has itimers and syscall() support, you can check out the old
example in
http://www.perl.com/CPAN/doc/misc/ancient/tutorial/eg/itimers.pl .
--
Love your enemies: they'll go crazy trying to figure out what you're up to.
------------------------------
Date: 19 Feb 1999 11:43:44 -0700
From: Tom Christiansen <perlfaq-suggestions@perl.com>
Subject: FAQ 8.20: How can I call my system's unique C functions from Perl?
Message-Id: <36cdb0e0@csnews>
(This excerpt from perlfaq8 - System Interaction
($Revision: 1.36 $, $Date: 1999/01/08 05:36:34 $)
part of the standard set of documentation included with every
valid Perl distribution, like the one on your system.
See also http://language.perl.com/newdocs/pod/perlfaq8.html
if your negligent system adminstrator has been remiss in his duties.)
How can I call my system's unique C functions from Perl?
In most cases, you write an external module to do it - see the answer to
"Where can I learn about linking C with Perl? [h2xs, xsubpp]". However,
if the function is a system call, and your system supports syscall(),
you can use the syscall function (documented in the perlfunc manpage).
Remember to check the modules that came with your distribution, and CPAN
as well - someone may already have written a module to do it.
--
You want it in one line? Does it have to fit in 80 columns? :-)
--Larry Wall in <7349@jpl-devvax.JPL.NASA.GOV>
------------------------------
Date: 19 Feb 1999 12:36:14 -0700
From: Tom Christiansen <perlfaq-suggestions@perl.com>
Subject: FAQ 8.21: Where do I get the include files to do ioctl() or syscall()?
Message-Id: <36cdbd2e@csnews>
(This excerpt from perlfaq8 - System Interaction
($Revision: 1.36 $, $Date: 1999/01/08 05:36:34 $)
part of the standard set of documentation included with every
valid Perl distribution, like the one on your system.
See also http://language.perl.com/newdocs/pod/perlfaq8.html
if your negligent system adminstrator has been remiss in his duties.)
Where do I get the include files to do ioctl() or syscall()?
Historically, these would be generated by the h2ph tool, part of the
standard perl distribution. This program converts cpp(1) directives in C
header files to files containing subroutine definitions, like
&SYS_getitimer, which you can use as arguments to your functions. It
doesn't work perfectly, but it usually gets most of the job done. Simple
files like errno.h, syscall.h, and socket.h were fine, but the hard ones
like ioctl.h nearly always need to hand-edited. Here's how to install
the *.ph files:
1. become super-user
2. cd /usr/include
3. h2ph *.h */*.h
If your system supports dynamic loading, for reasons of portability and
sanity you probably ought to use h2xs (also part of the standard perl
distribution). This tool converts C header files to Perl extensions. See
the perlxstut manpage for how to get started with h2xs.
If your system doesn't support dynamic loading, you still probably ought
to use h2xs. See the perlxstut manpage and the ExtUtils::MakeMaker
manpage for more information (in brief, just use make perl instead of a
plain make to rebuild perl with a new static extension).
--
Von Neumann: "Anyone attempting to generate random numbers by
deterministic means is, of course, living in a state of sin."
------------------------------
Date: 19 Feb 1999 08:54:15 -0700
From: Tom Christiansen <tchrist@mox.perl.com>
Subject: FMTEYEWTK: Sorting
Message-Id: <36cd8927@csnews>
(This document is part of the FMTEYEWTK series at
http://language.perl.com/doc/FMTEYEWTK/ )
Far More Than Everything You've Ever Wanted to Know About Sorting
------------------------------------------------------------------------
In comp.lang.perl, Randal Schwartz <merlyn@stonehenge.com> obfuscates:
>>>> "Hugo" == Hugo Andrade Cartaxeiro <hac@nexus.inesc.pt> writes:
:
:Hugo> print $str;
:Hugo> eir 11 9 2 6 3 1 1 81% 63% 13
:Hugo> oos 10 6 4 3 3 0 4 60% 70% 25
:Hugo> hrh 10 6 4 5 1 2 2 60% 70% 15
:Hugo> spp 10 6 4 3 3 1 3 60% 60% 14
:
:Hugo> and I like to sort it with the last field as the order key. I know
:
:Hugo> perl has some features to do it, but I can't make 'em work properly.
:
:Well, speaking Perl with a Lisp.... :-)
We all recall that Perl est un esprit LISP dans un corps C, n'est-ce pas?
:-) (``corps C'' sounds like ``corset'' in French.) [Philippe Verdret
<phd@eurolang.fr>]
CODE 0.0
:require 5; # if you don't have perl 5 by now, why NOT? :-)
:$str =
: join "\n",
: map { $_->[0] }
: sort { $a->[1] <=> $b->[1] }
: map { [$_, (split)[-1]] }
: split /\n/,
: $str;
:
:Hugo> Reply by mail (if any).
:
:(oops :-)
Oh for cryin' out loud, Randal! You expect a new perl programmer to make
heads or tails of that>? :-) You're postings JAPHs for solutions, which
isn't going to help a lot. You'll probably manage to scare these poor people
away from the language forever? :-)
BTW, you have a bug.
Couldn't you at least indent it properly? :-) This might help folks make
more sense of it. I'm going to use the
map EXPR,LIST
rather and then
map BLOCK LIST
of map here so that it looks more like a conventional subroutine call
(although I can't do that with sort):
CODE 0.1
sort( { $a->[1] => $b->[1] }
map( [ $_, (split(' ', $_))[-1] ],
split(/\n/, $str)
)
)
)
);
Hm... ok, not much better. Let's do this slowly and work our way through it.
First of all, let's look at your string again.
eir 11 9 2 6 3 1 1 81% 63% 13
oos 10 6 4 3 3 0 4 60% 70% 25
hrh 10 6 4 5 1 2 2 60% 70% 15
spp 10 6 4 3 3 1 3 60% 60% 14
It looks like it's got a bunch of lines in it, so the first thing we're
going to do is break it up into a list of lines.
@lines = split (/\n/, $str);
You need to do this because each the perl built-in sort() function expects a
list and returns a list. It works like this in its most simple incarnation.
@newlist = sort @oldlist;
Well, that's all well and good, but it's going to do the default sort,
meaning it'll sort it by ASCII. Imagine a function like this:
CODE 1.1
sub by_last_num_field {
@alist = split(/\s+/, $a);
@blist = split(/\s+/, $b);
if ( $alist[$#alist] > $blist[$#blist] ) {
return 1;
} elsif ( $blist[$#blist] > $alist[$#alist] ) {
return -1
} else {
return 0;
}
}
You could then say
@newlist = sort by_last_num_field @oldlist;
And it would work out for you.
It turns out that the $a and $b are each of the two values to be compared in
the original list. They're not going to passed in the way normal arguments
are, which is $_[0], $_[1], etc.
That's work out ok, but it's pretty complicated looking. It turns out that
for just a matter, perl has a couple of operators to help you: <=> for
numbers and cmp for strings.
While normally we give split a regular expression as its first argument,
there's one case where you might want to give it a string: if the first
argument is just a single space, then it will automatically discard leading
null fields, which would otherwise be preserved. (Trailing null fields are
discarded in any case (unless you give split() a third argument).)
So you could just do this:
CODE 1.2
sub by_last_num_field {
@alist = split(' ',$a);
@blist = split(' ',$b);
return $alist[$#alist] <=> $blist[$#blist];
}
Now you could also have written that function ``in-lined'', like so. I'm
going to leave out the word return this time:
CODE 1.3
@newlist = sort {
@alist = split(' ', $a);
@blist = split(' ', $b);
$alist[$#alist] <=> $blist[$#blist];
} @oldlist;
Ok so far? We can make use of perl5's ability to do negative subscripting to
get the last element:
CODE 1.4
@newlines = sort {
@alist = split(' ', $a);
@blist = split(' ', $b);
$alist[-1] <=> $blist[-1];
} @oldlines;
And in fact, we can do this without the temporary lists:
CODE 1.5
@newlines = sort {
(split(' ', $a))[-1] <=> (split(' ', $b))[-1];
} @oldlines;
But no matter how you write it, that's a wee tad expensive for large data
sets. Since Perl's sort is a hook into C's qsort (lookup qsort(3) in the
on-line Unix manual, if you're daring), that makes it N-logN sort. You're
going to be doing 2 splits for each of those. With your data set of four,
this isn't two bad: you'll average only 11 splits. With ten lines though,
you're already up to 46. With a mere one hundred lines, 921 splits, and with
a thousand lines, we're talking 13,816 splits on the average! That's way too
many.
So let's arrange to do just N splits. We'll have to pre-process them:
CODE 2.1
@last_field = ():
foreach $line (@oldlines) {
$last_elt = (split(' ', $line))[-1];
push (@last_field, $last_elt);
}
Now for a given line $n in @oldlines , its last field can be found in the
corresponding element of the @last_field array, that is, $last_field[$n] .
You could also just use the return value from split directly, without using
a temporary, like this:
CODE 2.2
@last_field = ():
foreach $line (@oldlines) {
push (@last_field, (split(' ', $line))[-1]);
}
You'll find that many perl programs are used to letting the $_ variable be
the default to quite a few different operations. This is unsettling to the
newcomer, but it's really not that bad. Amongst other things, $_ is the
default iterator variable on a foreach loop, the default string to split on
in a split() function, as well as the default variable to search or replace
in a match or substitution. (You'll find that it's also the implicit
iterator variables in the map() and grep() built-ins, which are themselves
something like short-cut ways of writing foreach loops.)
CODE 2.3
@last_field = ():
foreach (@oldlines) {
push(@last_field, (split(' '))[-1]);
}
In perl5, we can write that without so many parentheses if we'd like. This
time I'm also going to spell "foreach" as "for". It actually doesn't matter
to perl, but it might to the reader.
CODE 2.4
@last_field = ():
for (@oldlines) {
push @last_field, (split ' ')[-1];
}
This entire process of reducing things to smaller and smaller amounts of
code to do the equivalent thing is an interesting exercise, but perilous.
While between 1.1 and 1.5 and between 2.1 and 2.4, we actually did improve
the efficiency of the operation ever so slightly, we've also managed to make
it harder to debug and harder to maintain. It's nice to have intermediary
values sometimes in order to print them out part way through the
calculation, whether in the debugger or with normal print statements. It's
also harder for someone to come by later and figure out what you're doing.
If your aim is job security, then this might work out in your favor, but in
most other cases, it probably won't.
Ok, how do we couple the stuff we did in code set 2 with what we did in code
set 1? Remembering that any given element in @last_field corresponds to the
last field (the thing we want to sort on) in the @oldlines list. So we're
going to do something a bit strange and sort not real data but rather just
the indices. That is, we'll rearrange indices (like 0,1,2,3,4) to correspond
to the properly sorted order (like 3,2,1,4,0). This is probably the hardest
concept to grasp in this whole tutorial.
CODE 3.1
@orig_indices = 0 .. $#oldlines;
@sorted_indices = sort {
$last_field[$a] <=> $last_field[$b]
} @orig_indices;
@newlist = ();
foreach $idx (@sorted_indices) {
push(@newlines, $oldlines[$idx]);
}
Now when we're done, we've got everything into its proper order.
Occasionally, advanced perl programmers make use of what we call array
slices. If you say
CODE: 3.2.1:
@old_idx = (1,2,3);
@new_idx = (3,1,2);
@x[@new_idx] = @y[@old_idx];
It's the same as though you'd said:
CODE 3.2.2
@x[3,1,2] = @y[1,2,3];
Which is really nothing more than simply:
CODE 3.2.3
$x[3] = $y[1];
$x[1] = $y[2];
$x[2] = $y[3];
Except that it's shorter to write. Notice how when you're doing slice
operations you have to subscript your list with an @ not a $? That's because
the $ or @ sign actually indicates what it is that you're getting back, not
what you're looking at. In this case, you're assigning a (sub-)list to a
(sub-)list, so you need the @ signs.
This means we can re-write 3.1 as
CODE 3.3
@orig_indices = 0 .. $#oldlines;
@sorted_indices = sort {
$last_field[$a] <=> $last_field[$b]
} @orig_indices;
@newlines = @oldlines[@sorted_indices];
Now, since sort will return the indices, we can avoid some temporaries and
write that as:
CODE 3.4
@newlines = @oldlines[ sort {
$last_field[$a] <=> $last_field[$b]
} 0 .. $#oldlines
];
If we'd used shorter variable names, we could have then written it all on
one line:
CODE 3.5
@new = @old[ sort { $f[$a] <=> $f[$b] } 0 .. $#old ];
Now what? Well, we have to get back the user's orginal data.
CODE 4.1
$str = join ("\n", @newlines);
But you'll find that you're shy a final newline (that's Randal's bug).
You could solve this by passing another null argument to join() ,
CODE 4.2
$str = join ("\n", @newlines, '');
or by concatenating a newline to the result:
CASE 4.3
$str = join ("\n", @newlines) . "\n";
Combining all these, we have final solution that looks like this:
CODE 5.1
@old = split (/\n/, $str);
@f = ():
for (@old) {
push @f, (split ' ')[-1];
}
@new = @old[ sort { $f[$a] <=> $f[$b] } 0 .. $#old ];
$str = join ("\n", @new) . "\n";
We can merge a few of these and have something shorter.
CODE 5.2
@old = @f = ():
for (split (/\n/, $str)) {
push @old, $_;
push @f, (split ' ')[-1];
}
$str = join ("\n", @old[sort{ $f[$a] <=> $f[$b] } 0..$#old ]) . "\n";
It's true tha this is verging on the inscrutable, but we're trying to get
closer to Randal's obfuscated perl entry he presented way at the beginning
of this.
Nonetheless, it's still quite a bit different from Randal's. One thing he's
done is make use of the map() function. This is a perl5 feature that uses
its first expression argument as code to execute repeatedly against each of
its following list arguments, setting $_ to each element, and returning the
result of the expression to create a new output list.
CODE 5.2.1
@single = 1..10;
@treble = map $_ * 3, @single;
You may also write this expression as a block, and omit the comma, which is
what Randal did:
CODE 5.2.2
@single = 1..10;
@treble = map { $_ * 3 } @single;
Or even omit the temporary:
CODE 5.2.3
@treble = map { $_ * 3 } 1..10;
You can use the map() function to shorten up 5.2 a bit by pulling out the
for loop:
CODE 5.3
@old = split (/\n/, $str);
@f = map { (split ' ')[-1] } @old;
$str = join ("\n", @old[sort{ $f[$a] <=> $f[$b] } 0..$#old ]) . "\n";
You could even make use of the result of the assignment statement as the
argument to map and shrink it sill more:
CODE 5.4
@f = map { (split ' ')[-1] } @old = split (/\n/, $str);
$str = join ("\n", @old[sort{ $f[$a] <=> $f[$b] } 0..$#old ]) . "\n";
Now, Randal didn't want to use any temporary variables at all. Let's look at
his jumbled code again:
CODE 0.0
$str =
join "\n",
map { $_->[0] }
sort { $a->[1] <=> $b->[1] }
map { [$_, (split)[-1]] }
split /\n/,
$str;
He's been kinda nasty in that there are no parens or indentation. I offered
this version which adds them:
CODE 0.1
$str = join("\n",
map($_->[0],
sort( {$a->[1] <=> $b->[1] }
map( [$_, (split(' ', $_))[-1] ],
split(/\n/, $str)
)
)
)
);
But you could retain Randal's block form as well:
CODE 0.2:
$str = join("\n",
map( { $_->[0] }
sort( {$a->[1] <=> $b->[1] }
map( { [$_, (split(' ', $_))[-1] ] }
split(/\n/, $str)
)
)
)
);
While in some ways code 0.2 resembles code 5.4, in many others it doesn't.
What are all those crazy arrows and square brackets doing there? Well,
that's a really interesting question.
Let's take this thing inside out. Here's the innermost portion of it:
CODE 6.1
map { [$_, (split(' ', $_))[-1] ] } split(/\n/, $str)
What's going on is that he's using the return from split to be the arguments
to map() over. Ok, but what's that map() code block doing???
CODE 6.2
[ $_, (split(' ', $_))[-1] ]
What 6.2 does happens to do is to create a reference to a list of two
elements. It's as though you said this
CODE 6.2.1
my @x = ( $_, (split(' ', $_))[-1] );
return \@x;
However, the list reference he's creating by using the [ .... ] notation
isn't a named list at all. If you wrote the inner loop as a real
honest-to-goodness loop, it might look like this: (remember that 6.3 is just
the long form of 6.1)
CODE 6.3
@listrefs = ();
for (split(/\n/, $str)) {
$newref = [ $_, (split(' ', $_))[-1] ];
push (@listrefs, $newref);
}
And then you'd have all your references in @listrefs. This means that for a
given line within $str (call it line number $n ), $listrefs [$n]->[0] is
going to be the whole line itself (without its newline), while
$listrefs[$n]->[1] is going to be its last field. We could actually write
those without the pointer arrows, because adjacent brackets may omit them,
as in $listrefs[$n][0] , but we're going to need them for what later, so you
should get used to seeing them.
Well, what's the next level do? It's a sort. Consider this:
CODE 7.1
@newrefs = sort { $a->[1] <=> $b->[1] } @listrefs;
Since $a and $b will be each element of @listrefs, they will each be
references to lists (kinda like pointers to lists). Remember that each of
these refers to a list of two elements, whose zeroth element $a->[0] is the
entire data, but whose first element $a->[1] is the sort key. So we'll
compare the sort keys.
Put we don't want the sort keys. We want the original data. You could write
a loop to do it:
CODE 7.2
@data = ():
foreach $ref (@newrefs) {
push(@data, $ref->[0]);
}
Or you could be like Randal and avoid temporaries by using a map() instead
of a loop:
CODE 7.3
@data = map { $_->[0] } @newrefs;
Combining up various stages, we get this:
CODE 7.4
@lines = split( /\n/, $str);
@listrefs = map { [$_, (split(' ', $_))[-1] ] } @lines;
@newrefs = sort { $a->[1] <=> $b->[1] } @listrefs;
@data = map { $_->[0] } @newrefs;
$str = join("\n", @data) . "\n";
It may not look like it, but this is very, very close to what Randal wrote
back in 0.0; he just didn't keep around his temporary results in names
variables. It's quite likely that he even developed the 0.0 code starting
from something that looked more like 7.4. If you don't, you'll have an
exceedingly difficult time debugging it.
But you know something? You don't have to be all so super-clever to do this.
There's a much more straightforward solution:
CODE 8.1
$tmpfile = "/tmp/sortage.$$";
open (TMP, ">$tmpfile");
print TMP $str;
close(TMP);
$str = `sort +10n $tmpfile`;
unlink $tmpfile;
See? We just write it to a file, then let the system sort program do the
work for you. Isn't that much easier to understand? If you can easily
express your sort in something that the system sort program will understand,
it usually makes more sense to use it. It'll run faster and be easier to
write as well.
Of course, we've made committed some minor errors here by omitting error
checking. This would be better:
CODE 8.2
$tmpfile = "/tmp/sortage.$$";
open (TMP, ">$tmpfile") || die "can't open $tmpfile for writing: $!";
(print TMP $str) || die "can't write to $tmpfile: $!";
close(TMP) || die "can't write to $tmpfile: $!";
$str = `sort +10n $tmpfile`;
if ($?) { die "sort exited with a $?" }
unlink($tmpfile) || die "can't unlink $tmpfile: $!";
That line to print() looks funny, eh? And what about the parens on the
unlink? You probably thought that you could have written both of those as
CODE 8.2.1
print TMP $str || die "can't write to $tmpfile: $!";
unlink $tmpfile || die "can't unlink $tmpfile: $!";
But in fact, you couldn't. The || binds more tightly than you want. It would
have been as though you'd written:
CODE 8.2.2
print TMP ($str || die "can't write to $tmpfile: $!");
unlink ($tmpfile || die "can't unlink $tmpfile: $!");
Which certainly isn't good. That's why in perl5 we have or, which is a very
low-precedence operator. We probably would write it like this:
CODE 8.3
$tmpfile = "/tmp/sortage.$$";
open TMP, ">$tmpfile" or die "can't open $tmpfile for writing: $!";
print TMP $str or die "can't write to $tmpfile: $!";
close TMP or die "can't close $tmpfile: $!";
$str = `sort +10n $tmpfile`;
if ($?) { die "sort exited with a $?" }
unlink $tmpfile or die "can't unlink $tmpfile: $!";
Another trick sometimes used is to use to the filename itself as an indirect
file handle.
CODE 8.4
$tmpfile = "/tmp/sortage.$$";
open $tmpfile, ">$tmpfile" or die "can't open $tmpfile for writing: $!";
print $tmpfile $str or die "can't write to $tmpfile: $!";
close $tmpfile or die "can't close $tmpfile: $!";
$str = `sort +10n $tmpfile`;
if ($?) { die "sort exited with a $?" }
unlink $tmpfile or die "can't unlink $tmpfile: $!";
Another trick sometimes used is to use to the filename itself as an indirect
file handle. This has a greater advantage when you are reading from the
file, and issuing diagnostics using warn() or die(), which include the line
(or chunk) position and filehandle anme of the last file read from. This way
you'd also get the
CODE 8.5
$tmpfile = "/tmp/sortage.$$";
open $tmpfile, ">$tmpfile" or die "can't open $tmpfile for writing: $!";
print $tmpfile $str or die "can't write to $tmpfile: $!";
close $tmpfile or die "can't close $tmpfile: $!";
# do something else to the file here, then...
open $tmpfile, "<$tmpfile" or die "can't open $tmpfile for reading: $!";
while (<$tmpfile>) {
if ( $something_or_other ) {
warn "oops message";
}
}
Now the warn will include the actual filename rather than just the
filehandle, like TMP.
Now, you probably shouldn't bother go to too much trouble in figuring out a
good filename. There is a POSIX function which will do this for you.
CODE 8.6
use POSIX;
$tmpfile = POSIX::tmpnam();
open TMP, ">$tmpfile" or die "can't open $tmpfile for writing: $!";
print TMP $str or die "can't write to $tmpfile: $!";
close TMP or die "can't close $tmpfile: $!";
$str = `sort +10n $tmpfile`;
if ($?) { die "sort exited with a $?" }
unlink $tmpfile or die "can't unlink $tmpfile: $!";
Just one final thing. This seems like just the application for a
bi-directional open. You might be tempted to write this using the open2()
function.
CODE 9.1
use IPC::Open2;
$kidpid = open2('INPUT', 'OUTPUT', 'sort +10n');
print OUTPUT $str or die "can't write to $tmpfile: $!";
close OUTPUT or die "can't close output pipe: $!";
while (<INPUT>) {
print "got line $.: $_";
}
close INPUT or die "can't close input pipe: $!";
or maybe to read the whole thing back again, undef the RS:
CODE 9.2
use FileHandle;
use IPC::Open2;
$kidpid = open2('INPUT', 'OUTPUT', 'sort +10n');
print OUTPUT $str or die "can't write to $tmpfile: $!";
close OUTPUT or die "can't close output pipe: $!";
INPUT->input_record_separator(undef);
$str = <INPUT>;
close INPUT or die "can't close input pipe: $!";
Now, it turns out that in this case it actually works, at least on my
system. I wrote all my input do the pipe first, which gladly read it all in
up until end of file, and then spit it out. That's how sort work. But cat
isn't so friendly, and works far more like most programs.
CODE 9.3
use FileHandle;
use IPC::Open2;
$kidpid = open2('INPUT', 'OUTPUT', 'cat');
print OUTPUT $str or die "can't write to $tmpfile: $!";
close OUTPUT or die "can't close output pipe: $!";
INPUT->input_record_separator(undef);
$str = <INPUT>;
close INPUT or die "can't close input pipe: $!";
If you do this and $str is big enough, you'll block forever. And even doing
a piece at a time:
CODE 9.4
use IPC::Open2;
$kidpid = open2('INPUT', 'OUTPUT', 'cat');
while (1) {
print OUTPUT $str;
$newstr =<INPUT>;
print "just read <<< $newstr >>>\n";
}
This approach won't work because in general, you can't control your child
processes notions of buffering, which are entirely useless for simulating
interactive work as you're attempting here. The only programs this works on
are ones that are designed for it, like dc. There aren't many system
programs like it.
I strongly advise against using open2 for almost anything, even though I'm
its author. UNIX buffering will just drive you up the wall. You'll end up
quite disappointed. It's better to use temp files as we did back in the code
examples in section 9 if you really want to deal with an external process in
which you wish to control both its input and its output.
There's one final lesson I'd like to impart. It's not a bad idea to order
your coding priorities like so:
1. correctness
2. maintainability
3. efficiency
Notice at no point did ``4. cleverness'' enter the picture. Of course, if
you're writing Obfuscated C contest entries, or tricksy little JAPHs, then
it makes perfect sense to invert these priorities, like so:
1. cleverness
2. efficiency
3. maintainability
4. correctness
But that's only good for tricks, not solid code, which is why code like that
in 5.1, 8.3, or even 7.4 will in the long run make easier both your life and
the lives of those around you.
While a LISP or Scheme programmer may be entirely conformatable with arrange
their code as Randal did in 0.0, I haven't personal found that most of your
average off-the-street UNIX, DOS, VMS, or IBM programmers take very quickly
to that style of functional programming -- or even the above average ones,
for that matter. It's nice in its own pristine way not to use any temporary
variables and all, but it's still not to most people's liking.
And no, I'm not ragging on Randal, merely teasing a bit. He's just trying to
be clever, and that's what he does. I'm just submitting a sample chapter for
his perusal for inclusion the mythical Alpaca Book :-)
--tom
--
Fifty years of programming language research, and we end up with C++ ???
--Richard A. O'Keefe
------------------------------
Date: 12 Dec 98 21:33:47 GMT (Last modified)
From: Perl-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Special: Digest Administrivia (Last modified: 12 Dec 98)
Message-Id: <null>
Administrivia:
Well, after 6 months, here's the answer to the quiz: what do we do about
comp.lang.perl.moderated. Answer: nothing.
]From: Russ Allbery <rra@stanford.edu>
]Date: 21 Sep 1998 19:53:43 -0700
]Subject: comp.lang.perl.moderated available via e-mail
]
]It is possible to subscribe to comp.lang.perl.moderated as a mailing list.
]To do so, send mail to majordomo@eyrie.org with "subscribe clpm" in the
]body. Majordomo will then send you instructions on how to confirm your
]subscription. This is provided as a general service for those people who
]cannot receive the newsgroup for whatever reason or who just prefer to
]receive messages via e-mail.
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.misc (and this Digest), send your
article to perl-users@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
The Meta-FAQ, an article containing information about the FAQ, is
available by requesting "send perl-users meta-faq". The real FAQ, as it
appeared last in the newsgroup, can be retrieved with the request "send
perl-users FAQ". Due to their sizes, neither the Meta-FAQ nor the FAQ
are included in the digest.
The "mini-FAQ", which is an updated version of the Meta-FAQ, is
available by requesting "send perl-users mini-faq". It appears twice
weekly in the group, but is not distributed in the digest.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V8 Issue 4941
**************************************
