[29499] in Perl-Users-Digest
Perl-Users Digest, Issue: 743 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Fri Aug 10 21:09:39 2007
Date: Fri, 10 Aug 2007 18:09:06 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Fri, 10 Aug 2007 Volume: 11 Number: 743
Today's topics:
Re: Certfication in AWK or SHell programming <mjcohen@acm.org>
Re: file info on XP <news@lawshouse.org>
Re: file info on XP <RedGrittyBrick@SpamWeary.foo>
Re: File::Find problem? usenet@DavidFilmer.com
Re: File::Find problem? <news@lawshouse.org>
Re: File::Find problem? <news@lawshouse.org>
Re: File::Find problem? <rkb@i.frys.com>
Re: how to tranpose a huge text file xhoster@gmail.com
Re: how to tranpose a huge text file <nospam-abuse@ilyaz.org>
Re: how to tranpose a huge text file <tzz@lifelogs.com>
Re: how to tranpose a huge text file <nospam-abuse@ilyaz.org>
Re: how to tranpose a huge text file <tzz@lifelogs.com>
Re: Need help writing a basic script print to text file <bill@ts1000.us>
Re: pass by reference <bpatton@ti.com>
Re: Perl On Apache <spamtrap@dot-app.org>
Re: Perl On Apache <bill@ts1000.us>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Fri, 10 Aug 2007 14:35:04 -0700
From: mjc <mjcohen@acm.org>
Subject: Re: Certfication in AWK or SHell programming
Message-Id: <1186781704.949862.53180@i38g2000prf.googlegroups.com>
On Aug 9, 4:24 am, "news.t-online.de" <suregeo...@yahoo.de> wrote:
> ehabaziz2...@gmail.com wrote:
> > Hello,
> > can I have any kind of certfifcation in Awk and SED and if the answer
> > is NO What is the nearest certfication to SED,AWK and even any
> > certfifcation to shell programming?
> > Thanks
>
> simply give AWK a call, or SEDhttp://www.awk.com/http://www.sed.com/
More useful is to look up www.gawk.com.
This gets you to "Network Solutions" with the following sponsered
links:
Unix Resume
Don't just search for a job find
www.Dice.com
OpenPGP Server Encryption
ANI Direct sells and supports the Authora EDGE products. 800-444-7576
www.anidirect.com
Job Scheduling Automation
Intelligent Job Scheduling for Windows, Linux, UNIX & OpenVMS
www.activebatch.com
UNIX and Linux Support
Support and access UNIX and Linux systems securely over the Internet.
www.itivity.tridia.com
UNIX Testing Tools
Quickly Automate UNIX Testing Get Started in 60 Min! Free Trial.
www.redstonesoftware.com
Learn Shell Programming
Expert instructors. Hands-On labs. All-Inclusive deals. Contact us.
www.technow.com
www.LinuxCBT.com
Comprehensive, Cost-effective CBTs Trusted by Organizations Globally
www.LinuxCBT.com
HP9000/3000 for less
Full Systems, Upgrades, service, & support.
www.baypointetech.com
Shell Script
Automate desktop management without writing scripts - Free Trial!
www.scriptlogic.com
ETL Open Source
The worldwide first Open Source software for Data Integration (ETL)
www.talend.com
The technow.com link gets you to this:
TN-215:
Introduction to Shell Programming
Learn Shell Programming Fast with Hands-on Training
Overview
This intensive hands-on class teaches students to develop customized
UNIX=99 commands, read & write Bourne, Korn & Bash Shell scripts &
automate critical server functions. The course includes extensive lab
exercises, including hands-on development & debugging of shell
scripts, which will give students the ability to be more efficient and
productive in less time.
Duration: 5 Days
What You Will Learn
Shell Review
Shell Interpretation of Quotes & Backslash
The Korn Shell
sed & awk
Programming Basics
Data Constructs
Programming Looping
Your Environment
More on Parameters
Loose Ends
So, if you take this course, this might satisfy what the OP wants.
------------------------------
Date: Fri, 10 Aug 2007 21:10:02 +0100
From: Henry Law <news@lawshouse.org>
Subject: Re: file info on XP
Message-Id: <1186776561.19234.1@proxy02.news.clara.net>
Jerry wrote:
> Hi!
>
> Is there a way to find out who made the last change, user id, to a file and
> if it is IP or local or remote?
>
> Can it be done on Linuix from Perl?
This is more of an operating system question: does the file system
maintain that information? If so you'll probably find a Perl module
that will retrieve it. But I think the answer to the first question is
"no". Work on that first.
--
Henry Law Manchester, England
------------------------------
Date: Fri, 10 Aug 2007 21:35:15 +0100
From: RedGrittyBrick <RedGrittyBrick@SpamWeary.foo>
Subject: Re: file info on XP
Message-Id: <ypSdnRoU68ScUSHbRVnyigA@bt.com>
Jerry wrote:
> Hi!
>
> Is there a way to find out who made the last change, user id, to a file and
> if it is IP or local or remote?
Usually no. However some operating systems have an "audit" feature that
you can enable and can configure to log that kind of activity.
> Can it be done on Linuix
http://www.cyberciti.biz/tips/linux-audit-files-to-see-who-made-changes-to-a-file.html
> from Perl?
Almost certainly. You'll probably have to code it yourself.
------------------------------
Date: Fri, 10 Aug 2007 20:14:11 -0000
From: usenet@DavidFilmer.com
Subject: Re: File::Find problem?
Message-Id: <1186776851.648594.191100@i13g2000prf.googlegroups.com>
On Aug 10, 1:04 pm, Monty <dale.schm...@offutt.af.mil> wrote:
>>> I wonder if the problem is in the relative pathing?
I'll bet it is.
Are you running the program from the same root path that the find is
looking at? Do you chdir() either to or away from that root path in
your program?
--
The best way to get a good answer is to ask a good question.
David Filmer (http://DavidFilmer.com)
------------------------------
Date: Fri, 10 Aug 2007 21:08:26 +0100
From: Henry Law <news@lawshouse.org>
Subject: Re: File::Find problem?
Message-Id: <1186776465.19234.0@proxy02.news.clara.net>
Monty wrote:
> Oh yes, none of these files are symbolic links.
Please post your real code (don't re-type it), together with the output
you get when you run that very code.
This, for example, is based on your code and works on my Win XP machine.
(I've replaced your "hdr" extension with "doc" for convenience).
#! /usr/bin/perl
use strict;
use warnings;
use File::Find;
find (\&process_file,".");
sub process_file {
if (/\.doc$/) {
print "$File::Find::name ...";
print "exists, " if -e $File::Find::name;
print "is readable, " if -r $File::Find::name;
open HDR_FILE, "<$File::Find::name" or die "Can't open file for
input: $!\n";
print "opened OK\n";
close HDR_FILE;
}
}
Output:
F:\WIP>clpm.pl
./IAM Capability Strategy V1.11.doc ...exists, is readable, opened OK
./Annual General Meeting 07.doc ...exists, is readable, opened OK
and so on.
--
Henry Law Manchester, England
------------------------------
Date: Fri, 10 Aug 2007 21:13:58 +0100
From: Henry Law <news@lawshouse.org>
Subject: Re: File::Find problem?
Message-Id: <1186776798.20151.0@proxy00.news.clara.net>
Henry Law wrote:
> Output:
> F:\WIP>clpm.pl
> ../IAM Capability Strategy V1.11.doc ...exists, is readable, opened OK
> ../Annual General Meeting 07.doc ...exists, is readable, opened OK
> and so on.
For some reason Thunderbird has put two dots where my original output
has one. The console shows one dot (the current directory).
./IAM Capability Strategy V1.11.doc ...exists, is readable, opened OK
./Annual General Meeting 07.doc ...exists, is readable, opened OK
(Unless TBird munges it again!)
--
Henry Law Manchester, England
------------------------------
Date: Fri, 10 Aug 2007 13:15:40 -0700
From: Ron Bergin <rkb@i.frys.com>
Subject: Re: File::Find problem?
Message-Id: <1186776940.277376.267520@e9g2000prf.googlegroups.com>
On Aug 10, 12:21 pm, "J. Gleixner" <glex_no-s...@qwest-spam-
no.invalid> wrote:
> Monty wrote:
> > open HDR_FILE, "<$File::Find::name" or die "Can't open file
> > for input: $!\n";
Are there spaces in the filename and/or path?
Since the default is to cd into each dir, you should be able ti use $_
instead of $File::Find::name
Try changing the open call to this:
open(HDR_FILE, '<', "$_") || die "Can't open file or input: $!";
------------------------------
Date: 10 Aug 2007 20:52:50 GMT
From: xhoster@gmail.com
Subject: Re: how to tranpose a huge text file
Message-Id: <20070810165253.351$nL@newsreader.com>
Jie <jiehuang001@gmail.com> wrote:
> I have a huge text file with 1000 columns and about 1 million rows,
> and I need to transpose this text file so that row become column and
> column become row. (in case you are curious, this is a genotype file).
>
> Can someone recommend me an easy and efficient way to transpose such a
> large dataset, hopefully with Perl ?
>
> Thank you very much!
Out of curiosity, I made this fairly general purpose program for
transposing files (as long as they are tab separated, and rectangular). It
will revert to using multiple passes if it can't open enough temp files to
do it all in one pass. There are more efficient ways of doing it in that
case, but they are more complicated and I'm lazy. On my computer, it seems
to be even faster than reading all the data into memory and buidling
in-memory strings, and of course uses a lot less memory.
It doesn't work to C<seek> on ARGV, so I had to open the (single) input
file explicitly instead. Writes to STDOUT.
use strict;
use warnings;
open my $in, $ARGV[0] or die "$ARGV[0] $!";
my @cols=split /\t/, scalar <$in>;
my $cols=@cols;
my $i=0; ## the first unprocessed column
while ($i<$cols) {
my @fh;
my $j;
## open as many files as the fd limit will let us
foreach ($j=$i;$j<$cols; $j++) {
open $fh[@fh], "+>",undef or do {
die "$j $!" unless $!{EMFILE};
pop @fh;
last;
};
};
$j--;
##warn "working on columns $i..$j";
seek $in,0,0 or die $!;
while (<$in>) { chomp;
my $x=0;
print {$fh[$x++]} "\t$_" or die $! foreach (split/\t/)[$i..$j];
}
foreach my $x (@fh) {
seek $x,0,0 or die $!;
$_=<$x>;
s/^\t//; # chop the unneeded leading tab
print "$_\n"
}
$i=$j+1;
}
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
------------------------------
Date: Fri, 10 Aug 2007 21:11:57 +0000 (UTC)
From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
Subject: Re: how to tranpose a huge text file
Message-Id: <f9ikat$j8d$1@agate.berkeley.edu>
[A complimentary Cc of this posting was sent to
Ted Zlatanov
<tzz@lifelogs.com>], who wrote in article <m27io39zg5.fsf@lifelogs.com>:
> To avoid the 1000-open-files solution, you can do the following:
> while (<F>)
> {
> chomp;
> my @data = split ' '; # you may want to ensure the size of @data is the same every time
> $size = scalar @data; # but $size will have the LAST size of @data
> print B join("\n", @data), "\n";
> }
This is a NULL operation. You just converted " " to "\n".
Essentially, nothing changed. [And $size is not used.]
> Now you can write each inverted output line by looking in break.txt,
> reading every line, chomp() it, and append it to your current output
> line if it's divisible by 1000 (so 0, 1000, 2000, etc. will match).
> Write "\n" to end the current output line.
Good. So what you suggest, is 1000 passes over a 4GB file. Good luck!
Hope this helps,
Ilya
------------------------------
Date: Fri, 10 Aug 2007 17:03:41 -0500
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: how to tranpose a huge text file
Message-Id: <m2absz82te.fsf@lifelogs.com>
On Fri, 10 Aug 2007 21:11:57 +0000 (UTC) Ilya Zakharevich <nospam-abuse@ilyaz.org> wrote:
IZ> [A complimentary Cc of this posting was sent to
IZ> Ted Zlatanov
IZ> <tzz@lifelogs.com>], who wrote in article <m27io39zg5.fsf@lifelogs.com>:
>> To avoid the 1000-open-files solution, you can do the following:
>> while (<F>)
>> {
>> chomp;
>> my @data = split ' '; # you may want to ensure the size of @data is the same every time
>> $size = scalar @data; # but $size will have the LAST size of @data
>> print B join("\n", @data), "\n";
>> }
IZ> This is a NULL operation. You just converted " " to "\n".
IZ> Essentially, nothing changed.
I disagree, but it's somewhat irrelevant, see at end...
IZ> [And $size is not used.]
It's necessary later when you are jumping $size line forward (I used
1000 in the example later). It's also handy to check if @data is not
the right size compared to the last line. Sorry I didn't mention that.
>> Now you can write each inverted output line by looking in break.txt,
>> reading every line, chomp() it, and append it to your current output
>> line if it's divisible by 1000 (so 0, 1000, 2000, etc. will match).
>> Write "\n" to end the current output line.
IZ> Good. So what you suggest, is 1000 passes over a 4GB file. Good luck!
I suggested a database, actually. I specifically said I don't recommend
doing this with file operations. I agree my file-based approach isn't
better than what you and others have suggested, but it does avoid the
multiple open files, and it has low memory usage.
Ted
------------------------------
Date: Fri, 10 Aug 2007 22:28:34 +0000 (UTC)
From: Ilya Zakharevich <nospam-abuse@ilyaz.org>
Subject: Re: how to tranpose a huge text file
Message-Id: <f9ioqi$ke5$1@agate.berkeley.edu>
[A complimentary Cc of this posting was sent to
Ted Zlatanov
<tzz@lifelogs.com>], who wrote in article <m2absz82te.fsf@lifelogs.com>:
> >> Now you can write each inverted output line by looking in break.txt,
> >> reading every line, chomp() it, and append it to your current output
> >> line if it's divisible by 1000 (so 0, 1000, 2000, etc. will match).
> >> Write "\n" to end the current output line.
>
> IZ> Good. So what you suggest, is 1000 passes over a 4GB file. Good luck!
>
> I suggested a database, actually.
And why do you think this would decrease the load on head seeks?
Either the data fits in memory (then database is not needed), or it is
read from disk (which would, IMO, imply the same amount of seeks with
database as with any other file-based operation).
One needs not a database, but a program with build-in caching
optimized for non-random access to 2-dimensional arrays. AFAIK,
imagemagick is mostly memory-based. On the other side of spectrum,
GIMP is based on tile-caching algorithms; if there were a way to
easily hook into this algorithm (with no screen display involved), one
could handle much larger datasets.
Yet another way might be compression; suppose that there are only
(e.g.) 130 "types" of entries; then one can compress the matrix into
1GB of data, which should be handled easily by almost any computer.
Hope this helps,
Ilya
------------------------------
Date: Fri, 10 Aug 2007 19:31:01 -0500
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: how to tranpose a huge text file
Message-Id: <m2tzr67vzu.fsf@lifelogs.com>
On Fri, 10 Aug 2007 22:28:34 +0000 (UTC) Ilya Zakharevich <nospam-abuse@ilyaz.org> wrote:
IZ> [A complimentary Cc of this posting was sent to
IZ> Ted Zlatanov
IZ> <tzz@lifelogs.com>], who wrote in article <m2absz82te.fsf@lifelogs.com>:
>> >> Now you can write each inverted output line by looking in break.txt,
>> >> reading every line, chomp() it, and append it to your current output
>> >> line if it's divisible by 1000 (so 0, 1000, 2000, etc. will match).
>> >> Write "\n" to end the current output line.
>>
IZ> Good. So what you suggest, is 1000 passes over a 4GB file. Good luck!
>>
>> I suggested a database, actually.
IZ> And why do you think this would decrease the load on head seeks?
IZ> Either the data fits in memory (then database is not needed), or it is
IZ> read from disk (which would, IMO, imply the same amount of seeks with
IZ> database as with any other file-based operation).
Look, databases are optimized to store large amounts of data
efficiently. You can always create a hand-tuned program that will do
one task (e.g. transposing a huge text file) well, but you're missing
the big picture: future uses of the data. I really doubt the only thing
anyone will ever want with that data is to transpose it.
IZ> One needs not a database, but a program with build-in caching
IZ> optimized for non-random access to 2-dimensional arrays. AFAIK,
IZ> imagemagick is mostly memory-based. On the other side of spectrum,
IZ> GIMP is based on tile-caching algorithms; if there were a way to
IZ> easily hook into this algorithm (with no screen display involved), one
IZ> could handle much larger datasets.
You and everyone else are overcomplicating this.
Rewrite the original input file for fixed-length records. Then you just
need to seek to a particular offset to read a record, and the problem
becomes transposing a matrix piece by piece. This is fairly simple.
IZ> Yet another way might be compression; suppose that there are only
IZ> (e.g.) 130 "types" of entries; then one can compress the matrix into
IZ> 1GB of data, which should be handled easily by almost any computer.
You need 5 bits per item: it has 16 possible values ([ACTG]{2}), plus
"--".
A database table, to come back to my point, would store these items as
enums. Then you, the user, don't have to worry about the bits per item
in the storage, and you can just use the database.
Ted
------------------------------
Date: Fri, 10 Aug 2007 17:21:12 -0700
From: Bill H <bill@ts1000.us>
Subject: Re: Need help writing a basic script print to text file
Message-Id: <1186791672.301164.246720@k79g2000hse.googlegroups.com>
On Aug 10, 2:52 pm, chewdon <neilverl...@gmail.com> wrote:
> I am trying to write a cgi script that will print text to a txt file
> with the name of the txt file being the date and time it was created.
> I am new to this and the only script I have been able to write is
> Hello World!. From what I have found on the internet I tried to write
> this but of course it does not work. This is what I have written.
>
> #!/usr/local/bin/perl
>
> #sends user back to my home page
>
> Location: "http://www.mywebsite.com/index.html
>
> print "text file data"; >> textfile.txt
>
> This seems like it should be a pretty simple script but I can not
> figure it out. Can anybody please tell me how to create this script?
>
> Thanks
Heres a brute force example:
#!/usr/local/bin/perl
$filename = time();
$filename += ".txt";
open(FILE,">$filename");
print FILE "any and all text you want to write\n";
close(FILE);
print "Location: http://www.mywebsite.com/index.html\n\n";
exit;
It's missing a bunch of things, flock, die etc - but brute force will
create a file that is named for the complete time (actually number of
seconds since the epoch).
Bill H
------------------------------
Date: Fri, 10 Aug 2007 12:53:27 -0500
From: Billy Patton <bpatton@ti.com>
Subject: Re: pass by reference
Message-Id: <f9i8mn$nkq$1@home.itg.ti.com>
anno4000@radom.zrz.tu-berlin.de wrote:
> Billy Patton <bpatton@ti.com> wrote in comp.lang.perl.misc:
>> Larry wrote:
>>> The following code snippet shows how I do "pass by reference" in
>>> Perl. It works fine, but I'm wondering... is there a less verbose way
>>> to do this?
>
> [...]
>
>> Do you really want side effects ?
>> Passing by reference does this.
>> Guess this is too simple to make that decision.
>>
>> sub squareMe($) {
>> my ($arg) = @_;
>> $arg *= $arg;
>> }
>>
>> my $n=7;
>> $n = squareMe $n; # no side effects
>> I hate it when my variables get changed somewhere else and I have to
>> track it down for debugging!
>
> That raises the question why you are using the mutator "*=" at all. It
> doesn't do anything useful.
>
> sub square_me {
> my $arg = shift;
> $arg*$arg;
> }
>
> would be exactly equivalent, as would be
>
> sub square_me { $_[0]*$_[0] }
>
> Anno
The beauty of perl "Always more than one way to skin a cat" :)
------------------------------
Date: Fri, 10 Aug 2007 16:21:15 -0400
From: Sherm Pendley <spamtrap@dot-app.org>
Subject: Re: Perl On Apache
Message-Id: <m2643ncf9g.fsf@dot-app.org>
Louis <t051315@hotmail.com> writes:
> I have a web application that at some point I would like to run a perl
> script on the web server to update an excel spreadsheet on the server.
>
> My problem is..Apache seems to be running it as a cgi script(?). It
> displays the script on the browser!
On the contrary, that shows that Apache is not running your script; if it
were doing so, you'd be seeing your script's output, not your script.
> Is there a way to do this? What is needed to configure Apache to run
> it as a "normal" script?
Apache doesn't run "normal" scripts. Your Perl script must run as either a
CGI (easier) or as a mod_perl server extension ((usually) faster). Both
CGI and mod_perl environments impose requirements above and beyond those
imposed on "normal" scripts run in a terminal, such as returning a proper
set of HTTP headers, and (usually, but not always) HTML output.
To get started with CGI in Perl, have a look at the CGI meta-faq:
<http://www.perl.org/CGI_MetaFAQ.html>
sherm--
--
Web Hosting by West Virginians, for West Virginians: http://wv-www.net
Cocoa programming in Perl: http://camelbones.sourceforge.net
------------------------------
Date: Fri, 10 Aug 2007 17:08:25 -0700
From: Bill H <bill@ts1000.us>
Subject: Re: Perl On Apache
Message-Id: <1186790905.471174.254430@g4g2000hsf.googlegroups.com>
On Aug 10, 2:13 pm, Louis <t051...@hotmail.com> wrote:
> I have a web application that at some point I would like to run a perl
> script on the web server to update an excel spreadsheet on the server.
>
> My problem is..Apache seems to be running it as a cgi script(?). It
> displays the script on the browser!
>
> Is there a way to do this? What is needed to configure Apache to run it
> as a "normal" script?
>
> Thank you.
Louis - you need to enable it. Look in your httpd.conf file, either
in : /usr/local/etc/apache/httpd.conf or in /etc/apache/httpd.conf -
For more information on this file look at www.apache.org. Curious what
instalation are you using that didnt enable perl under apache on
installation?
Bill H
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 743
**************************************
