[29504] in Perl-Users-Digest
Perl-Users Digest, Issue: 748 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sun Aug 12 16:09:43 2007
Date: Sun, 12 Aug 2007 13:09:10 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sun, 12 Aug 2007 Volume: 11 Number: 748
Today's topics:
Directory change and system <bill@ts1000.us>
Re: Directory change and system <mritty@gmail.com>
Re: how to tranpose a huge text file <tzz@lifelogs.com>
Optimized count of files in tree <patrick@frii.fr>
Re: Optimized count of files in tree <mritty@gmail.com>
Re: Optimized count of files in tree <patrick@frii.fr>
Re: Optimized count of files in tree <m@rtij.nl.invlalid>
Re: Optimized count of files in tree <bik.mido@tiscalinet.it>
Pagination II webmaster@valleywebnet.com
Re: Pagination webmaster@valleywebnet.com
Re: Pagination <ts@dionic.net>
Re: Pagination <noreply@gunnar.cc>
Re: Pagination webmaster@valleywebnet.com
Re: Pagination <spamtrap@dot-app.org>
Re: pass by reference <bik.mido@tiscalinet.it>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sun, 12 Aug 2007 09:48:21 -0700
From: Bill H <bill@ts1000.us>
Subject: Directory change and system
Message-Id: <1186937301.480325.112740@r34g2000hsd.googlegroups.com>
I am using system in script to run ImageMagick top convert a file to a
different format, but for it to work I have to be in the directory
that the file is in before doing the system. What would be the best
way of saving the current directory, change to the new one, doing my
system and then going back to the old one?
Bill H
------------------------------
Date: Sun, 12 Aug 2007 16:56:56 -0000
From: Paul Lalli <mritty@gmail.com>
Subject: Re: Directory change and system
Message-Id: <1186937816.550620.156840@22g2000hsm.googlegroups.com>
On Aug 12, 12:48 pm, Bill H <b...@ts1000.us> wrote:
> I am using system in script to run ImageMagick top convert a file to a
> different format, but for it to work I have to be in the directory
> that the file is in before doing the system. What would be the best
> way of saving the current directory, change to the new one, doing my
> system and then going back to the old one?
use Cwd;
my $old_dir = getcwd();
chdir $new_dir or die "Cannot change to $new_dir: $!";
system($IM_cmd);
chdir $old_dir or die "Cannot change back to $old_dir: $!";
Paul Lalli
------------------------------
Date: Sun, 12 Aug 2007 07:29:22 -0500
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: how to tranpose a huge text file
Message-Id: <m2fy2p542l.fsf@lifelogs.com>
On Sun, 12 Aug 2007 06:19:37 +0000 (UTC) Ilya Zakharevich <nospam-abuse@ilyaz.org> wrote:
IZ> [A complimentary Cc of this posting was sent to
IZ> Ted Zlatanov
IZ> <tzz@lifelogs.com>], who wrote in article <m2d4xu6ptv.fsf@lifelogs.com>:
>> You mean "SELECT column FROM table" (which is what you need in order to
>> write each line of the transposed file)? I think that's a pretty common
>> access pattern, and would perform well in most databases.
IZ> I have very little experience with databases. However, my hunch
IZ> (based on my experience with other types of software) is that they are
IZ> not as optimized as an optimistic point of view may imply.
I respect your experience and skills you have demonstrated repeatedly in
this newsgroup and in Perl's source code base. I think you may want to
investigate database technology further; software like PostgreSQL and
SQLite has gathered significant traction and respect, and is adaptable
to many technical tasks.
IZ> Doing one "SELECT column FROM table" (which results in a megaarray)
IZ> may be not *that* slow - given quick enough hardware. Doing it 1000
IZ> times would unravel all the missing optimizations in the
IZ> implementation of the database.
You don't have to get all the results of a SELECT at once. You can
fetch each row of the result set, which does not generate a large array.
I don't know the internal mechanics of result set management in the
various databases on the market, nor do I want to; from practical
experience with large result sets under Oracle I can tell you that
performance is quite good there. Do you want benchmarks to be convinced?
Ted
------------------------------
Date: Sun, 12 Aug 2007 15:46:36 +0200
From: Patrick <patrick@frii.fr>
Subject: Optimized count of files in tree
Message-Id: <46bf0f3a$0$422$426a34cc@news.free.fr>
Hello,
In an application I write in Perl, I must count the total number of
files (not directories) in a complete tree.
What is the most efficient way to do it ?
My current code is :
use File::Recurse;
my $nb = 0;
recurse { -f && $nb++ } $dir;
With this code, I can scan 10,000 files in 15 seconds.
I want to know if it exists a best way (= quicker) to do it ?
Thanks for your help.
Patrick
------------------------------
Date: Sun, 12 Aug 2007 06:57:19 -0700
From: Paul Lalli <mritty@gmail.com>
Subject: Re: Optimized count of files in tree
Message-Id: <1186927039.144077.275720@22g2000hsm.googlegroups.com>
On Aug 12, 9:46 am, Patrick <patr...@frii.fr> wrote:
> In an application I write in Perl, I must count the total number of
> files (not directories) in a complete tree.
>
> What is the most efficient way to do it ?
>
> My current code is :
>
> use File::Recurse;
> my $nb = 0;
> recurse { -f && $nb++ } $dir;
>
> With this code, I can scan 10,000 files in 15 seconds.
>
> I want to know if it exists a best way (= quicker) to do it ?
I don't know the answer for sure, but looking at File::Recurse, it
seems to be a bit bloated for what you want to do. In addition to
recursing through the directory structure, it also checks a hash of
options for each file found, and stores information about each entry
to be later returned from the recurse() function.
I would try just using the standard File::Find module and see if it's
any faster.
use File::Find;
my $nb = 0;
find($dir, sub { -f and $nb++ });
You may also wish to use the standard Benchmark module to actually
compare the two techniques.
Paul Lalli
------------------------------
Date: Sun, 12 Aug 2007 16:44:14 +0200
From: Patrick <patrick@frii.fr>
Subject: Re: Optimized count of files in tree
Message-Id: <46bf1cbb$0$413$426a34cc@news.free.fr>
Paul Lalli a =E9crit :
> On Aug 12, 9:46 am, Patrick <patr...@frii.fr> wrote:
>> In an application I write in Perl, I must count the total number of
>> files (not directories) in a complete tree.
>>
>> What is the most efficient way to do it ?
>>
>> My current code is :
>>
>> use File::Recurse;
>> my $nb =3D 0;
>> recurse { -f && $nb++ } $dir;
>>
>> With this code, I can scan 10,000 files in 15 seconds.
>>
>> I want to know if it exists a best way (=3D quicker) to do it ?
>=20
> I don't know the answer for sure, but looking at File::Recurse, it
> seems to be a bit bloated for what you want to do. In addition to
> recursing through the directory structure, it also checks a hash of
> options for each file found, and stores information about each entry
> to be later returned from the recurse() function.
>=20
> I would try just using the standard File::Find module and see if it's
> any faster.
>=20
> use File::Find;
> my $nb =3D 0;
> find($dir, sub { -f and $nb++ });
>=20
> You may also wish to use the standard Benchmark module to actually
> compare the two techniques.
>=20
> Paul Lalli
>=20
Thanks for your answer but I have tried with File::Find : I got exactly=20
the same time for the same tree : 16 seconds for 10,000 files.
I have also tried with a "manual" solution :
sub getFileNb {
my $nb =3D 0;
return if ! opendir DIR,$_;
my @list =3D readdir DIR;
closedir DIR;
$nb +=3D grep { -f "$dir/$_" } @list;
my @subdirs =3D grep { /^[^\.]/ && -d "$dir/$_" } @list;
foreach ( @subdirs ) {
$nb +=3D &getFileNb("$dir/$_");
}
return $nb;
}
The result is about the same : 16 seconds for my 10,000 files.
I search for a really better performance ...
Patrick
------------------------------
Date: Sun, 12 Aug 2007 17:40:57 +0200
From: Martijn Lievaart <m@rtij.nl.invlalid>
Subject: Re: Optimized count of files in tree
Message-Id: <pan.2007.08.12.15.40.57@rtij.nl.invlalid>
On Sun, 12 Aug 2007 16:44:14 +0200, Patrick wrote:
> Thanks for your answer but I have tried with File::Find : I got exactly
> the same time for the same tree : 16 seconds for 10,000 files.
>
> I have also tried with a "manual" solution :
>
> sub getFileNb {
> my $nb = 0;
>
> return if ! opendir DIR,$_;
> my @list = readdir DIR;
> closedir DIR;
>
> $nb += grep { -f "$dir/$_" } @list;
> my @subdirs = grep { /^[^\.]/ && -d "$dir/$_" } @list;
>
> foreach ( @subdirs ) {
> $nb += &getFileNb("$dir/$_");
> }
>
> return $nb;
> }
>
> The result is about the same : 16 seconds for my 10,000 files.
>
> I search for a really better performance ...
Looks like IO is the bottleneck. If you are on unix, do a
$ time find . -type f | wc -l
and see if that is significantly faster. If not, get a faster harddisk,
put the files on another (fast(er)) harddisk, switch to raid, add more
memory so you have more buffers, tume OS parameters, use another type of
filesystem. Mix and match to taste.
Note that
$ time yourscript.pl will give you insight in how much time is spend
waiting on I/O. Real-(user+sys) is the time spend waiting on other tasks
and IO. On a lightly loaded system, this will be mainly IO.
Look at this:
$ time find . -type f | wc -l
49956
real 0m14.594s
user 0m0.216s
sys 0m1.609s
The find command took only a fraction of the total time. One second was
spend in the kernel and 14 seconds was spend on IO.
HTH,
M4
------------------------------
Date: Sun, 12 Aug 2007 19:07:33 +0200
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: Optimized count of files in tree
Message-Id: <nbfub3hjglaof74gc403ham6uuom5qr138@4ax.com>
On Sun, 12 Aug 2007 16:44:14 +0200, Patrick <patrick@frii.fr> wrote:
>I have also tried with a "manual" solution :
>
>sub getFileNb {
> my $nb =3D 0;
>
> return if ! opendir DIR,$_;
> my @list =3D readdir DIR;
> closedir DIR;
>
> $nb +=3D grep { -f "$dir/$_" } @list;
> my @subdirs =3D grep { /^[^\.]/ && -d "$dir/$_" } @list;
You're stat()ing twice. It is time consuming, you'd better do it once.
>
> foreach ( @subdirs ) {
> $nb +=3D &getFileNb("$dir/$_");
> }
The &-form of sub call is obsolete and not likely to do what you mean.
Just avoid it.
>The result is about the same : 16 seconds for my 10,000 files.
>
>I search for a really better performance ...
I wouldn't go for a recursive solution then, but for an iterative one.
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: Sun, 12 Aug 2007 16:50:08 -0000
From: webmaster@valleywebnet.com
Subject: Pagination II
Message-Id: <1186937408.035474.227210@g4g2000hsf.googlegroups.com>
Thanks to everyone here, I am closer to getting this working but am
still having a problem....
The script runs and outputs a page, but in the db I have set up all of
the categories have enough entries to fill multiple pages. So, the
script runs and always goes to page 2 for some reason.
And if I click on one of the other links for a different page, then I
get nothing on that one.
Can anyone spot the problem?
Thanks in advance,
Jim
#!/usr/bin/perl -w
use DBI;
use CGI;
use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
use CGI qw(:standard);
use POSIX;
use HTML::Template;
use strict;
use diagnostics;
my $limit;
my $q = new CGI;
my $find = $q->param("search");
if (!$limit) {$limit = 5;}
my $dbname = "DBI:mysql:farthing_valleyweb:localhost";
my $dbusername = "farthing_farthin";
my $dbpassword = "ginajim";
my( $ID, $category, $name, $description, $contact, $phone, $fax,
$address, $city, $state, $zip, $email, $url, $keywords );
my $dbh = DBI->connect($dbname, $dbusername, $dbpassword)
or die ("Connection to database failed: $!\n");
my $sql = "select * from valley where category like ?";
my $sth = $dbh->prepare($sql) or die("Error in SQL1\n");
$sth->execute($find) or die "Error in SQL2 $!\n";
my $results = $sth->rows;
my $results_per_page = 5;
my $pagesrequired = ceil($results / $results_per_page);
my $sql = "select * from valley where category like ?
limit $limit, $results_per_page";
my $sth = $dbh->prepare($sql) or die("Error in SQL3\n");
$sth->execute($find) or die ("Error in SQL4\n");
$sth->bind_columns( \$ID, \$category, \$name, \$description, \
$contact, \$phone, \$fax, \$address, \$city, \$state, \$zip, \$email, \
$url, \$keywords );
while( $sth->fetch() ) {
print "<b>$name</b><br />
$phone<br />
$address<br />
$city, $state $zip<p />\n";
}
for (my $i = 0; $i <= $pagesrequired -1; $i++) {
if ($i == 0) {
if ($limit != 0) {
print "<a href=\"search.cgi?limit=0&find=$find\">";
print $i + 1;
print "</a>";
}
else {print $i + 1;}
}
if ($i > 0) {
if ($limit != ($i * $results_per_page)) {
print " | <a href=\"search.cgi?limit=";
print ($i * $results_per_page);
print "&find=$find\">\n";
print $i + 1, "</a>";
}
else {print " | ", $i + 1;}
}
}
------------------------------
Date: Sun, 12 Aug 2007 01:39:21 -0700
From: webmaster@valleywebnet.com
Subject: Re: Pagination
Message-Id: <1186907961.711396.100700@57g2000hsv.googlegroups.com>
On Aug 11, 5:03 pm, Robert Hicks <sigz...@gmail.com> wrote:
> On Aug 11, 2:40 pm, webmas...@valleywebnet.com wrote:
>
> > I have a DB with ~1100 entries that I have made searchable. However,
> > the results can be VERY long. So, I want a way of paginating the
> > output.
>
> > I have seen oodles of these types of scripts for PHP, but nothing for
> > PERL, and my PERL skills are rusty.
>
> > Anyone seen somethng like this for PERL, or have an example lying
> > around?
>
> > Thanks!
> > Jim
>
> HTML::Pager
> Data::Pager
> Data::Pageset
>
> Try one of those...
>
> HTH
>
> Robert
Unfortunately, I can't use any of those, my server doesn't have them
installed and getting them to install something new is like pulling
teeth from an alligator.
Anyone have any other suggestions?
Jim
------------------------------
Date: Sun, 12 Aug 2007 10:03:07 +0100
From: Tim Southerwood <ts@dionic.net>
Subject: Re: Pagination
Message-Id: <46beccb2$0$640$5a6aecb4@news.aaisp.net.uk>
webmaster@valleywebnet.com coughed up some electrons that declared:
>
> Unfortunately, I can't use any of those, my server doesn't have them
> installed and getting them to install something new is like pulling
> teeth from an alligator.
>
> Anyone have any other suggestions?
> Jim
Can you not "install" them adjacent to your script in a lib/ directory, then
path that into your script with a "use lib ..." and maybe a bit of "use
FindBin" for flexibility?
That's what I often do if a centralised installation is more trouble than
it's worth.
Cheers
Tim
------------------------------
Date: Sun, 12 Aug 2007 11:02:46 +0200
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: Pagination
Message-Id: <5i80loF3mdetrU1@mid.individual.net>
webmaster@valleywebnet.com wrote:
> On Aug 11, 5:03 pm, Robert Hicks <sigz...@gmail.com> wrote:
>> On Aug 11, 2:40 pm, webmas...@valleywebnet.com wrote:
>>> I have a DB with ~1100 entries that I have made searchable. However,
>>> the results can be VERY long. So, I want a way of paginating the
>>> output.
>>>
>>> I have seen oodles of these types of scripts for PHP, but nothing for
>>> PERL, and my PERL skills are rusty.
>>> Anyone seen somethng like this for PERL, or have an example lying
>>
>> HTML::Pager
>> Data::Pager
>> Data::Pageset
>>
>> Try one of those...
>
> Unfortunately, I can't use any of those, my server doesn't have them
> installed and getting them to install something new is like pulling
> teeth from an alligator.
Sounds tough... Then how about installing one of them yourself?
http://perldoc.perl.org/perlfaq8.html#How-do-I-keep-my-own-module%2flibrary-directory%3f
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: Sun, 12 Aug 2007 10:57:11 -0000
From: webmaster@valleywebnet.com
Subject: Re: Pagination
Message-Id: <1186916231.935772.164170@r34g2000hsd.googlegroups.com>
On Aug 12, 5:03 am, Tim Southerwood <t...@dionic.net> wrote:
> webmas...@valleywebnet.com coughed up some electrons that declared:
>
>
>
> > Unfortunately, I can't use any of those, my server doesn't have them
> > installed and getting them to install something new is like pulling
> > teeth from an alligator.
>
> > Anyone have any other suggestions?
> > Jim
>
> Can you not "install" them adjacent to your script in a lib/ directory, then
> path that into your script with a "use lib ..." and maybe a bit of "use
> FindBin" for flexibility?
>
> That's what I often do if a centralised installation is more trouble than
> it's worth.
>
> Cheers
>
> Tim
I didn't think of that, excellent suggestion.
Thanks!
Jim
------------------------------
Date: Sun, 12 Aug 2007 15:02:51 -0400
From: Sherm Pendley <spamtrap@dot-app.org>
Subject: Re: Pagination
Message-Id: <m2643ka84k.fsf@dot-app.org>
webmaster@valleywebnet.com writes:
> Unfortunately, I can't use any of those, my server doesn't have them
> installed and getting them to install something new is like pulling
> teeth from an alligator.
>
> Anyone have any other suggestions?
Install it yourself - you only need root access if you want to install a
module into a directory to which only root can write, such as /usr/lib.
In a nutshell, just download the module tarball, unpack it, and specify
an install location to which you have access when you run Makefile.PL. For
example:
perl Makefile.PL PREFIX=/home/mylogin
The above assumes you've already downloaded and unpacked the tarball, of
course. You can also configure the CPAN shell to use a configuration step
like this automatically.
For details, see:
perldoc perlmodinstall
perldoc CPAN
sherm--
--
Web Hosting by West Virginians, for West Virginians: http://wv-www.net
Cocoa programming in Perl: http://camelbones.sourceforge.net
------------------------------
Date: Sun, 12 Aug 2007 10:57:42 +0200
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: pass by reference
Message-Id: <jjitb3lpgib3rtm3g1ul9a4476ruc3d6fp@4ax.com>
On Sat, 11 Aug 2007 14:50:14 -0700, "Clenna Lumina"
<savagebeaste@yahoo.com> wrote:
>>> $_ is a name, just not a pronounceable one!
>>
>> In English, it's "it".
>
>Yes, in the context of for loops*, map, and such, from what I understand
>that's exactly what "it" is :)
>
>* Though it doesn't seem to be the came of a while loop onless you're
>reading a handle (file, pipe, socket, etc) using the <HANDLE> systax
>that sets the line to $_ each time around.
It's more or less still "it", the rationale being that a natural
language concept like a pronoun can fit nicely in the model of a
programming one, but it won't map just *exactly*: Perl *resembles*
English in some points, but obviously it is *not* English:
: If I convert a program into English, I don't get Shakespeare, but
: Cobol. A subset of English fit for morons and guaranteed not to make
: use of its expressive powers.
: - David Kastrup in comp.text.tex, "Re: pdflatex or dvipdf ?"
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 748
**************************************
