[32730] in Perl-Users-Digest
Perl-Users Digest, Issue: 3994 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Fri Jul 19 14:09:28 2013
Date: Fri, 19 Jul 2013 11:09:06 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Fri, 19 Jul 2013 Volume: 11 Number: 3994
Today's topics:
Re: names, values, boxes and microchips <ben@morrow.me.uk>
Re: names, values, boxes and microchips <rweikusat@mssgmbh.com>
Re: Open source Catalyst applications, for study? <ben@morrow.me.uk>
Restart Perl Application upon KDE Restart <josef.moellers@invalid.invalid>
Re: Restart Perl Application upon KDE Restart <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Re: Restart Perl Application upon KDE Restart <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Re: Restart Perl Application upon KDE Restart <news@lawshouse.org>
Re: the fastest way to create a directory <kkeller-usenet@wombat.san-francisco.ca.us>
Re: the fastest way to create a directory <justin.1303@purestblue.com>
Re: the fastest way to create a directory <cwilbur@chromatico.net>
Re: the fastest way to create a directory <news@lawshouse.org>
Re: the fastest way to create a directory <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Re: the fastest way to create a directory <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Re: the fastest way to create a directory <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Re: the fastest way to create a directory <rweikusat@mssgmbh.com>
Re: the fastest way to create a directory <rweikusat@mssgmbh.com>
Re: the fastest way to create a directory <rweikusat@mssgmbh.com>
Re: the fastest way to create a directory <hjp-usenet3@hjp.at>
Re: the fastest way to create a directory <rweikusat@mssgmbh.com>
Re: the fastest way to create a directory <ben@morrow.me.uk>
Re: the fastest way to create a directory <rweikusat@mssgmbh.com>
Re: the fastest way to create a directory <kkeller-usenet@wombat.san-francisco.ca.us>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Fri, 19 Jul 2013 17:07:47 +0100
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: names, values, boxes and microchips
Message-Id: <jr4oba-g85.ln1@anubis.morrow.me.uk>
Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
> Instead of trying another explanation of the 'it is a name given to a
> value' versus 'it is a register' issue, I thought I should rather try
> an example.
>
> Introductory remarks: The subroutine whose code is included below
> takes two sorted lists of 'DNS [reverse] domain objects' as arguments
> (represented as references to arrays[*]) and is supposed to produce an
> output list of all objects on the first input list which don't
> 'conflict' with the items on the filter list. One application this is
> put to is to determine which RFC1918 reverse zones still need to be
> 'terminated' on some DNS server, given a list of reverse domains
> configured by a customer which may cover some or all of the RFC1918
> address space(s).
[...]
> sub filter_against
> {
> my ($in, $filter) = @_;
> my ($in_item, $in_pos, $filter_item, $filter_pos);
> my (@out, $step_in, $step_filter, $rc);
>
> $in_pos = 0;
> $in_item = $in->[0];
> $step_in = sub {
> last LOOP if ++$in_pos == @$in;
> $in_item = $in->[$in_pos];
> };
>
> $filter_pos = 0;
> $filter_item = $filter->[0];
> $step_filter = sub {
> last LOOP if ++$filter_pos == @$filter;
> $filter_item = $filter->[$filter_pos];
> };
>
> LOOP: {
> ($rc) = $in_item->net_compare($filter_item);
>
> p_alloc('in %s, filter %s, rc %u',
> $in_item, $filter_item, $rc);
>
> given ($rc) {
> when (R_AFTER) {
> push(@out, $in_item);
> &$step_in;
> }
>
> when ([R_BEFORE, R_SUB]) {
> &$step_filter;
> }
>
> when ([R_SAME, R_SUPER]) {
> p_alloc('%s: dropping %s (%s)', __func__,
> $in_item, $filter_item);
>
> &$step_in;
> }
> }
>
> redo;
> }
>
> push(@out, @$in[$in_pos .. $#$in]);
> return \@out;
> }
There are times when it's necessary to write in that sort of state-
machine style, but it's always clearer to avoid it when you can.
use List::MoreUtils qw/part/;
my %rc = (
R_AFTER => 0,
R_SAME => 1,
R_SUPER => 1,
R_BEFORE => 2,
R_SUB => 2,
);
sub filter_against {
my ($in, $filter) = @_;
my @out;
for my $f (@$filter) {
(my $out, my $drop, $in) = part {
my ($rc) = $_->net_compare($f);
$rc{$rc} // die "bad return from net_compare";
} @$in;
push @out, @$out;
p_alloc("%s: dropping %s (%s)", __func__, $_, $f)
for @$drop;
}
return \@out;
}
This does more compares than yours, but given small-lists-infrequently-
compared I doubt that matters. If it does it should be fixable with
firstidx and splice, but that would have made the algorithm less clear.
I would rather have written the loop as a map, but Perl doesn't have
map-with-named-iterator. I don't think the use of $in as a state
variable can easily be eliminated, though there might be some 'reduce'
variant which can do so.
Ben
------------------------------
Date: Fri, 19 Jul 2013 17:38:44 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: names, values, boxes and microchips
Message-Id: <87bo5y5wtn.fsf@sapphire.mobileactivedefense.com>
Ben Morrow <ben@morrow.me.uk> writes:
> Quoth Rainer Weikusat <rweikusat@mssgmbh.com>:
[...]
>> sub filter_against
[...]
> There are times when it's necessary to write in that sort of state-
> machine style, but it's always clearer to avoid it when you can.
>
> use List::MoreUtils qw/part/;
>
> my %rc = (
> R_AFTER => 0,
> R_SAME => 1,
> R_SUPER => 1,
> R_BEFORE => 2,
> R_SUB => 2,
> );
>
> sub filter_against {
> my ($in, $filter) = @_;
>
> my @out;
> for my $f (@$filter) {
> (my $out, my $drop, $in) = part {
> my ($rc) = $_->net_compare($f);
> $rc{$rc} // die "bad return from net_compare";
> } @$in;
>
> push @out, @$out;
> p_alloc("%s: dropping %s (%s)", __func__, $_, $f)
> for @$drop;
> }
>
> return \@out;
> }
>
> This does more compares than yours, but given small-lists-infrequently-
> compared I doubt that matters.
Judgeing from a cursory look at the documentation, it also does a lot
of other stuff it shouldn't be doing and/or doesn't need to do,
including that I have serious doubts if this works at all. But I'm not
really in the mood to figure out if this hypercomplicated attempt at
reverse dicksizing (Look! Mine is shorter than yours!) is semantically
equivalent to the fairly straight-forward list merge based algorithm
I'm using. In any case, this is the kind of code I'd delete without
even trying to fix it as soon as I'd have to look at it for the first
time.
------------------------------
Date: Fri, 19 Jul 2013 16:22:51 +0100
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: Open source Catalyst applications, for study?
Message-Id: <b72oba-ps4.ln1@anubis.morrow.me.uk>
Quoth Jim Cochrane <allergic-to-spam@no-spam-allowed.invalid>:
> I'm looking for some well-designed, relatively modern applications that
> use the Catalyst web framework, with source code available. My goal is
> to study these applications to learn more about current good practices
> in designing and implementing a Catalyst application. I'd appreciate
> any pointers to such applications.
>
> Also, if you know of a newsgroup that is a more appropriate place to
> post this question, please let me know.
Not a newsgroup, but #catalyst on irc.perl.org is worth looking at.
Ben
------------------------------
Date: Thu, 18 Jul 2013 17:20:33 +0200
From: Josef Moellers <josef.moellers@invalid.invalid>
Subject: Restart Perl Application upon KDE Restart
Message-Id: <b4qfe2Fbd7qU1@mid.individual.net>
Under KDE, certain applications register with ksmserver to be restarted
at the next startup, e.g. when I exit without closing firefox,
thunderbird, kate, they are all restarted (more or less successful) when
I relogin.
They are remembered in ~/.kde/share/config/ksmserverrc
Is there a way (e.g. using Net::DBus) to have a perl script restarted? I
doubt that I can just edit ~/.kde/share/config/ksmserverrc.
Thanks,
Josef
------------------------------
Date: Thu, 18 Jul 2013 23:13:21 +0300
From: "George Mpouras" <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Subject: Re: Restart Perl Application upon KDE Restart
Message-Id: <ks9i9d$1bf8$1@news.ntua.gr>
define you script here
/etc/rc.local
or better converted to linux service
vi /etc/init.d/myscript
sub stop {...}
sub start {...}
sub status {...}
sub reload {...}
sysv-rc-conf --level 345 myscript on
------------------------------
Date: Thu, 18 Jul 2013 23:24:07 +0300
From: "George Mpouras" <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Subject: Re: Restart Perl Application upon KDE Restart
Message-Id: <ks9itj$1d3g$1@news.ntua.gr>
you could also start your script like
nohup /usr/local/myscript.pl 2> /var/log/myscript.log 1>&2 &
------------------------------
Date: Thu, 18 Jul 2013 21:46:26 +0100
From: Henry Law <news@lawshouse.org>
Subject: Re: Restart Perl Application upon KDE Restart
Message-Id: <qsqdnTHib9S-yXXMnZ2dnUVZ8hadnZ2d@giganews.com>
On 18/07/13 21:13, George Mpouras wrote:
> define you script here
>
> /etc/rc.local
George, I don't think that's what the OP wants. If you read the post
you'll see
> e.g. when I exit without closing firefox,
> thunderbird, kate, they are all restarted
So I think what's required is some way to register a Perl application
with KDE such that if it is shut down ungracefully (i.e. by the system
rather than the user) then KDE will know to restart it.
It seems that he also wants some "snapshot" capability which will allow
KDE to apply some logic to the restart; in the case of the browser this
means opening the pages that were open at the point of shutdown. What
that might mean in the case of the Perl application I couldn't say.
That said, I have no idea what the answer is.
--
Henry Law Manchester, England
------------------------------
Date: Wed, 17 Jul 2013 22:37:23 -0700
From: Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us>
Subject: Re: the fastest way to create a directory
Message-Id: <jhbkbax3eq.ln2@goaway.wombat.san-francisco.ca.us>
On 2013-07-17, George Mpouras <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam> wrote:
> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
So, either you run your test scripts as root, or your user account has
write permissions to a system directory. Neither of these situations
sounds particularly smart. Given your complete inability to post
properly to usenet, and your inability to refute Peter's benchmarks with
some of your own, I don't see how you can expect anyone to take your
code seriously.
--keith
--
kkeller-usenet@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information
------------------------------
Date: Thu, 18 Jul 2013 09:15:11 +0100
From: Justin C <justin.1303@purestblue.com>
Subject: Re: the fastest way to create a directory
Message-Id: <fpkkba-g67.ln1@zem.masonsmusic.co.uk>
On 2013-07-18, Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us> wrote:
> On 2013-07-17, George Mpouras <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam> wrote:
>> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
>
> So, either you run your test scripts as root, or your user account has
> write permissions to a system directory. Neither of these situations
> sounds particularly smart. Given your complete inability to post
> properly to usenet, and your inability to refute Peter's benchmarks with
> some of your own, I don't see how you can expect anyone to take your
> code seriously.
I don't think anyone takes his code seriously.
Justin.
--
Justin C, by the sea.
------------------------------
Date: Thu, 18 Jul 2013 15:49:12 -0400
From: Charlton Wilbur <cwilbur@chromatico.net>
Subject: Re: the fastest way to create a directory
Message-Id: <87fvvbfy2v.fsf@new.chromatico.net>
>>>>> "GM" == George Mpouras
>>>>> <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
>>>>> writes:
GM> yes there are about thousands dirs. try to undestand the
GM> code. looks simple but it is not
It's not simple. It's also not measurably more efficient than shelling
out to mkdir -p or invoking File::Path's make_path function.
Thou shalt make thy program's purpose and structure clear to thy fellow
man; for thy creativity is better used in solving problems than in
creating beautiful new impediments to understanding.
Charlton
--
Charlton Wilbur
cwilbur@chromatico.net
------------------------------
Date: Thu, 18 Jul 2013 21:47:23 +0100
From: Henry Law <news@lawshouse.org>
Subject: Re: the fastest way to create a directory
Message-Id: <qsqdnTDib9TGyXXMnZ2dnUVZ8hYAAAAA@giganews.com>
On 18/07/13 20:49, Charlton Wilbur wrote:
> Thou shalt make thy program's purpose and structure clear to thy fellow
> man; for thy creativity is better used in solving problems than in
> creating beautiful new impediments to understanding.
I like it! Is that a Charltonism? Attribution please if not.
--
Henry Law Manchester, England
------------------------------
Date: Fri, 19 Jul 2013 00:02:23 +0300
From: "George Mpouras" <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Subject: Re: the fastest way to create a directory
Message-Id: <ks9l5b$1jcm$1@news.ntua.gr>
Believe it or the following makes 18 uneccassery disk accesses
use Errno qw(EEXIST ENOTDIR);
my $access=0;
mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
print "disk touches : $access\n";
sub mkdir_p
{
my ($cur, $next, $remain);
$remain = $_[0];
while ($remain) {
($next, $remain) = $remain =~ /^(\/*[^\/]*)(.*)/;
$cur .= $next;
$access++;
mkdir($cur) or do {
return if $! != EEXIST;
next if $remain;
-d($cur) or $! = ENOTDIR, return;
};
}
return 1;
}
------------------------------
Date: Fri, 19 Jul 2013 01:01:01 +0300
From: "George Mpouras" <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Subject: Re: the fastest way to create a directory
Message-Id: <ks9oj8$1slf$1@news.ntua.gr>
#!/usr/bin/perl
# Some benchmark for what purposed until now ...
use Errno qw(EEXIST ENOTDIR);
use Time::HiRes;
use File::Path;
my $start_time=[];
my $err;
foreach my $code (qw/
perl_module_file_path
mkdir_p_Rainer_Weikusa
Mkdir_recursive
/)
{
*Test_the_code = \&{$code};
system('/bin/rm -rf /tmp/test_this') if -d '/tmp/test_this';
$start_time = [ Time::HiRes::gettimeofday ];
print "Testing $code ... ";
foreach (1..5_000)
{
my $dir = join('/','/tmp/test_this',split '',$_);
Test_the_code($dir) or die("$!");
}
print "finished at ", Time::HiRes::tv_interval($start_time) ," sec\n";
}
# Using Perl module File::Path
sub perl_module_file_path
{
File::Path::mkpath( $_[0], {'error' => \$err} );
@{$err} ? 0 : 1
}
sub mkdir_p_Rainer_Weikusa
{
my ($cur, $next, $remain);
$remain = $_[0];
while ($remain) {
($next, $remain) = $remain =~ /^(\/*[^\/]*)(.*)/;
$cur .= $next;
mkdir($cur) or do {
return if $! != EEXIST;
next if $remain;
-d($cur) or $! = ENOTDIR, return;
};
}
return 1;
}
# George Bouras
sub Mkdir_recursive_George_Bouras
{
return 1 if $_[0] eq '' || -d $_[0];
Mkdir_recursive( $_[0] =~/^(.*?)[\\\/][^\\\/]+$/ ) || return undef;
mkdir $_[0] || return undef
}
------------------------------
Date: Fri, 19 Jul 2013 01:07:42 +0300
From: "George Mpouras" <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
Subject: Re: the fastest way to create a directory
Message-Id: <ks9ovp$1tjv$1@news.ntua.gr>
#!/usr/bin/perl
# Some benchmark for what purposed until now ...
# Some typos corrected.
use strict;
use warnings;
use Errno qw(EEXIST ENOTDIR);
use Time::HiRes;
use File::Path;
my $start_time=[];
my $err;
foreach my $code (qw/
perl_module_file_path
mkdir_p_Rainer_Weikusa
Mkdir_recursive_George_Bouras
/)
{
my $testcode = \&$code;
system('/bin/rm -rf /tmp/test_this') if -d '/tmp/test_this';
$start_time = [ Time::HiRes::gettimeofday ];
print "Testing $code ... ";
foreach (1..5_000)
{
my $dir = join('/','/tmp/test_this',split '',$_);
$testcode->($dir) or die("$!");
}
print "finished at ", Time::HiRes::tv_interval($start_time) ," sec\n";
}
# Using Perl module File::Path
sub perl_module_file_path
{
File::Path::mkpath( $_[0], {'error' => \$err} );
@{$err} ? 0 : 1
}
sub mkdir_p_Rainer_Weikusa
{
my ($cur, $next, $remain);
$remain = $_[0];
while ($remain) {
($next, $remain) = $remain =~ /^(\/*[^\/]*)(.*)/;
$cur .= $next;
mkdir($cur) or do {
return if $! != EEXIST;
next if $remain;
-d($cur) or $! = ENOTDIR, return;
};
}
return 1;
}
# George Bouras
sub Mkdir_recursive_George_Bouras
{
return 1 if $_[0] eq '' || -d $_[0];
Mkdir_recursive_George_Bouras( $_[0] =~/^(.*?)[\\\/][^\\\/]+$/ ) || return
undef;
mkdir $_[0] || return undef
}
------------------------------
Date: Fri, 19 Jul 2013 10:46:05 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: the fastest way to create a directory
Message-Id: <87fvva2882.fsf@sapphire.mobileactivedefense.com>
"George Mpouras"
<nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
writes:
> Believe it or the following makes 18 uneccassery disk accesses
[...]
> use Errno qw(EEXIST ENOTDIR);
> my $access=0;
> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
> print "disk touches : $access\n";
If it does, you're kernel is seriously disabled. Leaving this aside,
that's the expected result when you call it uselessly three times in a
row with an argument like that.
------------------------------
Date: Fri, 19 Jul 2013 10:51:05 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: the fastest way to create a directory
Message-Id: <87bo5y27zq.fsf@sapphire.mobileactivedefense.com>
Rainer Weikusat <rweikusat@mssgmbh.com> writes:
> "George Mpouras"
> <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
> writes:
>> Believe it or the following makes 18 uneccassery disk accesses
>
> [...]
>
>> use Errno qw(EEXIST ENOTDIR);
>> my $access=0;
>> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
>> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
>> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
>> print "disk touches : $access\n";
>
> If it does, you're kernel is seriously disabled. Leaving this aside,
> that's the expected result when you call it uselessly three times in a
> row with an argument like that.
BTW: If that's an actual problem you have, a reasonably simple way to
deal with that would be to use a hash to keep track of pathnames
already known to exist provided keeping them all in memory isn't a
problem.
------------------------------
Date: Fri, 19 Jul 2013 12:09:32 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: the fastest way to create a directory
Message-Id: <87bo5yst5f.fsf@sapphire.mobileactivedefense.com>
Rainer Weikusat <rweikusat@mssgmbh.com> writes:
> "George Mpouras"
> <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
> writes:
>> Believe it or the following makes 18 uneccassery disk accesses
>
> [...]
>
>> use Errno qw(EEXIST ENOTDIR);
>> my $access=0;
>> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
>> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
>> mkdir_p("/opt/d1/d2/d3/d4/d5") or die("$!");
>> print "disk touches : $access\n";
>
> If it does, you're kernel is seriously disabled. Leaving this aside,
> that's the expected result when you call it uselessly three times in a
> row with an argument like that.
Other branch of the tree: No matter if the code was written to work
well if most directories already exist (yours) or most directories
don't exist (mine), the stat isn't needed because mkdir already checks
for that. The only downside of that is that the longest pathname needs
special-case treatment because EEXIST can refer to any kind of
filesystem object.
Since this is sort-of a cute little problem, here's another iterative
one, this time biased in the other direction:
use Errno qw(EEXIST ENOENT ENOTDIR);
sub mkdir_p
{
my ($prefix, $suffix, @tbc);
$prefix = $_[0];
mkdir($prefix) and return 1;
unless ($! == ENOENT) {
if ($! == EEXIST) {
return 1 if -d($prefix);
$! = ENOTDIR;
}
return;
}
while (($prefix, $suffix) = $prefix =~ /(.*?)([^\/]+\/*)$/) {
push(@tbc, $suffix);
last if mkdir($prefix) or $! == EEXIST;
return unless $! == ENOENT;
}
$prefix .= pop(@tbc), mkdir($prefix) or return
while @tbc;
return 1;
}
mkdir_p($ARGV[0]) or die($!);
------------------------------
Date: Fri, 19 Jul 2013 15:25:31 +0200
From: "Peter J. Holzer" <hjp-usenet3@hjp.at>
Subject: Re: the fastest way to create a directory
Message-Id: <slrnkuifib.ijg.hjp-usenet3@hrunkner.hjp.at>
On 2013-07-18 20:47, Henry Law <news@lawshouse.org> wrote:
> On 18/07/13 20:49, Charlton Wilbur wrote:
>> Thou shalt make thy program's purpose and structure clear to thy fellow
>> man; for thy creativity is better used in solving problems than in
>> creating beautiful new impediments to understanding.
>
> I like it! Is that a Charltonism? Attribution please if not.
It's a slight variation of Henry Spencer's 8th Commandment:
http://doc.cat-v.org/henry_spencer/ten-commandments
hp
--
_ | Peter J. Holzer | Fluch der elektronischen Textverarbeitung:
|_|_) | Sysadmin WSR | Man feilt solange an seinen Text um, bis
| | | hjp@hjp.at | die Satzbestandteile des Satzes nicht mehr
__/ | http://www.hjp.at/ | zusammenpaßt. -- Ralph Babel
------------------------------
Date: Fri, 19 Jul 2013 14:51:47 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: the fastest way to create a directory
Message-Id: <87vc468xos.fsf@sapphire.mobileactivedefense.com>
"Peter J. Holzer" <hjp-usenet3@hjp.at> writes:
> On 2013-07-18 20:47, Henry Law <news@lawshouse.org> wrote:
>> On 18/07/13 20:49, Charlton Wilbur wrote:
>>> Thou shalt make thy program's purpose and structure clear to thy fellow
>>> man; for thy creativity is better used in solving problems than in
>>> creating beautiful new impediments to understanding.
>>
>> I like it! Is that a Charltonism? Attribution please if not.
>
> It's a slight variation of Henry Spencer's 8th Commandment:
It's a serious abuse of that as the original refers to placement of
braces and indentation:
Thou shalt make thy program's purpose and structure clear to
thy fellow man by using the One True Brace Style, even if thou
likest it not, for thy creativity is better used in solving
problems than in creating beautiful new impediments to
understanding.
Ie, it's not about not confusing people who can't read code and don't
understand simple algorithms by placing them into a file A they happen
to look at for some weird reason instead of pulling something similar
in from a file B they're not looking at (for an equally weird
reason).
The applicable commandment would be the seventh:
Thou shalt study thy libraries and strive not to reinvent them
without cause, that thy code may be short and readable and thy
days pleasant and productive.
Something which comes to mind immediately when reading this would be
"Thou shalt not strive to learn how to program lest some people's
undeservedly comfortable retirement might not remain as comfortable as
they'd like it to be". Or mabye "thou shalt not burden thy fellow men
by forcing overly general solutions to simple problems you happen to
have written 600 years ago onto them -- use them enjoying thine
pleasant and productive days and leave others alone unless they ask
YOU for help".
------------------------------
Date: Fri, 19 Jul 2013 16:21:34 +0100
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: the fastest way to create a directory
Message-Id: <u42oba-ps4.ln1@anubis.morrow.me.uk>
Quoth "Peter J. Holzer" <hjp-usenet3@hjp.at>:
> On 2013-07-17 09:49, Rainer Weikusat <rweikusat@mssgmbh.com> wrote:
> > "Peter J. Holzer" <hjp-usenet3@hjp.at> writes:
> >> On 2013-07-15 22:40, Ben Morrow <ben@morrow.me.uk> wrote:
> >>> Quoth George Mpouras <nospam.gravitalsun.noadsplease@hotmail.noads.com>:
> >>>> sub Mkdir_recursive
> >>>> {
> >>>> return 1 if $_[0] eq '' || -d $_[0];
> >>>> Mkdir_recursive( $_[0] =~/^(.*?)[\\\/][^\\\/]+$/ ) || return undef;
> >>>> mkdir $_[0] || return undef
> >>>> }
> >>>
> >>> You'd be better off calling mkdir blind and keying off $! if it fails.
> >>> That way you save a stat in the case where the creation succeeds.
> >>
> >> That shouldn't make a noticeable difference. If the stat does cause any
> >> disk accesses, those would also have been caused by the mkdir, and if it
> >> doesn't (i.e. everything is already in the cache) the time for the stat
> >> calls is completely swamped by the mkdir's.
> >
> > Both stat and mkdir are system calls and 'one system call' is going to
> > be faster than 'two system calls'.
>
> As stated, that's trivially untrue ('one call to exec(2)' will *not* be
> faster than 'two calls to time(2)' except under pathological
> circumstances), but even if I translate that into 'an additional system
> call will take additional time', it's not necessarily true.
>
> In this case I think that the stat system call would normally add a
> little time but it will be completely swamped by the time spent in
> mkdir:
>
> Each successful mkdir call will cause at least 5 disk accesses on a
> typical Linux file system: 1 for the journal, 2 for the inode and
> content of the parent directory and 2 for the inode and content of the
> new directory (Oh, I forgot the bitmaps, add another 2 or 4 ...). These
> will happen *after* mkdir returns because of the writeback cache, and
> the kernel will almost certainly succeed in coalescing at least some and
> maybe many of those writes, but if you create a lot of directories
> (George wrote about "thousands", in my tests I created about 150000)
> these writes will eventually dominate.
Since people've been quoting Henry Spencer:
10. Thou shalt foreswear, renounce, and abjure the vile heresy which
claimeth that ``All the world's a VAX'', and have no commerce with
the benighted heathens who cling to this barbarous belief, that the
days of thy program may be long even though the days of thy current
machine be short.
I believe on my machine (FreeBSD/ZFS) mkdir does a single write to the
intent log, which is on an SSD. The writes to spinning rust come (a good
deal) later, and are thoroughly batched.
By my reading of SUSv4, mkdir(2) is not required to do any immediate IO
at all.
> Now, in addition to writing new blocks, where does the pair
> stat($d); mkdir($d)
> spend time?
>
> If the ancestor directories of $d are in cache (that would be the normal
> case), both stat and mkdir will walk exactly the same in-memory
> structure until they fail to find $d. So, yes, that part will be
> uselessly duplicated, but it's very fast compared to actually writing a
> new directory to the disk, so the extra time is negligible.
>
> If the ancestor directories of $d are not in cache, stat will load them
> into the cache, which may take a noticable time. But that time will then
> be saved by mkdir which can now use the cache instead of loading the
> directories itself: So again the difference is one walk through
> in-memory structures, which is insignificant compared to loading the
> structures from disk and then writing a new directory (which will happen
> anyway).
If they are not in cache to start with, the first stat or mkdir (of the
deepest directory) will prepopulate the cache for all the rest, so that
IO can be ignored.
> The ratios will be different depending on the relative speed of RAM and
> storage: Maybe SSDs are fast enough that the additional walk through the
> cache is noticable, but I doubt it. Of course anybody is free to post
> benchmark results to prove me wrong.
You've missed an important overhead: the system call itself. This is
never cheap, and depending on architecture can be very expensive. Given
that the rest of the process ought to be working almost entirely out of
cache, this is likely to be significant.
Also, if you are on NFS then, depending on the client-side caching
policy, stat may well invoke extra RPCs, which are also never cheap.
Ben
------------------------------
Date: Fri, 19 Jul 2013 16:40:21 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: the fastest way to create a directory
Message-Id: <87ip068snu.fsf@sapphire.mobileactivedefense.com>
Charlton Wilbur <cwilbur@chromatico.net> writes:
>>>>>> "GM" == George Mpouras
>>>>>> <nospam.gravitalsun.antispam@spamno.hotmail.anispam.com.nospam>
>>>>>> writes:
>
> GM> yes there are about thousands dirs. try to undestand the
> GM> code. looks simple but it is not
>
> It's not simple. It's also not measurably more efficient than shelling
> out to mkdir -p or invoking File::Path's make_path function.
While make_path isn't a particularly good implementation, it is by far
not that bad. I tested this with recreating the /usr directory
hierarchy of 'some computer' (11,749 directories) below /tmp using
File::Path::make_path, George's recursive function, the 2nd mkdir_p I
posted and 'shelling out to mkdir -p', measureing times via
NYTProf. The results were (% relative to make_path):
system('mkdir', '-p', ...) 10.7s 803.6%
make_path/ mkpath 1.3115s 100%
Mkdir_recursive 0.976s 74.42%
mkdir_p 0.611s 46.58%
------------------------------
Date: Fri, 19 Jul 2013 08:39:33 -0700
From: Keith Keller <kkeller-usenet@wombat.san-francisco.ca.us>
Subject: Re: the fastest way to create a directory
Message-Id: <l63obaxotf.ln2@goaway.wombat.san-francisco.ca.us>
On 2013-07-19, Peter J. Holzer <hjp-usenet3@hjp.at> wrote:
>
> It's a slight variation of Henry Spencer's 8th Commandment:
>
> http://doc.cat-v.org/henry_spencer/ten-commandments
Commandment Seven is clearly applicable to this thread as well!
--keith
--
kkeller-usenet@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 3994
***************************************
