[23483] in Perl-Users-Digest
Perl-Users Digest, Issue: 5696 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Wed Oct 22 14:06:04 2003
Date: Wed, 22 Oct 2003 11:05:08 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Wed, 22 Oct 2003 Volume: 10 Number: 5696
Today's topics:
c programmer in need of perl advise (Mike Deskevich)
Re: c programmer in need of perl advise <gpatnude@adelphia.net>
Re: c programmer in need of perl advise <michael.p.broida@boeing_oops.com>
Re: c programmer in need of perl advise <gpatnude@adelphia.net>
Re: c programmer in need of perl advise chance@austin.rr.com
Re: c programmer in need of perl advise <asu1@c-o-r-n-e-l-l.edu>
Client/Server Help! (nospam)
Re: Client/Server Help! <nobull@mail.com>
Re: Client/Server Help! <asu1@c-o-r-n-e-l-l.edu>
Re: Client/Server Help! <asu1@c-o-r-n-e-l-l.edu>
compress folders <ja@nospam.foo>
Re: Further on Taint - exact code that has the problem <ddunham@redwood.taos.com>
Re: interprocess queue (was Re: win32 interprocess comm (Mike Deskevich)
Re: Is flock Needed for Logs? (Malcolm Dew-Jones)
Re: Perl and IIS - script runs but 'The page cannot be <jwillmore@remove.adelphia.net>
Re: Regex to extract row data from text (Copy of data i <xx087@freenet.carleton.ca>
Re: Rows not being returned ? <jwillmore@remove.adelphia.net>
Re: Rows not being returned ? <noreply@gunnar.cc>
Re: Rows not being returned ? <xx087@freenet.carleton.ca>
Re: Taint - having some real trouble here, taint/perl e <ddunham@redwood.taos.com>
Re: Taint - having some real trouble here, taint/perl e (Ben)
Re: Taint - having some real trouble here, taint/perl e (Ben)
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: 22 Oct 2003 09:26:31 -0700
From: mikedeskevich@yahoo.com (Mike Deskevich)
Subject: c programmer in need of perl advise
Message-Id: <71734b21.0310220826.52f8a9cf@posting.google.com>
i have a quick (hopefully) question for the perl gurus out there. i
have a bunch of data files that i need to read in and do some
processing. the data files are simple two columns of (floating point)
numbers, but the size of the file can range from 1000 to 10,000 lines.
i need to save the data in an array for post processing, so i can't
just read a line and throw the data away. my main question is: is
there a faster way to read the data than how i'm currently doing it
(i'm a c programmer, so i'm sure that i'm not using perl as
efficiently as i can)
here's how i read my data files
$ct=0;
while (<DATAFILE>)
{
($xvalue[$ct],$yvalue[$ct])=split;
$ct++;
}
#do stuff with xvalue and yvalue
is there a more efficient way to read in two columns of numbers? it
turns out that i have a series of these data files to process and i
think that most of the time is being used in either perl start up
time, or data reading time, the post processing is happening pretty
fast (i think)
thanks,
mike
------------------------------
Date: Wed, 22 Oct 2003 16:55:21 GMT
From: "Greg Patnude" <gpatnude@adelphia.net>
Subject: Re: c programmer in need of perl advise
Message-Id: <ZDylb.124175$qj6.7331906@news1.news.adelphia.net>
Its called "slurping" a file -- the so-called "pros" claim it is not really
"recommended" but I do it all the time with absolutely no consequence and no
obvious performance hit until the files exceed 6 MB or so ...
if (open (DATA, "$FILENAME")) {
@DATA = <DATA>;
}
you can read more about it in the Perl FAQ -->
http://www.perldoc.com/perl5.6/pod/perlfaq5.html#How-can-I-read-in-an-entire-file-all-at-once-
--
Greg Patnude / The Digital Demention
2916 East Upper Hayden Lake Road
Hayden Lake, ID 83835
(208) 762-0762
"Mike Deskevich" <mikedeskevich@yahoo.com> wrote in message
news:71734b21.0310220826.52f8a9cf@posting.google.com...
> i have a quick (hopefully) question for the perl gurus out there. i
> have a bunch of data files that i need to read in and do some
> processing. the data files are simple two columns of (floating point)
> numbers, but the size of the file can range from 1000 to 10,000 lines.
> i need to save the data in an array for post processing, so i can't
> just read a line and throw the data away. my main question is: is
> there a faster way to read the data than how i'm currently doing it
> (i'm a c programmer, so i'm sure that i'm not using perl as
> efficiently as i can)
>
> here's how i read my data files
>
> $ct=0;
> while (<DATAFILE>)
> {
> ($xvalue[$ct],$yvalue[$ct])=split;
> $ct++;
> }
> #do stuff with xvalue and yvalue
>
> is there a more efficient way to read in two columns of numbers? it
> turns out that i have a series of these data files to process and i
> think that most of the time is being used in either perl start up
> time, or data reading time, the post processing is happening pretty
> fast (i think)
>
> thanks,
> mike
------------------------------
Date: Wed, 22 Oct 2003 17:00:29 GMT
From: "Michael P. Broida" <michael.p.broida@boeing_oops.com>
Subject: Re: c programmer in need of perl advise
Message-Id: <3F96B7AD.57EAF1A6@boeing_oops.com>
Mike Deskevich wrote:
>
> i have a quick (hopefully) question for the perl gurus out there. i
> have a bunch of data files that i need to read in and do some
> processing. the data files are simple two columns of (floating point)
> numbers, but the size of the file can range from 1000 to 10,000 lines.
> i need to save the data in an array for post processing, so i can't
> just read a line and throw the data away. my main question is: is
> there a faster way to read the data than how i'm currently doing it
> (i'm a c programmer, so i'm sure that i'm not using perl as
> efficiently as i can)
>
> here's how i read my data files
>
> $ct=0;
> while (<DATAFILE>)
> {
> ($xvalue[$ct],$yvalue[$ct])=split;
> $ct++;
> }
> #do stuff with xvalue and yvalue
>
> is there a more efficient way to read in two columns of numbers? it
> turns out that i have a series of these data files to process and i
> think that most of the time is being used in either perl start up
> time, or data reading time, the post processing is happening pretty
> fast (i think)
I would suggest NOT using index $ct. You can use
"push" to add elements to an array. AFTER all the
elements have been added, "scalar(@arrayname)" will
give you the number of entries in the array.
Dunno if that will help much, but it couldn't hurt.
Mike
------------------------------
Date: Wed, 22 Oct 2003 17:16:53 GMT
From: "Greg Patnude" <gpatnude@adelphia.net>
Subject: Re: c programmer in need of perl advise
Message-Id: <9Yylb.124183$qj6.7338738@news1.news.adelphia.net>
$#ARRY will give you the number of array elements also ...
--
Greg Patnude / The Digital Demention
2916 East Upper Hayden Lake Road
Hayden Lake, ID 83835
(208) 762-0762
"Michael P. Broida" <michael.p.broida@boeing_oops.com> wrote in message
news:3F96B7AD.57EAF1A6@boeing_oops.com...
> Mike Deskevich wrote:
> >
> > i have a quick (hopefully) question for the perl gurus out there. i
> > have a bunch of data files that i need to read in and do some
> > processing. the data files are simple two columns of (floating point)
> > numbers, but the size of the file can range from 1000 to 10,000 lines.
> > i need to save the data in an array for post processing, so i can't
> > just read a line and throw the data away. my main question is: is
> > there a faster way to read the data than how i'm currently doing it
> > (i'm a c programmer, so i'm sure that i'm not using perl as
> > efficiently as i can)
> >
> > here's how i read my data files
> >
> > $ct=0;
> > while (<DATAFILE>)
> > {
> > ($xvalue[$ct],$yvalue[$ct])=split;
> > $ct++;
> > }
> > #do stuff with xvalue and yvalue
> >
> > is there a more efficient way to read in two columns of numbers? it
> > turns out that i have a series of these data files to process and i
> > think that most of the time is being used in either perl start up
> > time, or data reading time, the post processing is happening pretty
> > fast (i think)
>
> I would suggest NOT using index $ct. You can use
> "push" to add elements to an array. AFTER all the
> elements have been added, "scalar(@arrayname)" will
> give you the number of entries in the array.
>
> Dunno if that will help much, but it couldn't hurt.
>
> Mike
------------------------------
Date: Wed, 22 Oct 2003 17:19:15 GMT
From: chance@austin.rr.com
Subject: Re: c programmer in need of perl advise
Message-Id: <bn6ea4$fb6$1@localhost.localdomain>
Mike Deskevich <mikedeskevich@yahoo.com> wrote:
> i have a quick (hopefully) question for the perl gurus out there. i
> have a bunch of data files that i need to read in and do some
> processing. the data files are simple two columns of (floating point)
> numbers, but the size of the file can range from 1000 to 10,000 lines.
> i need to save the data in an array for post processing, so i can't
> just read a line and throw the data away. my main question is: is
> there a faster way to read the data than how i'm currently doing it
> (i'm a c programmer, so i'm sure that i'm not using perl as
> efficiently as i can)
> here's how i read my data files
> $ct=0;
> while (<DATAFILE>)
> {
> ($xvalue[$ct],$yvalue[$ct])=split;
> $ct++;
> }
> #do stuff with xvalue and yvalue
> is there a more efficient way to read in two columns of numbers? it
> turns out that i have a series of these data files to process and i
> think that most of the time is being used in either perl start up
> time, or data reading time, the post processing is happening pretty
> fast (i think)
I'm no perl guru. I'm really a C programmer who doesn't suck too bad
at perl. Above is pretty much how I'd do it.
If speed is really a problem I've got 2 suggestions:
1) if don't have @xvalue and @yvalue already allocated, you'll be doing a
lot of dynamic memory allocation. That could be costing you.
If you knew in advance the length you could do :
$xvalue[$vector_len - 1] = 0.0
if the lengths of your vectors really are unknowable, then you could
at least start pre-allocating chunks in advance and doubling the
size each time you 'run out of room', although then your going to
have to do some pain in the ass bookeeping. If you have a pretty
good upper bound the right thing to do might be to go ahead and
say $xvalue[10000] = 0.0. Then at the end set $#x to $ct-1 (modulo
my fencepost errors).
2) some kind of sscanf like function probably exists somewhere.
Might be more specialized, and hence faster than using split.
A real guru may have a much better answer
--
I used to think government was a necessary evil.
I'm not so sure about the necessary part anymore.
------------------------------
Date: 22 Oct 2003 17:42:31 GMT
From: "A. Sinan Unur" <asu1@c-o-r-n-e-l-l.edu>
Subject: Re: c programmer in need of perl advise
Message-Id: <Xns941C8B742A787asu1cornelledu@132.236.56.8>
mikedeskevich@yahoo.com (Mike Deskevich) wrote in
news:71734b21.0310220826.52f8a9cf@posting.google.com:
> here's how i read my data files
>
> $ct=0;
> while (<DATAFILE>)
> {
> ($xvalue[$ct],$yvalue[$ct])=split;
> $ct++;
> }
In this case, the $xvalue and $yvalue arrays are constantly being
resized. Eliminating that may increase performance, but you might want to
actually measure that. I doubt there is going to be a huge difference
with only 10000 records.
#! C:/Perl/bin/perl.exe
use strict;
use warnings;
my $fn = shift || 'data';
my $curr_max = 1000;
my @xvalue = ();
$#xvalue = $curr_max;
my @yvalue = ();
$#yvalue = $curr_max;
open(DATAFILE, "<", $fn) || die "Cannot open $fn: $!\n";
do {
my $i = 0;
while (<DATAFILE>) {
($xvalue[$i], $yvalue[$i]) = split;
++$i;
$curr_max = 2*$curr_max if($i >= $curr_max);
$#xvalue = $curr_max;
$#yvalue = $curr_max;
}
$#xvalue = $i;
$#yvalue = $i;
};
close(DATAFILE) || die "Cannot close input file: $!\n";
__END__
--
A. Sinan Unur
asu1@c-o-r-n-e-l-l.edu
Remove dashes for address
Spam bait: mailto:uce@ftc.gov
------------------------------
Date: Wed, 22 Oct 2003 12:09:59 -0400
From: "bob Smith" <bobsmith(nospam)@bobsmith.com>
Subject: Client/Server Help!
Message-Id: <bn6c8d$gpt$1@news1.usf.edu>
I am trying to setup a client that sends info to a server and then the
server responsds back to the client with a response from the info it sent
across the open socket. I am new and I know I missing something. I have
looked at several examples and have taken from them all but can't get all to
work together.
1. I've gotten the client to send to the server and the server recieves
2. I've gotten the client to connect and the server to send to it.
I can't get those 2 things to happen within the same program. I know that I
need to probably say from the client "Hey I am done sending and I am ready
to recieve" but I haven't figured out how to do it. Here is my code below.
Any help would be appreciated even if you can point me to a simple example.
Client code:
#!/usr/bin/perl -w
use IO::Socket;
$sock = new IO::Socket::INET (PeerAddr => 'localhost',
PeerPort => 5150,
Proto => 'tcp'
);
die "Socket could not be created. Reason: $!\n" unless $sock;
foreach (1 .. 10) {
print "Hello $_: \n";
print $sock "Msg $_: How are you?\n";
#$sock->flush();
}
print "quit";
print $sock "quit";
$sock->flush();
my $line;
while (defined($line = <$sock>)) {
print $line;
}
close ($sock);
Server code:
#!/usr/bin/perl -w
use Socket;
my $host = 'localhost';
my $port = 5150;
my $proto = getprotobyname('tcp');
my $iaddr = inet_aton($host);
my $paddr = sockaddr_in($port, $iaddr);
socket(SERVER, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
setsockopt(SERVER, SOL_SOCKET, SO_REUSEADDR, 1) or die "setsock: $!";
bind(SERVER, $paddr) or die "bind: $!";
listen(SERVER, SOMAXCONN) or die "listen: $!";
my $client_addr;
while ($client_addr = accept(CLIENT, SERVER)) {
while (defined($buf = <CLIENT>)) {
print "$buf\n";
}
my ($client_port, $client_ip) = sockaddr_in($client_addr);
my $client_ipnum = inet_ntoa($client_ip);
my $client_host = gethostbyaddr($client_ip, AF_INET);
print "got a connection from: $client_host","[$client_ipnum]\n";
print CLIENT "Smile from the server\n";
close CLIENT;
}
------------------------------
Date: 22 Oct 2003 17:48:46 +0100
From: Brian McCauley <nobull@mail.com>
Subject: Re: Client/Server Help!
Message-Id: <u9ekx52p75.fsf@wcl-l.bham.ac.uk>
"bob Smith" <bobsmith(nospam)@bobsmith.com> writes:
> I know that I need to probably say from the client "Hey I am done
> sending and I am ready to recieve" but I haven't figured out how to
> do it.
> Client code:
> print $sock "quit";
> $sock->flush();
So "quit" is your protocol's "Hey I am done sending and I am ready to
recieve".
> Server code:
> while (defined($buf = <CLIENT>)) {
> print "$buf\n";
> }
Two problems there:
One: you are reading in line mode - so the client would need to send
"quit\n" not just "quit".
Two: you are not breaking out of the loop when you get the "quit".
You may also want to consider shutdown().
None of this really has to do with Perl - it's much the same in any
language using sockets.
I have a nagging doubt that there's also a problem using "\n" across
platforms.
------------------------------
Date: 22 Oct 2003 17:11:45 GMT
From: "A. Sinan Unur" <asu1@c-o-r-n-e-l-l.edu>
Subject: Re: Client/Server Help!
Message-Id: <Xns941C863C98EEDasu1cornelledu@132.236.56.8>
"bob Smith" <bobsmith(nospam)@bobsmith.com> wrote in
news:bn6c8d$gpt$1@news1.usf.edu:
> I can't get those 2 things to happen within the same program. I know
> that I need to probably say from the client "Hey I am done sending and
> I am ready to recieve" but I haven't figured out how to do it.
perldoc -f shutdown
Here is an untested example:
---- SimpleServer.pl ----
#! C:/Perl/bin/perl.exe
use strict;
use warnings;
use IO::Socket::INET;
my $server = IO::Socket::INET->new(
'LocalAddr' => '127.0.0.1',
'LocalPort' => '50000',
'ReuseAddr' => 1,
'Blocking' => 1,
'Proto' => 'tcp',
'Listen' => 10,
);
die "Cannot create listening socket: $!\n" unless $server;
while( 1 ) {
while(my $conn = $server->accept()) {
printf("New connection from %s:%s\n",
$conn->peerhost(), $conn->peerport());
while(<$conn>) {
print "MSG: $_\n";
}
$conn->shutdown(0);
$conn->printflush('Thank you for your message');
$conn->shutdown(2);
}
}
__END__
---- SimpleClient.pl ----
#! C:/Perl/bin/perl.exe
use strict;
use warnings;
use IO::Socket::INET;
my $server = shift || '127.0.0.1';
my $port = shift || 50000;
my $msg = join("\n", @_) || 'Hi there server dude ...';
print "Connecting to $server:$port to send \"$msg\"\n";
my $conn = IO::Socket::INET->new(
'PeerAddr' => $server,
'PeerPort' => $port,
'Blocking' => 1,
'Proto' => 'tcp'
);
die "Cannot connect: $!\n" unless $conn;
$conn->printflush($msg);
$conn->shutdown(1);
while(<$conn>) {
print;
}
$conn->shutdown(2);
__END__
---- SimpleClient.pl ----
use strict;
use warnings;
use IO::Socket::INET;
my $server = '127.0.0.1';
my $port = 50000;
my $msg = 'Hello there server dude ...';
print "Connecting to $server:$port to send \"$msg\"\n";
my $conn = IO::Socket::INET->new(
'PeerAddr' => $server,
'PeerPort' => $port,
'Blocking' => 1,
'Proto' => 'tcp'
);
die "Cannot connect: $!\n" unless $conn;
$conn->printflush($msg);
$conn->shutdown(1);
while(<$conn>) {
print;
}
$conn->shutdown(2);
__END__
--
A. Sinan Unur
asu1@c-o-r-n-e-l-l.edu
Remove dashes for address
Spam bait: mailto:uce@ftc.gov
------------------------------
Date: 22 Oct 2003 17:14:19 GMT
From: "A. Sinan Unur" <asu1@c-o-r-n-e-l-l.edu>
Subject: Re: Client/Server Help!
Message-Id: <Xns941C86AC06DAasu1cornelledu@132.236.56.8>
"A. Sinan Unur" <asu1@c-o-r-n-e-l-l.edu> wrote in
news:Xns941C863C98EEDasu1cornelledu@132.236.56.8:
> my $msg = join("\n", @_) || 'Hi there server dude ...';
Please replace this line with
my $msg = 'Hi there server dude ...';
in the example in my previous response.
--
A. Sinan Unur
asu1@c-o-r-n-e-l-l.edu
Remove dashes for address
Spam bait: mailto:uce@ftc.gov
------------------------------
Date: Wed, 22 Oct 2003 10:21:22 -0700
From: "Josie Armindez" <ja@nospam.foo>
Subject: compress folders
Message-Id: <bn6eam$j3h$0@pita.alt.net>
I want to go into /home and tar all of the user directories.
Here is what I have so far:
my $backup;
my $count;
$count = @folders;
my @folders = `ls`;
foreach ( @folders ) {
#$backup = `tar -cvf @folders.tar @folders`; # this doesn't work
}
$test_count = @folders;
print 'total ', scalar @folders, " folders\n";
How can I perform the tar operation on each of the folders in the array?
TIA
Josie
------------------------------
Date: Wed, 22 Oct 2003 16:04:49 GMT
From: Darren Dunham <ddunham@redwood.taos.com>
Subject: Re: Further on Taint - exact code that has the problem
Message-Id: <BUxlb.2707$B91.2605@newssvr14.news.prodigy.com>
Ben <2stormts@nemontel.net> wrote:
> This is a followup to my earlier post, sorry it's not in the same
> thread, but it's been several hours and google still hasn't coughed up
> the first post, so I can't get it into the same thread.
> $cmd = "pogmaker";
> $command = "$cmd $part $befo $fute $cols $redd >output.txt &";
> $command =~ /(.*)/;
> $command = $1;
> $ENV{"PATH"}="/bin:/usr/bin";
> delete @ENV{'IFS','CDPATH','ENV','BASH_ENV'};
> # print "Taint checks seem to be on<br>"
> # unless eval { local $^W; unlink "$^X$^T"; 1 };
> $rez = system($command);
> FYI, the print "Taint.... does not indicate that taint is on by
> printing.
Does the script output anything else to tell you that it's actually
running?
> pogmaker is in /usr/bin, has execute permission for anyone, owned by
> root.
> pogmaker absolutely will not run.
What is the exit code from the system command? (see $?).
--
Darren Dunham ddunham@taos.com
Unix System Administrator Taos - The SysAdmin Company
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >
------------------------------
Date: 22 Oct 2003 09:32:45 -0700
From: mikedeskevich@yahoo.com (Mike Deskevich)
Subject: Re: interprocess queue (was Re: win32 interprocess communication on one machine)
Message-Id: <71734b21.0310220832.7f6c0f13@posting.google.com>
this may not be ideal for your solution since you want threads, but it
may work and be reasonably easy to implement until you find a cleaner
solution.
make your message queue a file and have the master just append to the
file. and start children as their own process (either the master can
spawn them, or something else can start them). the children will read
the queue and start working on the first item (and delete it so the
next child doesn't get the same instruction). you'll have to get into
a little of file locking, but that's not terrribly hard.
it should be general enough to work on any platform too.
hope this helps.
mike
"Nathaniel Hekman" <hekmanATgeo-slopeDOTcom@no.spam> wrote in message news:<5HWkb.8738$f7.473921@localhost>...
> Thank you Anand. I played around with that a bit. Signals apparently don't
> work well on Windows, and pipes only work between a single parent/child
> pair.
>
> I probably wasn't clear on this in my original post (because I didn't
> realise it was an issue until I tried your ideas) but really what I'm
> looking for is a message queue. I want several siblings all watching a
> common queue. The parent puts a job on the queue, and the first child
> available takes it off the queue and starts processing it. Meanwhile the
> parent puts another job on the queue and when another child is available it
> takes it off and processes it, and so on.
>
> Anything out there like this? Or suggestions of alternate approaches?
>
>
> Nate
>
> "Anand" <some@one.com> wrote in message
> news:AdYjb.3214$8x2.1892443@newssrv26.news.prodigy.com...
> > You can use unnamed pipes for parent-child communication. Another
> > alternative would be to use signals.
> >
> > --Anand
> >
> > Nathaniel Hekman wrote:
> > > I have a parent process which spawns off several child processes using
> > > Win32::Process. The parent is just a controller, the children are perl
> > > scripts doing the actual work I'm trying to accomplish. The parent has
> a
> > > long list of jobs to do, launches three children to handle the first
> three
> > > jobs, and whenever any child finishes the parent launches another child
> with
> > > the next job, and so on until they're all done. (Ideally I'd like the
> > > children to be threads instead of separate processes, but from what I
> can
> > > see ActivePerl doesn't support threads yet on Windows.)
> > >
> > > Unfortunately, if I launch these child processes too quickly one after
> > > another, I get errors on the screen saying "could not extract perl
> runtime"
> > > or something along those lines. If I put a 2-3 second delay between
> > > launching each child I don't get that problem, but of course it slows
> down
> > > the job.
> > >
> > > I'd like to change the design, and launch three children immediately,
> then
> > > feed them jobs. When one finishes a job it keeps running and the parent
> > > gives another job to do. That would avoid shutting down and recreating
> > > processes, and would avoid having to use that 2-second delay.
> > >
> > > I realise I could do this using sockets, but that seems like overkill
> (and a
> > > security risk) since the children and parent are running on the same
> > > computer. What else would you suggest?
> > >
> > > Thanks for any ideas.
> > >
> > >
> > > Nate Hekman
> > > Calgary, Alberta, Canada
> > >
> > >
> >
------------------------------
Date: 22 Oct 2003 10:33:06 -0800
From: yf110@vtn1.victoria.tc.ca (Malcolm Dew-Jones)
Subject: Re: Is flock Needed for Logs?
Message-Id: <3f96bf52@news.victoria.tc.ca>
Michele Dondi (bik.mido@tiscalinet.it) wrote:
: On Thu, 16 Oct 2003 12:35:51 +0100, "Alan J. Flavell"
: <flavell@ph.gla.ac.uk> wrote:
: >[unattributed quote - original poster was Malcolm Dew-Jones:]
: >> > My understanding is that a single `write()' of less than a certain (fairly
: >> > large) size is supposed to be atomic on unix (and presumably on linux) -
: >> > that means atomic relative to other write() calls.
: >
: >Indeed. A lot of unix-based software appears to rely on the semantics
: >of opening for append, and atomically appending a record, without
: >using locking. You can't rely on the *sequence* in which asynchronous
: >processes write their records, but they don't fragment records and
: >they don't damage file structure. There's a great deal of software
: >that would go horribly wrong if that stopped working.
: I will trust your word and your experience, however, for anyone
: interested, here are the results of a test I made (following the idea
: of another poster):
: corrupt.pl:
: #!/usr/bin/perl -l
: use strict;
: use warnings;
:
: $|++;
: unlink 'log.txt';
: open my $out1, '>>log.txt' or die $!;
: open my $out2, '>>log.txt' or die $!;
:
: my $stop=time+5;
: my $count;
: while (time<$stop) {
: $count++;
: print $out1 "out1: $count";
: print $out2 "out2: $count";
: }
: __END__
: test.pl:
: #!/usr/bin/perl -ln
: use strict;
: use warnings;
:
: our $n;
: ++$n, print if !/^out[12]: \d+$/;
: END { print "$n/$." }
: __END__
: An here are the results of './test.pl log.txt' respectively on
: Linux[1] and Windows[2]:
: report.lnx.txt:
: 5189/4354160
: 4445/3845960
: 5326/4447462
: 4604/3955650
: 4186/3669472
: report.win.txt:
: 62/223608
: 102/247178
: 86/238094
: 21/195220
: 38/210018
: I don't think these data are statistically significative, however I
: find for the average of the ratios of the given numbers and for the
: corresponding standard deviation the following values:
: Linux: 1.17e-3, 2.41e-5
: Win: 2.64e-4, 1.26e-4
: Any comments?
Sure
#!/usr/bin/perl -l
# not-corrupt.pl:
use strict;
use warnings;
$|++;
my $stop=time+30;
my $count;
while (time<$stop) {
$count++;
print "out1: $count";
}
$ cat > log.txt
$ ./not-corrupt.pl >> log.txt & ./not-corrupt.pl >> log.txt &
./not-corrupt.pl >> log.txt & ./not-corrupt.pl >> log.txt &
same test routine as above
$ ./test.pl < log.txt
/1490113
Eye ball examination shows the outputs are intermingled
(E.g. a few lines)
out1: 569
out1: 570
out1: 16
out1: 11
out1: 12
out1: 17
out1: 882
out1: 883
out1: 18
out1: 571
so. no corruption when the routines pipe to standard output.
Perldoc says "$| ... STDOUT will typically be line buffered if output is
to the terminal"
If stdio uses line mode buffering, then each line will be passed as a
single entity to write() and the lines will be written uncorrupted.
If you use C then you control the buffering and a C program can append to
a log with no problem.
If you use perl then you must be careful because of the buffering, over
which you have less control.
Our perl programs, that log in this manner, run once and write at most one
or two lines. Presumably the size of the prints are smaller than the
stdio buffer, and so the entire print out ends up being written with a
single write(), and the so log files are uncorrupted.
However, if you have a perl program that logs many lines then the stdio
buffers will end up being flushed at (effectively) random times without
regard for line breaks, and so the logs will end up being corrupted.
But we can control that, so lets try a test where we force the data to be
flushed at the end of each line. (The best way would be to line mode the
stdio buffering, but I don't know how to do that.)
#!/usr/bin/perl
# flog
use warnings;
use strict;
# unlink 'log.txt';
open (my $out1, ">>log.txt") or die $!;
open (my $out2, ">>log.txt") or die $!;
my $now = time;
my $count;
until( time > $now+10 ) {
$count++;
print $out1 "out1: $count\n";
print $out2 "out2: $count\n";
select $out1; $|=1; $|=0;
select $out2; $|=1; $|=0;
}
close $out1;
close $out2;
$ cat > log.txt
$ ./flog & ./flog & ./flog & ./flog & ./flog &
# note minor typo in print, test.pl altered to check for two spaces
$ ./test.pl < log.txt
/582740
So, when five versions of the program were running, including two
writes per process, and after both writes then we force a flush, then
the data is uncorrupted.
One more test, how many lines can we print and then flush and still be ok?
I added the following lines after the second print
if ($count % 50 == 0)
{
select $out1; $|=1; $|=0;
select $out2; $|=1; $|=0;
}
examining log.txt confirms that each output stream is being flushed after
50 lines.
$ ./test.pl < log.txt
/4110602
You'll notice that the buffering was more efficient (we have four times as
many lines written in the same time) and as before we have still avoided
corruption.
Final conclusion - you can safely log by appending as long as you make
sure the data is written in line mode. In C you can set the buffering to
do this. In perl you need to be more careful, but as long as line mode is
used then it works. If logging continually then flush the lines
frequently enough to ensure that stdio is not flushing for you at some
less opportune time. If logging only a single line then the program will
flush just at the end, which is effectively the same as line mode
buffering for small amounts of data, so programs that write single lines
and then exit should normally be able to to safely log with out explicit
locking.
------------------------------
Date: Wed, 22 Oct 2003 15:42:37 GMT
From: James Willmore <jwillmore@remove.adelphia.net>
Subject: Re: Perl and IIS - script runs but 'The page cannot be displayed'
Message-Id: <20031022114233.1492c7aa.jwillmore@remove.adelphia.net>
On 22 Oct 2003 04:27:54 -0700
stewart@webslave.dircon.co.uk (stew dean) wrote:
> James Willmore <jwillmore@remove.adelphia.net> wrote in message
> news:<20031021114710.504263f7.jwillmore@remove.adelphia.net>...
> > On 21 Oct 2003 03:53:24 -0700
> > stewart@webslave.dircon.co.uk (stew dean) wrote:
<snip>
What happened when you ran the script at the command line? There is
_little_ difference between running the script on the command line and
running it through the web server - especially if you use the CGI
module (hint - '-debug').
Where is the code you speak of? You have yet to post any offending
code.
The documentation for _Perl_ (specifically, perlfaq9) gives a tip or
two on debugging a buggy Perl CGI script.
And yes, I was insulting to you. Sorry for the transgression. By the
same token, if you run a web server, you should at least know where
the logs are located - or you shouldn't be a web server admin or even
touch a web server for development purposes. I may not agree with
some of the admins or the developers where I work, but at least they
know where the logs are located :-)
For the time being - and to give a hint - try using CGI::Carp (just
not in a production environment).
HTH and have a nice day :-)
--
Jim
Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.
a fortune quote ...
People usually get what's coming to them ... unless it's been
<mailed.
------------------------------
Date: 22 Oct 2003 17:26:21 GMT
From: Glenn Jackman <xx087@freenet.carleton.ca>
Subject: Re: Regex to extract row data from text (Copy of data included)
Message-Id: <slrnbpdff0.9ko.xx087@smeagol.ncf.ca>
TimBenz <timbenz@timbenz.com> wrote:
> Here is a representative piece,
>
> NAME OF ISSUER TITLE OF CUSIP MARKET AMOUNT SH/PRINV
> DISC OTHER VOTING AUTHORITY
>
> 21ST CENTURY INS GRP COMMON 90130N103 974 70700 SH SOLE
> 70700 0 0
> 3COM CORP COMMON 885535104 5156 873949 SH SOLE
> 873949 0 0
> 3M COMPANY COMMON 88579Y101 36846 533460 SH SOLE
> 527760 0 5700
> 3M COMPANY COMMON 88579Y101 2735 39596 SH OTHER
> 39596 0 0
> IBM CORP COMMON 88179Y101 735 35110 SH SOLE
> 35110 0 0
Looks like fixed width fields, as opposed to delimited.
Does the "COMMON" always start at the 31st character?
If so, use substr() to extract the data.
--
Glenn Jackman
NCF Sysadmin
glennj@ncf.ca
------------------------------
Date: Wed, 22 Oct 2003 15:51:08 GMT
From: James Willmore <jwillmore@remove.adelphia.net>
Subject: Re: Rows not being returned ?
Message-Id: <20031022115109.0311bbe4.jwillmore@remove.adelphia.net>
On 22 Oct 2003 06:27:00 -0700
sylviestone@canada.com (Sylvie Stone) wrote:
<snip>
> my $one = $sth->fetchrow_array();
<snip>
You either need to do:
#place results into _array_ @one - access @one as an array
my @one = $sth->fetchrow_array();
-or-
#place results, as a _reference_, into $one - access $one as a
#reference
my $one = $sth->fetchrow_arrayref();
Right now, you're not getting anything to use in $one. Plus, is there
only _one_ row of data you are interested in? If so, this will work
to get the first row of data returned. If not, you'll need to loop
through the returned values.
HTH
--
Jim
Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.
a fortune quote ...
A fool's brain digests philosophy into folly, science into
<superstition, and art into pedantry. Hence University
<education. -- G. B. Shaw
------------------------------
Date: Wed, 22 Oct 2003 17:53:47 +0200
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: Rows not being returned ?
Message-Id: <bn69ak$sun86$1@ID-184292.news.uni-berlin.de>
Sylvie Stone wrote:
> Can someone PLEASE tell me why this is not returning the $one
> variable ?
What do you mean by "returning the $one variable"?
> I'l pulling results from a survey database table where the answers
> was a range from 1 - 10.
I may be stupid, but I still don't understand what it is you are
trying to do. Maybe it would be easier to grasp if you posted the
table structure.
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: 22 Oct 2003 17:19:20 GMT
From: Glenn Jackman <xx087@freenet.carleton.ca>
Subject: Re: Rows not being returned ?
Message-Id: <slrnbpdf1r.9ko.xx087@smeagol.ncf.ca>
Sylvie Stone <sylviestone@canada.com> wrote:
> my $sth = $dbh->prepare('select count(*) from $SURVEYTABLE where info
> = ?');
[...]
> $sth->execute( $nums );
> my $one = $sth->fetchrow_array();
> print "one is $one<br>num is $nums<br>\n";
What is the result of that print statement?
You want:
my @ary = $sth->fetchrow_array; # this returns an _array_, after all
my $one = $ary[0];
or:
my $aryref = $sth->fetchrow_arrayref;
my $one = $aryref->[0];
You should read the DBI docs.
--
Glenn Jackman
NCF Sysadmin
glennj@ncf.ca
------------------------------
Date: Wed, 22 Oct 2003 16:01:08 GMT
From: Darren Dunham <ddunham@redwood.taos.com>
Subject: Re: Taint - having some real trouble here, taint/perl experts, please help
Message-Id: <8Rxlb.2705$B91.1164@newssvr14.news.prodigy.com>
Abigail <abigail@abigail.nl> wrote:
> Darren Dunham (ddunham@redwood.taos.com) wrote on MMMDCCIII September
> MCMXCIII in <URL:news:V6hlb.2509$wY3.1498@newssvr25.news.prodigy.com>:
> ^^
> ^^ > This also times out and kills the web page AND the running command
> ^^ > 'client':
> ^^
> ^^ > exec("/usr/src/client/client",$p1,$p2,$p3,$p4);
> ^^ > exit;
> ^^
> ^^ That exit line is never executed. The perdoc on exec will tell you
> ^^ why. If you want to do that, use a fork as above.
> If you look up the perldoc on exec yourself, you see in the first
> paragraph that it *is* possible that 'exec' returns.
Quite correct. The bare 'exit' made me assume (perhaps incorrectly)
that the intent was to run the exec and then exit the child on normal
return. My statement was directed at the general case and not the
complete case.
--
Darren Dunham ddunham@taos.com
Unix System Administrator Taos - The SysAdmin Company
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >
------------------------------
Date: 22 Oct 2003 10:39:43 -0700
From: 2stormts@nemontel.net (Ben)
Subject: Re: Taint - having some real trouble here, taint/perl experts, please help
Message-Id: <5c8f38ff.0310220939.4610f7dc@posting.google.com>
"A. Sinan Unur" <asu1@c-o-r-n-e-l-l.edu> wrote in message news:<Xns941BA67685B38asu1cornelledu@132.236.56.8>...
> What is the error message?
There is no error message. system() returns zero, and $! is empty. But
my command is definitely not running. Keep in mind that I *can* make
it run just by using the usage form system("cmd","param","param",etc).
It works fine, and the input is thoroughly laundered anyway, it can't
really be fed anything from the perl script that would make it quietly
go away without an error code.
I've printed out the strings, which are entirely vanilla, and there is
no difference, except of course I can't add the output redirection
(>output.txt) or the detach (&) for the shell expansion that I can in
the form:
system("cmd param1 param2 >output.txt &");
...which is what returns me the 0 and empty $!, yet does not invoke my
command at all.
Since I wrote my original cry for assistance, using a helpful hint
here, I went to a fork and close streams approach. Although this does
not give me the diagnostic output file (>output.txt) it does detach
properly and so the main functionality I needed is there. If I need
the output.txt file I can open it directly and use fprintf instead of
printf to pump the messages out to it.
So my problem has been worked around, but the reason system() is not
working in this instance still eludes me.
--Ben
------------------------------
Date: 22 Oct 2003 10:55:45 -0700
From: 2stormts@nemontel.net (Ben)
Subject: Re: Taint - having some real trouble here, taint/perl experts, please help
Message-Id: <5c8f38ff.0310220955.17aac6ab@posting.google.com>
Darren Dunham <ddunham@redwood.taos.com> wrote in message news:<V6hlb.2509$wY3.1498@newssvr25.news.prodigy.com>...
> What leads you to think that it "acts like it is tainted"?
The behaviour - if I use ("literalCMD",$param,$param,etc) form of
system(), it works. If I use the ("$aggregateCMD $param $param etc.")
form, it doesn't. This behaviour is essentially described in the
perlsec docs; taint stops you if you use the latter, but not the
former, if it thinks you're tainted. That behaviour is what is
happening, so I was thinking taint. I did attempt to see if I could
detect taint being on just prior to invocation, but I could not. There
is some verbiage at the top of perlsec that talks about perl turning
on taint, apparently at the time the command is invoked: "Perl
automatically enables a set of special security checks, called taint
mode, when it detects its program running with differing real and
effective user or group IDs." I thought this might be my problem, in
which case it would seem that it happens at runtime, and no check just
prior to running would catch it. Hence my request for why taint might
be on, even though I had not asked for it.
> What error messages are you getting?
No error messages. system() returns 0 and !$ is empty. The command is
simply never invoked. I've verified this several ways - output files,
sleeping the command a bit at start and then using process searches,
and of course, it doesn't do anything it's supposed to. It works
perfectly if it gets started at all, though, either using fork or the
multiparameter form of system or exec.
> However, if you want to "detach" it, either run it in a shell
> explicitly..
>
> ("/usr/bin/sh","-c","/usr/src/client/client",$p1,$p2,$p3,$p4,&)
>
> or fork/exec yourself.
I did that. fork() works for me, it detaches and runs to completion.
Thank you for the suggestion.
> > exec("/usr/src/client/client",$p1,$p2,$p3,$p4);
> > exit;
>
> That exit line is never executed. The perdoc on exec will tell you
> why. If you want to do that, use a fork as above.
I was concerned about the exec() call failing, as things weren't
working right. Since if exec() fails, it does return, I think it's
important to leave that in there. Correct me if I'm wrong, please.
> Why do you think it's tainted?
I don't, actually. I think might *perl* think it's tainted. As to why,
I have no idea. I've done all the required hoop-jumping to
"decontaminate" my variables, path and environment, I didn't ask for
taint checking in the first place, and this used to work 100% under
redhat 6 and Apache 1.3. It's the behaviour that led me to think
taint, as I described above.
--Ben
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 5696
***************************************
