[16148] in Perl-Users-Digest
Perl-Users Digest, Issue: 3560 Volume: 9
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Mon Jul 10 16:07:45 2000
Date: Mon, 10 Jul 2000 13:07:35 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Message-Id: <963259655-v9-i3560@ruby.oce.orst.edu>
Content-Type: text
Perl-Users Digest Mon, 10 Jul 2000 Volume: 9 Number: 3560
Today's topics:
Definitive flock code (?) gevens@my-deja.com
Re: Definitive flock code (?) (John Harden Borwick)
Re: Definitive flock code (?) <gellyfish@gellyfish.com>
Re: Definitive flock code (?) gevens@my-deja.com
Re: Definitive flock code (?) <gellyfish@gellyfish.com>
Re: Definitive flock code (?) gevens@my-deja.com
Re: Definitive flock code (?) (Bart Lateur)
Re: Definitive flock code (?) <flavell@mail.cern.ch>
Re: Definitive flock code (?) gevens@my-deja.com
Re: Definitive flock code (?) <flavell@mail.cern.ch>
Re: Definitive flock code (?) (brian d foy)
Re: Definitive flock code (?) <flavell@mail.cern.ch>
Re: Definitive flock code (?) (brian d foy)
Re: Definitive flock code (?) <flavell@mail.cern.ch>
Re: Definitive flock code (?) (Randal L. Schwartz)
Re: Definitive flock code (?) (Mark-Jason Dominus)
Re: Definitive flock code (?) <flavell@mail.cern.ch>
Digest Administrivia (Last modified: 16 Sep 99) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Mon, 03 Jul 2000 18:19:03 GMT
From: gevens@my-deja.com
Subject: Definitive flock code (?)
Message-Id: <8jqle6$tge$1@nnrp1.deja.com>
Hello all.
I have spent many hours trying to determine the best way to use flock
when it comes to the simple process of writing to a file. It appears to
attract a lot of questions.
In the hope that others will be able to take this code as one way of
ensuring that their files are kept safe from two processes trying to
write to a file at the same time, I present this sample and invite your
comments :
The code is to append a userid and password to the end of a flat file.
It uses a separate 'semaphore file' to control the locking.
$passfile = "passfile.txt" ;
$semaphore = "passfile.lck" ;
open S, "> $semaphore" or die "Unable to open semaphore file: $!\n" ;
my $lockwait == 0;
until(flock S, LOCK_EX) {
$lockwait ++ ;
die "flock timed out\n" if($lockwait == 20) ;
sleep 1 ;
}
open (wdata, ">> $passfile") or die "Unable to open passfile: $!\n" ;
print wdata "$input{'username'}:" ;
print wdata "$password\n" ;
close (wdata) ;
close (S) ;
I welcome your advice and suggestions for improvement.
Sent via Deja.com http://www.deja.com/
Before you buy.
------------------------------
Date: 3 Jul 2000 19:49:11 GMT
From: jhborwic@unity.ncsu.edu (John Harden Borwick)
Subject: Re: Definitive flock code (?)
Message-Id: <8jqqnn$564$1@uni00nw.unity.ncsu.edu>
In article <8jqle6$tge$1@nnrp1.deja.com>, <gevens@my-deja.com> wrote:
[ snip ]
>my $lockwait == 0;
>
>until(flock S, LOCK_EX) {
> $lockwait ++ ;
> die "flock timed out\n" if($lockwait == 20) ;
> sleep 1 ;
> }
[ snip ]
Where 'use Fcntl qw(:flock);' is presumed.
Without speaking to the rest of your code, you need
until(flock S, LOCK_EX|LOCK_NB)
From 'perldoc -f flock':
If LOCK_NB is bitwise-or'ed with LOCK_SH or
LOCK_EX then `flock' will return immediately
rather than blocking waiting for the lock (check
the return status to see if you got it).
--
John Borwick
------------------------------
Date: Tue, 04 Jul 2000 11:52:29 GMT
From: Jonathan Stowe <gellyfish@gellyfish.com>
Subject: Re: Definitive flock code (?)
Message-Id: <1ek85.1281$iP2.115812@news.dircon.co.uk>
On Mon, 03 Jul 2000 18:19:03 GMT, gevens@my-deja.com Wrote:
> Hello all.
>
> I have spent many hours trying to determine the best way to use flock
> when it comes to the simple process of writing to a file. It appears to
> attract a lot of questions.
>
> In the hope that others will be able to take this code as one way of
> ensuring that their files are kept safe from two processes trying to
> write to a file at the same time, I present this sample and invite your
> comments :
>
> The code is to append a userid and password to the end of a flat file.
>
> It uses a separate 'semaphore file' to control the locking.
>
> $passfile = "passfile.txt" ;
> $semaphore = "passfile.lck" ;
>
> open S, "> $semaphore" or die "Unable to open semaphore file: $!\n" ;
>
> my $lockwait == 0;
>
> until(flock S, LOCK_EX) {
> $lockwait ++ ;
> die "flock timed out\n" if($lockwait == 20) ;
> sleep 1 ;
> }
>
There is no point in doing this unless you specify LOCK_NB as flock() will
block until it can gain the lock.
> open (wdata, ">> $passfile") or die "Unable to open passfile: $!\n" ;
>
I cant see any benefit in locking some other file than the one you want to
control access to ....
------------------------------
Date: Tue, 04 Jul 2000 13:35:22 GMT
From: gevens@my-deja.com
Subject: Re: Definitive flock code (?)
Message-Id: <8jsp6q$cq3$1@nnrp1.deja.com>
Hi Johnathan,
The purpose of locking a ‘semaphore’ file is not immediately apparent,
however, having had a look at a link posted in response to a previous
question re: flock, it seems to be a good practice.
http://www.plover.com/~mjd/perl/TPC/1998/Hardware-
notes.html#Semaphore_Files
This is not exactly relevant to my coding example in particular, but in
cases where you open a file using the single ‘>’ it seems that people
are suffering when :
1) The first process opens the file ‘>’ with the intention of setting
the lock (which is not yet obtained).
2) A second process reads the file – which is now empty
3) The first process successfully acquires the lock.
Too late ! The second process is working with no data.
Opening the separate semaphore file and setting the lock against it
prevents this scenario.
As far as whether LOCK_EX goes, any advice whether it waits correctly
and any pitfalls to be aware of would be much appreciated.
Thanks for your interest.
Sent via Deja.com http://www.deja.com/
Before you buy.
------------------------------
Date: Wed, 05 Jul 2000 10:17:20 GMT
From: Jonathan Stowe <gellyfish@gellyfish.com>
Subject: Re: Definitive flock code (?)
Message-Id: <QWD85.1365$iP2.128053@news.dircon.co.uk>
On Tue, 04 Jul 2000 13:35:22 GMT, gevens@my-deja.com Wrote:
> Hi Johnathan,
>
> The purpose of locking a ‘semaphore’ file is not immediately apparent,
> however, having had a look at a link posted in response to a previous
> question re: flock, it seems to be a good practice.
>
> http://www.plover.com/~mjd/perl/TPC/1998/Hardware-
> notes.html#Semaphore_Files
>
> This is not exactly relevant to my coding example in particular, but in
> cases where you open a file using the single ‘>’ it seems that people
> are suffering when :
>
> 1) The first process opens the file ‘>’ with the intention of setting
> the lock (which is not yet obtained).
> 2) A second process reads the file – which is now empty
> 3) The first process successfully acquires the lock.
>
Then dont open the file in that mode then, use '+<' for instance and then
truncate and seek
>
> Opening the separate semaphore file and setting the lock against it
> prevents this scenario.
>
Er yes.
> As far as whether LOCK_EX goes, any advice whether it waits correctly
> and any pitfalls to be aware of would be much appreciated.
>
It will block until the lock is succesfully achieved unless you supply
the LOCK_NB flag yes. The flock may return immediately without having
achieved a lock without LOCK_NB if there was an error condition, a 'flock'
may require system resources that are unavailable for instance, the
success of the flock should be checked and only retried if the error indicates
that trying again might prove successful. You should also do this if
you are using LOCK_NB.
/J\
------------------------------
Date: Wed, 05 Jul 2000 11:27:19 GMT
From: gevens@my-deja.com
Subject: Re: Definitive flock code (?)
Message-Id: <8jv62b$1eo$1@nnrp1.deja.com>
Thanks Johnathan,
It seems to be such an important area, and yet I am still not 100% sure
which code will be the most solid. Perl's philosopy of there 'always
being more than one way to do it' is proving all too true!
I started with a code snippet for comment...would it be cheeky to ask
if anyone has something that they use as a matter of course for flock
when:
1) opening a file
2) setting the lock
3) checking the lock ... on your advice
4 writing an entry
Thanks and best regards.
Sent via Deja.com http://www.deja.com/
Before you buy.
------------------------------
Date: Wed, 05 Jul 2000 12:33:45 GMT
From: bart.lateur@skynet.be (Bart Lateur)
Subject: Re: Definitive flock code (?)
Message-Id: <39642ad4.3027575@news.skynet.be>
Jonathan Stowe wrote:
>Then dont open the file in that mode then, use '+<' for instance and then
>truncate and seek
First: "+<" will fail if the file doesn't exist. I think I have used
"+>>" before -- and yes, I could still overwrite the old contents.
Second: reverse the order: first seek, then truncate. If you do it the
other way around, it might not work.
--
Bart.
------------------------------
Date: Wed, 5 Jul 2000 16:33:37 +0200
From: "Alan J. Flavell" <flavell@mail.cern.ch>
Subject: Re: Definitive flock code (?)
Message-Id: <Pine.GHP.4.21.0007051609010.10709-100000@hpplus03.cern.ch>
On Wed, 5 Jul 2000, Bart Lateur wrote:
> Jonathan Stowe wrote:
>
> >Then dont open the file in that mode then, use '+<' for instance and then
> >truncate and seek
>
> First: "+<" will fail if the file doesn't exist. I think I have used
> "+>>" before -- and yes, I could still overwrite the old contents.
>
> Second: reverse the order: first seek, then truncate.
Isn't it odd that the group revisits this question with relative
frequency, and yet only a tiny fraction (yours of course belonging to
that honourable minority) of the answers offered seem to be even
remotely logical. You'd think there would be a decent answer by now.
Trouble is, even the FAQ lists too many possibilities. Perhaps it
needs an extra FAQ entry that specifically answers this
frequently-needed option to update a file if it exists, initialise it
if it doesn't.
Let's see if I've got this right...
* open with +>> (it atomically creates the file if necessary, opens
it if present, does not clobber it)
* flock for exclusive access
* seek to the beginning and read the contents, if any.
* seek to the beginning, truncate, write new contents.
* close (and implicitly unlock).
Am I doing alright?
It seems a chore to have to open the file at the end, only to
reposition before reading, but it seems the only safe way if open() is
used.
Is it OK to use sysopen with O_RDWR|O_CREAT , instead, and otherwise
as described?
------------------------------
Date: Wed, 05 Jul 2000 16:31:14 GMT
From: gevens@my-deja.com
Subject: Re: Definitive flock code (?)
Message-Id: <8jvnsi$f6r$1@nnrp1.deja.com>
Hi Alan,
I agree that there is a need to collectively contribute towards a nice
solution for this most straightforward task.
If only to knock this on the head once and for all.
Your previous post indeed appears to be the way of removing the
requirement for a 'semaphore' file and could well be in line as an
offering towards my goal of providing a 'definitive' method (without
wanting to invite those wishing to split semantic hairs too finely).
I will post a code snippet containing my interpretation of this method
shortly and will look forward to any comments it attracts.
Again, any other comments / suggestions / offerings gladly accepted.
Sent via Deja.com http://www.deja.com/
Before you buy.
------------------------------
Date: Wed, 5 Jul 2000 20:08:52 +0200
From: "Alan J. Flavell" <flavell@mail.cern.ch>
Subject: Re: Definitive flock code (?)
Message-Id: <Pine.GHP.4.21.0007051948210.10709-100000@hpplus03.cern.ch>
On Tue, 4 Jul 2000 gevens@my-deja.com wrote:
> http://www.plover.com/~mjd/perl/TPC/1998/Hardware-
> notes.html#Semaphore_Files
I entirely agree that there _are_ situations where the use
of a separate semaphore file like that was worth the effort.
But MJD sets us an interesting puzzle here.
Notice the comment for
open F, "> $FILE" or die ...;
flock F, LOCK_EX;
that
"Unfortunately, this is wrong. The `writing' branch has already
clobbered the contents of the file before it acquired the lock."
Right.
Now notice the replacement recipe:
open S, "> $SEMAPHORE" or die ...;
and the helpful suggestion
"You can also arrange to have the process leave its PID in the
semaphore file so you know who has the resource locked,"
But the process that is _attempting_ to gain control of the semaphore
has just "clobbered" the semaphore file, which supposedly had
previously contained that useful PID information.
So, I'm left respectfully wondering, which is it to be? Either the
file contents _are_ being clobbered, in which case the semaphore idea
is a good one, but the PID will get clobbered just when one most needs
it. Or, ... (well, see what I mean?).
I remember sort-of visiting this issue here on a previous occasion,
though I don't think MJD's page came up in the discussion. I was left
uneasy before, and, now that I look at it again, I find I'm still
uneasy on this point. Timid as I am to make such comments about MJD's
advice, which I've generally found to be way above my class.
Just for the record, the recipe that I posted earlier today (the one
with the '+>>' open, I mean) has been working fine for me, in a fairly
heavily hammered CGI script.
It strikes me however that if something went wrong, it could block
indefinitely. Perhaps next time around I'll make some provision for
parachuting-out and reporting that the file is busy.
cheers
------------------------------
Date: Wed, 05 Jul 2000 17:05:40 -0400
From: brian@smithrenaud.com (brian d foy)
Subject: Re: Definitive flock code (?)
Message-Id: <brian-ya02408000R0507001705400001@news.panix.com>
In article <Pine.GHP.4.21.0007051948210.10709-100000@hpplus03.cern.ch>, "Alan J. Flavell" <flavell@mail.cern.ch> posted:
> Notice the comment for
>
> open F, "> $FILE" or die ...;
> flock F, LOCK_EX;
>
> that
>
> "Unfortunately, this is wrong. The `writing' branch has already
> clobbered the contents of the file before it acquired the lock."
in this case you can use a temporary file to create the new file
then movie it into place when it is ready. since you should
choose a unique filename somehow related to the process (using $$),
there should be no chance that another process will clobber your
file.
--
brian d foy
CGI Meta FAQ <URL:http://www.smithrenaud.com/public/CGI_MetaFAQ.html>
Perl Mongers <URL:http://www.perl.org/>
------------------------------
Date: Wed, 5 Jul 2000 23:41:56 +0200
From: "Alan J. Flavell" <flavell@mail.cern.ch>
Subject: Re: Definitive flock code (?)
Message-Id: <Pine.GHP.4.21.0007052333570.3180-100000@hpplus03.cern.ch>
On Wed, 5 Jul 2000, brian d foy wrote:
> in this case you can use a temporary file to create the new file
> then movie it into place when it is ready.
That's excellent advice, on unix, for the situation under discussion.
I somehow had the idea that it was a less-portable technique, though.
> since you should
> choose a unique filename somehow related to the process (using $$),
Agreed. Thanks.
But assuming one goes for the create-temp-file-and-move procedure,
it's still necessary to do the locking, isn't it? Otherwise you
could have two or more processes reading the same old file, and then
applying their updates on top of each other. My head's going around
with lock strategies now - do you have a clear scenario of how to
handle that? Does this call for the semaphore file strategem?
------------------------------
Date: Wed, 05 Jul 2000 21:33:14 -0400
From: brian@smithrenaud.com (brian d foy)
Subject: Re: Definitive flock code (?)
Message-Id: <brian-ya02408000R0507002133140001@news.panix.com>
In article <Pine.GHP.4.21.0007052333570.3180-100000@hpplus03.cern.ch>, "Alan J. Flavell" <flavell@mail.cern.ch> posted:
> On Wed, 5 Jul 2000, brian d foy wrote:
>
> > in this case you can use a temporary file to create the new file
> > then movie it into place when it is ready.
> I somehow had the idea that it was a less-portable technique, though.
less portable? if you are on a system in which you can't create or
move files, then you have larger problems. was there something else
you had in mind?
> > since you should
> > choose a unique filename somehow related to the process (using $$),
> But assuming one goes for the create-temp-file-and-move procedure,
> it's still necessary to do the locking, isn't it?
yes, but you don't clobber the file?
> Otherwise you
> could have two or more processes reading the same old file, and then
> applying their updates on top of each other. My head's going around
> with lock strategies now - do you have a clear scenario of how to
> handle that? Does this call for the semaphore file strategem?
you then have to let only one process do an update at a time. if
it gets more complicated than that, you might want to consider if
the flow of your information really requires that the entire file
be updated or if you should be using flat files at all.
--
brian d foy
CGI Meta FAQ <URL:http://www.smithrenaud.com/public/CGI_MetaFAQ.html>
Perl Mongers <URL:http://www.perl.org/>
------------------------------
Date: Thu, 6 Jul 2000 12:17:09 +0200
From: "Alan J. Flavell" <flavell@mail.cern.ch>
Subject: Re: Definitive flock code (?)
Message-Id: <Pine.GHP.4.21.0007061201520.27673-100000@hpplus03.cern.ch>
On Wed, 5 Jul 2000, brian d foy wrote:
> less portable? if you are on a system in which you can't create or
> move files, then you have larger problems. was there something else
> you had in mind?
Well, AIUI in unix you execute mv and the operation is atomic: even if
something goes wrong at that instant, the files are left in a
consistent state. The worst that can happen is the update is lost.
I had the idea that this wasn't feasible on certain other OSes? You
couldn't rename the new file to the name already occupied by the old
one, so you'd have to delete or rename the old one first, thus leaving
a time window where the file is inconsistent. You might be left with
the old file deleted, and the new file still sitting at its
semi-random temporary name.
=head2 How can I reliably rename a file?
Well, usually you just use Perl's rename() function. But that may
not work everywhere, in particular, renaming files across file
systems. If your operating system supports a mv(1) program or its
moral equivalent, this works:
rename($old, $new) or system("mv", $old, $new);
It may be more compelling to use the File::Copy module instead.
> Does this call for the semaphore file strategem?
>
> you then have to let only one process do an update at a time.
Sure, that's the principle, I wasn't in doubt about that. I was
pondering which of various ways to implement it.
> it gets more complicated than that, you might want to consider if
> the flow of your information really requires that the entire file
> be updated or if you should be using flat files at all.
Certainly. The scenario that I had in mind was one where the file
needed to be completely rewritten/replaced (I thought that had been
implicit in the previous discussion, but you're right, it's aways
better to state one's terms "up front").
cheers
------------------------------
Date: 06 Jul 2000 12:40:00 -0700
From: merlyn@stonehenge.com (Randal L. Schwartz)
Subject: Re: Definitive flock code (?)
Message-Id: <m18zvfm4j3.fsf@halfdome.holdit.com>
>>>>> "Alan" == Alan J Flavell <flavell@mail.cern.ch> writes:
Alan> But assuming one goes for the create-temp-file-and-move procedure,
Alan> it's still necessary to do the locking, isn't it? Otherwise you
Alan> could have two or more processes reading the same old file, and then
Alan> applying their updates on top of each other. My head's going around
Alan> with lock strategies now - do you have a clear scenario of how to
Alan> handle that? Does this call for the semaphore file strategem?
I've posted the strategy here (or somewhere close to here) before.
Look for my posting that includes both "rename" and "flock". The
trick is to ensure that the flock you have on the filehandle is the
same file as the file that currently occupies that name, and if not,
start over.
Oh heck... I just did the deja myself. Here it is again:
{
open FILE, $file;
my ($dev,$ino) = stat(FILE);
flock FILE, 1;
my ($dev1, $ino1) = stat($file);
redo unless "$dev $ino" eq "$dev1 $ino1";
}
open FILE_OUT, ">$file.tmp";
### ... process FILE into FILE_OUT
close FILE_OUT;
rename "$file.tmp", "$file";
close FILE;
That'll do it.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
------------------------------
Date: Sat, 08 Jul 2000 05:29:05 GMT
From: mjd@plover.com (Mark-Jason Dominus)
Subject: Re: Definitive flock code (?)
Message-Id: <3966bc20.5c9b$e6@news.op.net>
In article <Pine.GHP.4.21.0007051948210.10709-100000@hpplus03.cern.ch>,
Alan J. Flavell <flavell@mail.cern.ch> wrote:
>But the process that is _attempting_ to gain control of the semaphore
>has just "clobbered" the semaphore file, which supposedly had
>previously contained that useful PID information.
But the PID information doesn't matter. It's just a convenience.
If it gets destroyed, you are no worse off than if you hadn't written
it in the first place.
------------------------------
Date: Sat, 8 Jul 2000 12:41:03 +0200
From: "Alan J. Flavell" <flavell@mail.cern.ch>
Subject: Re: Definitive flock code (?)
Message-Id: <Pine.GHP.4.21.0007081236480.276-100000@hpplus03.cern.ch>
Sorry about this: I'm trying not to be pointlessly stubborn...
On Sat, 8 Jul 2000, Mark-Jason Dominus wrote:
> >But the process that is _attempting_ to gain control of the semaphore
> >has just "clobbered" the semaphore file, which supposedly had
> >previously contained that useful PID information.
>
> But the PID information doesn't matter.
I had the idea that it would be useful in debugging. I.e to discover
which PID last took the lock, when some other process is unable to get
it. But that seems to be just the situation where it would get
clobbered, on this basis.
> It's just a convenience.
Sure...
> If it gets destroyed, you are no worse off than if you hadn't written
> it in the first place.
I've no argument with that.
cheers
------------------------------
Date: 16 Sep 99 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 16 Sep 99)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
| NOTE: The mail to news gateway, and thus the ability to submit articles
| through this service to the newsgroup, has been removed. I do not have
| time to individually vet each article to make sure that someone isn't
| abusing the service, and I no longer have any desire to waste my time
| dealing with the campus admins when some fool complains to them about an
| article that has come through the gateway instead of complaining
| to the source.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V9 Issue 3560
**************************************
