[24557] in Perl-Users-Digest
Perl-Users Digest, Issue: 6735 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Sun Jun 27 00:05:45 2004
Date: Sat, 26 Jun 2004 21:05:04 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Sat, 26 Jun 2004 Volume: 10 Number: 6735
Today's topics:
Correct file locking techniques <ducott@hotmail.com>
Re: Correct file locking techniques <usenet@morrow.me.uk>
Re: Correct file locking techniques <invalid-email@rochester.rr.com>
Re: Embedding perl in Java (Shalini Joshi)
Re: monitoring changing web pages <spamtrap@dot-app.org>
Odd quick writing data to a file <ducott@hotmail.com>
Re: Odd quick writing data to a file <invalid-email@rochester.rr.com>
Re: Regexp for ls -l listing <tadmc@augustmail.com>
Re: repeated calculations everyday (Shalini Joshi)
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 26 Jun 2004 23:46:17 GMT
From: "Robert TV" <ducott@hotmail.com>
Subject: Correct file locking techniques
Message-Id: <dVnDc.910046$Ig.605785@pd7tw2no>
Hi, I am asking foir advice on which form of code i've written below is the
correct and safe way to lock a file. I've done some reading on" use Fcntl
qw(:DEFAULT :flock);" and I understand the basics. I've also read back on
many posts regarding file locking and there seems to be much debate. For
this code, I simply want to increase a value.
#!/usr/bin/perl
use Fcntl qw(:DEFAULT :flock);
use CGI::Carp qw(fatalsToBrowser);
open (NUMBER, "<number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
$value = <NUMBER>;
close(NUMBER);
$value = $value++;
open (NUMBER, ">number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
print NUMBER $value;
close(NUMBER);
The above code seems to work ok. My question is when the first user locks
and reads the file, does the locking prevent other users from reading the
file? The $value is then increased and printed back to the file.
Another concern of mine is ... User#1 locks file, reads file value, then
close and releases lock. While User#1 is writing the new value back to the
file, another user came and read the unlocked file BEFORE the value was
increased. So he gets the same value as User#1. User#2 then increases the
value and writes the same data as User#1. In the end, the locking technique
failed ... am I right?
Another method I came up with is locking and entire read/write session
within a what I will call "parent lock" Here is the code:
#!/usr/bin/perl
use Fcntl qw(:DEFAULT :flock);
use CGI::Carp qw(fatalsToBrowser);
open (PARENT, "<locker.txt") or die "Can't open file: $!"; #an empty file
flock (PARENT, LOCK_EX) or die "Can't lock file: $!";
open (NUMBER, "<number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_SH) or die "Can't lock file: $!";
$value = <NUMBER>;
close(NUMBER);
$value = $value++;
open (NUMBER, ">number.txt") or die "Can't open file: $!";
flock (NUMBER, LOCK_SH) or die "Can't lock file: $!";
print NUMBER $value;
close(NUMBER);
close(PARENT); #closes the parent lock
Now User #1 locks the parent file and moves on to the read/write session.
All other users get stuck back at the parent lock until released. I had to
use LOCK_SH for the read/write locks for some reason ... if all locks were
set to EX the program would stall. Is this technique better? Is it
effective? Or perhaps they are both ineffective?
In this next example, I don't want to increase a value, just append to the
file on a new line. So I do this:
#!/usr/bin/perl
use Fcntl qw(:DEFAULT :flock);
use CGI::Carp qw(fatalsToBrowser);
open (DATA, ">>locker.txt") or die "Can't open file: $!";
flock (DATA,LOCK_EX) or die "Can't lock file: $!";
print DATA "$data\n";
close(DATA);
Im going to assume that this is ok.
It's just when increasing a value that concerns me ... multipul users
reading same value when the read lock is released. Your suggestions are
greatly welcomes. TIA
Robert
------------------------------
Date: 27 Jun 2004 01:30:09 +0100
From: Ben Morrow <usenet@morrow.me.uk>
Subject: Re: Correct file locking techniques
Message-Id: <40de1511_2@mk-nntp-2.news.uk.tiscali.com>
Quoth "Robert TV" <ducott@hotmail.com>:
> Hi, I am asking foir advice on which form of code i've written below is the
> correct and safe way to lock a file. I've done some reading on" use Fcntl
> qw(:DEFAULT :flock);" and I understand the basics. I've also read back on
> many posts regarding file locking and there seems to be much debate. For
> this code, I simply want to increase a value.
>
> #!/usr/bin/perl
>
> use Fcntl qw(:DEFAULT :flock);
> use CGI::Carp qw(fatalsToBrowser);
>
> open (NUMBER, "<number.txt") or die "Can't open file: $!";
> flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
> $value = <NUMBER>;
> close(NUMBER);
>
> $value = $value++;
>
> open (NUMBER, ">number.txt") or die "Can't open file: $!";
> flock (NUMBER, LOCK_EX) or die "Can't lock file: $!";
> print NUMBER $value;
> close(NUMBER);
>
> The above code seems to work ok. My question is when the first user locks
> and reads the file, does the locking prevent other users from reading the
> file?
No. It simply prevents other processes from *obtaining a lock* on the
file.
> The $value is then increased and printed back to the file.
>
> Another concern of mine is ... User#1 locks file, reads file value, then
> close and releases lock. While User#1 is writing the new value back to the
> file, another user came and read the unlocked file BEFORE the value was
> increased. So he gets the same value as User#1. User#2 then increases the
> value and writes the same data as User#1. In the end, the locking technique
> failed ... am I right?
You are. You must read/modify/write under the same, exclusive, lock;
anyone who is just reading can use a shared lock.
> Another method I came up with is locking and entire read/write session
> within a what I will call "parent lock" Here is the code:
This would be correct, but there is no need for two layers of locks. The
inner lock is made completely redundant by the outer.
Correct technique is:
Processes that will only read:
open read-only
flock LOCK_SH
read the data
close
Processes that will read and modify:
open read-write ('+<')
flock LOCK_EX
read the data
make the modifications
seek to the start of the file (perldoc -f seek)
truncate the file (perldoc -f truncate)
write the new data
close
Alternatively, you can use a lockfile instead of proper flock locking;
this can be simpler, but you have to be careful to clean up after
yourself:
sysopen a lockfile with mode O_CREAT | O_EXCL
if the open fails, sleep for a bit and try again
open/close/read/write the data file as you will
unlink the lockfile
One problem with this is you may wish to check the age of the lockfile
and summerarily (sp?) unlink it if it appears to be too old...
> I had to use LOCK_SH for the read/write locks for some reason ... if
> all locks were set to EX the program would stall.
On some systems you cannot LOCK_EX a file unless you opened it for
writing.
> In this next example, I don't want to increase a value, just append to the
> file on a new line. So I do this:
>
> #!/usr/bin/perl
>
> use Fcntl qw(:DEFAULT :flock);
> use CGI::Carp qw(fatalsToBrowser);
>
> open (DATA, ">>locker.txt") or die "Can't open file: $!";
> flock (DATA,LOCK_EX) or die "Can't lock file: $!";
> print DATA "$data\n";
> close(DATA);
>
> Im going to assume that this is ok.
Note that your readers will need to lock as well.
Ben
--
All persons, living or dead, are entirely coincidental.
ben@morrow.me.uk Kurt Vonnegut
------------------------------
Date: Sun, 27 Jun 2004 02:01:17 GMT
From: Bob Walton <invalid-email@rochester.rr.com>
Subject: Re: Correct file locking techniques
Message-Id: <40DE2A67.9000105@rochester.rr.com>
Robert TV wrote:
> Hi, I am asking foir advice on which form of code i've written below is the
> correct and safe way to lock a file. I've done some reading on" use Fcntl
I have found that the most portable and least hassle way to lock files
is to use a lock file. This is a file which is used exclusively for the
purpose of establishing a lock -- it can be empty, or have any contents
you want, but the data in the lock file would be unused and unimportant.
This technique is particularly useful when doing stuff which is a bit
out of the ordinary, such as locking DBM-type files to which you wish to
tie a hash, for example.
If you want to write a file, open the lock file for write (that will, at
least on some systems, wipe out the contents of the lock file), then
establish an exclusive lock on it. Then go do whatever you wanted the
exclusive lock for (write another file, tie a hash to a DBM-type file,
or any other operation requiring exclusive access to a resource). Then,
when you are all done with your lock, close the lock file.
If you want to read a file, open the lock file for reading, and
establish a shared lock on it. Go read your data file or do whatever
you wanted the shared lock for. When done, close the lock file.
To the best of my knowledge, that works flawlessly on at least local
files on every OS that supports locking (that, by the way, rules out
Windoze 95/98/98SE). However, you should carefully and thoroughly read
everything your OS has to say about file locking, particularly if you
are considering locking files accessed via a network.
You will want to note that locking a file does not prevent other
processes from read/writing/deleting/whatevering a file. *All* it does
is establish a lock -- the other processes have to cooperate with the
lock in order for it to work. An exclusive lock means other processes
seeking a lock will block until the exclusive lock is released; a shared
lock means other processes seeking an exclusive lock will block until
all the shared locks are released.
I find it convenient to write a couple of subs to do the locking, like
[untested]:
sub lockex{
open LOCK,">lockfile.txt"
or die "Oops, couldn't open lockfile.txt for write, $!";
flock(LOCK,LOCK_EX)
or die "flock bombed out in lockex, $!";
}
sub locksh{
open LOCK,"lockfile.txt"
or die "Oops, couldn't open lockfile.txt for read, $!";
flock(LOCK,LOCK_SH)
or die "flock bombed out in locksh, $!";
}
sub unlock{
close LOCK or die "Oops, couldn't close lockfile.txt, $!";
}
HTH.
...
> Robert
--
Bob Walton
Email: http://bwalton.com/cgi-bin/emailbob.pl
------------------------------
Date: 26 Jun 2004 17:59:33 -0700
From: shalinij1@yahoo.com (Shalini Joshi)
Subject: Re: Embedding perl in Java
Message-Id: <283d6b7e.0406261659.78dc4e43@posting.google.com>
Hey!
Thanks a lot for the information.
I appreciate it..
Regards
shalini
Lincoln Yeoh <junkto@tm.net.my> wrote in message news:<qbkgd09p78sbef435f7kdj1pvptvqm3p4a@4ax.com>...
> On 20 Jun 2004 23:00:25 -0700, shalinij1@yahoo.com (Shalini Joshi)
> wrote:
>
> >Hey!
> >
> >Yeah Ben, thanks again..i managed to use perlcc to produce an
> >executable on the unix system. It didnt work on windows though(like it
> >had been pointed out in the doc)so used the B module to produce the
> >bytecode.
> >
> >Yeah will let my boss know about the source code hiding issues..thanks
> >again.
>
> For windows - try par and for executable compression try upx
>
> http://search.cpan.org/~autrijus/PAR/lib/PAR.pm
>
> There are many commercial products that use perl. Often without
> obfuscation. Whether people have the source code or not doesn't stop
> them from copying your stuff and using it. I use UPX+PAR just coz it
> bundles things up in a single convenient file and users do not need to
> install perl. But some AV scanners[1] seem to take a bit longer to
> scan stuff like this.
>
> It only makes it harder for people to figure out or extend/modify it.
> It doesn't make it impossible.
>
> The people who don't pay and bother to figure out your source code
> often aren't going to pay for it - they either have less need for it
> (they're smart enough to write much of it themselves), or they aren't
> going to pay for it anyway.
>
> Whereas the people who've already paid for it may regard having the
> source code as a big plus. I'd personally prefer buying a program
> that's easy to fix and modify, coz often programs outlast support from
> the vendors that supply them.
>
> Given a suitably viral license it may be disadvantageous for people to
> copy and reuse your code without permission. You may wish to consult
> your lawyers about that.
>
> That said, I'm not too fond about these license things. What are the
> limits of such things anyway? How valid are they? Someone should
> implement a software license that requires users to howl at the moon
> _publicly_ every month or brush their teeth after every meal.
>
> [1] I wonder how AV scanners deal with obfuscated polymorphic
> perl/lisp/etc viruses/worms...
------------------------------
Date: Sat, 26 Jun 2004 19:20:40 -0400
From: Sherm Pendley <spamtrap@dot-app.org>
Subject: Re: monitoring changing web pages
Message-Id: <15qdnZJz8qpUmUPdRVn-uw@adelphia.com>
krakle wrote:
> You don't need a module to do so..
You don't precisely *need* a module, true, but there are certainly modules
that will help make parts of it much easier.
As you say, cron is the way to schedule a periodic job, assuming a *nix
platform. Anacron might also be useful, if the machine running the script
sleeps and/or powers down occasionally.
> Have the script check the file modification time/date on that specific
> page once a day.
LWP.
> If it changes, have the script email the group of people.
Quite a few modules to choose from here - search CPAN for "Mailer", "SMTP",
and "MIME".
sherm--
--
Cocoa programming in Perl: http://camelbones.sourceforge.net
Hire me! My resume: http://www.dot-app.org
------------------------------
Date: Sun, 27 Jun 2004 02:26:22 GMT
From: "Robert TV" <ducott@hotmail.com>
Subject: Odd quick writing data to a file
Message-Id: <ifqDc.865811$Pk3.398962@pd7tw1no>
I seem to be experiencing something odd when writing data to a file ... I
say because I have not had a problem like this before. I have 3 values to
write to the file ... I want each value on its own line. I use this code
(not working correctly)
#create the file
open (FILE,">>datafile.inc") or die "Can't open file: $!";
close(FILE);
#write data to file
open (FILE,">>datafile.inc") or die "Can't open file: $!";
print FILE "$value1\n";
print FILE "$value2\n";
print FILE "$value3\n";
close(FILE);
After this operation is complete, I can find the file on the server. When I
open it, all data is on a single line. For some reason, it's ignoring my
"\n"s. I also tried this:
#create the file
open (FILE,">>datafile.inc") or die "Can't open file: $!";
close(FILE);
#write value1 to file
open (FILE,">>datafile.inc") or die "Can't open file: $!";
print FILE "$value1\n";
close(FILE);
#write value2 to file
open (FILE,">>datafile.inc") or die "Can't open file: $!";
print FILE "$value2\n";
close(FILE);
#write value3 to file
open (FILE,">>datafile.inc") or die "Can't open file: $!";
print FILE "$value3\n";
close(FILE);
Same thing, all data is still on a single line. Funny thing is i've working
with simple flatfile datafiles before without this problem. Maybe my brain
isn't firing correctly right now. I will add this though ... if I replace my
"\n"s with "\r\n"s I get the data on multipul lines ... BUT ... "\r"s screw
up other parts of my code (not shown) Any ideas?
Robert
------------------------------
Date: Sun, 27 Jun 2004 02:49:34 GMT
From: Bob Walton <invalid-email@rochester.rr.com>
Subject: Re: Odd quick writing data to a file
Message-Id: <40DE35B8.2020907@rochester.rr.com>
Robert TV wrote:
> I seem to be experiencing something odd when writing data to a file ... I
> say because I have not had a problem like this before. I have 3 values to
> write to the file ... I want each value on its own line. I use this code
> (not working correctly)
>
> #create the file
> open (FILE,">>datafile.inc") or die "Can't open file: $!";
> close(FILE);
>
> #write data to file
> open (FILE,">>datafile.inc") or die "Can't open file: $!";
> print FILE "$value1\n";
> print FILE "$value2\n";
> print FILE "$value3\n";
> close(FILE);
>
> After this operation is complete, I can find the file on the server. When I
> open it, all data is on a single line. For some reason, it's ignoring my
> "\n"s. I also tried this:
I'll bet your server is running some flavor of *nix, and you're running
Windoze -- is that right? When you transfer your file back to your PC,
be sure and use the ASCII mode of FTP (assuming, of course, that you are
using FTP to do your file transfers). Or use a good Windoze editor that
is capabile of handling files with *nix line endings (like VIM, for
example).
Or are you perhaps looking at your file in a web browser as an HTML
file? Remember that all whitespace is the same in HTML -- a newline is
treated as if it were a space character.
...
> Robert
--
Bob Walton
Email: http://bwalton.com/cgi-bin/emailbob.pl
------------------------------
Date: Sat, 26 Jun 2004 17:15:58 -0500
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: Regexp for ls -l listing
Message-Id: <slrncdrtcu.6ap.tadmc@magna.augustmail.com>
zzapper <david@tvis.co.uk> wrote:
> $ftp->dir("$file");
perldoc -q vars
What's wrong with always quoting "$vars"?
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: 26 Jun 2004 18:58:12 -0700
From: shalinij1@yahoo.com (Shalini Joshi)
Subject: Re: repeated calculations everyday
Message-Id: <283d6b7e.0406261758.2e7a9c88@posting.google.com>
So here is what I have of the code uptil now.
The thing that has me stumped is how do access the BASE array that I
stored using the storable module. When i try and print out the
particular element, it works but doesnt work ..gives me a HASH{xxxx}
error...also as I mention right at the end I store it as BASE[0][$i]
where $i is an element of an array of unique 9 digit numbers. (not
necessarily sorted etc)...Is there any way I can get it to print based
on the length of the hard-linked array..as opposed to the index or
something???
Thanks for the help!
Regards,
Shalini
if ((localtime)[3] ==1)
{
print "Today is the first day of the month!\n";
print "We are going to read all Account Position Files!\n";
#For each client we read the Account Position File: thus all account
position files on the server.
opendir DIR, '/home/sj2122/perlstuff' or die "$!";
my @files = grep {/[^(03)]+03\.[^\.]+\.txt$/} readdir(DIR);
closedir(DIR);
foreach $i (@files)
{
# Read every position file
print "$i\n";
open ACCOUNT_POSITION, "<$i" or die "Can't open $i $!";
local $_ = <ACCOUNT_POSITION>;
if (/^APR/)
{
push @position_records,$_;
}
while (<ACCOUNT_POSITION>)
{ #get all lines of the record
# stop if we encounter an APR line
if (/^APR/)
{
push @position_records, $_;
}
}
close ACCOUNT_POSITION;
}
foreach $i (@position_records)
{
%parameters_owner= (
dealer_number => substr($i,6,7),
CUSIP => substr($i,22,9),
cust_accnt_number => substr($i,38,20),
total_shares => substr($i,59,15),
);
if ($parameters_owner{dealer_number}==7654321)
{
print "Yahoo! We need to add this CUSIP number to selected records
in Price File!\n";
push @arrayofparams, {%parameters_owner};
push @cusip, $parameters_owner{CUSIP};
push @customers, $parameters_owner{cust_accnt_number};
}
}
my %hash = map {$_ => 1} @cusip;
@cusip_new = keys %hash; # to get unique CUSIP numbers
my %hash2 = map {$_ => 1} @customers;
@customers_new = keys %hash2;
foreach $i (@cusip_new)
{
my $sharetotal=0;
for $y (0 .. $#arrayofparams)
{
for $valuesonly (values %{ $arrayofparams[$y]} )
{
if ($valuesonly eq $i)
{
print "$valuesonly\n";
print "$arrayofparams[$y]{total_shares}\n";
$sharetotal += $arrayofparams[$y]{total_shares};
}
}
}
print "Total shares for $i is $sharetotal\n";
$BASE[0]{$i}= $sharetotal; # For day 1, total for the various
CUSIPs
print "$BASE[0]{$i}\n";
}
store(\@BASE, 'base_values'); # For later access.
open RECORDS, "< AETNA1.txt " or die "Can't read file: $!";
local $_ = <RECORDS>;
while (<RECORDS>)
{ #get all lines of the record
# stop if we encounter an FPR line
if (/^FPR/)
{
push @records, $_;
}
}
close RECORDS;
my $date= substr(@records[0],22,8);
my $cpday= &UnixDate($date,"%d");
my $cpmonth=&UnixDate($date,"%m");
my $cpyear=&UnixDate($date,"%y");
my $cpdate=$cpmonth.$cpday.$cpyear;
my $cpfilename= "AD".$cpyear.$cpmonth.$cpday."\.PRI";
my $wanted_handle = IO::File->new($cpfilename, O_WRONLY|O_CREAT)
or die "Couldn't open $cpfilename for writing:$!\n";
foreach $i (@records)
{
%parameters= (
CUSIP_P => substr($i,6,9),
fund_name => substr($i,59,38),
processing_date => $cpdate,
NAV_P => substr($i,30,9),
Fund_Type_P => "MU",
Public_Offering_Price => "000".substr($i,39,9),
);
foreach $selectedcusip (@cusip_new)
{
if ($selectedcusip eq $parameters{CUSIP_P})
{
push @arraypriceparams, {%parameters};
}
}
}
for $x (0 .. $#arraypriceparams)
{
print $wanted_handle
"$arraypriceparams[$x]{CUSIP_P},$arraypriceparams[$x]{Fund_Type_P},$arraypriceparams[$x]{processing_date},$arraypriceparams[$x]{Public_Offering_Price}\n";
}
}
#If not the first day of the month, read DIRECT FINANCIAL ACTIVITY
FILE
else
{
my $mday = (localtime)[3];
print "Today we are going to read the Direct Financial Acitivity
File and see the status of our CUSIPs!\n";
# Retrieving all base values
my $base_ref = retrieve('base_values');
#my @BASE = @{$base_ref};
#foreach $i (@{$base_ref})
#{
# print "$i{AET001A70}";
#}
print "$$base_ref[0]{AET001A20}\n";
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 6735
***************************************
