[31437] in Perl-Users-Digest
Perl-Users Digest, Issue: 2689 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Mon Nov 23 18:09:41 2009
Date: Mon, 23 Nov 2009 15:09:07 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Mon, 23 Nov 2009 Volume: 11 Number: 2689
Today's topics:
Re: access pref. and temp dir on windows <tadmc@seesig.invalid>
Re: access pref. and temp dir on windows <jurgenex@hotmail.com>
Re: access pref. and temp dir on windows <dontmewithme@got.it>
Re: access pref. and temp dir on windows <rkb@i.frys.com>
Re: access pref. and temp dir on windows <ben@morrow.me.uk>
How to find existence of a hash value. <venutaurus539@gmail.com>
Re: How to find existence of a hash value. <jurgenex@hotmail.com>
Re: How to get at the perl options sharma__r@hotmail.com
Re: How to get at the perl options <tzz@lifelogs.com>
perl hash: low-level implementation details? <isaacyho@gmail.com>
Re: perl hash: low-level implementation details? <jurgenex@hotmail.com>
Re: perl hash: low-level implementation details? <isaacyho@gmail.com>
Re: perl hash: low-level implementation details? <news@danrumney.co.uk>
Re: perl hash: low-level implementation details? <uri@StemSystems.com>
Perl RE bug with keys(%+) <clint.olsen@gmail.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sun, 22 Nov 2009 21:24:03 -0600
From: Tad McClellan <tadmc@seesig.invalid>
Subject: Re: access pref. and temp dir on windows
Message-Id: <slrnhgjvt0.dq1.tadmc@tadbox.sbcglobal.net>
Larry <dontmewithme@got.it> wrote:
> I am so sorry...I meant to say: where are those directories on windows?
If you were programming in Python or Visual Basic, would
it be different directories?
--
Tad McClellan
email: perl -le "print scalar reverse qq/moc.noitatibaher\100cmdat/"
------------------------------
Date: Sun, 22 Nov 2009 21:39:52 -0800
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: access pref. and temp dir on windows
Message-Id: <8s7kg55evl5tg8842r694m3321s565gauv@4ax.com>
"Larry" <dontmewithme@got.it> wrote:
>
>"Jürgen Exner" <jurgenex@hotmail.com> ha scritto nel messaggio
>news:0d1jg59ufj7g35so58odsvl2im1u7aib2q@4ax.com...
>> "Larry" <dontmewithme@got.it> wrote:
>>> I'm coding a little script for windows. I'd like to know how to save
>>> some
>>>data on a given preference file and some other data on a temp file. Can it
>>>actually be done?
>>
>> Maybe I don't see the forest for the trees, but what is the problem?
>>
>> perldoc -f open
>> perldoc -f print
>
>I am so sorry...I meant to say: where are those directories on windows?
Same place as when programming in any other programming language. Why
would they change location just because you are using Perl?
>I tried accessing temp dir on win by usinf File::Spec
>
>where the dir for preferences?
Try "set" from the command line. Usually there are some environment
variables for those values.
jue
------------------------------
Date: Mon, 23 Nov 2009 10:52:59 +0100
From: "Larry" <dontmewithme@got.it>
Subject: Re: access pref. and temp dir on windows
Message-Id: <4b0a5b79$0$34598$4fafbaef@reader1.news.tin.it>
"Ben Morrow" <ben@morrow.me.uk> ha scritto nel messaggio
news:t82pt6-rti1.ln1@osiris.mauzo.dyndns.org...
>
> Quoth "Larry" <dontmewithme@got.it>:
>>
>> "Jürgen Exner" <jurgenex@hotmail.com> ha scritto nel messaggio
>> news:0d1jg59ufj7g35so58odsvl2im1u7aib2q@4ax.com...
>> > "Larry" <dontmewithme@got.it> wrote:
>> >> I'm coding a little script for windows. I'd like to know how to save
>> >> some
> What do you mean by 'preferences'?
>
> You may want File::HomeDir.
I guess on windows vista it should be: C:\Users\<user>\AppData\... (AppData
supposed to be hidden)
------------------------------
Date: Mon, 23 Nov 2009 07:51:53 -0800 (PST)
From: Ron Bergin <rkb@i.frys.com>
Subject: Re: access pref. and temp dir on windows
Message-Id: <05914f06-f3b4-4201-893a-806c98819205@m33g2000pri.googlegroups.com>
On Nov 23, 1:52=A0am, "Larry" <dontmewit...@got.it> wrote:
> "Ben Morrow" <b...@morrow.me.uk> ha scritto nel messaggionews:t82pt6-rti1=
.ln1@osiris.mauzo.dyndns.org...
>
>
>
> > Quoth "Larry" <dontmewit...@got.it>:
>
> >> "J=FCrgen Exner" <jurge...@hotmail.com> ha scritto nel messaggio
> >>news:0d1jg59ufj7g35so58odsvl2im1u7aib2q@4ax.com...
> >> > "Larry" <dontmewit...@got.it> wrote:
> >> >> =A0 I'm coding a little script for windows. I'd like to know how to=
save
> >> >> some
> > What do you mean by 'preferences'?
>
> > You may want File::HomeDir.
>
> I guess on windows vista it should be: C:\Users\<user>\AppData\... (AppDa=
ta
> supposed to be hidden)
Have you tried:
my $tmpdir =3D $ENV{TEMP};
------------------------------
Date: Mon, 23 Nov 2009 18:31:47 +0000
From: Ben Morrow <ben@morrow.me.uk>
Subject: Re: access pref. and temp dir on windows
Message-Id: <j11rt6-uvt1.ln1@osiris.mauzo.dyndns.org>
Quoth "Larry" <dontmewithme@got.it>:
>
> "Ben Morrow" <ben@morrow.me.uk> ha scritto nel messaggio
> news:t82pt6-rti1.ln1@osiris.mauzo.dyndns.org...
> >
> > Quoth "Larry" <dontmewithme@got.it>:
> >>
> >> "Jürgen Exner" <jurgenex@hotmail.com> ha scritto nel messaggio
> >> news:0d1jg59ufj7g35so58odsvl2im1u7aib2q@4ax.com...
> >> > "Larry" <dontmewithme@got.it> wrote:
> >> >> I'm coding a little script for windows. I'd like to know how to save
> >> >> some
>
> > What do you mean by 'preferences'?
> >
> > You may want File::HomeDir.
>
> I guess on windows vista it should be: C:\Users\<user>\AppData\... (AppData
> supposed to be hidden)
File::HomeDir->my_data returns what on a default XP install would be called
C:\Documents and Settings\<user>\Local Settings\Application Data
and you can retrieve the non-Local Settings path with
Win32::GetFolderPath(Win32::CSIDL_APPDATA())
Ben
------------------------------
Date: Sun, 22 Nov 2009 23:08:20 -0800 (PST)
From: venu madhav <venutaurus539@gmail.com>
Subject: How to find existence of a hash value.
Message-Id: <f4e811e5-c442-4762-a803-acf600af7a1f@a10g2000pre.googlegroups.com>
Hi all,
I've a problem in CGI/PERL. I've a text box in my web page
whose value is taken by the server written in CGI Script. The value of
that text box is stored in an XML file for further use.
My problem is that when I don't have any value in that text
box and when CGI tries to print that value, it is printing the HASH
value of the key ( text box here). How can I validate for the
existence of value in that hash?
Please let me know if you need any other information.
Thanks,
Venu.
------------------------------
Date: Sun, 22 Nov 2009 23:18:20 -0800
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: How to find existence of a hash value.
Message-Id: <rmdkg550barfffrqkaduuv6cee7gfmbq5s@4ax.com>
venu madhav <venutaurus539@gmail.com> wrote:
>How can I validate for the
>existence of value in that hash?
Depending upon what exactly you mean one of
perldoc -f exists
perldoc -f defined
should solve the problem.
jue
------------------------------
Date: Sun, 22 Nov 2009 22:09:46 -0800 (PST)
From: sharma__r@hotmail.com
Subject: Re: How to get at the perl options
Message-Id: <0d4448ae-a9b8-4b62-88e7-e3b88a1cbb70@o9g2000prg.googlegroups.com>
On Nov 23, 5:26=A0am, "C.DeRykus" <dery...@gmail.com> wrote:
> On Nov 22, 5:56=A0am, sharma...@hotmail.com wrote:
>
>
>
>
>
> > On Nov 20, 11:33=A0pm, "C.DeRykus" <dery...@gmail.com> wrote:
>
> > > On Nov 20, 9:14=A0am, "Uri Guttman" <u...@StemSystems.com> wrote:
>
> > > > >>>>> "sr" =3D=3D sharma r <sharma...@hotmail.com> writes:
>
> > > > ...
>
> > > > then write a short wrapper script (in shell or c or even perl) to
> > > > call/exec perl and the script. don't do this in perl itself or you =
run
> > > > into your problem. if you split this into two things it becomes eas=
y.
>
> > > I'm not sure why it was included in the strawberry distro
> > > but there's an example which could be easily modified:
>
> > > #include <unistd.h>
> > > #include <stdlib.h>
> > > #include <stdio.h>
> > > //
> > > // =A0wrap perl executable
> > > //
> > > int main (int argc, char* argv[]) {
>
> > > =A0 =A0 const char* perl_cmd =3D "C:/strawberry/perl/bin/perl.orig.ex=
e";
> > > =A0 =A0 char* perl_args[argc];
> > > =A0 =A0 int i;
> > > =A0 =A0 FILE *fp;
>
> > > =A0 =A0 // perl_args[0] =3D argv[0];
> > > =A0 =A0 // for( int i =3D 1; =A0i <=3D argc; =A0i++ ) {
>
> > > =A0 =A0 fp =3D fopen("c:/temp/prog.log", "a");
> > > =A0 =A0 if (fp =3D=3D NULL) {
> > > =A0 =A0 =A0 =A0 printf("C:/temp/prog.log couldn't be opened...");
> > > =A0 =A0 =A0 =A0 exit(1);
> > > =A0 =A0 }
> > > =A0 =A0 for( i =3D 0; =A0i <=3D argc; =A0i++ ) {
> > > =A0 =A0 =A0 =A0 printf("arg %d =3D %s\n", i, argv[i] );
> > > =A0 =A0 =A0 =A0 // perl_args[i+1] =3D argv[i];
> > > =A0 =A0}
> > > =A0 =A0fprintf(fp,"%s was called...\n",argv[1]);
> > > =A0 =A0argv[0] =3D perl_cmd;
> > > =A0 =A0fclose(fp);
>
> > > =A0 =A0//return execv( perl_cmd, perl_args );
> > > =A0 =A0printf("exec'ing binary %s with arg %s...: =A0%s\n", argv[0],a=
rgv
> > > [1] );
> > > =A0 =A0return execv( perl_cmd, argv );
>
> > > }
>
> > > --
> > > Charles DeRykus
>
> > This doesn't solve my problem. The scenario when the script gets
> > executed
> > by the perl binary, as:
>
> > =A0 =A0perl [perl options] myScrit.plx [script options]
>
> > would still pick the whatever perl version is ordained by
> > the PATH variable
>
> > No amount of wrapper logic would solve this unless there's found a way
> > to recall what options were provided to perl. We already have @ARGV
> > for
> > the the script options.
>
> Without modifying Perl to provide access to all its
> commandline options (which would be useful), there
> doesn't =A0appear to be an easy way.
>
> An ugly alternative might be to force the non-standard
> invocation to re-specify the perl commandline:
>
> [untested]
>
> BEGIN{
> =A0 =A0...
> =A0 =A0if ( !exists $ENV{_perl_wrapper} || !defined ... ) {
> =A0 =A0 =A0 print "Sorry, you'll need to re-specify any "
> =A0 =A0 =A0 =A0 =A0 =A0 "options you used to invoke perl>";
> =A0 =A0 =A0 chomp( my $opts =3D <> );
> =A0 =A0 =A0 ...
> =A0 =A0 =A0 exec("perl $opts -w -S \"@ARGV\" -- $0"); # ..
> =A0 =A0}
>
> }
>
> --
> Charles DeRykus- Hide quoted text -
>
> - Show quoted text -
Thanks to everyone who took the time to reply & clarify my problem.
Really appreciate it.
I have got some good leads all due to you guys.
--Rakesh
------------------------------
Date: Mon, 23 Nov 2009 15:53:28 -0600
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: How to get at the perl options
Message-Id: <877htg3o2v.fsf@lifelogs.com>
On Sun, 22 Nov 2009 22:41:47 +0000 Ben Morrow <ben@morrow.me.uk> wrote:
BM> Quoth Oliver 'ojo' Bedford <newsojo@web.de>:
>> Am Sun, 22 Nov 2009 05:56:03 -0800 schrieb sharma__r:
>>
>> >
>> > No amount of wrapper logic would solve this unless there's found a way
>> > to recall what options were provided to perl. We already have @ARGV for
>> > the the script options.
>>
>> Then you probably have to stick to non-portable ways of getting the
>> command-line. On Linux you can evaluate /proc/$$/cmdline. On other systems
>> you could use 'ps'.
BM> Or just write a little bit of XS to grab PL_origargv and PL_origargc.
BM> You need to make sure you look at them before anyone gets a chance to
BM> assign to $0, though, since those assignments reuse that memory (for
BM> obvious reasons).
Ilya, would that be sufficient in your opinion? Devel::OriginalARGV?
I'll write it if no one else wants to.
I can't think of plausible security issues, since the script can already
look at the process list in almost all cases as Oliver and others
pointed out. Any opinions?
Ted
------------------------------
Date: Mon, 23 Nov 2009 13:47:55 -0800 (PST)
From: zikester <isaacyho@gmail.com>
Subject: perl hash: low-level implementation details?
Message-Id: <3461b6e2-e274-4287-8a0a-51b3952be9a3@w19g2000pre.googlegroups.com>
I've written a perl program that takes 3GB worth of key/value pairs
( each are numbers in the range of 0-60million ), and builds a hash
with them. The hash itself seems to be taking more than 130GB ( linux
64-bit ) and counting--I had to kill the program b/c it was growing
too large ( we have some 130Gb memory machines at our lab ).
Is there an article/book that describes the inner workings of perl
data structures / hashes in particular? I just want to know why it's
taking so much memory.
Thanks
Isaac
------------------------------
Date: Mon, 23 Nov 2009 13:56:37 -0800
From: Jürgen Exner <jurgenex@hotmail.com>
Subject: Re: perl hash: low-level implementation details?
Message-Id: <2s0mg5lv85ianhpv3i4ono69kve280b8cf@4ax.com>
zikester <isaacyho@gmail.com> wrote:
>I've written a perl program that takes 3GB worth of key/value pairs
>( each are numbers in the range of 0-60million ), and builds a hash
>with them. The hash itself seems to be taking more than 130GB ( linux
>64-bit ) and counting--I had to kill the program b/c it was growing
>too large ( we have some 130Gb memory machines at our lab ).
>
>Is there an article/book that describes the inner workings of perl
>data structures / hashes in particular? I just want to know why it's
>taking so much memory.
One wild guess: the keys in a hash are always strings. I.e. even if you
use numbers Perl will use the string value of that number, not the
numerical value itself. Depending on the minimum size of a string in
perl this could add up quickly. How many key-value pairs are we talking
about?
Implementing a sparse array may be a better choice (is there a module
for that?).
jue
------------------------------
Date: Mon, 23 Nov 2009 14:41:41 -0800 (PST)
From: zikester <isaacyho@gmail.com>
Subject: Re: perl hash: low-level implementation details?
Message-Id: <01bd2085-6542-42dd-9620-7fb5dc0fabad@j9g2000prh.googlegroups.com>
On Nov 23, 1:56=A0pm, J=FCrgen Exner <jurge...@hotmail.com> wrote:
> zikester <isaac...@gmail.com> wrote:
> >I've written a perl program that takes 3GB worth of key/value pairs
> >( each are numbers in the range of 0-60million ), and builds a hash
> >with them. =A0The hash itself seems to be taking more than 130GB ( linux
> >64-bit ) and counting--I had to kill the program b/c it was growing
> >too large ( we have some 130Gb memory machines at our lab ).
>
> >Is there an article/book that describes the inner workings of perl
> >data structures / hashes in particular? =A0 I just want to know why it's
> >taking so much memory.
>
> One wild guess: the keys in a hash are always strings. I.e. even if you
> use numbers Perl will use the string value of that number, not the
> numerical value itself. Depending on the minimum size of a string in
> perl this could add up quickly. How many key-value pairs are we talking
> about?
>
> Implementing a sparse array may be a better choice (is there a module
> for that?).
>
> jue
Good point--we are talking about 510 million pairs of data. I
actually did reimplement this in C++ using a sparse array of vectors
and it ran in about 15Gb.
And good guess as to the hash value always being strings: is there
any documentation on the low-level workings on these types of topics?
I'd love to get more in-depth knowledge of what Perl is actually doing
at the "C" level.
------------------------------
Date: Mon, 23 Nov 2009 16:43:34 -0600
From: Dan Rumney <news@danrumney.co.uk>
Subject: Re: perl hash: low-level implementation details?
Message-Id: <hef350$mgh$1@aioe.org>
Jürgen Exner wrote:
> zikester <isaacyho@gmail.com> wrote:
>> I've written a perl program that takes 3GB worth of key/value pairs
>> ( each are numbers in the range of 0-60million ), and builds a hash
>> with them. The hash itself seems to be taking more than 130GB ( linux
>> 64-bit ) and counting--I had to kill the program b/c it was growing
>> too large ( we have some 130Gb memory machines at our lab ).
>>
>> Is there an article/book that describes the inner workings of perl
>> data structures / hashes in particular? I just want to know why it's
>> taking so much memory.
>
> One wild guess: the keys in a hash are always strings. I.e. even if you
> use numbers Perl will use the string value of that number, not the
> numerical value itself. Depending on the minimum size of a string in
> perl this could add up quickly. How many key-value pairs are we talking
> about?
>
> Implementing a sparse array may be a better choice (is there a module
> for that?).
All numbers from 0 to 60m as strings is only going to add up to a few
tens of MB.
Perhaps there's a bug in your script. Can you provide the source code?
Or... at the very least, a simplified version that still results in the
same behaviour
Dan
------------------------------
Date: Mon, 23 Nov 2009 17:58:10 -0500
From: "Uri Guttman" <uri@StemSystems.com>
Subject: Re: perl hash: low-level implementation details?
Message-Id: <87d43826il.fsf@quad.sysarch.com>
>>>>> "z" == zikester <isaacyho@gmail.com> writes:
z> On Nov 23, 1:56 pm, Jürgen Exner <jurge...@hotmail.com> wrote:
>> zikester <isaac...@gmail.com> wrote:
>> >I've written a perl program that takes 3GB worth of key/value pairs
>> >( each are numbers in the range of 0-60million ), and builds a hash
>> >with them. The hash itself seems to be taking more than 130GB ( linux
>> >64-bit ) and counting--I had to kill the program b/c it was growing
>> >too large ( we have some 130Gb memory machines at our lab ).
>>
>> >Is there an article/book that describes the inner workings of perl
>> >data structures / hashes in particular? I just want to know why it's
>> >taking so much memory.
>>
>> One wild guess: the keys in a hash are always strings. I.e. even if you
>> use numbers Perl will use the string value of that number, not the
>> numerical value itself. Depending on the minimum size of a string in
>> perl this could add up quickly. How many key-value pairs are we talking
>> about?
>>
>> Implementing a sparse array may be a better choice (is there a module
>> for that?).
>>
>> jue
z> Good point--we are talking about 510 million pairs of data. I
z> actually did reimplement this in C++ using a sparse array of vectors
z> and it ran in about 15Gb.
z> And good guess as to the hash value always being strings: is there
z> any documentation on the low-level workings on these types of topics?
z> I'd love to get more in-depth knowledge of what Perl is actually doing
z> at the "C" level.
hashes take up space for the key in string form and an SV for the
value and other overhead. you can check out the space usage with the
Devel::Size module.
since your numbers are all under 60M you can save some space by using a
packed 4 byte integer for the key instead of the stringified number
which can be up to 8 bytes long. this may not actually save any space
since the allocated string buffer will likely hold 8 bytes anyhow but it
is worth trying out. easy to check this out with devel::size on a small
data set.
and it would be very easy to make your own sparse matrix (i found
Math::MatrixSparse cpan but i don't know how efficient it is and it is
version .01). just write a simple hash for your numbers and index (mod
the array size) into a large array. handle collisions with one of
several easy algorithms (rehash until no collision, sequential scan for
next empty slot, or push to an array for that slot). that should lower
your storage needs by quite a bit as arrays use less ram than hashes.
if you are really tightfisted about space, you can even implement that
whole thing in a single (very large) scalar string. that will have
almost no perl overhead but you will need to use substr to index into
the slots (of your making) and deal with collisions and buckets (that
holds the actual value of a key). this isn't too hard but could be worth
it if you need the space.
finally, why aren't you using a disk based hash like some dbm flavor?
many exist and are pretty fast and do caching for speed. then the code
is just a tied hash and you no extra work. slower for sure than ram but
if you have lots of ram, it can do the job in decent time.
uri
--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
------------------------------
Date: Mon, 23 Nov 2009 13:51:33 -0800 (PST)
From: Clint O <clint.olsen@gmail.com>
Subject: Perl RE bug with keys(%+)
Message-Id: <595bedc2-546a-4881-98cc-621bf0c46d76@u16g2000pru.googlegroups.com>
Maybe this is a bug, maybe not. I am using the named capture buffers
to reduce bugs as I change grouping of my regular expressions over
time. In a lexical analysis application, I'm using it over a series
of alternations.
my $re = qr/ (?<ALT1>pattern) | (?<ALT2>pattern) | ...
One of the alternations happens to be nested:
my $foo = qr{
(?<CODEBEGIN>
\{
(?<CODE>
(?:
(?> [^{}\n]+ ) # Non-parens without
backtracking
|
(?&CODEBEGIN) # Recurse to start of
pattern
)*
)
\}
)
}x;
However, when I ask for the keys of %+, I only get back CODEBEGIN yet
the CODE capture is there when I ask for it. My hope was to use the
keys to determine what I matched so I didn't have to do a series of
tests on %+, but apparently I will have to continue doing this since
this method won't work.
This is Perl 5.10.0.
Thanks,
-Clint
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 2689
***************************************
