[25588] in Perl-Users-Digest
Perl-Users Digest, Issue: 7832 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Fri Feb 25 14:07:10 2005
Date: Fri, 25 Feb 2005 11:05:18 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Fri, 25 Feb 2005 Volume: 10 Number: 7832
Today's topics:
Re: BerkeleyDB locking question <Paul.Marquess@btinternet.com>
Re: Comparing huge XML Files xhoster@gmail.com
Re: Do sparse arrays take up full memory? (Anno Siegel)
Re: Do sparse arrays take up full memory? xhoster@gmail.com
How to decode this unicode-hex string <sun_tong@users.sourceforge.net>
Re: How to decode this unicode-hex string <phaylon@dunkelheit.at>
Re: How to decode this unicode-hex string <sun_tong@users.sourceforge.net>
Re: How to decode this unicode-hex string <phaylon@dunkelheit.at>
Re: maximum size of a hash table <tzz@lifelogs.com>
Re: maximum size of a hash table <postmaster@castleamber.com>
Re: maximum size of a hash table <spamtrap@dot-app.org>
Re: maximum size of a hash table <postmaster@castleamber.com>
Re: maximum size of a hash table <spamtrap@dot-app.org>
Re: maximum size of a hash table <spamtrap@dot-app.org>
Re: OOP Tutorial <nobull@mail.com>
Re: Parsing a chemical formal <tzz@lifelogs.com>
Re: Parsing a chemical formal <postmaster@castleamber.com>
Re: perl network sniffer <jamiethacker@gmail.com>
Re: perl network sniffer <phaylon@dunkelheit.at>
Re: perl network sniffer <gargoyle@no.spam>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Fri, 25 Feb 2005 16:17:25 -0000
From: "Paul Marquess" <Paul.Marquess@btinternet.com>
Subject: Re: BerkeleyDB locking question
Message-Id: <3891cmF5mc9t6U1@individual.net>
"Mark" <admin@asarian-host.net> wrote in message
news:rLqdnWh8kPpGrYLfRVnyhw@giganews.com...
> "Paul Marquess" <Paul.Marquess@btinternet.com> wrote in message
> news:386vufF5kehhoU1@individual.net...
>
> Odd; where is the shared memory, btw? I loaded several databases, the way
> you suggested; then did an "ipcs" to watch shared memory segments (I have
> 48MB of it available), yet I see no pages assigned to BerkeleyDB. I guess
> it is using mmap by default.
Correct.
> So, I tried opening my $env like this:
>
> -Flags => DB_CREATE|DB_INIT_CDB|DB_INIT_MPOOL|DB_SYSTEM_MEM
>
> But $BerkeleyDB::Error said: "Invalid argument". Obviously, I am still
> mising something. :) If you, or others, know what it is, I'd love to hear
> it.
Don't go there. The CDS interface has to be used with the options I
specified.
Paul
------------------------------
Date: 25 Feb 2005 16:56:34 GMT
From: xhoster@gmail.com
Subject: Re: Comparing huge XML Files
Message-Id: <20050225115634.836$Rt@newsreader.com>
"junnuthala" <junnuthula@yahoo.com> wrote:
> Thanks for all the replies.
>
> But for a 6MB XML file, having more than 300,000 elements,
That is only 20 bytes per element. It is hard to imagine XML with any
degree of sophistication using so little space per element.
> XML::Parser
> module is taking almost 35 minutes to get the parsed result as a tree.
This is an absurdly long time. I can parse 2 3.6MB files having 100,000
elements each in under a minute (2GHz P4, linux). It uses about 100MB of
memory to do so. Either you are swapping badly, you have an ancient
machine, or there is something pathological about your system.
> Any suggestions why the XML::Parser is taking so much time to parse a
> moderately big file?
I think parts of the XML::Parser can fall back onto pure Perl if some of
the binary libraries it needs are not installed. Make sure it is not doing
this. Make sure it is not running out of real RAM.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
------------------------------
Date: 25 Feb 2005 16:07:21 GMT
From: anno4000@lublin.zrz.tu-berlin.de (Anno Siegel)
Subject: Re: Do sparse arrays take up full memory?
Message-Id: <cvnifp$8fb$1@mamenchi.zrz.TU-Berlin.DE>
Ian <giblin@panix.com> wrote in comp.lang.perl.misc:
> I always thought that
>
> my @array;
> $array[0] = 1;
> $array[99] = 1;
>
> ....would only use two array elements' worth of memory. After reading
> some of the documentation, I'm not so sure. What's the real answer
> please?
An unoccupied element in an array uses one C pointer.
Anno
------------------------------
Date: 25 Feb 2005 17:08:32 GMT
From: xhoster@gmail.com
Subject: Re: Do sparse arrays take up full memory?
Message-Id: <20050225120832.197$IC@newsreader.com>
"Ian" <giblin@panix.com> wrote:
> I always thought that
>
> my @array;
> $array[0] = 1;
> $array[99] = 1;
>
> ....would only use two array elements' worth of memory. After reading
> some of the documentation, I'm not so sure. What's the real answer
> please?
The "real" answer is that it uses as much memory as it uses.
The fake answer is that generally a sparse array is going to use less
memory than a non-sparse array covering the same range (because the "empty"
elements do take some space, but not as much as a full blown scalar would),
but will take more memory than a hash using stringified numbers as the keys
will take.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
------------------------------
Date: Fri, 25 Feb 2005 11:30:37 -0500
From: * Tong * <sun_tong@users.sourceforge.net>
Subject: How to decode this unicode-hex string
Message-Id: <pan.2005.02.25.16.30.37.393574@users.sourceforge.net>
Hi,
When I select from non-English web sites and paste into my emacs,
sometimes I get a unicode-hex string like this: \u82f1\u6587, which was
"English" in Big5 encoding.
I'm wondering how I can decode such strings and return the 8-bit character.
So far I've been looking into the following Perl modules man pages an
tried each one of them: Unicode::UTF8simple, Unicode::String,
Unicode::Lite. None of them seems to be able to do that. They handle
unicode-hex strings like this: "U+00d6 U+00d0 U+00b9 U+00fa". The
difference between the above representation is that, the \u82f1 represent
one 8-bit character, while in Perl it is represented in two U+00xx values.
I had also played with tcl decodings, but wasn't successful. Please help.
Thanks a lot!
tong
--
Tong (remove underscore(s) to reply)
*niX Power Tools Project: http://xpt.sourceforge.net/
- All free contribution & collection
------------------------------
Date: Fri, 25 Feb 2005 17:42:09 +0100
From: phaylon <phaylon@dunkelheit.at>
Subject: Re: How to decode this unicode-hex string
Message-Id: <pan.2005.02.25.16.42.09.411917@dunkelheit.at>
* Tong * wrote:
> I'm wondering how I can decode such strings and return the 8-bit
> character.
Sometimes I think all some people read from this group before posting is
the name. Look at the thread right before yours.
--
http://www.dunkelheit.at/
The eternal mistake of mankind is to set up an attainable ideal.
-- Aleister Crowley
------------------------------
Date: Fri, 25 Feb 2005 11:54:29 -0500
From: * Tong * <sun_tong@users.sourceforge.net>
Subject: Re: How to decode this unicode-hex string
Message-Id: <pan.2005.02.25.16.54.28.838720@users.sourceforge.net>
On Fri, 25 Feb 2005 17:42:09 +0100, phaylon wrote:
>> I'm wondering how I can decode such strings and return the 8-bit
>> character.
>
> Sometimes I think all some people read from this group before posting is
> the name. Look at the thread right before yours.
Can you at least specify the thread subject if you want to help? Did you
mean the thread "How to convert latin1 to utf8"? Did you see that I've tried the
Unicode::String (and much more) before the posting? After all, have you
read the two threads carefully and seen the giant difference between them?
--
Tong (remove underscore(s) to reply)
*niX Power Tools Project: http://xpt.sourceforge.net/
- All free contribution & collection
------------------------------
Date: Fri, 25 Feb 2005 18:07:22 +0100
From: phaylon <phaylon@dunkelheit.at>
Subject: Re: How to decode this unicode-hex string
Message-Id: <pan.2005.02.25.17.07.22.251396@dunkelheit.at>
* Tong * wrote:
> Can you at least specify the thread subject if you want to help?
No, that's your job. My job is to code. But sometimes I make breaks. And,
I'm sorry if this is offensive to you, but I'm not willing to spend my
breaks doing someone other's work.
> Did you mean the thread "How to convert latin1 to utf8"?
Bingo.
> Did you see that I've tried the Unicode::String (and much more) before
> the posting?
Yeah. And I said there I would try out Encode, have you done that?
> After all, have you read the two threads carefully and seen the giant
> difference between them?
Nope, clear me up.
--
http://www.dunkelheit.at/
That is not dead, which can eternal lie,
and with strange aeons even death may die.
-- H.P. Lovecraft
------------------------------
Date: Fri, 25 Feb 2005 11:50:52 -0500
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: maximum size of a hash table
Message-Id: <4n7jkwtwgj.fsf@lifelogs.com>
On 24 Feb 2005, postmaster@castleamber.com wrote:
> So, I was just wondering if Perl has a similar "limit" (ie. the result
> of "its" hash function is always n bits).
In the Perl 5.8.6 hv.h, you can see:
#ifdef PERL_HASH_INTERNAL_ACCESS
#define PERL_HASH_INTERNAL(hash,str,len) \
STMT_START { \
register const char *s_PeRlHaSh_tmp = str; \
register const unsigned char *s_PeRlHaSh = (const unsigned char *)s_PeRlHaSh_tmp; \
register I32 i_PeRlHaSh = len; \
register U32 hash_PeRlHaSh = PL_rehash_seed; \
while (i_PeRlHaSh--) { \
hash_PeRlHaSh += *s_PeRlHaSh++; \
hash_PeRlHaSh += (hash_PeRlHaSh << 10); \
hash_PeRlHaSh ^= (hash_PeRlHaSh >> 6); \
} \
hash_PeRlHaSh += (hash_PeRlHaSh << 3); \
hash_PeRlHaSh ^= (hash_PeRlHaSh >> 11); \
(hash) = (hash_PeRlHaSh + (hash_PeRlHaSh << 15)); \
} STMT_END
#endif
which is used most of the time (there's also a PERL_HASH macro which
is slower, but with the same data types) in hv.c.
This seems, to my still-bleeding eyes, to say that the actual hash is
an unsigned 32-bit value and hence the answer to your question is 32.
If I've misunderstood something, please let me know :) I haven't done
C in ages.
Ted
------------------------------
Date: 25 Feb 2005 17:47:31 GMT
From: John Bokma <postmaster@castleamber.com>
Subject: Re: maximum size of a hash table
Message-Id: <Xns960877F2D101castleamber@130.133.1.4>
Anno Siegel wrote:
> John Bokma <postmaster@castleamber.com> wrote in comp.lang.perl.misc:
>> Tassilo v. Parseval wrote:
>>
>> > Also sprach John Bokma:
>>
>> > Yet, the number of those bits (as
>> > you put it) has no bearing on the capacity of a hash. You could use
>> > a hash function that only returns 2bit-wide numbers and still store
>> > an amount of items only limited by the available memory in the
>> > hash.
>>
>> Yes, but than one ends up with a hash table with O(n) look up time
>> :-) So if the hash function is limited to n bits there is a point
>> that the hash table acts more like 2^n big lists instead of a hash
>> table.
>
> That's no contradiction. A hash of size k (with this kind of
> collision management) *is* a way to distribute a long list of length n
> over k lists of average length n/k, no more, no less. Keeping the
> average length <= 1 is an option, but not the only way to run a hash.
I am more than aware of that, (they teach those things in Utrecht, you know
;-) ). I didn't state it was a contradiction, but only that O(n) hash look
up is not what I want out of a hash table.
I am happy with constant avg length of c, and hence O(1) look up.
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html
------------------------------
Date: Fri, 25 Feb 2005 12:57:17 -0500
From: Sherm Pendley <spamtrap@dot-app.org>
Subject: Re: maximum size of a hash table
Message-Id: <gJednRRpLepj-4LfRVn-tg@adelphia.com>
John Bokma wrote:
> I am more than aware of that, (they teach those things in Utrecht, you
> know ;-) ). I didn't state it was a contradiction, but only that O(n) hash
> look up is not what I want out of a hash table.
You could use your own hashing function, if Perl's built-in gives you such
poor results. XS code can pass a hash value to hv_fetch_ent() and
hv_store_ent() - normally you'd pass 0 to have Perl calculate it for you
using its built-in, but that's just a convenience, not a requirement.
You could write a couple of fetch/store routines in C, export them to Perl,
and provide a nice %hash wrapper for them with tie(). The rest of your Perl
code would never need to know the difference.
sherm--
--
Cocoa programming in Perl: http://camelbones.sourceforge.net
Hire me! My resume: http://www.dot-app.org
------------------------------
Date: 25 Feb 2005 17:58:17 GMT
From: John Bokma <postmaster@castleamber.com>
Subject: Re: maximum size of a hash table
Message-Id: <Xns960879BD041BCcastleamber@130.133.1.4>
Ted Zlatanov wrote:
> On 24 Feb 2005, postmaster@castleamber.com wrote:
>
>> So, I was just wondering if Perl has a similar "limit" (ie. the
>> result of "its" hash function is always n bits).
[ snip ]
> This seems, to my still-bleeding eyes,
Shouldn't that be StIlLBlEeDiNgEyEs << 3 ? :-D
> to say that the actual hash is
> an unsigned 32-bit value and hence the answer to your question is 32.
I was guessing that :-D.
> If I've misunderstood something, please let me know :) I haven't done
> C in ages.
Same here :-D. (I am now even more happy about this).
So to answer the OPs question, you can have at max 2^32 slots, and only
to store the hash value for each slot requires 16 GB ( 32 bit x 2 ^ 32 =
2 ^ 2 x 2 ^32 = 2 ^36 ).
So, until you can buy a few TB at your local supermarket, don't worry
too much :-D.
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html
------------------------------
Date: Fri, 25 Feb 2005 13:04:11 -0500
From: Sherm Pendley <spamtrap@dot-app.org>
Subject: Re: maximum size of a hash table
Message-Id: <-LKdnS-jnPcA9YLfRVn-hg@adelphia.com>
Ted Zlatanov wrote:
> This seems, to my still-bleeding eyes, to say that the actual hash is
> an unsigned 32-bit value and hence the answer to your question is 32.
Actually, there's a comment towards the top of handy.h that sheds a bit more
light. I32 & U32 are at *least* 32 bits, but on systems with 64-bit ints,
I32 & U32 are also 64 bits.
sherm--
--
Cocoa programming in Perl: http://camelbones.sourceforge.net
Hire me! My resume: http://www.dot-app.org
------------------------------
Date: Fri, 25 Feb 2005 13:16:02 -0500
From: Sherm Pendley <spamtrap@dot-app.org>
Subject: Re: maximum size of a hash table
Message-Id: <mbCdncE2WML59oLfRVn-3w@adelphia.com>
John Bokma wrote:
> So to answer the OPs question, you can have at max 2^32 slots
That's not entirely accurate. On 64-bit architectures, I32 & U32 are 64-bit
ints (see handy.h), so on those architectures you can have 2^64 slots.
Thus, even on a machine with a 64-bit address space, and a perl that's built
to support it, the limiting factor is RAM.
sherm--
--
Cocoa programming in Perl: http://camelbones.sourceforge.net
Hire me! My resume: http://www.dot-app.org
------------------------------
Date: Fri, 25 Feb 2005 18:14:56 +0000
From: Brian McCauley <nobull@mail.com>
Subject: Re: OOP Tutorial
Message-Id: <cvnpkt$69f$1@sun3.bham.ac.uk>
Leslie Viljoen wrote:
>> (snip)
>> (I stopped reading after 4/23 pages as I have to go now but I may take
>> another look later)
>>
>
> Thank-you so much for the help!
To carry on...
Using the term 'our variable' to mean 'package variable' is slang.
There's nothing (IMHO) wrong with using it so long as when you introduce
the slang term you also proint out the proper term.
You use the term 'constructor' where you mean 'class method' in the
phrase 'when a method is not a constructor, the first parameter given to
it is not the class, but the blessed object'.
Calling a constuctor on an existing object is generally considered a BAD
THING. Having a constuctor that can be called on an existing object and
does something other than act as a (copy) constructor is an even worse
thing. [...] I now see that that you go on to talk about copy
constructors. IMNSHO you should not mention the idea that a constructor
could also be a instance method until you get to talking about copy
constructors.
print "address: ".$objectref->{address}."\n";
is more idiomatically
print "address: $objectref->{address}\n";
"If we don't want a copy constructor, we can make simple constructors
that can be called from both packages and objects like this"... but it's
bad form to do so, so don't do it.
Using the name super_animal of a subclass of animal is confusing.
(Since SUPER is also used to mean parent class).
The constuct base an rebless paradigm is usually the wrong one.
# 'Wrong'
$objref = animal->new ($name); # Construct base
return bless ($objref, $class);
# 'Right'
$objref = $class->SUPER::new($name)
return $objref;
The difference is that if animal::new internally calls any instance
methods that are overridden in super_animal then in the 'Wrong' example
animal::new will call the methods in animal:: rather than in
super_animal::. Usually this is the wrong thing. Occasionally it is
the right thing in which case the explict construct base and rebless
becomes the right thing.
You say of DESTROY 'If we define it, it will be called when an object
goes out of scope.' This is wrong - it is when there are nolonger any
references to the object. This may be when the variable holding the
last serviving reference goes out of scope, then again it may not.
------------------------------
Date: Fri, 25 Feb 2005 12:43:16 -0500
From: Ted Zlatanov <tzz@lifelogs.com>
Subject: Re: Parsing a chemical formal
Message-Id: <4n3bvktu17.fsf@lifelogs.com>
On 25 Feb 2005, luotao@kammer.uni-hannover.de wrote:
> I'm wrting since days a perl programm. The programm contains a small
> routine, wich shall parse a chemical formal and return the name and
> portion of single atoms
> in the material as a array(or a hash)
...
> The $molecule contains the formal (i.E. H2O, FeCl3 or CaCl), Every
> Beginning letter of a element ist written in upper case. As you can
> see, I split first the $molecule with Letters in upper case, which
> means FeCl3 turns into {F,e,C,l3}, than I scan the splitted list,
> which is stored in the array @Literal, for capital letters, every
> capital letter will be pushed in a temporary Array. If the following
> item in array is not written in upper case, which means, that the
> Name of the atom contains more than one letter, it'll be also pushed
> in the same temporary Array, which will be later joined and puted in
> the output array. The final result of the Formal H20 should be
> {H2,O}, FeCl3 {Fe,Cl3} and so on....
I think you are not doing this correctly.
You are not parsing random letters, you are parsing chemical
elements' names in sequence. So don't just say "split on a letter."
Build a dictionary of element names (it's a finite list, although you
can anticipate new elements may need to be added at the end).
Something like this:
my %elements = { H => { number => 1, extra => data => you => need },
He => { number => 2, ...},
...
};
Then, build your regular expression to match elementa from your
%element hash.
This may be a LOT easier with the Parse::RecDescent module, which I
think is the right tool for this task. It can parse formulas like the
ones you describe, as long as you write a suitable grammar (you can
write a rule that will match elements from the %elements hash). It
will generate a suitable parse tree for you, which will be a lot more
functional that your {H2,O} format. For more information and help,
mail the recdescent list at recdescent@perl.org after you've read the
Parse::RecDescent documentation :)
Ted
------------------------------
Date: 25 Feb 2005 18:01:12 GMT
From: John Bokma <postmaster@castleamber.com>
Subject: Re: Parsing a chemical formal
Message-Id: <Xns96087A441C6EAcastleamber@130.133.1.4>
Abigail wrote:
> :: The $molecule contains the formal (i.E. H2O, FeCl3 or CaCl), Every
> my @atoms = $molecule =~ /([A-Z][a-z]*[0-9]*)/g;
Note that not all "atoms" are atoms, e.g. H2, Cl3 etc.
--
John Small Perl scripts: http://johnbokma.com/perl/
Perl programmer available: http://castleamber.com/
Happy Customers: http://castleamber.com/testimonials.html
------------------------------
Date: 25 Feb 2005 08:08:41 -0800
From: "jamie@nospam.com" <jamiethacker@gmail.com>
Subject: Re: perl network sniffer
Message-Id: <1109347721.068395.96030@z14g2000cwz.googlegroups.com>
<Haven't you seen HTTP::Proxy at CPAN? What's wrong with it? Let me
guess,
you wanted to hear "other opinions"? >
I misunderstood your response. That is exactly what I was looking for.
Thanks for the input.
------------------------------
Date: Fri, 25 Feb 2005 17:17:58 +0100
From: Robert Sedlacek <phaylon@dunkelheit.at>
Subject: Re: perl network sniffer
Message-Id: <pan.2005.02.25.16.17.58.5884@dunkelheit.at>
jamie@nospam.com wrote:
> I misunderstood your response.
Then I must have misunderstood your first post, where you're saying:
| I searched on CPAN and am unsure which of the packages would be most
| useful. Any help would be much appreciated.
?
--
http://www.dunkelheit.at/
That is not dead, which can eternal lie,
and with strange aeons even death may die.
-- H.P. Lovecraft
------------------------------
Date: Fri, 25 Feb 2005 16:52:34 GMT
From: gargoyle <gargoyle@no.spam>
Subject: Re: perl network sniffer
Message-Id: <mJITd.39479$a96.36893@bignews3.bellsouth.net>
On 2005-02-25, jamie@nospam.com <jamiethacker@gmail.com> wrote:
><Google for "Proxy">
>
> I realize that it would be simple enought to place a proxy in the
> "mix", however that is not possible with our current layout. The point
> is that I would like to do it, if possible, with perl. I would like
> the information to be inserted into a mysql db (which i have no
> problems doing). Is this not possible using perl?
Just capture the output of tcpdump or some sniffer (tethereal maybe?)
and use a regex to log HTTP packets to the database. You can even
configure the sniffer to ignore all non-HTTP packets, to reduce the cpu
overhead of your script...
It shouldn't take more than a few dozen lines of code to do this. I
wrote a similar script a while back that watched for certain kinds of
ARP packets in this fashion:
open (ARP, "tcpdump -ln arp |") or die;
while (<ARP>) {
/foo/ and do_this();
/bar/ and do_that();
}
Getting tcpdump to give you URLs may require some work though (looking
inside the tcp packet), but ethereal will just outright print it to
stdout if you tell it to, I think...
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 7832
***************************************
