[22117] in Perl-Users-Digest
Perl-Users Digest, Issue: 4339 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Fri Jan 3 18:11:06 2003
Date: Fri, 3 Jan 2003 15:10:14 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Fri, 3 Jan 2003 Volume: 10 Number: 4339
Today's topics:
Problem with huge dataset, 100000000 a magic number? <mthunter@students.uiuc.edu>
Re: Problem with huge dataset, 100000000 a magic number <ryan@cardweb.com>
Re: Problem with huge dataset, 100000000 a magic number <mthunter@students.uiuc.edu>
Re: Problem with huge dataset, 100000000 a magic number ctcgag@hotmail.com
Re: Problem with huge dataset, 100000000 a magic number <goldbb2@earthlink.net>
Re: Problem with huge dataset, 100000000 a magic number <mthunter@students.uiuc.edu>
Re: Problem with huge dataset, 100000000 a magic number <mthunter@students.uiuc.edu>
Re: Problem with huge dataset, 100000000 a magic number <mthunter@students.uiuc.edu>
Re: Problem with huge dataset, 100000000 a magic number <uri@stemsystems.com>
Re: Problem with huge dataset, 100000000 a magic number <mthunter@students.uiuc.edu>
Re: Problem with huge dataset, 100000000 a magic number ctcgag@hotmail.com
Re: Problem with huge dataset, 100000000 a magic number <mthunter@students.uiuc.edu>
Re: Problem with huge dataset, 100000000 a magic number <uri@stemsystems.com>
Re: Problem with huge dataset, 100000000 a magic number <mthunter@students.uiuc.edu>
Re: Problem with huge dataset, 100000000 a magic number ctcgag@hotmail.com
Re: Problem with huge dataset, 100000000 a magic number <mthunter@students.uiuc.edu>
Session Variables in Perl? <extendedpartition@NOSPAM.yahoo.com>
Re: Session Variables in Perl? <someone@somewhere.nl>
Re: Session Variables in Perl? <goldbb2@earthlink.net>
Re: Stat() function not working. Kind of... <ryan@cardweb.com>
Re: stdin binary or text ? <goldbb2@earthlink.net>
Re: The diamond operator <truthxayer@yahoo.com>
using, include, import? (Nina)
Re: using, include, import? <nobull@mail.com>
Re: using, include, import? <spam@thecouch.homeip.net>
Re: vectors & large amounts of data - time & space prob (Jonathan Scott Duff)
Re: XS memory management <eric.anderson@cordata.net>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Fri, 03 Jan 2003 19:02:59 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1bnok.l5m.mthunter@ux12.cso.uiuc.edu>
Greetings all,
I'm using perl to crunch some data generated from flow-tools. My script
works well until I get into really large data, where I end up with
problems. I may very well be running the machine out of memory, but
some recent debugging output produced this:
processing line 99900000
processing line 99950000
processing line 100000000
Report generation finished at Fri Jan 3 10:09:56 PST 2003
The script seems to have aborted right on or after line 100000000 (100
Million in American English). That seems way too weird to be
coincidence. Anybody have any ideas as to what could be wrong? I am
using freebsd 5-current from August with perl 5.6.1.
Thanks,
Mike
------------------------------
Date: Fri, 03 Jan 2003 14:14:42 -0500
From: Ryan <ryan@cardweb.com>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <FilR9.61$NS3.10396@news.abs.net>
Mike Hunter wrote:
> processing line 99900000
> processing line 99950000
> processing line 100000000
> Report generation finished at Fri Jan 3 10:09:56 PST 2003
>
> The script seems to have aborted right on or after line 100000000 (100
> Million in American English). That seems way too weird to be
> coincidence. Anybody have any ideas as to what could be wrong? I am
> using freebsd 5-current from August with perl 5.6.1.
without posting some code, I think you're unlikely to get a good answer.
You could be doing something that is running into certain limitations
in your platform, Perl itself, it's impossible to know based on what
you've provided.
Ryan
------------------------------
Date: Fri, 03 Jan 2003 19:28:49 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1bp92.pca.mthunter@ux12.cso.uiuc.edu>
On Fri, 03 Jan 2003 14:14:42 -0500, Ryan wrote:
> Mike Hunter wrote:
> without posting some code, I think you're unlikely to get a good answer.
> You could be doing something that is running into certain limitations
> in your platform, Perl itself, it's impossible to know based on what
> you've provided.
Thanks for responding. While I'm thinking of it, is there a way to get
perl to tell you how much free memory remains on the machine? Also, I
read somewhere about perl auto-forking to get more memory, but I have
subsequently been unable to find any info on that...does anybody know
anything about that?
The code is gonna be a bit out of context without the surrounding
flow-tools environment, but here's the segment you're interested in:
open DATA, "$flow_print_exe -l -f 5 < $tempdir/poi.$$ |" or
die "could not open flowprinted data pipe!";
print STDERR "Processing big data\n";
my %host_hash;
foreach my $host (@sorted_prelim)
{
$existance_hash{$host} = 1;
}
print "begginging while (<DATA)>)\n";
while (<DATA>)
{
print "processing line $.\n" if (($. % 50000) == 0);
if (/^[^0-9]/)
{
next;
}
my ($starttime, $endtime, $source_interface, $source_ip, $source_port,
$dest_interface, $dest_ip, $dest_port, $ip_prot, $tcp_flags,
$packets, $octets) = split /\s+/, $_;
my @dest_split = split /\./, $dest_ip;
my @source_split = split /\./, $source_ip;
my $dest_string = pack "CCCCS", $dest_split[0], $dest_split[1],
$dest_split[2], $dest_split[3], $dest_port;
my $source_string = pack "CCCCS", $source_split[0], $source_split[1],
$source_split[2], $source_split[3], $source_port;
#ip:host is packed, put in hash according to directionality
#format: flows in-pk in-oc out-pk out-oc total-pk total-oc
if ( exists $existance_hash{$dest_ip})
{
&row_add(\%host_hash, $dest_string, $source_string,
[1, $packets, $octets, 0, 0, $packets, $octets]);
}
if ( exists $existance_hash{$source_ip})
{
&row_add(\%host_hash, $source_string, $dest_string,
[1, 0, 0, $packets, $octets, $packets, $octets ]);
}
}
close DATA;
...
sub row_add
{
my ($hash_ref, $peer1_str, $peer2_str, $list_ref) = @_;
my %peer_hash; #often added
if (not exists $hash_ref->{$peer1_str})
{
#two extra for total
$peer_hash{"_total_"} = [0,0,0,0,0,0,0];
$hash_ref->{$peer1_str} = \%peer_hash;
}
if (not exists $hash_ref->{$peer1_str}->{$peer2_str})
{
$hash_ref->{$peer1_str}->{$peer2_str} = [0,0,0,0,0,0,0];
}
for (my $i = 0; $i <= $#$list_ref; $i++)
{
$hash_ref->{$peer1_str}->{$peer2_str}->[$i] +=
$list_ref->[$i];
$hash_ref->{$peer1_str}->{"_total_"}->[$i] +=
$list_ref->[$i];
}
}
------------------------------
Date: 03 Jan 2003 19:44:10 GMT
From: ctcgag@hotmail.com
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <20030103144410.333$z9@newsreader.com>
mthunter@uiuc.edu wrote:
> Greetings all,
>
> I'm using perl to crunch some data generated from flow-tools. My script
> works well until I get into really large data,
How really large is it?
> where I end up with
> problems. I may very well be running the machine out of memory, but
> some recent debugging output produced this:
>
> processing line 99900000
> processing line 99950000
> processing line 100000000
> Report generation finished at Fri Jan 3 10:09:56 PST 2003
Your script seems to have terminated with a message indicating
normal termination. The most straightforward conclusion is there is
nothing wrong with (the termination of) your script.
> The script seems to have aborted right on or after line 100000000 (100
> Million in American English).
Well, if the script aborted there, it probably wouldn't use it's last
breath to print a "Report generation finished" message (unless you told
it to do that, in which case you should know why you told it to that).
> That seems way too weird to be
> coincidence.
100_000_000 within 50_000 is not much of a coincidence, especially
considering people don't post here evertime something dies at 292_251_831.
> Anybody have any ideas as to what could be wrong?
Yes. You have a typo at line 23.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service New Rate! $9.95/Month 50GB
------------------------------
Date: Fri, 03 Jan 2003 15:22:00 -0500
From: Benjamin Goldberg <goldbb2@earthlink.net>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <3E15F0E8.70374D38@earthlink.net>
Mike Hunter wrote:
[snip]
> while (<DATA>)
> {
> print "processing line $.\n" if (($. % 50000) == 0);
[snip]
> }
>
> close DATA;
Insert a line saying:
print "processed $. lines\n";
Just *after* the while loop completes.
It seems to me that the most likely reason for the output you see is
that your $flow_print_exe program is producing more than 100000000, but
fewer than 100050000 lines. The only "coincidence" involved is that the
number of lines was less than 50 thousand more than 100 million -- *not*
that the number of lines was exactly 100 million (which would indeed
seem a bit odd).
--
$..='(?:(?{local$^C=$^C|'.(1<<$_).'})|)'for+a..4;
$..='(?{print+substr"\n !,$^C,1 if $^C<26})(?!)';
$.=~s'!'haktrsreltanPJ,r coeueh"';BEGIN{${"\cH"}
|=(1<<21)}""=~$.;qw(Just another Perl hacker,\n);
------------------------------
Date: Fri, 03 Jan 2003 20:10:33 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1brna.1j6.mthunter@ux12.cso.uiuc.edu>
On 03 Jan 2003 19:44:10 GMT, ctcgag@hotmail.com wrote:
> mthunter@uiuc.edu wrote:
> > Greetings all,
> >
> > I'm using perl to crunch some data generated from flow-tools. My script
> > works well until I get into really large data,
>
> How really large is it?
I think it's 7 gigs as binary data, which expands human-readable before
going into the perl script.
> > where I end up with
> > problems. I may very well be running the machine out of memory, but
> > some recent debugging output produced this:
> >
> > processing line 99900000
> > processing line 99950000
> > processing line 100000000
> > Report generation finished at Fri Jan 3 10:09:56 PST 2003
>
> Your script seems to have terminated with a message indicating
> normal termination. The most straightforward conclusion is there is
> nothing wrong with (the termination of) your script.
Actually that normal message was from the script that's running the
script in question. Within that script, $? does not seem to be getting
set.
> > The script seems to have aborted right on or after line 100000000 (100
> > Million in American English).
>
> Well, if the script aborted there, it probably wouldn't use it's last
> breath to print a "Report generation finished" message (unless you told
> it to do that, in which case you should know why you told it to that).
>
> > That seems way too weird to be
> > coincidence.
>
> 100_000_000 within 50_000 is not much of a coincidence, especially
> considering people don't post here evertime something dies at 292_251_831.
That's a good point.
> > Anybody have any ideas as to what could be wrong?
>
> Yes. You have a typo at line 23.
I'll look at that...
Mike
------------------------------
Date: Fri, 03 Jan 2003 20:14:42 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1brv3.1j6.mthunter@ux12.cso.uiuc.edu>
On Fri, 03 Jan 2003 15:22:00 -0500, Benjamin Goldberg wrote:
> Mike Hunter wrote:
> [snip]
> > while (<DATA>)
> > {
> > print "processing line $.\n" if (($. % 50000) == 0);
> [snip]
> > }
> >
> > close DATA;
>
> Insert a line saying:
>
> print "processed $. lines\n";
>
> Just *after* the while loop completes.
>
> It seems to me that the most likely reason for the output you see is
> that your $flow_print_exe program is producing more than 100000000, but
> fewer than 100050000 lines. The only "coincidence" involved is that the
> number of lines was less than 50 thousand more than 100 million -- *not*
> that the number of lines was exactly 100 million (which would indeed
> seem a bit odd).
You're probably right. I put that in. I'll let you know what happens.
Mike
------------------------------
Date: Fri, 03 Jan 2003 20:15:21 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1bs0a.1j6.mthunter@ux12.cso.uiuc.edu>
On 03 Jan 2003 19:44:10 GMT, ctcgag@hotmail.com wrote:
> mthunter@uiuc.edu wrote:
> > Greetings all,
> >
> > I'm using perl to crunch some data generated from flow-tools. My script
> > works well until I get into really large data,
>
> How really large is it?
>
> > where I end up with
> > problems. I may very well be running the machine out of memory, but
> > some recent debugging output produced this:
> >
> > processing line 99900000
> > processing line 99950000
> > processing line 100000000
> > Report generation finished at Fri Jan 3 10:09:56 PST 2003
>
> Your script seems to have terminated with a message indicating
> normal termination. The most straightforward conclusion is there is
> nothing wrong with (the termination of) your script.
>
>
> > The script seems to have aborted right on or after line 100000000 (100
> > Million in American English).
>
> Well, if the script aborted there, it probably wouldn't use it's last
> breath to print a "Report generation finished" message (unless you told
> it to do that, in which case you should know why you told it to that).
>
> > That seems way too weird to be
> > coincidence.
>
> 100_000_000 within 50_000 is not much of a coincidence, especially
> considering people don't post here evertime something dies at 292_251_831.
>
> > Anybody have any ideas as to what could be wrong?
>
> Yes. You have a typo at line 23.
Is this the line 23 you're talking about?
$packets, $octets) = split /\s+/, $_;
I don't see the typo.
Mike
------------------------------
Date: Fri, 03 Jan 2003 20:24:49 GMT
From: Uri Guttman <uri@stemsystems.com>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <x7ptre0ynn.fsf@mail.sysarch.com>
>>>>> "MH" == Mike Hunter <mthunter@students.uiuc.edu> writes:
MH> On 03 Jan 2003 19:44:10 GMT, ctcgag@hotmail.com wrote:
>>
>> Yes. You have a typo at line 23.
MH> Is this the line 23 you're talking about?
MH> $packets, $octets) = split /\s+/, $_;
MH> I don't see the typo.
it was a JOKE! your original post (which is what he responded to) had no
source code. so he 'found' the typo on line 23. it was a hint to post
code and not just complain that your program didn't work. how can
someone fix code that isn't seen? notice he quoted your original post
and not the later one which showed code.
uri
--
Uri Guttman ------ uri@stemsystems.com -------- http://www.stemsystems.com
----- Stem and Perl Development, Systems Architecture, Design and Coding ----
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
Damian Conway Perl Classes - January 2003 -- http://www.stemsystems.com/class
------------------------------
Date: Fri, 03 Jan 2003 20:29:14 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1bsqb.4l7.mthunter@ux12.cso.uiuc.edu>
On Fri, 03 Jan 2003 20:24:49 GMT, Uri Guttman wrote:
> >>>>> "MH" == Mike Hunter <mthunter@students.uiuc.edu> writes:
>
> MH> On 03 Jan 2003 19:44:10 GMT, ctcgag@hotmail.com wrote:
> >>
> >> Yes. You have a typo at line 23.
>
> MH> Is this the line 23 you're talking about?
>
> MH> $packets, $octets) = split /\s+/, $_;
>
> MH> I don't see the typo.
>
> it was a JOKE! your original post (which is what he responded to) had no
> source code. so he 'found' the typo on line 23. it was a hint to post
> code and not just complain that your program didn't work. how can
> someone fix code that isn't seen? notice he quoted your original post
> and not the later one which showed code.
Fair enough :) But people reference other posts all the time.
------------------------------
Date: 03 Jan 2003 21:29:22 GMT
From: ctcgag@hotmail.com
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <20030103162922.750$ba@newsreader.com>
mthunter@uiuc.edu wrote:
> On Fri, 03 Jan 2003 15:22:00 -0500, Benjamin Goldberg wrote:
> > Mike Hunter wrote:
> > [snip]
> > > while (<DATA>)
> > > {
> > > print "processing line $.\n" if (($. % 50000) == 0);
> > [snip]
> > > }
> > >
> > > close DATA;
> >
> > Insert a line saying:
> >
> > print "processed $. lines\n";
> >
> > Just *after* the while loop completes.
> You're probably right. I put that in. I'll let you know what happens.
And if that doesn't solve it, then change the progress line to
print "processing line $.\n" if (($. % 50000) == 0 or $. > 99_999_000);
to see exactly where it dies. (don't forget to set $| first ).
Also, check the return value of "close DATA;"
Finally, how are you invoking this script? Are you sure you're checking
exit status properly?
And yes, typo in line 23 was a joke. I hadn't yet seen your follow-up
with code.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service New Rate! $9.95/Month 50GB
------------------------------
Date: Fri, 03 Jan 2003 21:41:04 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1c111.fjb.mthunter@ux12.cso.uiuc.edu>
On 03 Jan 2003 21:29:22 GMT, ctcgag@hotmail.com wrote:
> > You're probably right. I put that in. I'll let you know what happens.
>
> And if that doesn't solve it, then change the progress line to
> print "processing line $.\n" if (($. % 50000) == 0 or $. > 99_999_000);
> to see exactly where it dies. (don't forget to set $| first ).
Are you suggesting I set $| just for the debugging, or are you
suggesting it would be a good thing to do in this situation in general?
If they latter, what's your reasoning?
> Also, check the return value of "close DATA;"
Will do. Thanks.
> Finally, how are you invoking this script? Are you sure you're checking
> exit status properly?
The script is being called from within another perl script, and I'm
checking $?
> And yes, typo in line 23 was a joke. I hadn't yet seen your follow-up
> with code.
:)
Mike
------------------------------
Date: Fri, 03 Jan 2003 21:48:20 GMT
From: Uri Guttman <uri@stemsystems.com>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <x7k7hl29cs.fsf@mail.sysarch.com>
>>>>> "MH" == Mike Hunter <mthunter@students.uiuc.edu> writes:
MH> On Fri, 03 Jan 2003 20:24:49 GMT, Uri Guttman wrote:
>> >>>>> "MH" == Mike Hunter <mthunter@students.uiuc.edu> writes:
>>
MH> On 03 Jan 2003 19:44:10 GMT, ctcgag@hotmail.com wrote:
>> >>
>> >> Yes. You have a typo at line 23.
>>
MH> Is this the line 23 you're talking about?
>>
MH> $packets, $octets) = split /\s+/, $_;
>>
MH> I don't see the typo.
>>
>> it was a JOKE! your original post (which is what he responded to) had no
>> source code. so he 'found' the typo on line 23. it was a hint to post
>> code and not just complain that your program didn't work. how can
>> someone fix code that isn't seen? notice he quoted your original post
>> and not the later one which showed code.
MH> Fair enough :) But people reference other posts all the time.
but he quoted only from your original. the typo comment is a known joke
about people who don't post code. the line number is sometimes 42. if
there was a real typo, any regular here would have quoted the code and
pointed it out.
uri
--
Uri Guttman ------ uri@stemsystems.com -------- http://www.stemsystems.com
----- Stem and Perl Development, Systems Architecture, Design and Coding ----
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
Damian Conway Perl Classes - January 2003 -- http://www.stemsystems.com/class
------------------------------
Date: Fri, 03 Jan 2003 21:59:09 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1c22u.ka7.mthunter@ux12.cso.uiuc.edu>
On Fri, 03 Jan 2003 21:48:20 GMT, Uri Guttman wrote:
> >> it was a JOKE! your original post (which is what he responded to) had no
> >> source code. so he 'found' the typo on line 23. it was a hint to post
> >> code and not just complain that your program didn't work. how can
> >> someone fix code that isn't seen? notice he quoted your original post
> >> and not the later one which showed code.
>
> MH> Fair enough :) But people reference other posts all the time.
>
> but he quoted only from your original. the typo comment is a known joke
> about people who don't post code. the line number is sometimes 42. if
> there was a real typo, any regular here would have quoted the code and
> pointed it out.
If it had been 42 I would have gotten it straight away.
------------------------------
Date: 03 Jan 2003 22:55:03 GMT
From: ctcgag@hotmail.com
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <20030103175503.670$b3@newsreader.com>
mthunter@uiuc.edu wrote:
> On 03 Jan 2003 21:29:22 GMT, ctcgag@hotmail.com wrote:
>
> > > You're probably right. I put that in. I'll let you know what
> > > happens.
> >
> > And if that doesn't solve it, then change the progress line to
> > print "processing line $.\n" if (($. % 50000) == 0 or $. >
> > 99_999_000); to see exactly where it dies. (don't forget to set $|
> > first ).
>
> Are you suggesting I set $| just for the debugging, or are you
> suggesting it would be a good thing to do in this situation in general?
> If they latter, what's your reasoning?
Mostly for debugging, because without it an abnormal termination can
swallow some of your pending output, making you think it died much earlier
than it did. (Of course, that can be avoided by using warn rather than
print for debugging purposes, assuming STDERR isn't overly polluted with
other warnings).
But in general, when I'm not writing command line filters or CGIs, most
prints to implicit STDOUT are either debugging prints or progress
monitoring prints, so I end up doing $|++ a lot (but not as much as I
should, physician heal thyself.)
> > Finally, how are you invoking this script? Are you sure you're
> > checking exit status properly?
>
> The script is being called from within another perl script, and I'm
> checking $?
using what: system, backticks, do, parallel::jobs, piped open ?
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service New Rate! $9.95/Month 50GB
------------------------------
Date: Fri, 03 Jan 2003 23:01:07 GMT
From: Mike Hunter <mthunter@students.uiuc.edu>
Subject: Re: Problem with huge dataset, 100000000 a magic number?
Message-Id: <slrnb1c5n4.t56.mthunter@ux12.cso.uiuc.edu>
On 03 Jan 2003 22:55:03 GMT, ctcgag@hotmail.com wrote:
> > > Finally, how are you invoking this script? Are you sure you're
> > > checking exit status properly?
> >
> > The script is being called from within another perl script, and I'm
> > checking $?
>
> using what: system, backticks, do, parallel::jobs, piped open ?
system
Mike
------------------------------
Date: Fri, 03 Jan 2003 10:45:56 -0600
From: Extended Partition <extendedpartition@NOSPAM.yahoo.com>
Subject: Session Variables in Perl?
Message-Id: <m5fb1vg6is925capm02mtqs7t4kk8ei1q5@4ax.com>
Hello Everyone,
I've been looking aroung GOOGLE.COM for a tutorial on how to use
session variables in my Perl web application. It seems that almost
everything I find is fairly useless. Could someone kindly point me in
the right direction?
Thanks in Advance,
Anthony Saffer
------------------------------
Date: Fri, 3 Jan 2003 21:59:00 +0100
From: "Stefan" <someone@somewhere.nl>
Subject: Re: Session Variables in Perl?
Message-Id: <3e15fa73$0$2243$e4fe514c@dreader6.news.xs4all.nl>
"Extended Partition" <extendedpartition@NOSPAM.yahoo.com> schreef in bericht
news:m5fb1vg6is925capm02mtqs7t4kk8ei1q5@4ax.com...
> Hello Everyone,
>
> I've been looking aroung GOOGLE.COM for a tutorial on how to use
> session variables in my Perl web application. It seems that almost
> everything I find is fairly useless. Could someone kindly point me in
> the right direction?
>
> Thanks in Advance,
> Anthony Saffer
Hay Anthony,
What did you find so far ? And more important, what kind of functionality
do you have in mind ? Any special requirements ? Be as specific and
informative as you can and you'll see you'll get lots of usefull answers.
About your original question; the problem is that a new instance is started
with every new client-click. But what about a session-manager-process ? A
continuously running daemon which administrates whatever you want using an
unique (and very random) session-id. This may be part of the URL passed from
and to the client. The web-application and session-manager can communicate
using ordinary IPC methods (see www.perldoc.com).
Good luck,
Stefan
------------------------------
Date: Fri, 03 Jan 2003 17:04:52 -0500
From: Benjamin Goldberg <goldbb2@earthlink.net>
Subject: Re: Session Variables in Perl?
Message-Id: <3E160904.65AC276F@earthlink.net>
Extended Partition wrote:
>
> Hello Everyone,
>
> I've been looking aroung GOOGLE.COM for a tutorial on how to use
> session variables in my Perl web application. It seems that almost
> everything I find is fairly useless. Could someone kindly point me in
> the right direction?
Under normal circumstances, perl doesn't have session variables. Every
time you start a perl program as a CGI handler, it is a new instance of
perl.
That said, there are a number of ways of maintaining persistant data,
some of which are perl-specific, some of which are not.
If you are using mod_perl, mod_speedycgi, or mod_persistantperl, then
each invocation of your script is *not* a new process, but is actually a
subroutine call. Thus, you can maintain persistant data in a global
variable, declared with our() or use vars. Beware, though, that if
there're more than one httpd process, each one will have it's own
embeded mod_<whatever> instance, which can mean you'll have N different
independent pieces of persistant state, one per process.
If you are running mod_perl under Apache, you can use Apache::Session or
Apache::SessionX. This is designed to let you keep per-webuser data,
and avoids some of the problems that come from having more than one web
server process.
You can use shared memory with or without perl, and you'll have
persistant data which exists until your machine reboots, and only one
instance of it no matter how many web server processes there are. The
perl functions for accessing shared memory are shmget, shmread,
shmwrite, and shmctl. Beware, though, that they offer no
synchronization, which means you'll need to use semaphores to guard
access to the shared memory pieces (semctl, semget, and semop).
You can also use and of the IPC::Cache, IPC::MM, IPC::ShareLite,
IPC::Sharable, or IPC::SharedCache modules for managing shared memory.
Note that neither IPC::Cache nor IPC::MM provide a locking facility, so
you'll have to write that part yourself, if you use one of them.
With mod_fastcgi, you can have a piece of persistant data simply by
declaring it before the while loop which accepts fastcgi connections.
This works regardless of whether you use perl and CGI::Fast, or whether
you write a fastcgi client in C.
You can have persistant data by storing your stuff in a file or a
database at the end of your script, and retrieving it at the beginning.
For this, I would suggest you check out CPAN's cache modules:
http://search.cpan.org/search?query=cache&mode=dist
In a web environment (cgi or mod_<whatever>) then you can send cookies
to the browser, and read then back.
A good combination is to use a cookie to identify the browser, and then
use that cookie as a key to a cache of some sort. See:
http://www.stonehenge.com/merlyn/WebTechniques/col61.html
--
$..='(?:(?{local$^C=$^C|'.(1<<$_).'})|)'for+a..4;
$..='(?{print+substr"\n !,$^C,1 if $^C<26})(?!)';
$.=~s'!'haktrsreltanPJ,r coeueh"';BEGIN{${"\cH"}
|=(1<<21)}""=~$.;qw(Just another Perl hacker,\n);
------------------------------
Date: Fri, 03 Jan 2003 14:53:48 -0500
From: Ryan <ryan@cardweb.com>
Subject: Re: Stat() function not working. Kind of...
Message-Id: <hTlR9.65$NS3.10988@news.abs.net>
Mac wrote:
> Thank you "purl guru" that fixed the problem. Tad go jump in a lake, dickhead.
That "dickhead" gave you the best advice one can give. He instructed
you to read the documentation, showed you how, and then to be nice
pointed you to the exact place in the documentation to get the specific
answer you were looking for.
Had you followed his advice, you would have a. learned how to use
perldoc, b. found the answer you needed right away and understood what
you were doing wrong, c. would have had the satisfaction of correcting
the problem yourself, and finally d. would be more prepared to solve
future problems you will undoubtably run into.
And here's the real kicker. Tad was really trying to help you. He knew
what the problem was instantly, and could have just replied simply
"perldoc -f readdir", but he took the time to read the readdir page
himself so he could point you to the proper place in the doc...
Hopefully you have learned something from this.
Ryan
------------------------------
Date: Fri, 03 Jan 2003 15:40:29 -0500
From: Benjamin Goldberg <goldbb2@earthlink.net>
Subject: Re: stdin binary or text ?
Message-Id: <3E15F53D.2B421C1A@earthlink.net>
Villy Kruse wrote:
> Benjamin Goldberg wrote:
>
> >cedric wrote:
> >>
> >> Hi !
> >> I would like to know if in perl stdin is opened in binary or text
> >> mode?
> >
> >In the absence of -Mopen=..., perl's stdin is opened in text mode.
> >
> >This means, if you're on dos/windows, any CRLF that's read from stdin
> >in will appear in the string as just "\n".
> >
> >And if you're on the newest versions of RedHat (or something else
> >which sets $ENV{LANG}), and if you're using perl5.8, then text mode
> >means that stdin is UTF8.
>
> The LANG environment variable should contain the text utf8 or similar
> for this to take effect, shouldn't it?
Yes.
Perl checks whether any of the LANG, LC_ALL, and LC_CTYPE environment
variables start with either 'utf8' or 'utf-8' (case insensitivly). It
also checks if nl_langinfo(CODESET) starts with 'utf8', if that function
is available (this likely gets it's info from the environment variables,
too; I'm not sure on this though.).
--
$..='(?:(?{local$^C=$^C|'.(1<<$_).'})|)'for+a..4;
$..='(?{print+substr"\n !,$^C,1 if $^C<26})(?!)';
$.=~s'!'haktrsreltanPJ,r coeueh"';BEGIN{${"\cH"}
|=(1<<21)}""=~$.;qw(Just another Perl hacker,\n);
------------------------------
Date: Fri, 03 Jan 2003 14:31:59 -0800
From: TruthXayer <truthxayer@yahoo.com>
Subject: Re: The diamond operator
Message-Id: <3E160F5F.6000008@yahoo.com>
James Tate wrote:
> Hello,
>
> I have looked through perldoc.com and have read what the have about
> the diamond operator.
Read more carefully, its explained in perldoc
Evaluating a filehandle in angle brackets yields the next
line from that file (newline, if any, included), or undef at
end of file. Ordinarily you must assign that value to a
variable, but there is one situation where an automatic
assignment happens. If and ONLY if the input symbol is the
only thing inside the conditional of a while or for(;;)
loop, the value is automatically assigned to the variable
$_. In these loop constructs, the assigned value (whether
assignment is automatic or explicit) is then tested to see
if it is defined. The defined test avoids problems where
line has a string value that would be treated as false by
perl e.g. "" or "0" with no trailing newline.
Anyway, the following lines are equivalent to each other:
while (defined($_ = <STDIN>)) { print; }
while ($_ = <STDIN>) { print; }
while (<STDIN>) { print; }
for (;<STDIN>;) { print; }
print while defined($_ = <STDIN>);
print while ($_ = <STDIN>);
print while <STDIN>;
and this also behaves similarly, but avoids the use of $_ :
while (my $line = <STDIN>) { print $line }
If you really mean such values to terminate the loop they
should be tested for explicitly:
while (($_ = <STDIN>) ne '0') { ... }
while (<STDIN>) { last unless $_; ... }
In other boolean contexts, <filehandle> without explicit
defined test or comparison will solicit a warning if -w is
in effect.
The filehandles STDIN, STDOUT, and STDERR are predefined.
(The filehandles stdin, stdout, and stderr will also work
except in packages, where they would be interpreted as local
identifiers rather than global.) Additional filehandles may
be created with the open() function. See the open() entry
in the perlfunc manpage for details on this.
>
> An example they give me is:
>
>>while (defined($line = <>)) {
>> chomp($line);
>> print "It was $line that I saw!\n";
>>}
>
>
> I don't really understand why the diamond operator is there in the
> while loop...
>
> Thank you. :)
>
> --
> Posted via http://dbforums.com
------------------------------
Date: 3 Jan 2003 09:28:53 -0800
From: arya7@hotmail.com (Nina)
Subject: using, include, import?
Message-Id: <d2b4fb35.0301030928.2d99ef7f@posting.google.com>
Hi,
I want to be able to move out small pieces of code from one perl file
into an external file, and just simply include it from the original
file.
Something that works like #include in C/C++. Mostly it's going to be
variable initializations or configuration of the program...
How to do this?
thanks!
- Nina
E.g.
#!/usr/bin/perl
# START ### Move the following out
$SOMEVAR = 5;
$OTHERVAR = 10;
# END ###
# rest of the remaining code follows
...
# END
I want to be able to do
#!/usr/bin/perl
#include "config.pl"
...
# END
------------------------------
Date: 03 Jan 2003 18:02:52 +0000
From: Brian McCauley <nobull@mail.com>
Subject: Re: using, include, import?
Message-Id: <u9vg16862a.fsf@wcl-l.bham.ac.uk>
arya7@hotmail.com (Nina) writes:
> I want to be able to move out small pieces of code from one perl file
> into an external file, and just simply include it from the original
> file.
>
> Something that works like #include in C/C++.
That would be one of do / require or use depending on your exact requirements.
Probaly you are looking for is 'require'.
Probably what you really should use is 'use'.
--
\\ ( )
. _\\__[oo
.__/ \\ /\@
. l___\\
# ll l\\
###LL LL\\
------------------------------
Date: Fri, 03 Jan 2003 13:40:15 -0500
From: Mina Naguib <spam@thecouch.homeip.net>
Subject: Re: using, include, import?
Message-Id: <3E15D90F.6090903@thecouch.homeip.net>
-----BEGIN xxx SIGNED MESSAGE-----
Hash: SHA1
Nina wrote:
| Hi,
|
| I want to be able to move out small pieces of code from one perl file
| into an external file, and just simply include it from the original
| file.
|
| Something that works like #include in C/C++. Mostly it's going to be
| variable initializations or configuration of the program...
|
| How to do this?
|
| thanks!
Hello Nina
The best way to do this is with a "require" statement. See "perldoc -f
require" or
http://www.perldoc.com/perl5.8.0/pod/func/require.html
The above plus a "package" statement can be used to localize your
configuration variables to it's own namespace, which is a clean,
desirable thing to do.
Best of luck.
-----BEGIN xxx SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQE+FdkPeS99pGMif6wRAmoZAJ9NuksJHJCA+xmQ/PX2tllOnZkDWQCg9s+q
2a9nIaPzyxHo9PRYpSAe5NA=
=b5o3
-----END PGP SIGNATURE-----
------------------------------
Date: 3 Jan 2003 12:03:13 -0800
From: duff@pobox.com (Jonathan Scott Duff)
Subject: Re: vectors & large amounts of data - time & space problems
Message-Id: <9f992f90.0301031203.6a15cd40@posting.google.com>
ctcgag@hotmail.com wrote in message news:<20030103000902.531$NH@newsreader.com>...
> ctcgag@hotmail.com wrote:
> >
> > 100,000 hashes each with 200 key/value pairs
> > or 2e7 key/value pairs. A 6 character hash key is ~65 bytes
> > each, (actually less, because once Perl's seen a certain key, each
> > further use of it only takes ~49 bytes.), a floating point value that
> > is never used in a string context is ~35 bytes, so that comes to about
> > 2 gig. However, a fp value that has been used in a string context
> > is ~95 bytes, which would add up to a total of 3.2 Gig, still
> > comfortably less than the 7 Gig-death you report.
>
> Correction, a 6 character hash key that has already been used (in
> a different hash) uses ~12 bytes, not ~49 bytes, on my machine.
>
> So, assuming only 1e5 distinct words are used as keys in these
> 2e7 hash slots, that drops the memory usage down to 1G.
Also keep in mind the way that perl allocates memory as it grows its
data structures. When perl runs out of space, it allocates twice as
much space. So, for instance, if you were growing an array and perl
initially allocated 8 slots for the elements of the array, then when
you add the ninth element, there will be 16 slots allocated.
I'm just guessing that the same applies for hashes as this is my vague
recollection of how things work (and how I suppose they still work).
-Scott
------------------------------
Date: Fri, 03 Jan 2003 15:41:41 -0500
From: "Eric Anderson" <eric.anderson@cordata.net>
Subject: Re: XS memory management
Message-Id: <pan.2003.01.03.20.41.38.546397@cordata.net>
On Fri, 03 Jan 2003 03:17:41 +0000, Martien Verbruggen wrote:
> You don't mention the name of the project or the function concerned, so
> it's a bit hard for anyone to guess what it does, and how it is wrapped.
The project is AxKit (the XML application server for mod_perl by the
Apache group). You can find it at http://www.axkit.org. There is a
function called axBuildURI which is defined as follows (only if libxml2 is
compiled with it):
char * axBuildURI(pool *p, const char *URI, const char *base)
{
return ap_pstrdup(p, (char *)xmlBuildURI(URI, base));
}
if libxml2 is not compiled with it the function is defined as
char * axBuildURI(pool *p, const char *URI, const char *base)
{
if (URI[0] != '/') {
return ap_pstrdup(p, ap_make_full_path(p, ap_make_dirstr_parent(p, base), URI));
}
else {
return (char*)URI;
}
}
The purpose of the function seem so to be to build a URI when given a base
path and a relative URI. For the most part is is simply wrapping the
function xmlBuildURI from the libxml2 library. I looked up the function in
that library and it says that it is the callers job to free the memory
pointed to by the pointer returned.
But as you can see from the code it simply does a strdup (a special Apache
version). From what I can tell there is no way that the string returned
from xmlBuildURI is ever freed since it is simply copied and never
referenced again.
But before I wrote the authors about that I wanted to make sure I
understood things correctly. Since this is a Perl XS module I wanted to
make sure that memory wasn't different somehow.
If you want to see the code in context look at the following URL:
http://cvs.apache.org/viewcvs.cgi/xml-axkit/AxKit.xs?rev=1.6&content-type=text/vnd.viewcvs-markup
You will notice that there is another function called build_uri that
actually is the one calling this function and from what I can tell is
providing the actual interface that perl code can call.
Anyway, thanks for you input. I'm not really working on this project. I'm
just looking at the code to try to understand Perl XS, mod_perl, and
apache better.
--
Eric Anderson
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
The Perl-Users Digest is a retransmission of the USENET newsgroup
comp.lang.perl.misc. For subscription or unsubscription requests, send
the single line:
subscribe perl-users
or:
unsubscribe perl-users
to almanac@ruby.oce.orst.edu.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
To request back copies (available for a week or so), send your request
to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
where x is the volume number and y is the issue number.
For other requests pertaining to the digest, send mail to
perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
sending perl questions to the -request address, I don't have time to
answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 4339
***************************************
