[28831] in Perl-Users-Digest
Perl-Users Digest, Issue: 75 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Thu Jan 25 18:10:50 2007
Date: Thu, 25 Jan 2007 15:10:13 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Thu, 25 Jan 2007 Volume: 11 Number: 75
Today's topics:
Re: Statistics Extraction <doni.sekar@gmail.com>
Re: Statistics Extraction <doni.sekar@gmail.com>
Re: Statistics Extraction <bik.mido@tiscalinet.it>
Style questions <cbigam@somewhereelse.nucleus.com>
Re: Style questions <spamtrap@dot-app.org>
Tool to execute thousands of commands in parallel <ignoramus26157@NOSPAM.26157.invalid>
Re: Tool to execute thousands of commands in parallel <noreply@gunnar.cc>
Re: Tool to execute thousands of commands in parallel <glex_no-spam@qwest-spam-no.invalid>
Re: Tool to execute thousands of commands in parallel <ignoramus26157@NOSPAM.26157.invalid>
Re: Tool to execute thousands of commands in parallel <bpatton@ti.com>
Re: Using perl modules (NOSPAM)
Re: Using perl modules <bik.mido@tiscalinet.it>
Re: weird perl script <cwilbur@chromatico.net>
Re: weird perl script <sbryce@scottbryce.com>
Re: weird perl script <bik.mido@tiscalinet.it>
Re: weird perl script jcharth@hotmail.com
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: 25 Jan 2007 14:00:25 -0800
From: "doni" <doni.sekar@gmail.com>
Subject: Re: Statistics Extraction
Message-Id: <1169762424.044335.175680@13g2000cwe.googlegroups.com>
On Jan 25, 12:13 am, Josef Moellers
<josef.moell...@fujitsu-siemens.com> wrote:
> It is a common understanding in this group that we don't write code for
> you, but we're willing to comment on any code that you post. Our
> comments may very well contain code e.g. when we want to suggest a
> different approach, as I did in my reply.
> So, play around with the code that was posted and come back if you have
> (unworking) code. Post that code and we're glad to suggest modifications.
> --
Joseph,
Can you take a look at the code below...
MAC, PHY and Network stats gets generated every 30 minutes and are
copied to a file.
I want to extract all the stats information that is copied to a file
into an hash of hashes and print them.
The file has multiple MAC, PHY and Network Statistics in it and I am
extracting the MAC data in a MAC hash of hashes, PHY data in a PHY hash
of hashes and Network data in a Network hash of hashes.
Here is how the data that is represented in a file. There will be
multiple MAC, PHY and Network Statistics in the file but I am showing
only one of them in here as an example.
For example, if there were 10 MAC Statistics in the file, this is how I
want the data from the "frames with invalid header" stats message to be
copied to a hash.
$mac{'frames with invalid header'}{1} = 0
$mac{'frames with invalid header'}{2} = 0
....... $mac{'frames with invalid header'}{10} = 0
Here is the program I wrote to do the above operation but its not
working as expected. Can anyone let me know where I went wrong.
Thanks,
doni
### Open the netstat file copied to this location ###
my $ex_data = 'netstat.log';
open (NETSTAT,$ex_data) || die("Cannot Open File: $!");
my %mac = (); # Will be the HOH for MAC Statistics
my %phy = (); # Will be the HoH for PHY Statistics
my %bbu = (); # Will be the HoH for BBU Statistics
my %dli = (); # Will be the HoH for DLI Statistics
my $stats; # State Variable
my $i = 0; # Count variable for MAC
my $j = 0; # Count Variable for PHY
my $a = 0; # Count Variable for Network
my $x = 0; # Count Varialbe for Hash
while (<NETSTAT>) {
chomp;
if (/^(\S+)\s+statistics:/) {
$stats = $1;
if ($stats =~ /mac/i)
{ $i++; }
elsif ($stats =~ /phy/i)
{ $j++; }
else
{ $a++; }
}
elsif (/^\s+(\d+)\s+(\S.*)/)
{
($value, $key) = ($1, $2);
if ($stats =~ /mac/i) {
next unless defined $stats;
$mac{$key}{$i} = $value;
}
if ($stats =~ /phy/i) {
next unless defined $stats;
$phy{$key}{$j} = $value;
}
if ($stats =~ /network/i) {
next unless defined $stats;
$network{$key}{$a} = $value;
}
}
else { }
}
foreach $key (keys (%mac)) {
foreach $x (keys %{$mac{$key} })
{
while ($x <= $i) {
print "Message Value is: $mac{$key}{$i}\n";
$i++;
}
}
}
close(NETSTAT) || die("Cannot close $ex_data: $!");
------------------------------
Date: 25 Jan 2007 14:09:49 -0800
From: "doni" <doni.sekar@gmail.com>
Subject: Re: Statistics Extraction
Message-Id: <1169762989.773817.145480@h3g2000cwc.googlegroups.com>
On Jan 25, 2:00 pm, "doni" <doni.se...@gmail.com> wrote:
> }foreach $key (keys (%mac)) {
> foreach $x (keys %{$mac{$key} })
> {
> while ($x <= $i) {
> print "Message Value is: $mac{$key}{$i}\n";
> $i++;
IT IS $x++ and not $i++. I changed that mistake in the
code but still it doesnt work...
> }
>
> }
>
> }close(NETSTAT) || die("Cannot close $ex_data: $!");
------------------------------
Date: Thu, 25 Jan 2007 23:40:32 +0100
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: Statistics Extraction
Message-Id: <nkbir2pf8rnaq48584dncptft85grl9947@4ax.com>
On 25 Jan 2007 14:00:25 -0800, "doni" <doni.sekar@gmail.com> wrote:
>The file has multiple MAC, PHY and Network Statistics in it and I am
>extracting the MAC data in a MAC hash of hashes, PHY data in a PHY hash
>of hashes and Network data in a Network hash of hashes.
Why don't you use a %stat HoHoH where the primary keys are qw/mac phy
network/? That will make your life easier. Thus once you have $kind
holding one of them in your code you will do:
$stat{$kind}{$key}[ $count{$kind}++ ] = $value;
where the single %count hash replaces the three or four different
counters you were holding. Well, actually I'm suggesting a HoHoA
because although I *didn't* look in detail into your code I noticed
that in your original HoH's the secondary key was an integer that you
incremented along the way, and then an array would be perhaps better
fit in this sense. BTW, you can also use save yourself the need to
hold counters yourself and use autovivification like thus:
push @{ $stat{$kind}{$key} }, $value;
HTH,
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: Thu, 25 Jan 2007 22:48:18 GMT
From: "Colin B." <cbigam@somewhereelse.nucleus.com>
Subject: Style questions
Message-Id: <45b93355@news.nucleus.com>
Hey all;
I'm not a programmer, and never have been. I can grind out a decent shell
script, and beat my head against perl until things work, but I'm seldom
happy with the inelegance of the solution. Does anyone have any good style
guides for perl?
And more specifically, I've got the following general case that keeps coming
up routinely.
foreach (`some_unix_command_that_puts_out_multiple_lines`) {
chomp;
split;
(do_stuff_on_the_separate_fields);
}
Anything to fix up there?
Thanks,
Colin
------------------------------
Date: Thu, 25 Jan 2007 17:52:17 -0500
From: Sherm Pendley <spamtrap@dot-app.org>
Subject: Re: Style questions
Message-Id: <m2r6tiu3by.fsf@Sherm-Pendleys-Computer.local>
"Colin B." <cbigam@somewhereelse.nucleus.com> writes:
> Does anyone have any good style
> guides for perl?
Everyone who has Perl has one:
perldoc perlstyle
sherm--
--
Web Hosting by West Virginians, for West Virginians: http://wv-www.net
Cocoa programming in Perl: http://camelbones.sourceforge.net
------------------------------
Date: Thu, 25 Jan 2007 13:18:52 -0600
From: Ignoramus26157 <ignoramus26157@NOSPAM.26157.invalid>
Subject: Tool to execute thousands of commands in parallel
Message-Id: <lL2dnXwZF6sBnyTYnZ2dnUVZ_rSjnZ2d@giganews.com>
I am writing some perl scripts right now that execute certain
commands. Though the scripts work, they take a while each to execute
and I am consuming a lot of computer time just waiting.
Each command can be either done with perl (using LWP::UserAgent) or in
shell (using wget). Right now I am using the former.
What I would like to do is to be able to run them in parallel, but no
more than N at a time. Typical would be 20,000 commands, run no more
than 4-5 at a time.
So. Is there some sort of an engine, either as a perl module, or a
standalone execution system, that can be fed with commands and execute
them in parallel, making sure that the maximum is not exceeded.
One solution is to generate giant makefiles and run make -j 5, but I
am afraid that make may choke on that many targets.
My perl is perl, v5.8.5 built for i386-linux-thread-multi
Any other solutions? Thanks.
i
------------------------------
Date: Thu, 25 Jan 2007 20:23:40 +0100
From: Gunnar Hjalmarsson <noreply@gunnar.cc>
Subject: Re: Tool to execute thousands of commands in parallel
Message-Id: <51sehqF1m43ijU1@mid.individual.net>
Ignoramus26157 wrote:
> What I would like to do is to be able to run them in parallel, but no
> more than N at a time.
Check out the CPAN module Parallel::ForkManager.
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
------------------------------
Date: Thu, 25 Jan 2007 13:40:37 -0600
From: "J. Gleixner" <glex_no-spam@qwest-spam-no.invalid>
Subject: Re: Tool to execute thousands of commands in parallel
Message-Id: <45b9073a$0$61929$815e3792@news.qwest.net>
Ignoramus26157 wrote:
> I am writing some perl scripts right now that execute certain
> commands. Though the scripts work, they take a while each to execute
> and I am consuming a lot of computer time just waiting.
>
> Each command can be either done with perl (using LWP::UserAgent) or in
> shell (using wget). Right now I am using the former.
[...]
LWP::Parallel::UserAgent
Here's an article from Randal showing how to check bookmarks in parallel:
http://www.stonehenge.com/merlyn/UnixReview/col56.html
------------------------------
Date: Thu, 25 Jan 2007 14:32:01 -0600
From: Ignoramus26157 <ignoramus26157@NOSPAM.26157.invalid>
Subject: Re: Tool to execute thousands of commands in parallel
Message-Id: <Tc-dnSwLBodcjiTYnZ2dnUVZ_uLinZ2d@giganews.com>
On Thu, 25 Jan 2007 20:23:40 +0100, Gunnar Hjalmarsson <noreply@gunnar.cc> wrote:
> Ignoramus26157 wrote:
>> What I would like to do is to be able to run them in parallel, but no
>> more than N at a time.
>
> Check out the CPAN module Parallel::ForkManager.
>
Gunnar, this is perfect for my needs. Thank you. You solved my
problem.
igor
------------------------------
Date: Thu, 25 Jan 2007 14:59:31 -0600
From: Billy Patton <bpatton@ti.com>
Subject: Re: Tool to execute thousands of commands in parallel
Message-Id: <epb5nj$om7$1@home.itg.ti.com>
Ignoramus26157 wrote:
> I am writing some perl scripts right now that execute certain
> commands. Though the scripts work, they take a while each to execute
> and I am consuming a lot of computer time just waiting.
>
> Each command can be either done with perl (using LWP::UserAgent) or in
> shell (using wget). Right now I am using the former.
>
> What I would like to do is to be able to run them in parallel, but no
> more than N at a time. Typical would be 20,000 commands, run no more
> than 4-5 at a time.
>
> So. Is there some sort of an engine, either as a perl module, or a
> standalone execution system, that can be fed with commands and execute
> them in parallel, making sure that the maximum is not exceeded.
>
> One solution is to generate giant makefiles and run make -j 5, but I
> am afraid that make may choke on that many targets.
>
> My perl is perl, v5.8.5 built for i386-linux-thread-multi
>
> Any other solutions? Thanks.
>
> i
If your running that many simple commands from inside your perl script,
you might consider using a makefile
giving make
make -j --max-load 1.0
--max-load 5.0 for a 5 board system.
What this does is to load the system up to max. It doesn't stop at just
5 jobs. If you system can handle 20 it will load it up.
------------------------------
Date: Thu, 25 Jan 2007 10:04:02 -0600
From: "Mumia W. (NOSPAM)" <paduille.4060.mumia.w+nospam@earthlink.net>
Subject: Re: Using perl modules
Message-Id: <epalus$dk1$1@aioe.org>
On 01/25/2007 08:56 AM, LHradowy wrote:
> I am having difficulty using perl modules, I know once I start and
> figure this out, it would make my life simplier.
>
> My module is used for logging into a database.
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
> package login_acdb;
>
> my $file = "/etc/accesscare/$ENV{'ORACLE_SID'}/voice/ac.login";
> open LOGINID, "< $file" or die "Could not open $file: $!";
> my $oraLoginId = <LOGINID>;
"My" creates a variable that is constrained to the current block or
file. If you want $orgLoginId to be accessible to other parts of the
program, declare it with "our":
our $oraLoginId = <LOGINID>;
After you're finished reading the login id, you can probably close the file.
> chomp $oraLoginId;
> 1;
>
> The file looks like: oracle/oracle321
>
Huh?
> I have added it to my PERL5LIB environment so it is in the @INC path.
>
> But now I am having problems with how to call it.
>
> Here is where I call it in my script:
> sub doSqlSelect {
> my @buffer = qx { sqlplus -S $login_acdb::oraLoginId <<EOF;
> set heading off
> set pagesize 0
> set feedback off
> SELECT feature||','||description
> FROM vt_feature
> ORDER by 1;
> EOF
> };
> [...]
The commands probably need to be within quotes, and I would create the
commands string separately:
sub doSqlSelect {
my $sql = <<' EOF';
set heading off
set pagesize 0
set feedback off
SELECT feature||','||description
FROM vt_feature
ORDER by 1;
EOF
$sql =~ s/(["\$])/\\$1/g;
my @buffer = qx { echo "$login_acdb::oraLoginId" "$sql" };
}
The indentation helps keeps things more readable, and the quotes around
$sql are needed to prevent the shell from misinterpreting the contents.
Since you are passing a lot of data through the shell, you must be very
careful about properly quoting and escaping your data, or your program
will be exploitable to hackers who want to get shell access on your machine.
I doubt that my little substitution above is adequate, but I hope you
get the idea.
--
Windows Vista and your freedom in conflict:
http://www.securityfocus.com/columnists/420/2
------------------------------
Date: Thu, 25 Jan 2007 19:09:48 +0100
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: Using perl modules
Message-Id: <edshr21ev4sf7pgpifpjclte432iag305c@4ax.com>
On 25 Jan 2007 15:43:34 GMT, anno4000@radom.zrz.tu-berlin.de wrote:
>> My module is used for logging into a database.
>> #!/usr/bin/perl
>
>A module normally doesn't need a shebang line.
In fact, I usually put
# -*- Perl -*-
at the top, so that my editor recognizes it and enters in Perl mode.
Well, actually I've also set up an association for .pm so it's not
really necessary in this sense, but just like the shebang line is much
like a 'hello' in actual scripts, I feel at ease with it as a 'hello'
for modules.
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: 25 Jan 2007 11:09:51 -0500
From: Charlton Wilbur <cwilbur@chromatico.net>
Subject: Re: weird perl script
Message-Id: <871wljxf3k.fsf@mithril.chromatico.net>
>>>>> "A" == Andy <anedza@infotek-consulting.com> writes:
A> Some PERL modules that furnish email functionality are actually
The language is Perl; the executable is perl. It's a shibboleth, but
one that correlates very well with correct advice and attention to detail.
A> dependant upon other existing non-PERL email mediating
A> facilities to be present that are O/S resident. sendmail is an
A> example of that - its available only on UNIX and not windows
A> platforms.
Googling for "sendmail windows" shows that this is simply not correct.
Sendmail is frequently installed on Unix platforms by default, and
must be installed seaparately on Windows.
A> sendmail is not itself an email server; it only
A> formats email headers and messages which it forwards to an SMTP
A> server.
The word you are fumbling for here is mail transfer agent, or MTA.
Sendmail is, in fact, a fully-functional SMTP server.
A> An email PERL module (such as MAIL::xxxx) that relies
A> on sendmail would only pass parameters (such as recipient email
A> address) to sendmail which would then format these into the
A> commands of the email protocol used to send or receive email.
Many Perl modules, including those in the Mail:: hierarchy (note: Perl
is case-sensitive, so MAIL::Foo and Mail::Foo are different modules)
can either use the sendmail binary or an SMTP server. Exactly which
each module supports is discussed in the documentation for the module,
but a number of them support both.
A> Another thing to consider is whether your authentication is
A> passing the security layer for your POP3 server. Because you
A> experience a "hang", its likely the email PERL modules you are
A> using are searching for an email mediator that isn't on the
A> platform you are running on. If it was an authentication
A> problem, you would probably receive an unauthorized message of
A> some kind right away.
"an email mediator"?
If the script works as intended on one platform but hangs on another,
the most likely problem is a network timeout. The best solution is to
examine the data that are actually being sent, possibly with a network
sniffer; alternately, to mimic the process manually using command-line
tools, as if it's a network configuration problem (such as network
traffic not being allowed in or out on a certain port) it will show up
immediately under those circumstances.
Charlton
--
Charlton Wilbur
cwilbur@chromatico.net
------------------------------
Date: Thu, 25 Jan 2007 09:29:33 -0700
From: Scott Bryce <sbryce@scottbryce.com>
Subject: Re: weird perl script
Message-Id: <WvidnXqMwa5xRyXYnZ2dnUVZ_oipnZ2d@comcast.com>
Michele Dondi wrote:
> I suppose you
> dynamically render a page with perl. The rendered page contains some
> Javascript code.
Which could explain why the script works on some platforms and not
others. The OP should consider whether the JavaScript causing the problem.
------------------------------
Date: Thu, 25 Jan 2007 19:34:08 +0100
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: weird perl script
Message-Id: <1tthr2prvmcp86ploa4ets876odj8hlit8@4ax.com>
On Thu, 25 Jan 2007 16:51:14 +0100, Michele Dondi
<bik.mido@tiscalinet.it> wrote:
>so one spots the problem in the first place. Incidentally, what is a
>"java script in the perl script" supposed to be? I suppose you
>dinamically render a page with perl. The rendered page contains some
>Javascript code. Perl is not concerned with the latter any more...
Incidentally (and on an OT basis) Javascript is a language that bears
very few relations to a "Java script". In fact up until a few days
before its official release it was planned to be called Livescript, it
was changed as a marketing trick to exploit Java's popularity at that
time. But they are otherwise mostly unrelated languages.
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: 25 Jan 2007 11:18:17 -0800
From: jcharth@hotmail.com
Subject: Re: weird perl script
Message-Id: <1169752697.324331.242380@l53g2000cwa.googlegroups.com>
Thanks. I think i found the problem i am using cgi and everytime the
parameter was blank, things got messed up so i put the parameter in an
if statement before assing the value to a variable.
On Jan 25, 1:34 pm, Michele Dondi <bik.m...@tiscalinet.it> wrote:
> On Thu, 25 Jan 2007 16:51:14 +0100, Michele Dondi
>
> <bik.m...@tiscalinet.it> wrote:
> >so one spots the problem in the first place. Incidentally, what is a
> >"java script in the perl script" supposed to be? I suppose you
> >dinamically render a page with perl. The rendered page contains some
> >Javascript code. Perl is not concerned with the latter any more...Incidentally (and on an OT basis) Javascript is a language that bears
> very few relations to a "Java script". In fact up until a few days
> before its official release it was planned to be called Livescript, it
> was changed as a marketing trick to exploit Java's popularity at that
> time. But they are otherwise mostly unrelated languages.
>
> Michele
> --
> {$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
> (($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
> .'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
> 256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 75
*************************************
