[28928] in Perl-Users-Digest
Perl-Users Digest, Issue: 172 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Tue Feb 27 00:18:59 2007
Date: Mon, 26 Feb 2007 21:19:29 -0800 (PST)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Mon, 26 Feb 2007 Volume: 11 Number: 172
Today's topics:
here is the famous godwin script, donated to the usenet <anonymous@panta-rhei.eu.org>
Re: here is the famous godwin script, donated to the us <bik.mido@tiscalinet.it>
Re: How to boost performance of my crude script? <poo@pee.com>
Re: How to boost performance of my crude script? <1usa@llenroc.ude.invalid>
Re: How to boost performance of my crude script? <poo@pee.com>
Re: How to boost performance of my crude script? <wahab-mail@gmx.de>
Re: How to boost performance of my crude script? <1usa@llenroc.ude.invalid>
Re: How to boost performance of my crude script? <wahab-mail@gmx.de>
Re: How to boost performance of my crude script? <1usa@llenroc.ude.invalid>
Re: How to boost performance of my crude script? <poo@pee.com>
Net::SSH::Perl - Broken Pipe azzi.george@gmail.com
Net::SSH::Perl install Hangs <randall.belk@gmail.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: 25 Feb 2007 04:18:47 -0000
From: Anonymous via Panta Rhei <anonymous@panta-rhei.eu.org>
Subject: here is the famous godwin script, donated to the usenet populous!
Message-Id: <WI4BHGN539138.2213773148@anonymous.poster>
#!/usr/bin/perl
require 5;
$VERSION = '1.0';
use File::Slurp;
use warnings;
use Getopt::Std;
use List::Util ('shuffle');
use News::Article;
use strict 'refs';
my(%option, $inputFilename, $outputFilename, @addBody, $code);
my $ctr = 0;
my(@dropGroups) = ();
my(@followups) = ();
# generate messages to enforce Godwin's law
# example:
# $ grep -r -l -i '\(nazi\|hitler\)' /var/spool/news/alt/what/ever |xargs godwin
# By default this script adds Followup-To: alt.dev.null,
# X-No-Archive: Yes, and a X-Code header to keep it from replying
# to its own messages.
# This script just generates messages, it's up to you how you get
# them onto usenet. Warning, if you post a lot of these so you
# can be traced, you'll probably get TOSsed!
# -G alt.foo,alt.bar
# drop some Newsgroups
# -N
# do not FU alt.dev.null
# -a
# do not set X-No-Archive: Yes
# -b FILE
# use text from FILE instead of default data
# -c
# suppress X-Code: random header
# -d DIRECTORY
# put output file here instead of pwd
# -f 'Some Name foo@bar'
# use this From header instead of the default
# -u alt.foo,alt.bar
# add FUs
# -L FILE
# ignore files not newer than FILE
# This script is hereby donated to the PUBLIC DOMAIN.
getopts 'd:af:b:u:G:NcL:', \%option;
my $maxAge = $option{L} ? (-M $option{L}) : 0;
my $outputDirectory = $option{'d'} || '.';
my $from = $option{'f'} || 'Godwin <godwin@...>';
if ($option{'G'}) {
@dropGroups = split(/,/, $option{'G'}, 0);
}
push @followups, 'alt.dev.null' unless $option{'N'};
push @followups, split(/,/, $option{'u'}, 0) if $option{'u'};
if ($option{'b'}) {
@addBody = read_file($option{'b'});
}
else {
@addBody = <DATA>;
}
foreach $inputFilename (@ARGV) {
if (($option{L}) && (-M $inputFilename >= $maxAge)) {
print "$inputFilename ignored (too old)\n";
}
else {
my $input = 'News::Article'->new($inputFilename);
if ($input->header('X-Code')) {
print "$inputFilename ignored (X-Code)\n";
}
else {
my $output = 'News::Article'->new;
$output->set_headers('Subject', reSubject($input),
'From', $from,
'References', reRefs($input));
$output->set_headers('X-No-Archive', 'Yes') unless $option{'a'};
unless ($option{'c'}) {
$code = `mcookie`;
chomp $code;
$output->set_headers('X-Code', $code);
}
$output->set_body(reBody($input));
my(@groupings) = reGroup($input);
my $groups;
while (@groupings) {
$outputFilename = getFilename($outputDirectory, $inputFilename);
$groups = shift @groupings;
$output->set_headers('Newsgroups', $groups);
$output->set_headers('Followup-To', fu($groups)) if @followups > 0;
open OUTFILE, ">$outputFilename";
$output->write(\*OUTFILE);
close OUTFILE;
print "$inputFilename --> $outputFilename\n";
}
}
}
}
sub reSubject {
my $subject = $_[0]->header('Subject');
$subject = 'Re: ' . $subject unless $subject =~ /\s*(\[.*\])?\s*Re:/i;
return $subject;
}
sub reRefs {
my(@refs) = $_[0]->header('Message-ID');
my(@oldRefs) = split(/\s+/, $_[0]->header('References') || '', 0);
unshift @refs, pop @oldRefs if @oldRefs;
unshift @refs, shift @oldRefs if @oldRefs;
return join(' ', @refs);
}
sub getFilename {
my $orig = $_[1];
$orig =~ s[^.*/][];
my $fn;
do {
$fn = $_[0] . '/' . $orig . '-' . $$ . $ctr++ . '.output'
} while -f $fn;
return $fn;
}
sub reGroup {
my(@groups) = shuffle(split(/,/, $_[0]->header('Newsgroups'), 0));
if ($option{'G'}) {
@groups = dropGroups(@groups);
}
my(@output) = ();
while (@groups > 3) {
push @output, join(',', @groups[0, 1, 2]);
shift @groups;
shift @groups;
shift @groups;
}
push @output, join(',', @groups);
return @output;
}
sub dropGroups {
my(%groups) = map({$_, 1;} @_);
foreach my $item (@dropGroups) {
delete $groups{$item};
}
return keys %groups;
}
sub fu {
my(@groups) = split(/,/, $_[0], 0);
unshift @groups, @followups;
if (@groups > 3) {
@groups = @groups[0, 1, 2];
}
return join(',', shuffle(@groups));
}
sub reBody {
my(@old) = $_[0]->body;
my(@new) = map({"> $_";} grep({$_ =~ /nazi|hitler/i;} @old));
push @new, '', @addBody;
return @new;
}
__DATA__
This thread is terminated by Godwin's Law.
Further posts are prohibited.
Thank you for co-operating.
~~~~~~~~~~~~~~~~~~~~~
This message was posted via one or more anonymous remailing services.
The original sender is unknown. Any address shown in the From header
is unverified. You need a valid hashcash token to post to groups other
than alt.test and alt.anonymous.messages. Visit www.panta-rhei.eu.org
for abuse and hashcash info.
------------------------------
Date: Sat, 24 Feb 2007 15:19:36 +0100
From: Michele Dondi <bik.mido@tiscalinet.it>
Subject: Re: here is the famous godwin script, donated to the usenet populous!
Message-Id: <l6i0u2hrekmo14j1j1behq0r3mnobnbvgs@4ax.com>
On Sat, 24 Feb 2007 12:40:37 +0100, "john.swilting"
<john.swilting@wanadoo.fr> wrote:
>> $orig =~ s[^.*/][];
[snip]
>not understoud
>$orig = s[^.*/][]
Please do not retype, just copy and paste instead. If you retype, try
to do so accurately
$orig =~ s[^.*/][]; # and
$orig = s[^.*/][] # are VERY different!
>it is an infinite loop
No, it's not. Not infinite, let alone a loop.
Michele
--
{$_=pack'B8'x25,unpack'A8'x32,$a^=sub{pop^pop}->(map substr
(($a||=join'',map--$|x$_,(unpack'w',unpack'u','G^<R<Y]*YB='
.'KYU;*EVH[.FHF2W+#"\Z*5TI/ER<Z`S(G.DZZ9OX0Z')=~/./g)x2,$_,
256),7,249);s/[^\w,]/ /g;$ \=/^J/?$/:"\r";print,redo}#JAPH,
------------------------------
Date: Sat, 24 Feb 2007 09:59:24 -0500
From: DB <poo@pee.com>
Subject: Re: How to boost performance of my crude script?
Message-Id: <8LudnRpjTfhRz33YnZ2dnUVZ_tijnZ2d@adelphia.com>
Thanks for the comments. I did edit the actual code a bit before posting
(trying to avoid line wrapping) and that's why some of the variables are
mis-spelled etc. The code really does work, ugly as it is. I'm sure the
style criticisms are all very valid. I know it's a mess but it works.
I *should* have posted a bare-bones version, which Micro has so nicely
provided:
> open FH1, '<single_large_file.txt';
> while( <FH1> ) {
>
> if( xyz )
> open FH2, '<single_large_file.txt';
> while( <FH2> ) {
> match_something()
> }
> close FH2
> }
> }
> ...
since this removes the distraction of my poor coding practices, allowing
me to focus on my real concern of making things faster. My crude code
runs, but very slowly - as in taking hours to complete on my humble box,
an Athlon 2GHz with 1/2 GB RAM.
The 'xyz' in the code above amounts to having to reparse the text file
one time when a certain condition occurs, and that happens for about 1
in 10 of the 250,000 lines of text (about 12MB).
Given this hardware, would I see more speed benefit from loading the
data into memory as Mirco suggests, or by using a SQL database as Sinan
suggests?
Thanks again for looking.
DB
------------------------------
Date: Sat, 24 Feb 2007 16:59:51 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: How to boost performance of my crude script?
Message-Id: <Xns98E17A117A916asu1cornelledu@127.0.0.1>
DB <poo@pee.com> wrote in
news:8LudnRpjTfhRz33YnZ2dnUVZ_tijnZ2d@adelphia.com:
>> open FH1, '<single_large_file.txt';
>> while( <FH1> ) {
>>
>> if( xyz )
>> open FH2, '<single_large_file.txt';
>> while( <FH2> ) {
>> match_something()
>> }
>> close FH2
>> }
>> }
>> ...
>
...
>
> The 'xyz' in the code above amounts to having to reparse the text file
> one time when a certain condition occurs, and that happens for about 1
> in 10 of the 250,000 lines of text (about 12MB).
>
> Given this hardware, would I see more speed benefit from loading the
> data into memory as Mirco suggests, or by using a SQL database as
> Sinan suggests?
IMHO, you are asking the wrong question. The right question would be
"which one of the potential remedies can I implement faster (as well as
correctly).
Below is a contrived example of two-pass processing. It is contrived
because I don't understand the nature of your processing.
Before running the example, I generated an input file using:
#!/usr/bin/perl
use strict;
use warnings;
my @actions = qw( ADD COPY DELETE MOVE );
my @data = qw( sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf);
for ( 1 ... 500_000 ) {
my $action = $actions[ @actions * rand ];
print join("\t", $action, @data), "\n";
}
__END__
The data file is roughly 30 MB in size and looks like:
C:\DOCUME~1\asu1\LOCALS~1\Temp\1> tail data.txt
DELETE sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
COPY sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
DELETE sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
ADD sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
MOVE sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
MOVE sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
COPY sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
COPY sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
MOVE sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
COPY sdjfh sdjfhskd sdhfjhkui ruyfeih fhfhedu feifieh iehrf
The processing script will collect all the lines that begin with the
same action together. (Obviously, one can do this in just a single pass,
but, as I said, this is a contrived tow-pass example. In fact, making
two passes over the data may not even be needed in your case. I just
don't know).
The two pass version of the script is:
#!/usr/bin/perl
use strict;
use warnings;
my (@data, %output);
while ( <> ) {
chomp;
push @data, [ split /\t/ ];
}
for my $event ( @data ) {
my ($action, @payload) = @{ $event };
push @{ $output{$action} }, \@payload;
}
for my $type ( keys %output ) {
print "$type\n";
for my $item ( @{ $output{$type} } ) {
print "@$item\n";
}
}
__END__
This version takes about 25 seconds. If a single pass version is used:
#!/usr/bin/perl
use strict;
use warnings;
my (%output);
while ( <> ) {
chomp;
my ($action, @payload) = split /\t/;;
push @{ $output{$action} }, \@payload;
}
for my $type ( keys %output ) {
print "$type\n";
for my $item ( @{ $output{$type} } ) {
print "@$item\n";
}
}
__END__
total time falls to 12 seconds.
All timings on a 1Ghz AMD 64 Windows XPSP2 1GB RAM, AS Perl 5.8.8.8210
So, even though these scripts do not correspond to the task you have at
hand, clearly, major speed gains are possible by not re-reading the file
each time you need to re-scan the data. Further, with proper use of Perl
data structures, you can eliminate some if not all passes over the data.
Most of the times above are probably spent in I/O.
If the speed gains from what you consider is the simplest alternative do
not satisfy you, you can then look for other alternatives.
Sinan
--
A. Sinan Unur <1usa@llenroc.ude.invalid>
(remove .invalid and reverse each component for email address)
comp.lang.perl.misc guidelines on the WWW:
http://augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: Sat, 24 Feb 2007 19:20:12 -0500
From: DB <poo@pee.com>
Subject: Re: How to boost performance of my crude script?
Message-Id: <CdydnU2-iMCgS33YnZ2dnUVZ_q6vnZ2d@adelphia.com>
OK, I redid my script, this time first reading the data into an array of
arrays. It doesn't seem any faster than the previous version, so I'll
attempt to explain more clearly what I need the script to do.
My raw data file looks like this, with approx 250,000 lines:
item_number,description,price
item1,screw,1.00
item2,nail,2.00
item3,USE item6,0.00
item4,nut,2.45
item5,rivet,3.25
item6,pin,2.50
item7,USE item9,0.00
item8,wrench,4.45
item9,clip, 3.33
...
I want the output to look like:
item_number,description,price
item1,screw,1.00
item2,nail,2.00
item3,Replaced with item6 - pin,2.50
item4,nut,2.45
item5,rivet,3.25
item6,pin,2.50
item7,Replaced with item9 - clip,3.33
item8,wrench,4.45
item9,clip, 3.33
...
So I first parse through the data line by line, and if the 1st three
characters of the 2nd field are "USE", I then parse back through the
data and replace the remaining fields of that line with the fields from
the replacing item. Hopefully I've explained clearly.
Both versions of my script work, they just work slowly. It seems that
I/O may not have been a bottleneck. Maybe I should try the SQL approach,
or maybe I just have to live with a slow script? Can anyone shed light?
DB
------------------------------
Date: Sun, 25 Feb 2007 01:46:35 +0100
From: Mirco Wahab <wahab-mail@gmx.de>
Subject: Re: How to boost performance of my crude script?
Message-Id: <erqpc8$o5g$1@mlucom4.urz.uni-halle.de>
DB wrote:
> My raw data file looks like this, with approx 250,000 lines:
>
> item_number,description,price
> item1,screw,1.00
> item2,nail,2.00
Why not posting about 50 lines of *original data* +
some comments (clearly separated from the data) - of
what else might be around (what is not covered by
your example).
From your "pseudo example"-example I'd guess
you'd work on a hash which key is the first
word (or a computed expression from the first
two words) in a way that allows you to "look up"
an item if necessary.
For the data you provided, the following
would almost do:
use strict;
use warnings;
my @sequence; # read the file into an array
my $expected_size = 250000;
my %bighash;
keys(%bighash) = $expected_size;
open (my $fh, '<', 'data.txt') or "die without a blink, $!";
# preload and preprocess file
while( <$fh> ) {
chomp; # don't let line endings pass
next unless length; # next line if empty line
my @line = split /,/;
$bighash{ $line[0] } = @sequence; # array index in hash (lookup table)
push @sequence, \@line # save splitted line to array
}
close $fh;
# now resolve USE and write results
for my $line (@sequence) {
if( $line->[1] =~ /USE\s*(\w+)/ ) { # any USE thingy here
my $index = $bighash{ $1 }; # look it up
my @found = @{ $sequence[$index] }[1..2]; # drop field #0
splice @$line, 1, scalar @$line - 1, (@found, "from $line->[1]")
}
# print out resolved line
print "@$line \n";
}
Regards
Mirco
------------------------------
Date: Sun, 25 Feb 2007 01:45:15 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: How to boost performance of my crude script?
Message-Id: <Xns98E1D3254B202asu1cornelledu@127.0.0.1>
DB <poo@pee.com> wrote in
news:CdydnU2-iMCgS33YnZ2dnUVZ_q6vnZ2d@adelphia.com:
> OK, I redid my script, this time first reading the data into an array
> of arrays. It doesn't seem any faster than the previous version, so
> I'll attempt to explain more clearly what I need the script to do.
>
> My raw data file looks like this, with approx 250,000 lines:
>
> item_number,description,price
> item1,screw,1.00
> item2,nail,2.00
> item3,USE item6,0.00
> item4,nut,2.45
> item5,rivet,3.25
> item6,pin,2.50
> item7,USE item9,0.00
> item8,wrench,4.45
> item9,clip, 3.33
> ...
>
> I want the output to look like:
>
> item_number,description,price
> item1,screw,1.00
> item2,nail,2.00
> item3,Replaced with item6 - pin,2.50
> item4,nut,2.45
> item5,rivet,3.25
> item6,pin,2.50
> item7,Replaced with item9 - clip,3.33
> item8,wrench,4.45
> item9,clip, 3.33
You should really read the posting guidelines for this group. We have
wasted some time because you took so long to actually show data.
Anyway, here's a quick and dirty script (I ignored headers and opening
files etc to avoid cluttering the code):
#!/usr/bin/perl
use strict;
use warnings;
my %products;
while ( my $line = <DATA> ) {
chomp $line;
last unless length $line;
my ($item, @data) = split /,/, $line;
$products{$item} = { pos => $., data => \@data };
}
my @items = sort {
$products{$a}->{pos} <=> $products{$b}->{pos}
} keys %products;
for my $i ( @items ) {
if ( $products{$i}->{data}->[0] =~ /^USE\s+(\w+)/ ) {
$products{$i}->{data}->[0] = sprintf(
'Replaced with %s', $products{$1}->{data}->[0] );
}
print join(',', $i, @{ $products{$i}->{data} }), "\n";
}
__DATA__
item1,screw,1.00
item2,nail,2.00
item3,USE item6,0.00
item4,nut,2.45
item5,rivet,3.25
item6,pin,2.50
item7,USE item9,0.00
item8,wrench,4.45
item9,clip, 3.33
C:\Home\asu1\src\1> t
item1,screw,1.00
item2,nail,2.00
item3,Replaced with pin,0.00
item4,nut,2.45
item5,rivet,3.25
item6,pin,2.50
item7,Replaced with clip,0.00
item8,wrench,4.45
item9,clip, 3.33
--
A. Sinan Unur <1usa@llenroc.ude.invalid>
(remove .invalid and reverse each component for email address)
comp.lang.perl.misc guidelines on the WWW:
http://augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
------------------------------
Date: Sun, 25 Feb 2007 01:55:56 +0100
From: Mirco Wahab <wahab-mail@gmx.de>
Subject: Re: How to boost performance of my crude script?
Message-Id: <erqses$p0c$1@mlucom4.urz.uni-halle.de>
Mirco Wahab wrote:
> open (my $fh, '<', 'data.txt') or "die without a blink, $!";
WTF!
must read:
...) or die "without a blink, $!";
> splice @$line, 1, scalar @$line - 1, (@found, "from $line->[1]")
> }
> # print out resolved line
> print "@$line \n";
to match your output convention, this has to
be rewritten as:
...
splice @$line, 1, -1 + @$line, "Replaced with $1 - ". join ',', @found
}
# print out resolved line
print +(join ',', @$line),"\n";
...
This would exactly print the desired output.
Regards
M.
------------------------------
Date: Sun, 25 Feb 2007 14:24:19 GMT
From: "A. Sinan Unur" <1usa@llenroc.ude.invalid>
Subject: Re: How to boost performance of my crude script?
Message-Id: <Xns98E25FA9CAC59asu1cornelledu@127.0.0.1>
"A. Sinan Unur" <1usa@llenroc.ude.invalid> wrote in news:Xns98E1D3254B202asu1cornelledu@127.0.0.1:
> DB <poo@pee.com> wrote in
> news:CdydnU2-iMCgS33YnZ2dnUVZ_q6vnZ2d@adelphia.com:
>
>> OK, I redid my script, this time first reading the data into an array
>> of arrays. It doesn't seem any faster than the previous version, so
>> I'll attempt to explain more clearly what I need the script to do.
>>
>> My raw data file looks like this, with approx 250,000 lines:
>>
>> item_number,description,price
>> item1,screw,1.00
>> item2,nail,2.00
>> item3,USE item6,0.00
>> item4,nut,2.45
>> item5,rivet,3.25
>> item6,pin,2.50
>> item7,USE item9,0.00
>> item8,wrench,4.45
>> item9,clip, 3.33
>> ...
>>
>> I want the output to look like:
>>
>> item_number,description,price
>> item1,screw,1.00
>> item2,nail,2.00
>> item3,Replaced with item6 - pin,2.50
>> item4,nut,2.45
>> item5,rivet,3.25
>> item6,pin,2.50
>> item7,Replaced with item9 - clip,3.33
>> item8,wrench,4.45
>> item9,clip, 3.33
>
...
> Anyway, here's a quick and dirty script (I ignored headers and opening
> files etc to avoid cluttering the code):
Well, here is a another version which is slightly less cluttered and
actually produces the output you want:
#!/usr/bin/perl
use strict;
use warnings;
my %products;
while ( my $line = <DATA> ) {
chomp $line;
last unless length $line;
my ($item, @data) = split /,/, $line;
s/^\s+// for @data;
s/\s+$// for @data;
$products{$item} = { pos => $., data => \@data };
}
my @items = sort {
$products{$a}->{pos} <=> $products{$b}->{pos}
} keys %products;
for my $x ( @items ) {
my $ref = \$products{$x}->{data}->[0];
$$ref =~ s{^USE\s+(\w+)}
{sprintf('Replaced with %s - %s', $1, $products{$1}->{data}->[0])}e;
print join(',', $x, @{ $products{$x}->{data} }), "\n";
}
__DATA__
item1,screw,1.00
item2,nail,2.00
item3,USE item6,0.00
item4,nut,2.45
item5,rivet,3.25
item6,pin,2.50
item7,USE item9,0.00
item8,wrench,4.45
item9,clip, 3.33
C:\DOCUME~1\asu1\LOCALS~1\Temp> t
item1,screw,1.00
item2,nail,2.00
item3,Replaced with item6 - pin,0.00
item4,nut,2.45
item5,rivet,3.25
item6,pin,2.50
item7,Replaced with item9 - clip,0.00
item8,wrench,4.45
item9,clip,3.33
------------------------------
Date: Sun, 25 Feb 2007 13:32:47 -0500
From: DB <poo@pee.com>
Subject: Re: How to boost performance of my crude script?
Message-Id: <9aGdnZdaHP7NS3zYnZ2dnUVZ_ozinZ2d@adelphia.com>
Thanks for your patience. I was able to hack together a functioning
script base on Sinan's and Mirco's "create a hash indexed by item
numbers" method.
Needless to say, my version isn't nearly as elegant or pretty as their
versions, but I understand it, it works, and it works in seconds instead
of hours :)
DB
------------------------------
Date: 26 Feb 2007 19:02:15 -0800
From: azzi.george@gmail.com
Subject: Net::SSH::Perl - Broken Pipe
Message-Id: <1172545334.925089.327100@a75g2000cwd.googlegroups.com>
Dear Perl Programmers,
Good evening. Am trying to use Net::SSH::Perl to do a simple ls -l
command on a remote system. However, it keeps giving a Broken Pipe
error. Below is the script, along with the error.
Also, it is taking unusually long to just login. Confirmed there is a
connection using ps -ef on the remote machine. Any help given would be
greatly appreciated. Note that this is running on solaris 10 (not
sure if that makes a difference). Thank you in advance for your help.
Pierre
Perl Script:
-------------------------------------
#!/usr/bin/perl
use warnings;
use strict;
use Net::SSH::Perl;
my $user;
my $pass;
my $cmd = "ls -lrt /tmp/*.txt";
my $pwd = "ssh_info.txt";
my $host = "x.x.x.x";
open ('LOGIN', "$pwd") || die "$!\n";
while (<LOGIN>) {
($user, $pass) = split /:/, $_;
}
close 'LOGIN' || die "LOGIN FH: $! \n";
my $ssh = Net::SSH::Perl->new($host, debug =>1, protocol =>'2,1');
print "$host\n";
print "Command that will be run before logging in: $cmd\n";
$ssh->config->set('interactive', 1)
unless defined $ssh->config->get('interactive');
$ssh->login($user,$pass);
print "Command that is about to be run: $cmd\n";
my ($stdout, $stderr, $exit) = $ssh->cmd($cmd);
print "$tdout\n";
Error Received
-------------------------------------------------------------------
Reading configuration data /var/home/collect/.ssh/config
Reading configuration data /etc/ssh_config
Connecting to x.x.x.x, port 22.
Remote version string: SSH-2.0-OpenSSH_4.3
Remote protocol version 2.0, remote software version OpenSSH_4.3
Net::SSH::Perl Version 1.30, protocol version 2.0.
No compat match: OpenSSH_4.3.
Connection established.
x.x.x.x
Command that will be run before logging in: ls -lrt /tmp/*.txt
Sent key-exchange init (KEXINIT), wait response.
Algorithms, c->s: 3des-cbc hmac-sha1 none
Algorithms, s->c: 3des-cbc hmac-sha1 none
Entering Diffie-Hellman Group 1 key exchange.
Sent DH public key, waiting for reply.
Received host key, type 'ssh-dss'.
Host 'x.x.x.x' is known and matches the host key.
Computing shared secret key.
Verifying server signature.
Waiting for NEWKEYS message.
Enabling incoming encryption/MAC/compression.
Send NEWKEYS, enable outgoing encryption/MAC/compression.
Sending request for user-authentication service.
Broken Pipe
------------------------------
Date: 24 Feb 2007 18:55:22 -0800
From: "Randall" <randall.belk@gmail.com>
Subject: Net::SSH::Perl install Hangs
Message-Id: <1172372122.629277.137520@v33g2000cwv.googlegroups.com>
I'm trying to install Net::SSH::Perl on a FC5 box. The install goes
fine and until it gets to t/03-packet......ok 1/10
Then it just hangs. Any help would be appreciated. I have no idea
how to proceed.
Randall
cpan> install Net::SSH::Perl
CPAN: Storable loaded ok
Going to read /root/.cpan/Metadata
Database was generated on Fri, 23 Feb 2007 16:08:33 GMT
Running install for module Net::SSH::Perl
Running make for D/DB/DBROBINS/Net-SSH-Perl-1.30.tar.gz
CPAN: Digest::MD5 loaded ok
CPAN: Compress::Zlib loaded ok
Checksum for /root/.cpan/sources/authors/id/D/DB/DBROBINS/Net-SSH-
Perl-1.30.tar.gz ok
Scanning cache /root/.cpan/build for sizes
Net-SSH-Perl-1.30/
Net-SSH-Perl-1.30/lib/
Net-SSH-Perl-1.30/lib/Net/
Net-SSH-Perl-1.30/lib/Net/SSH/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth/KeyboardInt.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth/Rhosts.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth/Password.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth/PublicKey.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth/RSA.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth/Rhosts_RSA.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth/ChallengeResponse.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Agent.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util/RSA.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util/Authfile.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util/SSH1MP.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util/Hosts.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util/SSH1Misc.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util/Term.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util/SSH2MP.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher/IDEA.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher/CFB.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher/Blowfish.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher/RC4.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher/CBC.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher/DES3.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher/DES.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Handle/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Handle/SSH2.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Handle/SSH1.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Comp/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Comp/Zlib.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Mac.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/SSH2.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Cipher.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Config.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Util.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Key/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Key/RSA.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Key/RSA1.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Key/DSA.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/AuthMgr.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Auth.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Comp.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Kex.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/ChannelMgr.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Kex/
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Kex/DH1.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Key.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Buffer.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Constants.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Channel.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/SSH1.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl/Packet.pm
Net-SSH-Perl-1.30/lib/Net/SSH/Perl.pm
Net-SSH-Perl-1.30/t/
Net-SSH-Perl-1.30/t/06-circular.t
Net-SSH-Perl-1.30/t/04-config.t
Net-SSH-Perl-1.30/t/config
Net-SSH-Perl-1.30/t/03-packet.t
Net-SSH-Perl-1.30/t/05-cipher.t
Net-SSH-Perl-1.30/t/01-compile.t
Net-SSH-Perl-1.30/t/02-buffer.t
Net-SSH-Perl-1.30/t/test-common.pl
Net-SSH-Perl-1.30/Changes
Net-SSH-Perl-1.30/eg/
Net-SSH-Perl-1.30/eg/pssh
Net-SSH-Perl-1.30/eg/cmd.pl
Net-SSH-Perl-1.30/eg/pssh-keygen
Net-SSH-Perl-1.30/eg/pscp
Net-SSH-Perl-1.30/eg/remoteinteract2.pl
Net-SSH-Perl-1.30/eg/remoteinteract.pl
Net-SSH-Perl-1.30/MANIFEST
Net-SSH-Perl-1.30/README
Net-SSH-Perl-1.30/LICENSE
Net-SSH-Perl-1.30/META.yml
Net-SSH-Perl-1.30/MANIFEST.SKIP
Net-SSH-Perl-1.30/ToDo
Net-SSH-Perl-1.30/Makefile.PL
Removing previously used /root/.cpan/build/Net-SSH-Perl-1.30
CPAN.pm: Going to build D/DB/DBROBINS/Net-SSH-Perl-1.30.tar.gz
This is Net::SSH::Perl.
As of version 1.00, Net::SSH::Perl supports both the SSH1 and
SSH2 protocols natively. The two protocols have different
module prerequisitives, so you need to decide which protocol(s)
you plan to use. If you use one or the other, only those modules
for your chosen protocol will be installed; if you choose both,
all of the supporting modules will be installed. Please choose
the protocols you'd like to use from the following list ("Both"
is the default).
[1] SSH1
[2] SSH2
[3] Both SSH1 and SSH2
Which protocol(s) do you plan to use? [3] 3
Some of the Net::SSH::Perl ciphers depend on a Crypt:: module from
CPAN. You may already have the necessary modules installed, in which
case you don't need to bother with this step. Otherwise you'll need
to install at least one cipher to use Net::SSH::Perl. Please choose
at least one from the following list (Crypt::IDEA is the default).
[1] IDEA
[2] DES
[3] DES3
[4] Blowfish
[5] RC4
Enter your choices, separated by spaces: [1] 1
Checking for optional modules
Checking if your kit is complete...
Looks good
Writing Makefile for Net::SSH::Perl
cp lib/Net/SSH/Perl/Agent.pm blib/lib/Net/SSH/Perl/Agent.pm
cp lib/Net/SSH/Perl/Auth/KeyboardInt.pm blib/lib/Net/SSH/Perl/Auth/
KeyboardInt.pm
cp lib/Net/SSH/Perl/Util/RSA.pm blib/lib/Net/SSH/Perl/Util/RSA.pm
cp lib/Net/SSH/Perl/AuthMgr.pm blib/lib/Net/SSH/Perl/AuthMgr.pm
cp lib/Net/SSH/Perl/Auth/Rhosts.pm blib/lib/Net/SSH/Perl/Auth/
Rhosts.pm
cp lib/Net/SSH/Perl/Cipher/IDEA.pm blib/lib/Net/SSH/Perl/Cipher/
IDEA.pm
cp lib/Net/SSH/Perl/Util/Authfile.pm blib/lib/Net/SSH/Perl/Util/
Authfile.pm
cp lib/Net/SSH/Perl/Handle/SSH2.pm blib/lib/Net/SSH/Perl/Handle/
SSH2.pm
cp lib/Net/SSH/Perl/Util/SSH1MP.pm blib/lib/Net/SSH/Perl/Util/
SSH1MP.pm
cp lib/Net/SSH/Perl/Util/Hosts.pm blib/lib/Net/SSH/Perl/Util/Hosts.pm
cp lib/Net/SSH/Perl/Auth.pm blib/lib/Net/SSH/Perl/Auth.pm
cp lib/Net/SSH/Perl/Util/Term.pm blib/lib/Net/SSH/Perl/Util/Term.pm
cp lib/Net/SSH/Perl/Cipher/CBC.pm blib/lib/Net/SSH/Perl/Cipher/CBC.pm
cp lib/Net/SSH/Perl/Handle/SSH1.pm blib/lib/Net/SSH/Perl/Handle/
SSH1.pm
cp lib/Net/SSH/Perl/Comp.pm blib/lib/Net/SSH/Perl/Comp.pm
cp lib/Net/SSH/Perl/Auth/PublicKey.pm blib/lib/Net/SSH/Perl/Auth/
PublicKey.pm
cp lib/Net/SSH/Perl/Kex.pm blib/lib/Net/SSH/Perl/Kex.pm
cp lib/Net/SSH/Perl/ChannelMgr.pm blib/lib/Net/SSH/Perl/ChannelMgr.pm
cp lib/Net/SSH/Perl/Cipher/CFB.pm blib/lib/Net/SSH/Perl/Cipher/CFB.pm
cp lib/Net/SSH/Perl/Auth/RSA.pm blib/lib/Net/SSH/Perl/Auth/RSA.pm
cp lib/Net/SSH/Perl/Util/SSH1Misc.pm blib/lib/Net/SSH/Perl/Util/
SSH1Misc.pm
cp lib/Net/SSH/Perl/Comp/Zlib.pm blib/lib/Net/SSH/Perl/Comp/Zlib.pm
cp lib/Net/SSH/Perl/Mac.pm blib/lib/Net/SSH/Perl/Mac.pm
cp lib/Net/SSH/Perl/Kex/DH1.pm blib/lib/Net/SSH/Perl/Kex/DH1.pm
cp lib/Net/SSH/Perl/SSH2.pm blib/lib/Net/SSH/Perl/SSH2.pm
cp lib/Net/SSH/Perl/Auth/Password.pm blib/lib/Net/SSH/Perl/Auth/
Password.pm
cp lib/Net/SSH/Perl/Auth/Rhosts_RSA.pm blib/lib/Net/SSH/Perl/Auth/
Rhosts_RSA.pm
cp lib/Net/SSH/Perl/Cipher/Blowfish.pm blib/lib/Net/SSH/Perl/Cipher/
Blowfish.pm
cp lib/Net/SSH/Perl/Key/RSA1.pm blib/lib/Net/SSH/Perl/Key/RSA1.pm
cp lib/Net/SSH/Perl/Key.pm blib/lib/Net/SSH/Perl/Key.pm
cp lib/Net/SSH/Perl/Buffer.pm blib/lib/Net/SSH/Perl/Buffer.pm
cp lib/Net/SSH/Perl/Key/DSA.pm blib/lib/Net/SSH/Perl/Key/DSA.pm
cp lib/Net/SSH/Perl/Cipher/DES3.pm blib/lib/Net/SSH/Perl/Cipher/
DES3.pm
cp lib/Net/SSH/Perl/Cipher/RC4.pm blib/lib/Net/SSH/Perl/Cipher/RC4.pm
cp lib/Net/SSH/Perl/Cipher.pm blib/lib/Net/SSH/Perl/Cipher.pm
cp lib/Net/SSH/Perl/Config.pm blib/lib/Net/SSH/Perl/Config.pm
cp lib/Net/SSH/Perl/Constants.pm blib/lib/Net/SSH/Perl/Constants.pm
cp lib/Net/SSH/Perl/Util.pm blib/lib/Net/SSH/Perl/Util.pm
cp lib/Net/SSH/Perl/Util/SSH2MP.pm blib/lib/Net/SSH/Perl/Util/
SSH2MP.pm
cp lib/Net/SSH/Perl.pm blib/lib/Net/SSH/Perl.pm
cp lib/Net/SSH/Perl/Auth/ChallengeResponse.pm blib/lib/Net/SSH/Perl/
Auth/ChallengeResponse.pm
cp lib/Net/SSH/Perl/Channel.pm blib/lib/Net/SSH/Perl/Channel.pm
cp lib/Net/SSH/Perl/SSH1.pm blib/lib/Net/SSH/Perl/SSH1.pm
cp lib/Net/SSH/Perl/Cipher/DES.pm blib/lib/Net/SSH/Perl/Cipher/DES.pm
cp lib/Net/SSH/Perl/Key/RSA.pm blib/lib/Net/SSH/Perl/Key/RSA.pm
cp lib/Net/SSH/Perl/Packet.pm blib/lib/Net/SSH/Perl/Packet.pm
Manifying blib/man3/Net::SSH::Perl::Auth::KeyboardInt.3pm
Manifying blib/man3/Net::SSH::Perl::Agent.3pm
Manifying blib/man3/Net::SSH::Perl::Auth::Rhosts.3pm
Manifying blib/man3/Net::SSH::Perl::AuthMgr.3pm
Manifying blib/man3/Net::SSH::Perl::Cipher::IDEA.3pm
Manifying blib/man3/Net::SSH::Perl::Auth.3pm
Manifying blib/man3/Net::SSH::Perl::Cipher::CBC.3pm
Manifying blib/man3/Net::SSH::Perl::Comp.3pm
Manifying blib/man3/Net::SSH::Perl::Auth::PublicKey.3pm
Manifying blib/man3/Net::SSH::Perl::Kex.3pm
Manifying blib/man3/Net::SSH::Perl::ChannelMgr.3pm
Manifying blib/man3/Net::SSH::Perl::Auth::RSA.3pm
Manifying blib/man3/Net::SSH::Perl::Cipher::CFB.3pm
Manifying blib/man3/Net::SSH::Perl::Comp::Zlib.3pm
Manifying blib/man3/Net::SSH::Perl::Kex::DH1.3pm
Manifying blib/man3/Net::SSH::Perl::Mac.3pm
Manifying blib/man3/Net::SSH::Perl::SSH2.3pm
Manifying blib/man3/Net::SSH::Perl::Auth::Password.3pm
Manifying blib/man3/Net::SSH::Perl::Auth::Rhosts_RSA.3pm
Manifying blib/man3/Net::SSH::Perl::Cipher::Blowfish.3pm
Manifying blib/man3/Net::SSH::Perl::Key::RSA1.3pm
Manifying blib/man3/Net::SSH::Perl::Key.3pm
Manifying blib/man3/Net::SSH::Perl::Buffer.3pm
Manifying blib/man3/Net::SSH::Perl::Key::DSA.3pm
Manifying blib/man3/Net::SSH::Perl::Cipher::DES3.3pm
Manifying blib/man3/Net::SSH::Perl::Cipher::RC4.3pm
Manifying blib/man3/Net::SSH::Perl::Cipher.3pm
Manifying blib/man3/Net::SSH::Perl::Constants.3pm
Manifying blib/man3/Net::SSH::Perl::Config.3pm
Manifying blib/man3/Net::SSH::Perl::Util.3pm
Manifying blib/man3/Net::SSH::Perl.3pm
Manifying blib/man3/Net::SSH::Perl::Channel.3pm
Manifying blib/man3/Net::SSH::Perl::SSH1.3pm
Manifying blib/man3/Net::SSH::Perl::Cipher::DES.3pm
Manifying blib/man3/Net::SSH::Perl::Packet.3pm
Manifying blib/man3/Net::SSH::Perl::Key::RSA.3pm
/usr/bin/make -- OK
Running make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e"
"test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/01-compile.....
t/01-compile.....ok 1/1
t/01-compile.....ok
t/02-buffer......
t/02-buffer......ok 1/19
t/02-buffer......ok
t/03-packet......
t/03-packet......ok 1/10
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 172
**************************************
