[32743] in Perl-Users-Digest
Perl-Users Digest, Issue: 4007 Volume: 11
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Mon Aug 5 16:09:41 2013
Date: Mon, 5 Aug 2013 13:09:06 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Mon, 5 Aug 2013 Volume: 11 Number: 4007
Today's topics:
configuring STD* IO to use locale's encoding? <oneingray@gmail.com>
Re: configuring STD* IO to use locale's encoding? <peter@makholm.net>
Re: configuring STD* IO to use locale's encoding? <oneingray@gmail.com>
Re: configuring STD* IO to use locale's encoding? <peter@makholm.net>
Re: configuring STD* IO to use locale's encoding? <gravitalsun@hotmail.foo>
Re: fast scan <rweikusat@mssgmbh.com>
Re: fast scan <rweikusat@mssgmbh.com>
Re: fast scan <derykus@gmail.com>
Re: fast scan <gravitalsun@hotmail.foo>
Re: fast scan <rweikusat@mssgmbh.com>
Re: fast scan <rweikusat@mssgmbh.com>
Re: fast scan <derykus@gmail.com>
Re: fast scan <gravitalsun@hotmail.foo>
Re: fast scan <rweikusat@mssgmbh.com>
Re: fast scan <nospam.gravitalsun.noadsplease@hotmail.noads.com>
Re: fast scan <rweikusat@mssgmbh.com>
Re: fast scan <nospam.gravitalsun.noadsplease@hotmail.noads.com>
Re: fast scan <derykus@gmail.com>
Re: fast scan <rweikusat@mssgmbh.com>
Re: fast scan <gravitalsun@hotmail.foo>
Re: fast scan <derykus@gmail.com>
PAUSE vs. $VERSION <oneingray@gmail.com>
XS; and exception objects <oneingray@gmail.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Sat, 03 Aug 2013 17:48:30 +0000
From: Ivan Shmakov <oneingray@gmail.com>
Subject: configuring STD* IO to use locale's encoding?
Message-Id: <87txj6ll75.fsf@violet.siamics.net>
Do I understand it correctly that a proper way to configure the
"standard IO" filehandles to use the encoding specified by the
locale currently in effect would be like:
## NB: use locale; affects collation, etc., not IO handles
use Encode::Locale; # for :encoding(locale)
binmode ($_, ":encoding(locale)")
foreach (\*STDERR, \*STDIN, \*STDOUT);
Doesn't it warrant a pragma, BTW?
TIA.
PS. FWIW, print ("\N{U+0432}\N{U+0444}"); now appears to work correctly
with at least two different encodings. And so does warn ().
Also to note is that there seem to be configurations which
result in non-English, -- and non-ASCII, -- $! texts...
--
FSF associate member #7257
------------------------------
Date: Sun, 04 Aug 2013 08:39:52 +0200
From: Peter Makholm <peter@makholm.net>
Subject: Re: configuring STD* IO to use locale's encoding?
Message-Id: <87fvuqhscn.fsf@vps1.hacking.dk>
Ivan Shmakov <oneingray@gmail.com> writes:
> binmode ($_, ":encoding(locale)")
> foreach (\*STDERR, \*STDIN, \*STDOUT);
>
> Doesn't it warrant a pragma, BTW?
It seems that you could use
use open ':locale';
but it would make all ':encoding(locale)' the default for all
filehandles and not just the STD* ones.
//Makholm
------------------------------
Date: Sun, 04 Aug 2013 07:56:27 +0000
From: Ivan Shmakov <oneingray@gmail.com>
Subject: Re: configuring STD* IO to use locale's encoding?
Message-Id: <87haf5lwic.fsf@violet.siamics.net>
>>>>> Peter Makholm <peter@makholm.net> writes:
>>>>> Ivan Shmakov <oneingray@gmail.com> writes:
>> binmode ($_, ":encoding(locale)")
>> foreach (\*STDERR, \*STDIN, \*STDOUT);
>> Doesn't it warrant a pragma, BTW?
> It seems that you could use
> use open ':locale';
BTW, where is this pragma documented?
> but it would make all ':encoding(locale)' the default for all
> filehandles and not just the STD* ones.
... Which is perfectly sensible. Thanks!
--
FSF associate member #7257
------------------------------
Date: Sun, 04 Aug 2013 11:45:35 +0200
From: Peter Makholm <peter@makholm.net>
Subject: Re: configuring STD* IO to use locale's encoding?
Message-Id: <877gg1iybk.fsf@vps1.hacking.dk>
Ivan Shmakov <oneingray@gmail.com> writes:
> > use open ':locale';
>
> BTW, where is this pragma documented?
As all others pragmas you can find the documentation with 'perldoc
open'. The documentation is also available at
https://metacpan.org/module/open
On perldoc.perl.org the search interface prefers the entry from the
perlfunc manual page. You can view all results and find the
documentation under 'Core modules' or just look for 'Pragmas' in the
sidebar.
Both the open and the binmode entries in the perlfun manpage refers to
the open pragma. In the binmode entry it is quite clearly state:
The :bytes , :crlf , :utf8 , and any other directives of the form
:... , are called I/O layers. The open pragma can be used to establish
default I/O layers. See open.
But the reference in the open entry is a bit more obscure and could
probably use a rewording.
//Makholm
------------------------------
Date: Sun, 04 Aug 2013 16:12:50 +0300
From: George Mpouras <gravitalsun@hotmail.foo>
Subject: Re: configuring STD* IO to use locale's encoding?
Message-Id: <ktlk0k$1j7s$1@news.ntua.gr>
Στις 3/8/2013 20:48, ο/η Ivan Shmakov έγραψε:
> Do I understand it correctly that a proper way to configure the
> "standard IO" filehandles to use the encoding specified by the
> locale currently in effect would be like:
>
The following 3 pieces of code are equivalent, make your choises
Assuming that your Perl script is saved as UTF8 file.
# code1
print 'Καλημέρα';
# code2
use feature qw/say/;
use Encode;
use utf8;
binmode STDOUT, ':utf8';
print 'Καλημέρα';
# code2
use feature qw/say/;
use Encode;
use utf8;
print Encode::encode('utf8', 'Καλημέρα')
------------------------------
Date: Sat, 03 Aug 2013 14:48:11 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: fast scan
Message-Id: <87pptuopgk.fsf@sapphire.mobileactivedefense.com>
George Mpouras <gravitalsun@hotmail.foo> writes:
> # scan a network in 2 seconds using fork. Very simplistic but with
> potential !
[...]
> sub Scan
> {
> my @Pids;
>
> foreach my $ip (@_)
> {
> my $pid = fork();
> die "Could not fork because $!\n" unless defined $pid;
>
> if (0 == $pid)
> {
> my $ping = Net::Ping->new('icmp');
> say "host $ip is up" if $ping->ping($ip, $duration);
> $ping->close();
> exit
> }
> else
> {
> push @Pids, $pid
> }
> }
>
> foreach my $pid (@Pids) { waitpid($pid, 0) }
> }
If you run code like this on somebody else's network (or network and
computer), you will potentially learn more details about lart than you
ever wanted to. In order to achieve maximum disaster, run it on a
gateway for a busy network which suffers from 'traditional' BSD
network buffer management, preferably in a loop. If you manage to
reach mbuf exhaustion, you've produced a stable 'congestion collapse'
(slight misuse of the term) situation: The gateway will drop incoming
ethernet frames until it got rid of enough pings to reconsider this
descision. At this point, a reply tsunami will hit it (ping replies,
TCP retransmissions, ARP replies and ARP queries) and it will
immediately run out of mbufs again. Repeat until heat death of the
universe ...
NB: This is not a story I just invented or some kind of theoretical
conjecture. I've had the mispleasure to encounter this exact problem
on such a gateway some years ago.
------------------------------
Date: Sat, 03 Aug 2013 14:56:39 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: fast scan
Message-Id: <877gg2zxm0.fsf@sapphire.mobileactivedefense.com>
Rainer Weikusat <rweikusat@mssgmbh.com> writes:
> George Mpouras <gravitalsun@hotmail.foo> writes:
>> # scan a network in 2 seconds using fork. Very simplistic but with
>> potential !
>
> [...]
>
>> sub Scan
>> {
>> my @Pids;
>>
>> foreach my $ip (@_)
>> {
>> my $pid = fork();
>> die "Could not fork because $!\n" unless defined $pid;
>>
>> if (0 == $pid)
>> {
>> my $ping = Net::Ping->new('icmp');
>> say "host $ip is up" if $ping->ping($ip, $duration);
>> $ping->close();
>> exit
>> }
>> else
>> {
>> push @Pids, $pid
>> }
>> }
>>
>> foreach my $pid (@Pids) { waitpid($pid, 0) }
>> }
>
> If you run code like this on somebody else's network (or network and
> computer), you will potentially learn more details about lart than you
> ever wanted to.
For completeness: It is not entirely inconceivable that the fork bomb
manages to slow transmissions down so much that the 'DDoS suicide'
doesn't happen.
------------------------------
Date: Sat, 03 Aug 2013 21:00:22 -0700
From: Charles DeRykus <derykus@gmail.com>
Subject: Re: fast scan
Message-Id: <ktkjlm$vj8$1@speranza.aioe.org>
On 8/2/2013 5:39 PM, George Mpouras wrote:
> # scan a network in 2 seconds using fork. Very simplistic but with
> potential !
>
>
>
>
> #!/usr/bin/perl
> use strict;
> use warnings;
> use feature qw/say/;
> use Net::Ping;
> use Net::IP;
>
> my $threads = 255;
> my $duration = 2;
> my @ip_team = ();
> $|= 1;
>
>
> my $ip = new Net::IP ('192.168.0.1 - 192.168.0.254') or die "Could not
> initiate object because ". Net::IP::Error() ."\n";
>
>
> while ($ip) {
> push @ip_team, $ip++ ->ip();
> if ( $threads == @ip_team ) { Scan(@ip_team); @ip_team = () }
> }
>
> Scan(@ip_team);
>
>
>
> sub Scan
> {
> my @Pids;
>
> foreach my $ip (@_)
> {
> my $pid = fork();
> die "Could not fork because $!\n" unless defined $pid;
>
> if (0 == $pid)
> {
> my $ping = Net::Ping->new('icmp');
> say "host $ip is up" if $ping->ping($ip, $duration);
> $ping->close();
> exit
> }
> else
> {
> push @Pids, $pid
> }
> }
>
> foreach my $pid (@Pids) { waitpid($pid, 0) }
> }
A less resource intensive alternative with POE:
http://poe.perl.org/?POE_Cookbook/Pinging_Multiple_Hosts
--
Charles DeRykus
------------------------------
Date: Sun, 04 Aug 2013 13:42:07 +0300
From: George Mpouras <gravitalsun@hotmail.foo>
Subject: Re: fast scan
Message-Id: <ktlb61$qpe$1@news.ntua.gr>
>
>
> A less resource intensive alternative with POE:
>
> http://poe.perl.org/?POE_Cookbook/Pinging_Multiple_Hosts
>
I really do not know if it is really less resource intensive, and I am
not interesting to find out, but with the simple code you can always
have as many $threads you want.
What I was thinking is why I really need super bloated frameworks like
POE or similar if I can do much simpler the same using only basic
functions.
------------------------------
Date: Sun, 04 Aug 2013 14:54:43 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: fast scan
Message-Id: <87mwoxsgrg.fsf@sapphire.mobileactivedefense.com>
George Mpouras <gravitalsun@hotmail.foo> writes:
>> A less resource intensive alternative with POE:
>>
>> http://poe.perl.org/?POE_Cookbook/Pinging_Multiple_Hosts
>>
>
> I really do not know if it is really less resource intensive, and I am
> not interesting to find out, but with the simple code you can always
> have as many $threads you want.
Except that this is total nonsense here because all these different
processes end up stuffing IP datagrams into the TX queue for the same
network device which then sends them one after another. This means
you're basically just adding a lot of overhead because the scheduler
needs to deal with all the processes and other parts of the kernel
have to serialize them forcibly. Also, sending out ICMP echo requests
as fast as the kernel can manage to deal with the processes to
hundreds or thousands of hosts is really bad: That will end up as
hundreds or thousands of hosts hammering your single computer with
replies as fast as they can (if you ask 10,000 people to throw tennis
balls to you at the same time, you'll end up being stoned to death by
tennis balls).
> What I was thinking is why I really need super bloated frameworks like
> POE or similar if I can do much simpler the same using only basic
> functions.
A relatively simple way to do this would be to use two processes: One
which sends pings (rate-limited(!)) and another which blocks in recv
on a raw socket in order to process the replies. A single process
utilizing select and non-blocking sends and receive would be a
somewhat better choice.
------------------------------
Date: Sun, 04 Aug 2013 14:59:29 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: fast scan
Message-Id: <87iozlsgji.fsf@sapphire.mobileactivedefense.com>
Charles DeRykus <derykus@gmail.com> writes:
> On 8/2/2013 5:39 PM, George Mpouras wrote:
[insane networking code]
> A less resource intensive alternative with POE:
>
> http://poe.perl.org/?POE_Cookbook/Pinging_Multiple_Hosts
As far as I can tell, that suffers from the same "Hit me as fast as
you can, bazillions, 'cos really tired of his life!"
problem. Something equivalent can probably be implemented without POE
by writing (at worst) a little more text (my gut feeling says 'less',
actually).
------------------------------
Date: Sun, 04 Aug 2013 09:20:48 -0700
From: Charles DeRykus <derykus@gmail.com>
Subject: Re: fast scan
Message-Id: <ktlv1u$j9c$1@speranza.aioe.org>
On 8/4/2013 3:42 AM, George Mpouras wrote:
>>
>>
>> A less resource intensive alternative with POE:
>>
>> http://poe.perl.org/?POE_Cookbook/Pinging_Multiple_Hosts
>>
>
> I really do not know if it is really less resource intensive, and I am
> not interesting to find out, but with the simple code you can always
> have as many $threads you want.
>
> What I was thinking is why I really need super bloated frameworks like
> POE or similar if I can do much simpler the same using only basic
> functions.
I have to admit I hadn't dabbled with POE before. However, I quickly
installed two POE modules and had the above program running immediately.
So, even with a sub-idiot's grasp of POE's internals, you can leverage a
far more scalable, less resource-hogging code resource that can be
adapted to many different contexts.
It's true your simple fork example with no data sharing really doesn't
require POE or even threads. But later you may want something more
complicated which does. And there are many robust POE programs that you
can build upon without the riskiness and complexity of a fork (or an
even trickier thread) model. And, later, in another multi-tasking
scenario, what if you needed to fire off lots of MySQL queries, then
gather, analyze, maybe collate/reformat output. A viral fork model might
crash, not to mention seriously annoying other users... or re-kindle
thoughts about chucking it all for some simple goat herding in the Alps.
--
Charles DeRykus
------------------------------
Date: Mon, 05 Aug 2013 03:28:11 +0300
From: George Mpouras <gravitalsun@hotmail.foo>
Subject: Re: fast scan
Message-Id: <ktmrit$2b07$1@news.ntua.gr>
# dizzying fast with port scanner, cpu is almost 0%
# this is forking the forks !
#!/usr/bin/perl
use strict;
use warnings;
use feature qw/say/;
use Net::Ping;
my $threads = 80;
my $duration = 1;
my @ip_team = ();
my $db_dir = Unique_node_name('/tmp/.fastscan');
mkdir $db_dir or die "Could not create directory \"$db_dir\" because
\"$^E\"\n" unless -d $db_dir;
$|= 1;
my @ports = (21, 22, 80, 135, 443);
my ($o1a,$o1b, $o2a,$o2b, $o3a,$o3b, $o4a,$o4b) =
Check_and_define_octet( '192.168.0.[1-254]' );
foreach my $o1 ($o1a .. $o1b) {
foreach my $o2 ($o2a .. $o2b) {
foreach my $o3 ($o3a .. $o3b) {
foreach my $o4 ($o4a .. $o4b) {
push @ip_team, "$o1.$o2.$o3.$o4";
if ( $threads == @ip_team ) { Scan(@ip_team); @ip_team = () }
}}}}
Scan(@ip_team);
system("/bin/rm -rf $db_dir");
sub Scan
{
my @Pids;
foreach my $ip (@_)
{
my $pid = fork(); die "Could not fork because $!\n" unless defined $pid;
if (0 == $pid)
{
my
$ping= Net::Ping->new('icmp');
$ping->service_check(0);
if ( $ping->ping($ip, $duration) )
{
mkdir "$db_dir/$ip";
my @SubPids;
foreach my $port (@ports)
{
my $pid = fork(); die "Could not fork because $!\n" unless defined $pid;
if (0 == $pid)
{
my
$subping = Net::Ping->new('tcp', 2);
$subping->service_check(0);
$subping->port_number($port);
mkdir "$db_dir/$ip/$port" if $subping->ping($ip);
$subping->close;
exit 0
}
else
{
push @SubPids, $pid
}
}
foreach my $pid (@SubPids) { waitpid($pid, 0) }
say "$ip is up";
chdir "$db_dir/$ip";
foreach ( glob '*' ) { say "\t port $_ is open" }
}
$ping->close();
exit 0
}
else
{
push @Pids, $pid
}
}
foreach my $pid (@Pids) { waitpid($pid, 0) }
}
sub Unique_node_name
{
my ($dir,$file )= $_[0] =~/^(.*?)([^\/]*)$/;
if ( $dir=~/^\s*$/ ) { $dir = '.' } else { $dir =~s/\/*$// }
$file = 'node' if $file=~/^\s*$/;
return "$dir/$file" if ! -e "$dir/$file";
my $i=1; while ( -e "$dir/$i.$file" ) {$i++}
"$dir/$i.$file"
}
# Accepts a host definition like 192.168.[0-3].[1-254]
# and returns for every octet its first and stop number
# For example for the [10-12].1.86.[1-100]
# it will return
# 10,12, 1,1, 86,86, 1,100
#
sub Check_and_define_octet
{
my @O;
( my $hosts = $_[0] )=~s/\s+//g;
( $O[0]->[0] , $O[1]->[0], $O[2]->[0], $O[3]->[0] ) = $hosts
=~/^([^.]+)\.([^.]+)\.([^.]+)\.([^.]+)$/ or die "The host definition
argument is not like 192.168.[0-3].[1-254]\n";
my $i=0;
foreach my $start (1,0,0,1)
{
if ( $O[$i]->[0] =~/^\d+$/ )
{
@{$O[$i]}[0,1] = (( $O[$i]->[0] >= $start ) && ( $O[$i]->[0] < 255 ))
? @{$O[$i]}[0,0] : die "Octet \"$O[$i]->[0]\" should be an integer from
$start to 254\n"
}
elsif ( $O[$i]->[0] =~/\[(\d+)-(\d+)\]/ )
{
$O[$i]->[0] = (( $1 >= $start ) && ( $1 < 255 )) ? $1 : $start;
$O[$i]->[1] = (( $2 >= $start ) && ( $2 < 255 )) ? $2 : 254;
@{$O[$i]}[0,1] = $O[$i]->[0] > $O[$i]->[1] ? @{$O[$i]}[1,0] :
@{$O[$i]}[0,1]
}
else
{
die "Sorry but octet \"$O[$i]->[0]\" should be something like 12 or
[10 - 254]\n"
}
$i++
}
#use Data::Dumper; print Dumper \@O; exit;
@{$O[0]}, @{$O[1]}, @{$O[2]}, @{$O[3]}
}
------------------------------
Date: Mon, 05 Aug 2013 08:45:42 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: fast scan
Message-Id: <87bo5c4m3d.fsf@sapphire.mobileactivedefense.com>
George Mpouras <gravitalsun@hotmail.foo> writes:
> # dizzying fast with port scanner, cpu is almost 0%
> # this is forking the forks !
[more of this]
Something which also deserves to be mentioned here: The sole reason
for this insane fork-orgy is that George has to work around the
library he chose to use for 'network communication' which offers only
a synchronous 'send request and wait for reply' interface. Which is
actually not atypical for 'technical solutions' starting with 'Welches
Gurkenglass, das hier dumm herumsteht, koennte sich wohl dafuer
eignen, diesen Nagel in die Wand zu schlagen?': It starts with some
clueless, devil-may-care individual selecting the wrong tool for the
job at hand because he reaches for the closest one and then proceeds
as set of 'ingenious' workarounds for the deficiencies of that.
------------------------------
Date: Mon, 05 Aug 2013 15:58:23 +0300
From: George Mpouras <nospam.gravitalsun.noadsplease@hotmail.noads.com>
Subject: Re: fast scan
Message-Id: <kto7g4$91t$1@news.ntua.gr>
Στις 5/8/2013 10:45, ο/η Rainer Weikusat έγραψε:
> George Mpouras <gravitalsun@hotmail.foo> writes:
>> # dizzying fast with port scanner, cpu is almost 0%
>> # this is forking the forks !
>
> [more of this]
>
> Something which also deserves to be mentioned here: The sole reason
> for this insane fork-orgy is that George has to work around the
> library he chose to use for 'network communication' which offers only
> a synchronous 'send request and wait for reply' interface. Which is
> actually not atypical for 'technical solutions' starting with 'Welches
> Gurkenglass, das hier dumm herumsteht, koennte sich wohl dafuer
> eignen, diesen Nagel in die Wand zu schlagen?': It starts with some
> clueless, devil-may-care individual selecting the wrong tool for the
> job at hand because he reaches for the closest one and then proceeds
> as set of 'ingenious' workarounds for the deficiencies of that.
>
>
Playing around for fan I’ve done the same think using the modules
threads
threads::shared
The code was much smaller but to my surprise cpu usage jumped from
almost 0 to 60% and script all together consume about 1.5Gb ram, wow !
I think this is because the threads module create real threads probably
duplicating each memory. Fork somehow creates something more light …
------------------------------
Date: Mon, 05 Aug 2013 15:24:02 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: fast scan
Message-Id: <87txj4usfx.fsf@sapphire.mobileactivedefense.com>
George Mpouras <nospam.gravitalsun.noadsplease@hotmail.noads.com>
writes:
> Στις 5/8/2013 10:45, ο/η Rainer Weikusat έγραψε:
>> George Mpouras <gravitalsun@hotmail.foo> writes:
>>> # dizzying fast with port scanner, cpu is almost 0%
>>> # this is forking the forks !
>>
>> [more of this]
>>
>> Something which also deserves to be mentioned here: The sole reason
>> for this insane fork-orgy is that George has to work around the
>> library he chose to use for 'network communication' which offers only
>> a synchronous 'send request and wait for reply' interface. Which is
>> actually not atypical for 'technical solutions' starting with 'Welches
>> Gurkenglass, das hier dumm herumsteht, koennte sich wohl dafuer
>> eignen, diesen Nagel in die Wand zu schlagen?': It starts with some
>> clueless, devil-may-care individual selecting the wrong tool for the
>> job at hand because he reaches for the closest one and then proceeds
>> as set of 'ingenious' workarounds for the deficiencies of that.
[...]
> Playing around for fan I’ve done the same think using the modules
>
> threads
> threads::shared
>
> The code was much smaller but to my surprise cpu usage jumped from
> almost 0 to 60% and script all together consume about 1.5Gb ram, wow
> !
>
> I think this is because the threads module create real threads
> probably duplicating each memory. Fork somehow creates something more
> light …
The perl (bytecode) interpreter doesn't support
multithreading. Because of this, Perl threading support works such
that each thread gets its own copy of the interpreter created by
making a memory-to-memory copy of an already existing one. This
implies that creating threads is slow and multi-threaded Perl programs
need a lot of memory because despite running in a shared address space
they can't really share any of it.
In contrast to this, fork nowadays usually works by copying the page
table of the forking process and changing the memory access
permissions to read-only: Initially, both parent and child process
will share all memory. As soon as either of both tries to write to
something, a page fault occurs and the kernel handles that by
allocating a new page of memory, copying the page where the fault
occurred and then giving one to the child and the other to the parent
with access permissions changed back to read-write. This means copying
happens only when needed and only what really has to be copied is.
But for 'network scanning' this is an aside. The sensible way to do
that is to make it work asynchronously, ie, instead of waiting for a
reply after each request, keep sending whatever requests remain to be
sent (again, rate-limited) and process replies as they arrive.
------------------------------
Date: Mon, 05 Aug 2013 17:44:09 +0300
From: George Mpouras <nospam.gravitalsun.noadsplease@hotmail.noads.com>
Subject: Re: fast scan
Message-Id: <ktodmd$prj$1@news.ntua.gr>
Στις 5/8/2013 17:24, ο/η Rainer Weikusat έγραψε:
> George Mpouras <nospam.gravitalsun.noadsplease@hotmail.noads.com>
> writes:
>> Στις 5/8/2013 10:45, ο/η Rainer Weikusat έγραψε:
>>> George Mpouras <gravitalsun@hotmail.foo> writes:
>>>> # dizzying fast with port scanner, cpu is almost 0%
>>>> # this is forking the forks !
>>>
>>> [more of this]
>>>
>>> Something which also deserves to be mentioned here: The sole reason
>>> for this insane fork-orgy is that George has to work around the
>>> library he chose to use for 'network communication' which offers only
>>> a synchronous 'send request and wait for reply' interface. Which is
>>> actually not atypical for 'technical solutions' starting with 'Welches
>>> Gurkenglass, das hier dumm herumsteht, koennte sich wohl dafuer
>>> eignen, diesen Nagel in die Wand zu schlagen?': It starts with some
>>> clueless, devil-may-care individual selecting the wrong tool for the
>>> job at hand because he reaches for the closest one and then proceeds
>>> as set of 'ingenious' workarounds for the deficiencies of that.
>
> [...]
>
>> Playing around for fan I’ve done the same think using the modules
>>
>> threads
>> threads::shared
>>
>> The code was much smaller but to my surprise cpu usage jumped from
>> almost 0 to 60% and script all together consume about 1.5Gb ram, wow
>> !
>>
>> I think this is because the threads module create real threads
>> probably duplicating each memory. Fork somehow creates something more
>> light …
>
> The perl (bytecode) interpreter doesn't support
> multithreading. Because of this, Perl threading support works such
> that each thread gets its own copy of the interpreter created by
> making a memory-to-memory copy of an already existing one. This
> implies that creating threads is slow and multi-threaded Perl programs
> need a lot of memory because despite running in a shared address space
> they can't really share any of it.
>
> In contrast to this, fork nowadays usually works by copying the page
> table of the forking process and changing the memory access
> permissions to read-only: Initially, both parent and child process
> will share all memory. As soon as either of both tries to write to
> something, a page fault occurs and the kernel handles that by
> allocating a new page of memory, copying the page where the fault
> occurred and then giving one to the child and the other to the parent
> with access permissions changed back to read-write. This means copying
> happens only when needed and only what really has to be copied is.
>
> But for 'network scanning' this is an aside. The sensible way to do
> that is to make it work asynchronously, ie, instead of waiting for a
> reply after each request, keep sending whatever requests remain to be
> sent (again, rate-limited) and process replies as they arrive.
>
Very sensible explanation.
For curiosity I fire up some similar scans using nmap.
It is very fast while it respects cpu and network utilization.
I have to look at its code to grab a general idea of how it works.
------------------------------
Date: Mon, 05 Aug 2013 11:44:45 -0700
From: Charles DeRykus <derykus@gmail.com>
Subject: Re: fast scan
Message-Id: <ktorrs$uav$1@speranza.aioe.org>
On 8/4/2013 6:59 AM, Rainer Weikusat wrote:
> Charles DeRykus <derykus@gmail.com> writes:
>> On 8/2/2013 5:39 PM, George Mpouras wrote:
>
> [insane networking code]
>
>> A less resource intensive alternative with POE:
>>
>> http://poe.perl.org/?POE_Cookbook/Pinging_Multiple_Hosts
>
> As far as I can tell, that suffers from the same "Hit me as fast as
> you can, bazillions, 'cos really tired of his life!"
> problem. Something equivalent can probably be implemented without POE
> by writing (at worst) a little more text (my gut feeling says 'less',
> actually).
>
POE though is using non-blocking I/O and an event loop behind the scenes
which is certainly a reasonable approach IIUC. Admittedly there's a a
fair amount of overhead with POE's pseudo-kernel set-up, but, at least,
there's a single process rather than cloning perl interpreters or
forking a bazillion times.
And, what if the fork/thread bomber later finds he needs to save or
post-process the ip's that successfully pinged... He'll inevitably need
to delve into IPC or variable sharing/locking intricacies via
threads::shared, etc. POE's single process makes it trivial. Finally,
there's the extensive, robust POE networking examples already available
for templates...
--
Charles DeRykus
------------------------------
Date: Mon, 05 Aug 2013 20:28:16 +0100
From: Rainer Weikusat <rweikusat@mssgmbh.com>
Subject: Re: fast scan
Message-Id: <87li4guecv.fsf@sapphire.mobileactivedefense.com>
Charles DeRykus <derykus@gmail.com> writes:
> On 8/4/2013 6:59 AM, Rainer Weikusat wrote:
>> Charles DeRykus <derykus@gmail.com> writes:
>>> On 8/2/2013 5:39 PM, George Mpouras wrote:
>>
>> [insane networking code]
>>
>>> A less resource intensive alternative with POE:
>>>
>>> http://poe.perl.org/?POE_Cookbook/Pinging_Multiple_Hosts
>>
>> As far as I can tell, that suffers from the same "Hit me as fast as
>> you can, bazillions, 'cos really tired of his life!"
>> problem. Something equivalent can probably be implemented without POE
>> by writing (at worst) a little more text (my gut feeling says 'less',
>> actually).
>
> POE though is using non-blocking I/O and an event loop behind the
> scenes which is certainly a reasonable approach IIUC. Admittedly
> there's a a fair amount of overhead with POE's pseudo-kernel set-up,
> but, at least, there's a single process rather than cloning perl
> interpreters or forking a bazillion times.
This will make the issue I referred to as 'DDos suicide' in another
posting worse because the probe packets are being sent out faster: The
original posting was about pinging all hosts on the local LAN. And
this local LAN may neither be 'small' (a /24 or less) nor sparsely
populated. Let's assume that a thousand hosts reply: Somewhat
simplified, these have 1000x the bandwidth and other resources
available for sending their messages than the host who sent the probes
can muster for processing them.
------------------------------
Date: Mon, 05 Aug 2013 22:46:42 +0300
From: George Mpouras <gravitalsun@hotmail.foo>
Subject: Re: fast scan
Message-Id: <ktovf7$29qe$1@news.ntua.gr>
> POE though is using non-blocking I/O and an event loop behind the scenes
> which is certainly a reasonable approach IIUC. Admittedly there's a a
> fair amount of overhead with POE's pseudo-kernel set-up, but, at least,
> there's a single process rather than cloning perl interpreters or
> forking a bazillion times.
>
I have play with POE but not in depth, because I tired from its big
documentation and its many modules.
I think I will try once more but I will monitor carefully the system to
check if what it claims is true.
I mean the threads module in documentation is perfect but in real life
.... forget it, at least for real intensive tasks.
------------------------------
Date: Mon, 05 Aug 2013 13:04:12 -0700
From: Charles DeRykus <derykus@gmail.com>
Subject: Re: fast scan
Message-Id: <ktp0gq$bjn$1@speranza.aioe.org>
On 8/5/2013 12:28 PM, Rainer Weikusat wrote:
> Charles DeRykus <derykus@gmail.com> writes:
>> On 8/4/2013 6:59 AM, Rainer Weikusat wrote:
>> ...
>>
>> POE though is using non-blocking I/O and an event loop behind the
>> scenes which is certainly a reasonable approach IIUC. Admittedly
>> there's a a fair amount of overhead with POE's pseudo-kernel set-up,
>> but, at least, there's a single process rather than cloning perl
>> interpreters or forking a bazillion times.
>
> This will make the issue I referred to as 'DDos suicide' in another
> posting worse because the probe packets are being sent out faster: The
> original posting was about pinging all hosts on the local LAN. And
> this local LAN may neither be 'small' (a /24 or less) nor sparsely
> populated. Let's assume that a thousand hosts reply: Somewhat
> simplified, these have 1000x the bandwidth and other resources
> available for sending their messages than the host who sent the probes
> can muster for processing them.
>
Actually, it appears that POE has already dealt with than issue
(although tuning might take guesswork) with its own modulation setting:
http://search.cpan.org/~rcaputo/POE-Component-Client-Ping-1.173/lib/POE/Component/Client/Ping.pm
From above doc:
Parallelism => $limit
Parallelism sets POE::Component::Client::Ping's maximum number of
simultaneous ICMP requests. Higher numbers speed up the processing of
large host lists, up to the point where the operating system or network
becomes oversaturated and begin to drop packets.
--
Charles DeRykus
------------------------------
Date: Mon, 05 Aug 2013 20:07:41 +0000
From: Ivan Shmakov <oneingray@gmail.com>
Subject: PAUSE vs. $VERSION
Message-Id: <87siynkik2.fsf@violet.siamics.net>
As per the PAUSE recommendation [1], I check my modules to have
the proper $VERSION information:
$ perl -MExtUtils::MakeMaker \
-le 'print MM->parse_version (shift)' lib/Tree/Range.pm
0.21
$
... or, rather:
$ perl -e 'use common::sense;
use ExtUtils::MakeMaker;
print (MM->parse_version ($_), "\t", $_, "\n")
foreach (@ARGV);' \
$(find lib/Tree/ -name \*.pm)
0.21 lib/Tree/Range/base.pm
...
0.21 lib/Tree/Range.pm
$
[1] http://pause.perl.org/pause/query?ACTION=pause_04about#version
Now, the PAUSE namespace indexer fails to recognize it, like:
--cut--
module: Tree::Range::base
version: undef
in file: Tree-Range-0.21/lib/Tree/Range/base.pm
status: Not indexed because Tree-Range-0.1/lib/Tree/Range/base.pm
in O/ON/ONEGRAY/Tree-Range-0.1.tar.gz has a higher version
number (0.1)
--cut--
Do I understand it correctly that the "package PACKAGE VERSION"
form is a (relatively) recent addition to Perl and that the
version of ExtUtils::MakeMaker (or?) used at PAUSE is yet to be
updated to handle it?
TIA.
--
FSF associate member #7257
------------------------------
Date: Sun, 04 Aug 2013 17:37:16 +0000
From: Ivan Shmakov <oneingray@gmail.com>
Subject: XS; and exception objects
Message-Id: <878v0hl5mb.fsf_-_@violet.siamics.net>
>>>>> Ben Morrow <ben@morrow.me.uk> writes:
>>>>> Quoth Ivan Shmakov <oneingray@gmail.com>:
[...]
>> Unless I'm missing something, this is exactly what I do. The point
>> is that I'm hesitant to either generate exception objects /or/ call
>> Perl code from XS, so my XS ("low-level") interface returns error
>> codes, which the Perl wrapper methods then turn into proper
>> exceptions (which are indeed objects.)
> Ah, OK. I though you were trying to provide both return-value and
> exception interfaces so users could choose which they wanted.
FWIW, I'm going to document both, but I don't expect for the
users to actually use the return-value interface. (Other than
in the cases the respective Perl wrapper methods are found to be
faulty or otherwise flawed.)
> Throwing exceptions from XS is fairly straightforward; you build your
> object (however you like) and call Perl_croak_sv. However it would
> probably be easier to have a My::Exception->throw method, which
> builds and throws an exception; this means you just need to make a
> method call, which goes like this:
... Which is exactly what I'm trying to avoid. Debugging XS
code isn't all that easy to me, and I'm pretty sure that having
to debug XS code paths intertwined with Perl's won't be of any
help to me. Or to any inexperienced Perl user who may happen to
have to debug my code, for that matter.
[...]
> and an exception of this class reaches the top-level 'print a message
> and kill the program' exception handler, that message will not
> include the 'at file ... line ...' information. Adding that
> information is something die does if you pass it a string; if you
> pass it an object, it has nowhere to put the information.
> This means that your 'new' method needs to record that information,
> somehow, so it can include it in the stringification.
ACK, thanks for the information.
For now, I'm effectively "substituting" die () with croak (),
like:
package main;
require Carp;
$SIG{"__DIE__"}
= \&Carp::confess;
for the "at file" error messages weren't so far nearly as
helpful as complete stack traces, and my guess is that it also
solves the issue at hand. Thus, I'm going to postpone this one
until some later time.
> You can pull it out of 'caller', though that means working out how
> many stack frames to go back to get to the 'real' location of the
> error.
Yes, and it seems that there is indeed some choice.
> You can also use Carp::shortmess, which gives you the message 'croak'
> would have given you.
It doesn't seem documented, does it?
--
FSF associate member #7257
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
Back issues are available via anonymous ftp from
ftp://cil-www.oce.orst.edu/pub/perl/old-digests.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V11 Issue 4007
***************************************
