[26135] in Perl-Users-Digest
Perl-Users Digest, Issue: 8327 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Wed Aug 17 06:05:26 2005
Date: Wed, 17 Aug 2005 03:05:05 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Wed, 17 Aug 2005 Volume: 10 Number: 8327
Today's topics:
Implementing Interfaces and Type Safety (OOP Newbie in <vtatila@mail.student.oulu.fi>
Re: Implementing Interfaces and Type Safety (OOP Newbie <tassilo.von.parseval@rwth-aachen.de>
Obtaining verbose info for http transfers. <sisyphus1@nomail.afraid.org>
Re: Obtaining verbose info for http transfers. <simon@unisolve.com.au>
Re: Obtaining verbose info for http transfers. <sisyphus1@nomail.afraid.org>
Re: Obtaining verbose info for http transfers. <no@email.com>
Re: Obtaining verbose info for http transfers. <kuujinbo@hotmail.com>
perl and mysql random datas <""alexjaquet\"@[no spam]msn.com">
Re: perl and mysql random datas <no@email.com>
Re: Perl Solaris/Linux LASTLOG <drew@drew.net>
Re: Problem with Curses <babacio@free.fr>
Simulating smaller MTU? ie sending small packets. <norealaddress@nowhere.com>
Re: Simulating smaller MTU? ie sending small packets. <jgibson@mail.arc.nasa.gov>
Re: Simulating smaller MTU? ie sending small packets. <norealaddress@nowhere.com>
Re: Simulating smaller MTU? ie sending small packets. <tassilo.von.parseval@rwth-aachen.de>
Re: Simulating smaller MTU? ie sending small packets. <norealaddress@nowhere.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Wed, 17 Aug 2005 09:54:35 +0300
From: "Veli-Pekka Tätilä" <vtatila@mail.student.oulu.fi>
Subject: Implementing Interfaces and Type Safety (OOP Newbie in Perl)
Message-Id: <ddumvj$7nl$1@news.oulu.fi>
Hi,
I'm new to OOP in Perl having only used classes by others, read some book's
take on the subject and implemented closures for quick-n-dirty object-like
thingies. One thing I didn't see mentioned in Beginning PErl is Perl's take
on what Java calls interfaces and C++ programmers pure virtual abstract base
classes. That is in stead of a real class with state or useful methods, it
is something non-instanciable - a contract that concrete sub-classes will
have a certain set of methods in them. Is there a good way to implement an
interface in Perl or some perlish idiom for achieving the same thing?
I've noticed that most Perl classes don't rely too much on abstraction where
as in Java I'm almost always talking through some sort of interface. Don't
get me wrong, though, usually I like Perl's simple and practical ways of
achieving what you want, <grin>.
Also, if there's such a thing as an interface in Perl, how is it used and
what about type safety? In Java one would get a reference to the interface
and can assign to it any class implementing the interface. Java will then
automagically see that the right method gets called inside the class and we
have polymorphism. Further more, type checking is done at compile time for
the most part.
From what I've understood objects are actually blessed references in Perl
and when assigning to a reference, the type of the referent may vary freely.
Thus the best you can do is to keep assigning the right sub-classes derived
from a dummy interface class and hope that you don't make any mistakes in
the process, right? The only trouble with that is accidentally assigning
something that does not conform to the interface . Perl will only throw an
error at compile time should the missing method or methods get called. I've
noticed that even in trivial cases, typoed method names are not catched when
checking the syntax. PHP 4 does come to mind, argh, but fortunately Perl's
OOP stuff is not at all as toy-like as it used to be in PHP.
In contrast, should one try using operators, functions or references with
the wrong primitive type e.g. hash functions for arrays, you'll usually get
at least a compile-time error before the app is run. is there any way of
enforcing similar type safety for user-created objects?
I suppose you could use reflection to some effect with the methods in the
universal base class. Checking whether the referent is derived from the
interface by calling isa or asking if it does support a particular method
with the appropriately named method can.
Is there a better way than reflection i.e. relying on prototypes or being
able to strongly type references?
Lastly, speaking of Perl's OOP stuff, I remember a funny quote about modules
from the Camel book:
<cite>
Perl does not patrol private/public borders within its modules - unlike
languages such as C++, Ada, and Modula-17, Perl isn't infatuated with
enforced privacy. As we mentioned at the beginning of the chapter, a Perl
module would prefer that you stayed out of its living room because you
weren't invited, not because it has a shotgun.
</cite>
--
With kind regards Veli-Pekka Tätilä (vtatila@mail.student.oulu.fi)
Accessibility, game music, synthesizers and programming:
http://www.student.oulu.fi/~vtatila/
------------------------------
Date: Wed, 17 Aug 2005 11:33:22 +0200
From: "Tassilo v. Parseval" <tassilo.von.parseval@rwth-aachen.de>
Subject: Re: Implementing Interfaces and Type Safety (OOP Newbie in Perl)
Message-Id: <slrndg6132.2mu.tassilo.von.parseval@localhost.localdomain>
Also sprach Veli-Pekka Tätilä:
> I'm new to OOP in Perl having only used classes by others, read some book's
> take on the subject and implemented closures for quick-n-dirty object-like
> thingies. One thing I didn't see mentioned in Beginning PErl is Perl's take
> on what Java calls interfaces and C++ programmers pure virtual abstract base
> classes. That is in stead of a real class with state or useful methods, it
> is something non-instanciable - a contract that concrete sub-classes will
> have a certain set of methods in them. Is there a good way to implement an
> interface in Perl or some perlish idiom for achieving the same thing?
Yes, it can be simulated in a fairly simple manner. The idea is to write
a package where each method dies on invocation. However, what you cannot
easily have is compile-time checks that ensure that a class implementing
that interface does in fact implement (=override in this case) all
methods.
Here is a simple interface:
package Interface;
use Carp;
sub new {
my $class = shift;
croak "'$class' does not implement 'new'";
}
sub method1 {
my $self = shift;
my $class = ref $self;
croak "'$class' does not implement 'method1'";
}
# etc.
If you want to be smart and avoid typing repetitive code, have perl
generate the methods for you based on a list of methods that must be
implemented by subclasses:
package Interface;
use Carp;
my @MUST_IMPLEMENT = qw/new method1 method2/;
for (@MUST_IMPLEMENT) {
no strict 'refs'; # when strictures are enabled
*$_ = sub {
my $invoc = shift;
my $class = ref($invoc) || $invoc;
croak "'$class' does not implement '$_'";
};
}
1;
__END__
A class implementing the above interface would look like this:
package Implementation;
use base qw/Interface/; # subclass 'Interface'
sub new {
...
}
sub method1 {
...
}
# etc.
1;
__END__
> I've noticed that most Perl classes don't rely too much on abstraction where
> as in Java I'm almost always talking through some sort of interface. Don't
> get me wrong, though, usually I like Perl's simple and practical ways of
> achieving what you want, <grin>.
>
> Also, if there's such a thing as an interface in Perl, how is it used and
> what about type safety? In Java one would get a reference to the interface
> and can assign to it any class implementing the interface. Java will then
> automagically see that the right method gets called inside the class and we
> have polymorphism. Further more, type checking is done at compile time for
> the most part.
When it comes to object-orientedness, everything happens at runtime as
far as Perl is concerned. What you say about assigning an object to a
variable of an interface type in Java (and the appropriate type checks
at compile time) does not exist for Perl and I would assume there is no
way to do it at compile-time. There are ways, however, to ensure no such
assignments happens at run-time, but these need to be implemented
manually, via a tied interface. But then this is quite heavy stuff.
> From what I've understood objects are actually blessed references in Perl
> and when assigning to a reference, the type of the referent may vary freely.
> Thus the best you can do is to keep assigning the right sub-classes derived
> from a dummy interface class and hope that you don't make any mistakes in
> the process, right? The only trouble with that is accidentally assigning
> something that does not conform to the interface . Perl will only throw an
> error at compile time should the missing method or methods get called. I've
> noticed that even in trivial cases, typoed method names are not catched when
> checking the syntax. PHP 4 does come to mind, argh, but fortunately Perl's
> OOP stuff is not at all as toy-like as it used to be in PHP.
That is true. Mistyped method invocations are caught at runtime. They
have to be due to Perl's super-polymorphic nature. Consider that a
particular method might not exist at compile-time. It might be added
later at run-time...alas, a whole class might suddenly spring into
existance during run-time. Lack of compile-time checks is the price you
have to pay when using a dynamic language such as Perl.
> In contrast, should one try using operators, functions or references with
> the wrong primitive type e.g. hash functions for arrays, you'll usually get
> at least a compile-time error before the app is run. is there any way of
> enforcing similar type safety for user-created objects?
No. The reason why it works with functions is that functions don't have
inheritance involved. No dynamic dispatch then means that many more
things can be checked at compile-time. Note however that a mistyped
function name is still a run-time error. The errors at compile-time that
you mention are those concerning functions with a prototype:
sub func (\@) {
"I must receive an array";
my $ary_ref = shift; # it's passed as reference
}
my @ary = (1 .. 10);
func @ary; # ok
func 1 .. 10; # compile-time error
func $scalar; # as is this
func %hash; # or this
Perl-builtins have prototypes attached to them so there perl can make
some sanity checks on the validity of the arguments at compile-time.
> I suppose you could use reflection to some effect with the methods in the
> universal base class. Checking whether the referent is derived from the
> interface by calling isa or asking if it does support a particular method
> with the appropriately named method can.
Yes, this is what I referred to earlier with "but these need to be
implemented manually". Run-time checks such as isa() or ref() are the
ones most commonly found to catch type violations.
> Is there a better way than reflection i.e. relying on prototypes or being
> able to strongly type references?
Prototypes don't exist for methods. Strongly typing references is
possible to a certain limited extent. See 'perldoc fields'.
> Lastly, speaking of Perl's OOP stuff, I remember a funny quote about modules
> from the Camel book:
>
><cite>
> Perl does not patrol private/public borders within its modules - unlike
> languages such as C++, Ada, and Modula-17, Perl isn't infatuated with
> enforced privacy. As we mentioned at the beginning of the chapter, a Perl
> module would prefer that you stayed out of its living room because you
> weren't invited, not because it has a shotgun.
></cite>
That's precisely what Perl is about. It's a matter of getting used to
it. This very liberal approach may be frowned upon by the Ada language
designers, however each has its own virtues and shortcomings. The
non-enforced privacy in Perl doesn't exist because the designers of Perl
were too dumb to implement it. It was a deliberate decision.
Tassilo
--
use bigint;
$n=71423350343770280161397026330337371139054411854220053437565440;
$m=-8,;;$_=$n&(0xff)<<$m,,$_>>=$m,,print+chr,,while(($m+=8)<=200);
------------------------------
Date: Wed, 17 Aug 2005 14:25:48 +1000
From: "Sisyphus" <sisyphus1@nomail.afraid.org>
Subject: Obtaining verbose info for http transfers.
Message-Id: <4302bc85$0$30086$afc38c87@news.optusnet.com.au>
Hi,
One of the nice things about Net::FTP is that if you run it with debugging
switched on, you get a report of the actual communication that's taking
place between the local box and the remote ftp server.
Is there a module that provides the same sort of report in relation to http
downloads ?
I have a satellite broadband connection, and when surfing the web I find
it's about as slow as my old 28kbps dial-up connection was. (There's no
problem when it comes to http or ftp downloads of large files - it's just
that in general surfing it's fairly slow.) I suspect this has something to
do with the latency involved in the passing of communications via satellite
between my PC and the remote web server - but I don't know how many times
messages are passed back and forth, and it would be informative if there was
a perl module that would dispay this info (if only in terms of a report
similar to that offered by Net::FTP).
It's not uncommon for my browser to take 15-20 seconds to download a web
page that doesn't contain a lot of data (a few small pictures, an
advertisement or 2, maybe more than one frame, but not much in terms of
overall data to be transferred) - which implies that if this is explained in
terms of the aforementioned latency, then there's quite a few messages going
back and forth. (A ping of a remote site takes about 1.3 seconds.)
Cheers,
Rob
--
To reply by email send to optusnet.com.au instead of nomail.afraid.org
------------------------------
Date: Wed, 17 Aug 2005 14:54:36 +1000
From: Simon Taylor <simon@unisolve.com.au>
Subject: Re: Obtaining verbose info for http transfers.
Message-Id: <ddug4i$1k90$1@otis.netspace.net.au>
Hello Rob,
> One of the nice things about Net::FTP is that if you run it with debugging
> switched on, you get a report of the actual communication that's taking
> place between the local box and the remote ftp server.
Yes, it's rather slick.
> Is there a module that provides the same sort of report in relation to http
> downloads ?
I've sometimes used the LWP GET command as follows:
GET -Sdxu http://www.yahoo.com.au
Regards,
Simon Taylor
------------------------------
Date: Wed, 17 Aug 2005 15:51:39 +1000
From: "Sisyphus" <sisyphus1@nomail.afraid.org>
Subject: Re: Obtaining verbose info for http transfers.
Message-Id: <4302d0a9$0$21235$afc38c87@news.optusnet.com.au>
"Simon Taylor" <simon@unisolve.com.au> wrote in message
news:ddug4i$1k90$1@otis.netspace.net.au...
> Hello Rob,
>
> > One of the nice things about Net::FTP is that if you run it with
debugging
> > switched on, you get a report of the actual communication that's taking
> > place between the local box and the remote ftp server.
>
> Yes, it's rather slick.
>
> > Is there a module that provides the same sort of report in relation to
http
> > downloads ?
>
> I've sometimes used the LWP GET command as follows:
>
> GET -Sdxu http://www.yahoo.com.au
>
That's a nix command, right ? It's probably the sort of thing I'm looking
for ... but I'm on Win32 :-)
I half expected that LWP::UserAgent or HTTP::Request/Response might
implement the verbosity I'm after since they obviously know all about the
http protocol, but I can't find anything in their docs that helps in that
regard.
I vaguely recall having used some sniffer type (non-perl) program a few
years back .... I might have to google that up again if there's no
ready-made perl solution.
Cheers,
Rob
------------------------------
Date: Wed, 17 Aug 2005 09:50:36 +0100
From: Brian Wakem <no@email.com>
Subject: Re: Obtaining verbose info for http transfers.
Message-Id: <3mgc2rF171coiU2@individual.net>
Sisyphus wrote:
> I half expected that LWP::UserAgent or HTTP::Request/Response might
> implement the verbosity I'm after
> Cheers,
> Rob
use LWP::Debug qw(+);
--
Brian Wakem
Email: http://homepage.ntlworld.com/b.wakem/myemail.png
------------------------------
Date: Wed, 17 Aug 2005 18:32:08 +0900
From: ko <kuujinbo@hotmail.com>
Subject: Re: Obtaining verbose info for http transfers.
Message-Id: <ddv06t$fq$1@pin3.tky.plala.or.jp>
Sisyphus wrote:
> Hi,
>
> One of the nice things about Net::FTP is that if you run it with debugging
> switched on, you get a report of the actual communication that's taking
> place between the local box and the remote ftp server.
>
> Is there a module that provides the same sort of report in relation to http
> downloads ?
>
> I have a satellite broadband connection, and when surfing the web I find
> it's about as slow as my old 28kbps dial-up connection was. (There's no
> problem when it comes to http or ftp downloads of large files - it's just
> that in general surfing it's fairly slow.) I suspect this has something to
> do with the latency involved in the passing of communications via satellite
> between my PC and the remote web server - but I don't know how many times
> messages are passed back and forth, and it would be informative if there was
> a perl module that would dispay this info (if only in terms of a report
> similar to that offered by Net::FTP).
>
> It's not uncommon for my browser to take 15-20 seconds to download a web
> page that doesn't contain a lot of data (a few small pictures, an
> advertisement or 2, maybe more than one frame, but not much in terms of
> overall data to be transferred) - which implies that if this is explained in
> terms of the aforementioned latency, then there's quite a few messages going
> back and forth. (A ping of a remote site takes about 1.3 seconds.)
>
> Cheers,
> Rob
>
For a start, how about something like this:
use strict;
use warnings;
use LWP;
my $ua = LWP::UserAgent->new(
requests_redirectable => [],
max_redirect => 100,
);
verbose_http('http://hotmail.com/');
sub verbose_http {
push my @urls, shift;
while (my $url = shift @urls) {
my $r = $ua->get($url);
if ($r->is_redirect) {
print $r->headers->as_string . "\n";
my $redirect = $r->header('Location');
push @urls, $redirect;
} elsif ($r->is_success) {
print $r->content . "\n";
}
}
}
__END__
HTH - keith
------------------------------
Date: Wed, 17 Aug 2005 06:16:07 +0200
From: Alexandre Jaquet <""alexjaquet\"@[no spam]msn.com">
Subject: perl and mysql random datas
Message-Id: <4302ba0d$0$1159$5402220f@news.sunrise.ch>
Hi,
I'm looking for a technique to display datas who came from my mysql db
randomly, but it's not quit simple because thoses datas have to match
specific criteria like an attributes is setted to true.
exemple :
article
id_article ...
have_payed
date
Then I want to extract datas where have_payed = 1. and I need to get a
navigation bar who display 4 items by page
here is my code without random
sub loadUserIndex {
local our $string = "";
local our $index = '0';
local our $total = '0';
local our ($c)= sqlSelectMany("id_article",
"article","have_payed = '1'");
local our $id_command;
while(($id_command)=$c->fetchrow()) {
$total +=1;
}
local our $nb_page = arrondi ($total / 4, 1);
local our $min_index = '0';
local our $max_index = '4';
for (local our $i = '0'; $i < $nb_page;$i++) {
$string .= "<a
href=\"/cgi-bin/recordz.cgi?lang=$lang&page=random&session=$session_id&min_index=$min_index&max_index=$max_index\"
class=\"menulink\" class=&{ns4class};><-$i-></a>  <img
src=\"../images/next2.gif\">";
$min_index += 4;
}
return $string;
}
sub loadUserByIndex {
local our $index_start = $query->param ("min_index");
$index_start =~ s/\W//g; ;
local our $index_end = $query->param ("max_index");
$index_end =~ s/\W//g; ;
if (!$index_start ) {
$index_start = 0;
}
if (!$index_end ) {
$index_end = 4;
}
local our ($c)= sqlSelectMany("id_article,name"
"article","have_payed = '1' LIMIT $index_start, $index_end");
local our $string = "";
while( ($ARTICLE{'id_article'},$ARTICLE{'name'})=$c->fetchrow()) {
$string .= "<tr><td><a
href=\"/cgi-bin/recordz.cgi?lang=$lang&session=$session_id&action=userdetail&name=$ARTICLE{'id_article'}</td><td>$ARTICLE{'name'}</td></tr>";
}
#$string .="</table>";
return $string;
}
I see not where and how can I get random "id_article" and display it
with limitt because they will not be ordered by "id_article"
thanks for any suggestion
------------------------------
Date: Wed, 17 Aug 2005 09:49:05 +0100
From: Brian Wakem <no@email.com>
Subject: Re: perl and mysql random datas
Message-Id: <3mgc00F171coiU1@individual.net>
Alexandre Jaquet wrote:
>
> I see not where and how can I get random "id_article" and display it
> with limitt because they will not be ordered by "id_article"
>
> thanks for any suggestion
You can't have it both ways. It can't be ordered and random.
What's wrong with mysql's ORDER BY RAND() ?
--
Brian Wakem
Email: http://homepage.ntlworld.com/b.wakem/myemail.png
------------------------------
Date: Wed, 17 Aug 2005 00:17:38 -0400
From: drew <drew@drew.net>
Subject: Re: Perl Solaris/Linux LASTLOG
Message-Id: <h8udnSE_e9g8JZ_eRVn-jA@comcast.com>
thank you... works like a charm.
RedGrittyBrick wrote:
> drew wrote:
>
>> i have to process a large amount of data and i cannot perform this on
>> the invidual servers for either security sake or the way we have been
>> setup.
>>
>>
>> here is the script from the PERL COOKBOOK to show the closest approach
>>
>> #!/usr/bin/perl
>> # laston - find out when given user last logged on
>> use User::pwent;
>> use IO::Seekable qw(SEEK_SET);
>>
>> open (LASTLOG, "/var/log/lastlog") or die "can't open
>> /usr/adm/lastlog: $!";
>>
>> $typedef = 'L A12 A16'; # linux fmt; sunos is "L A8 A16"
>> $sizeof = length(pack($typedef, ()));
>>
>> for $user (@ARGV) {
>> $U = ($user =~ /^\d+$/) ? getpwuid($user) : getpwnam($user);
>> unless ($U) { warn "no such uid $user\n"; next; }
>> seek(LASTLOG, $U->uid * $sizeof, SEEK_SET) or die "seek failed: $!";
>> read(LASTLOG, $buffer, $sizeof) == $sizeof or next;
>> ($time, $line, $host) = unpack($typedef, $buffer);
>> printf "%-8s UID %5d %s%s%s\n", $U->name, $U->uid,
>> $time ? ("at " . localtime($time)) : "never logged in",
>> $line && " on $line",
>> $host && " from $host";
>> }
>>
>>
>>
>>
>>
>>
>> i have geathered all the passwd files/lastlog file for each server
>>
>> how do i get getpwuid / getpwnam to work on the the password file that
>> i gathered and then work on the lastlog of the associated server?
>>
>
> I'd not use getpwuid and getpwnam. I'd read the passwd files, split the
> records and store the UID and Name in a hash keyed by Login-ID. Then I'd
> use '$name{$user}' in place of 'getpwnam($user)'
>
> I'd either keep each passwd and lastlog pair in its own directory named
> by server, or, I'd maybe prefix the filenames with servername and
> iterate over a predefined list of server names.
>
> YMMV.
------------------------------
Date: Wed, 17 Aug 2005 08:26:43 +0200
From: Babacio <babacio@free.fr>
Subject: Re: Problem with Curses
Message-Id: <m2iry59izg.fsf@baba.ba>
Babacio.
> Hi,
>
> Sorry if the question looks stupid to some of you...
>
> Here is a piece of code:
> ####################################
> use Curses;
>
> initscr();
> # here I should use curses
> endwin();
>
> print "Hello!\n";
> print "What do you say? ";
> $x=<STDIN>;
> print "You said $x\n";
> #####################################
>
> The functions initscr() and endwin() are to be use before and after
> doing stuff with curses...
>
> When I run it (on Mac OS X / darwin), I have the following problem :
> the text of the three prints does not appear until the end of the
> program, so I enter the value of $x with nothing printed, and
> after that the three line appear.
>
> (...)
Even if you don't have a solution, could you at least tell me if the
behaviour is the same on other systems (Linux, FreeBSD) ?
Thanks.
------------------------------
Date: Tue, 16 Aug 2005 22:44:55 GMT
From: Ed W <norealaddress@nowhere.com>
Subject: Simulating smaller MTU? ie sending small packets.
Message-Id: <G%tMe.36033$TK3.1918@fe05.news.easynews.com>
Hi, for various reasons I'm writing a little stress test app which tries
to simulate the effects of varying sized TCP packets on the overall
transfer speed.
So I have written a little app which acts as a server, waits for a
connection and then spews data in fixed sized chunks of your choice. I
also turn off nagle, turn on autoflush, and as far as I can tell ask for
the data to go out immediately
What I observe (using an ethernet dump) is that once the receiver is not
keeping up with the speed the sender is spewing packets, the *sender*
(which in this case is linux 2.6.12) is starting to coallesce the packets
So for example if I ask it to send 1000 byte packets I can see from the
network trace that it starts to send lots of MTU sized packets instead
(larger).
This is not what I was expecting at all, in fact I had no idea that
there was some clever process in linux to coallesce small network
packets? Am I tripping over some perl buffering instead? Any thoughts
on where to look?
Note, that it's not a mis-measurement problem at the receiving side. A
Network trace is showing me that the packets are coming out at MTU sized
(in general, but with a smattering of packets the size I requested).
If I slow down the sending rate, or speedup the receiver then the
packets go through at the correct size...
Grateful for any help trying to work around this
Ed W
------------------------------
Date: Tue, 16 Aug 2005 17:20:34 -0700
From: Jim Gibson <jgibson@mail.arc.nasa.gov>
Subject: Re: Simulating smaller MTU? ie sending small packets.
Message-Id: <160820051720348794%jgibson@mail.arc.nasa.gov>
In article <G%tMe.36033$TK3.1918@fe05.news.easynews.com>, Ed W
<norealaddress@nowhere.com> wrote:
> Hi, for various reasons I'm writing a little stress test app which tries
> to simulate the effects of varying sized TCP packets on the overall
> transfer speed.
>
> So I have written a little app which acts as a server, waits for a
> connection and then spews data in fixed sized chunks of your choice. I
> also turn off nagle, turn on autoflush, and as far as I can tell ask for
> the data to go out immediately
>
> What I observe (using an ethernet dump) is that once the receiver is not
> keeping up with the speed the sender is spewing packets, the *sender*
> (which in this case is linux 2.6.12) is starting to coallesce the packets
>
> So for example if I ask it to send 1000 byte packets I can see from the
> network trace that it starts to send lots of MTU sized packets instead
> (larger).
>
> This is not what I was expecting at all, in fact I had no idea that
> there was some clever process in linux to coallesce small network
> packets? Am I tripping over some perl buffering instead? Any thoughts
> on where to look?
>
> Note, that it's not a mis-measurement problem at the receiving side. A
> Network trace is showing me that the packets are coming out at MTU sized
> (in general, but with a smattering of packets the size I requested).
>
> If I slow down the sending rate, or speedup the receiver then the
> packets go through at the correct size...
>
> Grateful for any help trying to work around this
That behavior is exactly what you should expect from any intelligent
I/O process. It is probably a function of the TCP/IP software, not just
linux. Buffering should occur both on input (read a big chunk and
parcel out small chunks as needed to reader) and output (receive small
chunks from writer and stuff into output buffer until buffer can be
written).
Turning on autoflush may cause the I/O process to immediately attempt
to send less than a full buffer, but if something prevents the output
process from sending, it should still accept small chunks from the
writer and stuff them in the output buffer (until it, too, gets full --
at which point the writer should block).
The only work-around I can see is make sure that the readers operate
faster than the writers.
----== Posted via Newsfeeds.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= East/West-Coast Server Farms - Total Privacy via Encryption =---
------------------------------
Date: Wed, 17 Aug 2005 00:35:31 GMT
From: Ed W <norealaddress@nowhere.com>
Subject: Re: Simulating smaller MTU? ie sending small packets.
Message-Id: <nDvMe.35775$PI2.20970@fe03.news.easynews.com>
> The only work-around I can see is make sure that the readers operate
> faster than the writers.
I have nearly found a workaround. If I use the following:
setsockopt($sock, &Socket::IPPROTO_TCP, &Socket::TCP_MAXSEG, 500);
Then I can change the size of the MSS for the connection (which is the
effect I'm basically after).
The problem is that this only seems to be working if I use it on the
listening socket before I accept any connections. It doesn't seem to
work if I call it on the accepted connection
Looking at the C docs however, suggests that this *ought* to work at any
time... Likewise attempts to turn on and off TCP_CORK (for fun) aren't
working on the open connection and this is definitely supposed to be
possible
Any ideas what I am missing?
Thanks
Ed W
------------------------------
Date: Wed, 17 Aug 2005 08:47:44 +0200
From: "Tassilo v. Parseval" <tassilo.von.parseval@rwth-aachen.de>
Subject: Re: Simulating smaller MTU? ie sending small packets.
Message-Id: <slrndg5ncg.qn.tassilo.von.parseval@localhost.localdomain>
Also sprach Ed W:
> Hi, for various reasons I'm writing a little stress test app which tries
> to simulate the effects of varying sized TCP packets on the overall
> transfer speed.
This is probably a moot venture. The smaller the packets are, the lower
the overall throughput is going to be. This is due to the fact that TCP
packet have to be acknowledged. If no piggy-bagging is used (which is
the case for a pure-receiver side), then an additional 40 bytes (IP +
TCP minimum header size) need to be sent out on each acknowledgement.
The number of ACKs sent depend on the window size. Normally a receiver
tries to minimize ACKs and window size update messages (Clark).
> So I have written a little app which acts as a server, waits for a
> connection and then spews data in fixed sized chunks of your choice. I
> also turn off nagle, turn on autoflush, and as far as I can tell ask for
> the data to go out immediately
>
> What I observe (using an ethernet dump) is that once the receiver is not
> keeping up with the speed the sender is spewing packets, the *sender*
> (which in this case is linux 2.6.12) is starting to coallesce the packets
>
> So for example if I ask it to send 1000 byte packets I can see from the
> network trace that it starts to send lots of MTU sized packets instead
> (larger).
But it's probably going to send these larger packets at a lower rate.
Did you also check the ACK packets from the receiver? The Nagle
algorithm tells the sender never to send small packets. Turning it off
means sending them immediately as long as the receiver's side can keep
up. Now, if the receiver is congested, I would assume that the sender
still buffers small packets and once an ACK packet arrives it sends out
as many data as there is space in the receiving window.
> This is not what I was expecting at all, in fact I had no idea that
> there was some clever process in linux to coallesce small network
> packets? Am I tripping over some perl buffering instead? Any thoughts
> on where to look?
No, you're tripping over a sane implementation of the TCP stack. TCP by
nature is slow and has some overhead which is reduced by various means,
most notably the Nagle (sender) and Clark (receiver) algorithms.
Furthermore, in order to avoid clogging the subnet between sender and
receiver, congestion control is carried out (see TCP slow start
algorithm).
> Note, that it's not a mis-measurement problem at the receiving side. A
> Network trace is showing me that the packets are coming out at MTU sized
> (in general, but with a smattering of packets the size I requested).
>
> If I slow down the sending rate, or speedup the receiver then the
> packets go through at the correct size...
In order to do your measurements, you should probably adjust parameters
on the receiving side. If you want smaller packets, try to set the
window size (TCP_WINDOW_CLAMP, I think). TCP_MAX_SEG also needs to be
set there as the MSS is announced by the receiver during the
three-way-handshake when the connection is established.
Tassilo
--
use bigint;
$n=71423350343770280161397026330337371139054411854220053437565440;
$m=-8,;;$_=$n&(0xff)<<$m,,$_>>=$m,,print+chr,,while(($m+=8)<=200);
------------------------------
Date: Wed, 17 Aug 2005 07:53:58 GMT
From: Ed W <norealaddress@nowhere.com>
Subject: Re: Simulating smaller MTU? ie sending small packets.
Message-Id: <p2CMe.259355$GX1.138454@fe01.news.easynews.com>
> This is probably a moot venture. The smaller the packets are, the lower
> the overall throughput is going to be. This is due to the fact that TCP
> packet have to be acknowledged.
Be careful with your generalisation. The point of my experiment is to
test an unreliable (and very slow) satellite network to determine
whether faster speed would be achieved using smaller MTU due to less
retranmissions. 1500 bytes represents up to 7 seconds of transmission
time...
> In order to do your measurements, you should probably adjust parameters
> on the receiving side. If you want smaller packets, try to set the
> window size (TCP_WINDOW_CLAMP, I think). TCP_MAX_SEG also needs to be
> set there as the MSS is announced by the receiver during the
> three-way-handshake when the connection is established.
I'm not sure I can see how window size affects things, but it's
interesting to see that I can influence it on a per connection basis?
I'm trying to change TCP_MAX_SEG and the docs imply it can be changed
once the connection is established, but at least using perl this doesn't
appear to work.
If I change it on a listening socket then I observe that the subsequent
tcp handshake uses the original max values, but that TCP then uses the
smaller values for sending data (ie it does what I expect). It would
just be useful to be able to change the MSS while the connection is
operating
It might for example be useful to change the MSS if we observe more
corrupted tcp packets arriving, or other similar algorithm.
Also, is it possible to observe how full the network buffers are?
getsockopt(xxx)? Again, it might be useful to observe this value in the
situation above and slow down sending when the buffers are filling up
(for example with these huge latencies I might want to have more control
over the amount of outstanding data)
Any thoughts?
Thanks
Ed W
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 8327
***************************************
