[26136] in Perl-Users-Digest
Perl-Users Digest, Issue: 8328 Volume: 10
daemon@ATHENA.MIT.EDU (Perl-Users Digest)
Wed Aug 17 11:05:18 2005
Date: Wed, 17 Aug 2005 08:05:05 -0700 (PDT)
From: Perl-Users Digest <Perl-Users-Request@ruby.OCE.ORST.EDU>
To: Perl-Users@ruby.OCE.ORST.EDU (Perl-Users Digest)
Perl-Users Digest Wed, 17 Aug 2005 Volume: 10 Number: 8328
Today's topics:
detecting pod in a file? <mikee@mikee.ath.cx>
Extraction of bits using unpack <ihate@spam.com>
Re: Extraction of bits using unpack (Anno Siegel)
Re: Extraction of bits using unpack <ihate@spam.com>
move information from HTTP::Request to CGI.pm <miroslav@suchy.cz>
Re: Obtaining verbose info for http transfers. <tadmc@augustmail.com>
Re: Problem with Curses <scobloke2@infotop.co.uk>
Re: Simulating smaller MTU? ie sending small packets. <tassilo.von.parseval@rwth-aachen.de>
Re: Simulating smaller MTU? ie sending small packets. <flavell@ph.gla.ac.uk>
Re: Simulating smaller MTU? ie sending small packets. <norealaddress@nowhere.com>
Re: Simulating smaller MTU? ie sending small packets. <norealaddress@nowhere.com>
Digest Administrivia (Last modified: 6 Apr 01) (Perl-Users-Digest Admin)
----------------------------------------------------------------------
Date: Wed, 17 Aug 2005 14:07:24 GMT
From: Mike <mikee@mikee.ath.cx>
Subject: detecting pod in a file?
Message-Id: <wwHMe.6801$rc6.6132@fe03.lga>
I want to detect if pod is embedded into a file or not.
Is there an easier way to detect the embedded pod other
than the sample code below?
Mike
my $file = '/tmp/file';
open(IN, "<$file") or die "$0: cannot read file '$file': $!";
my $pod = 1 if scalar(grep(/^=pod/o, <IN>)) > 0;
close(IN);
pod2html($file) if $pod;
------------------------------
Date: Wed, 17 Aug 2005 10:08:52 +0000 (UTC)
From: pip <ihate@spam.com>
Subject: Extraction of bits using unpack
Message-Id: <87k6ikc1u7.fsf@readonly.co.uk>
Hi all,
I am trying to extract some data from a binary file using unpack. All
is well where I have to extract simple data types such as shorts,
strings and ints but then I reach this in the docs I am using:
Here is the way a date is represented:
MSB LSB
__________________________________________________________
| year | Month | Day | Time |
| 12 bits | 4 bits | 5 bits | 11 bits |
|__________|____________|___________|____________________|
I have played with the hex string operator in unpack with little
joy. I also had a go at it with vec() but got nowhere.
Could someone please point me in the right direction for extracting
this data please?
Thanks,
pip
P.S if it makes a difference I am pulling the file from an x86 based
system and parsing it on an x86 system.
------------------------------
Date: 17 Aug 2005 11:04:44 GMT
From: anno4000@lublin.zrz.tu-berlin.de (Anno Siegel)
Subject: Re: Extraction of bits using unpack
Message-Id: <ddv5kc$pe1$1@mamenchi.zrz.TU-Berlin.DE>
pip <ihate@spam.com> wrote in comp.lang.perl.misc:
> Hi all,
>
> I am trying to extract some data from a binary file using unpack. All
> is well where I have to extract simple data types such as shorts,
> strings and ints but then I reach this in the docs I am using:
>
> Here is the way a date is represented:
>
> MSB LSB
> __________________________________________________________
> | year | Month | Day | Time |
> | 12 bits | 4 bits | 5 bits | 11 bits |
> |__________|____________|___________|____________________|
>
> I have played with the hex string operator in unpack with little
> joy. I also had a go at it with vec() but got nowhere.
>
> Could someone please point me in the right direction for extracting
> this data please?
You want the usual shifting and masking of integers, it seems. Get
the four bytes into an integer $x (unpack), and apply
my( $year, $month, $day, $time) = bit_split( $x, 12, 4, 5, 11);
with
sub bit_split {
my $bits = shift;
my @parts;
for ( reverse @_ ) {
my $mask = ( 1 << $_) - 1;
unshift @parts, $mask & $bits;
$bits >>= $_;
}
@parts;
}
Anno
--
If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the
"Reply" at the bottom of the article headers.
------------------------------
Date: Wed, 17 Aug 2005 11:19:05 +0000 (UTC)
From: pip <ihate@spam.com>
Subject: Re: Extraction of bits using unpack
Message-Id: <8764u4izec.fsf@readonly.co.uk>
Anno,
> pip <ihate@spam.com> wrote in comp.lang.perl.misc:
>> Hi all,
>>
>> I am trying to extract some data from a binary file using unpack. All
>> is well where I have to extract simple data types such as shorts,
>> strings and ints but then I reach this in the docs I am using:
>> [ ... snip ... ]
>
> my( $year, $month, $day, $time) = bit_split( $x, 12, 4, 5, 11);
>
> with
>
> sub bit_split {
> my $bits = shift;
> my @parts;
> for ( reverse @_ ) {
> my $mask = ( 1 << $_) - 1;
> unshift @parts, $mask & $bits;
> $bits >>= $_;
> }
> @parts;
> }
This worked a charm, I would never have worked this out without your
help. Now to go a read up on this bit shifting malarkey.
Cheers,
pip
------------------------------
Date: Wed, 17 Aug 2005 16:29:10 +0200
From: Miroslav Suchy <miroslav@suchy.cz>
Subject: move information from HTTP::Request to CGI.pm
Message-Id: <ddvhpb$vn9$1@mrazik2.dkm.cz>
Hello,
I have following code:
my $daemon = HTTP::Daemon->new(
ReuseAddr => 1,
LocalAddr => 'mydomain.com',
LocalPort => 8888,
) or die "Cannot start daemon: $!\n";
while (my $conn = $daemon->accept()) {
while (my $req = $conn->get_request()) {
if ($req->method() eq 'GET') { # I do not use GET
$q = CGI->new($url->query());
} elsif ($req->method() eq 'POST') { # I only use POST method
$q = CGI->new($req->content);
open FF, ">/tmp/debug"; #output to FF just for debug
print FF $req->content;
close FF;
}
print STDERR "params: ", join(',', $q->param), "\n";
}
}
And it does not work as I expect. This is content of file "/tmp/debug":
-----------------------------18004688081790386843384315354
Content-Disposition: form-data; name="id"
6
-----------------------------18004688081790386843384315354
Content-Disposition: form-data; name="pw"
aa
-----------------------------18004688081790386843384315354--
I expect following output from script:
params: id,pw
but I get this output:
params: -----------------------------18004688081790386843384315354
Content-Disposition: form-data, name
What I'm doing wrong? Thank you.
Miroslav Suchy
------------------------------
Date: Wed, 17 Aug 2005 07:55:33 -0500
From: Tad McClellan <tadmc@augustmail.com>
Subject: Re: Obtaining verbose info for http transfers.
Message-Id: <slrndg6cu5.l8o.tadmc@magna.augustmail.com>
Sisyphus <sisyphus1@nomail.afraid.org> wrote:
> One of the nice things about Net::FTP is that if you run it with debugging
> switched on, you get a report of the actual communication that's taking
> place between the local box and the remote ftp server.
>
> Is there a module that provides the same sort of report in relation to http
> downloads ?
Web Scraping Proxy
http://www.research.att.com/~hpk/wsp/
> I have a satellite broadband connection, and when surfing the web I find
> it's about as slow as my old 28kbps dial-up connection was. (There's no
> problem when it comes to http or ftp downloads of large files - it's just
> that in general surfing it's fairly slow.)
Spyware would provide those symptoms too...
--
Tad McClellan SGML consulting
tadmc@augustmail.com Perl programming
Fort Worth, Texas
------------------------------
Date: Wed, 17 Aug 2005 10:52:16 +0000 (UTC)
From: Ian Wilson <scobloke2@infotop.co.uk>
Subject: Re: Problem with Curses
Message-Id: <ddv4t0$ffe$1@nwrdmz01.dmz.ncs.ea.ibs-infra.bt.com>
Babacio wrote:
> Babacio.
>
>
>>Hi,
>>
>>Sorry if the question looks stupid to some of you...
>>
>>Here is a piece of code:
>>####################################
>>use Curses;
>>
>>initscr();
>># here I should use curses
>>endwin();
>>
>>print "Hello!\n";
>>print "What do you say? ";
>>$x=<STDIN>;
>>print "You said $x\n";
>>#####################################
>>
>>The functions initscr() and endwin() are to be use before and after
>>doing stuff with curses...
>>
>>When I run it (on Mac OS X / darwin), I have the following problem :
>>the text of the three prints does not appear until the end of the
>>program, so I enter the value of $x with nothing printed, and
>>after that the three line appear.
>>
>>(...)
>
>
> Even if you don't have a solution, could you at least tell me if the
> behaviour is the same on other systems (Linux, FreeBSD) ?
$ uname
Linux
$ ./curses.pl
Can't locate curses.pm in @INC
Maybe you should say where you got Curses.pm and which version you are
using?
------------------------------
Date: Wed, 17 Aug 2005 12:05:15 +0200
From: "Tassilo v. Parseval" <tassilo.von.parseval@rwth-aachen.de>
Subject: Re: Simulating smaller MTU? ie sending small packets.
Message-Id: <slrndg62ur.2mu.tassilo.von.parseval@localhost.localdomain>
Also sprach Ed W:
>> This is probably a moot venture. The smaller the packets are, the lower
>> the overall throughput is going to be. This is due to the fact that TCP
>> packet have to be acknowledged.
>
> Be careful with your generalisation. The point of my experiment is to
> test an unreliable (and very slow) satellite network to determine
> whether faster speed would be achieved using smaller MTU due to less
> retranmissions. 1500 bytes represents up to 7 seconds of transmission
> time...
Ah, there we are! TCP design was heavily based on the assumption of
using wires on the physical layer (which means: the medium is fairly
reliable). I made the same assumption. :-)
TCP over a wireless link is a completely different story. The current
design still works but it can be horrendously inefficient, especially
congestion control.
>> In order to do your measurements, you should probably adjust parameters
>> on the receiving side. If you want smaller packets, try to set the
>> window size (TCP_WINDOW_CLAMP, I think). TCP_MAX_SEG also needs to be
>> set there as the MSS is announced by the receiver during the
>> three-way-handshake when the connection is established.
>
> I'm not sure I can see how window size affects things, but it's
> interesting to see that I can influence it on a per connection basis?
Yes, per connection. The window size is a parameter sent with each ACK
message and it refers to the currently remaining size of the receiving
buffer. This is normally handled by the TCP stack. It is not a global
parameter you set once and then it remains there. But from the
description in tcp(7) I assume that TCP_WINDOW_CLAMP is an upper bound
and no sent packet may ever exceed it.
> I'm trying to change TCP_MAX_SEG and the docs imply it can be changed
> once the connection is established, but at least using perl this doesn't
> appear to work.
TCP_MAXSEG actually. To what value did you set it? If it exceeds the MTU
of your interface, the value is ignored. Also, the minimum size is, I
think, 556 which is a TCP requirement.
> If I change it on a listening socket then I observe that the subsequent
> tcp handshake uses the original max values, but that TCP then uses the
> smaller values for sending data (ie it does what I expect). It would
> just be useful to be able to change the MSS while the connection is
> operating
As far as I know this is not in the specifications of TCP. The segment
size is agreed on during connection establishment (actually, both sides
may use different values for the MSS).
> It might for example be useful to change the MSS if we observe more
> corrupted tcp packets arriving, or other similar algorithm.
The only way to do that is to shutdown the connection and establish a
new one with the updated values for the MSS.
> Also, is it possible to observe how full the network buffers are?
> getsockopt(xxx)? Again, it might be useful to observe this value in the
> situation above and slow down sending when the buffers are filling up
> (for example with these huge latencies I might want to have more control
> over the amount of outstanding data)
With a packet-sniffer, you certainly can. You find it in the window size
field of the TCP header that is sent back as acknowledgement from the
receiver. But as I said earlier, this is a dynamic value so I don't
think you can find that value on a per-socket basis. There is TCP_INFO
that returns some values. But the structure returned seems to have no
information on the last state of the receiver's window.
Since you are trying to optimize a TCP connection over a wireless link,
are you sure at all that reducing the packet size is a good idea? The
problem with wireless links is their lack of reliability (and latency).
When data get through corrupted (or not at all) a approach is to send
again immediately in the hope that it gets through this time. Also,
making packages smaller does not necessarily make the transmission more
reliable. Wireless is really just send-and-pray.
However, if you have problems with the buffer of the receiver filling up
too quickly, wouldn't that mean that the data got through beautifully?
If the receiver constantly has full buffers, it means the link is in
fact quite reliable and thus making it look similar to a wired link. In
this case you can just rely on the default behaviour of your TCP stacks
as they work well for reliable links.
Tassilo
--
use bigint;
$n=71423350343770280161397026330337371139054411854220053437565440;
$m=-8,;;$_=$n&(0xff)<<$m,,$_>>=$m,,print+chr,,while(($m+=8)<=200);
------------------------------
Date: Wed, 17 Aug 2005 11:30:38 +0100
From: "Alan J. Flavell" <flavell@ph.gla.ac.uk>
Subject: Re: Simulating smaller MTU? ie sending small packets.
Message-Id: <Pine.LNX.4.62.0508171118220.4050@ppepc56.ph.gla.ac.uk>
On Wed, 17 Aug 2005, Ed W wrote:
> > This is probably a moot venture. The smaller the packets are, the
> > lower the overall throughput is going to be. This is due to the
> > fact that TCP packet have to be acknowledged.
>
> Be careful with your generalisation. The point of my experiment is
> to test an unreliable (and very slow) satellite network to determine
> whether faster speed would be achieved using smaller MTU due to less
> retranmissions. 1500 bytes represents up to 7 seconds of
> transmission time...
Yes, but the acknowledgement doesn't have to be for every individual
packet. Check the "window" parameter. You may however need some very
large buffers if you hope to improve performance. One sees a similar
effect without the satellite, if trying to get good bulk data
throughtput on a transatlantic cable link: despite having at least
1Gbit/sec paths at all points between the hosts at each end, and the
hosts themselves being adequate to the purpose, the throughput looks
quite miserable unless some serious tuning of the TCP parameters is
done.
However, this would be better explored on a networking group, I think,
than right here on c.l.p.misc.
And google for * tcp tuning throughput * and similar combinations of
terms. Anything recent-ish which comes back with LBL.gov and/or
internet2 in the URL is likely to be worth a look.
> I'm not sure I can see how window size affects things, but it's
> interesting to see that I can influence it on a per connection
> basis?
The acknowledgements are serial numbered as to which packets they
relate to, so you can be acknowledging a packet which was quite some
time back while you have all the intervening packets "up the spout" or
in transit, at least for the major part of the transfer (at the ends
of course it sorts itself out).
hope this helps.
------------------------------
Date: Wed, 17 Aug 2005 11:11:24 GMT
From: Ed W <norealaddress@nowhere.com>
Subject: Re: Simulating smaller MTU? ie sending small packets.
Message-Id: <43031B65.7000507@nowhere.com>
You may however need some very
> large buffers if you hope to improve performance. One sees a similar
> effect without the satellite, if trying to get good bulk data
> throughtput on a transatlantic cable link: despite having at least
> 1Gbit/sec paths at all points between the hosts at each end, and the
> hosts themselves being adequate to the purpose, the throughput looks
> quite miserable unless some serious tuning of the TCP parameters is
> done.
I think you overestimate the problem. I am using Iridium...
On a clear day with no clouds, the satellite overhead and a following
wind it can do 2400 baud...
Retransmissions are what I need to reduce
Ed W
------------------------------
Date: Wed, 17 Aug 2005 11:24:22 GMT
From: Ed W <norealaddress@nowhere.com>
Subject: Re: Simulating smaller MTU? ie sending small packets.
Message-Id: <F7FMe.51011$7S7.9126@fe10.news.easynews.com>
> TCP_MAXSEG actually. To what value did you set it? If it exceeds the MTU
> of your interface, the value is ignored. Also, the minimum size is, I
> think, 556 which is a TCP requirement.
The docs say that you can use it to reduce the MSS over what was
negotiated at the link establishment. This seems to be the case I can
see the MTU being established at 1500 using a packet sniffer, but
setting TCP_MAXSEG then means packets go out at (say) 500 bytes
I haven't found an issue under Linux setting values from 300 bytes to
1400 bytes, so I don't know if there is any limit?
>>If I change it on a listening socket then I observe that the subsequent
>>tcp handshake uses the original max values, but that TCP then uses the
>>smaller values for sending data (ie it does what I expect). It would
>>just be useful to be able to change the MSS while the connection is
>>operating
>
>
> As far as I know this is not in the specifications of TCP. The segment
> size is agreed on during connection establishment (actually, both sides
> may use different values for the MSS).
It seems obvious though that there is no technical reason that we can't
agree on 1400 bytes being the largest we are allowed to send to the
remote and then sending 700 byte packets instead
My understanding is that this param can be changed at runtime. Also
there is TCP_CORK which is supposed to optimally pack data into packets
- again I can't seem to toggle this using the perl code
>>It might for example be useful to change the MSS if we observe more
>>corrupted tcp packets arriving, or other similar algorithm.
>
>
> The only way to do that is to shutdown the connection and establish a
> new one with the updated values for the MSS.
The docs imply not?
> Since you are trying to optimize a TCP connection over a wireless link,
> are you sure at all that reducing the packet size is a good idea? The
> problem with wireless links is their lack of reliability (and latency).
> When data get through corrupted (or not at all) a approach is to send
> again immediately in the hope that it gets through this time. Also,
> making packages smaller does not necessarily make the transmission more
> reliable. Wireless is really just send-and-pray.
I observe several to a few dozen random errors per minute. If you do a
little quick maths I think you can easily see that packets taking 7 secs
each to send (and retransmit in case of error) are much less efficient
than say 1 second packets.
I have just done some testing and this is very much bourne out practice
with the smaller packets (I tested at 300, 500 and 600 bytes) suffer
only very small amounts of slowdown and are much more robust in the case
of retransmit. I think a quick model in Excel would also show that this
is the case?
> However, if you have problems with the buffer of the receiver filling up
> too quickly, wouldn't that mean that the data got through beautifully?
> If the receiver constantly has full buffers, it means the link is in
> fact quite reliable and thus making it look similar to a wired link.
Wrong buffer. I'm worried about the sending buffer at my end. I can't
see how the remote buffer would ever be anything but empty since that
would imply the application was sleeping on the job and not consuming
the transmitted input
In my case I was to keep the untransmitted data as small as possible.
Over my satellite system if I transmit 65Kb, and then later some urgent
data comes along it ends up on the tail of the other data. This means
that some minutes will pass by before I can even start to get the urgent
data out.
This is also compounded at the remote ISP end which might have several
megabyte buffers and be queuing up tons of data to squeeze down this
tiny pipe. Obviously I can't control any QOS settings at the remote end
because it's not under my control
Anyway, we are off track. What I really need to do now is figure out
how to control the sending packet size effectively. Variable MSS will
be a big help if it can be done?
Ed W
In
> this case you can just rely on the default behaviour of your TCP stacks
> as they work well for reliable links.
>
> Tassilo
------------------------------
Date: 6 Apr 2001 21:33:47 GMT (Last modified)
From: Perl-Users-Request@ruby.oce.orst.edu (Perl-Users-Digest Admin)
Subject: Digest Administrivia (Last modified: 6 Apr 01)
Message-Id: <null>
Administrivia:
#The Perl-Users Digest is a retransmission of the USENET newsgroup
#comp.lang.perl.misc. For subscription or unsubscription requests, send
#the single line:
#
# subscribe perl-users
#or:
# unsubscribe perl-users
#
#to almanac@ruby.oce.orst.edu.
NOTE: due to the current flood of worm email banging on ruby, the smtp
server on ruby has been shut off until further notice.
To submit articles to comp.lang.perl.announce, send your article to
clpa@perl.com.
#To request back copies (available for a week or so), send your request
#to almanac@ruby.oce.orst.edu with the command "send perl-users x.y",
#where x is the volume number and y is the issue number.
#For other requests pertaining to the digest, send mail to
#perl-users-request@ruby.oce.orst.edu. Do not waste your time or mine
#sending perl questions to the -request address, I don't have time to
#answer them even if I did know the answer.
------------------------------
End of Perl-Users Digest V10 Issue 8328
***************************************
