[16528] in Athena Bugs
Re: NFS parameters in attach.conf
daemon@ATHENA.MIT.EDU (Theodore Y. Ts'o)
Thu Dec 3 13:10:06 1998
Date: Thu, 3 Dec 1998 13:10:02 -0500
From: "Theodore Y. Ts'o" <tytso@MIT.EDU>
To: "Thomas H. Grayson" <thg@MIT.EDU>
Cc: bugs@MIT.EDU, network@MIT.EDU, f_l@MIT.EDU, karen@MIT.EDU, jf@MIT.EDU,
phils@MIT.EDU
In-Reply-To: Thomas H. Grayson's message of Wed, 02 Dec 1998 17:36:25 EST,
<199812022236.RAA22057@m1-115-7.mit.edu>
Date: Wed, 02 Dec 1998 17:36:25 EST
From: "Thomas H. Grayson" <thg@MIT.EDU>
options {nfs}:.* rsize=1024,wsize=1024,soft
The 'rsize' and 'wsize' parameters refer to the size in bytes of
the read and write buffers. These 1K buffer sizes are very
small. The Solaris man page for 'mount_nfs' indicates that the
defaults for both parameters are 8192 for NFS V2 and 32768 for
NFS V3. These low parameters have a significant negative impact
on NFS performance.
The rsize and wsize parameters refer to size of the packets actually
sent out by NFS when reading and writing. When you try to send a UDP
datagram larger than the size of the MTU of the network, the packet has
to get fragmented into packets which are smaller than the MTU. The
problem with packet fragmentation is that on a busy network, one of the
fragments may get dropped by the router. When that happens, *all* of
the fragments have to get retransmitted, thus further increasing the
load on the network and on the router.
The rsize and wsize were set explicitly to be 1024 because before we did
that, the default was 8192, which works very well on the local Ethernet
LAN. (And in fact, Sun tuned their defaults assuming that the clients
and servers were on the same LAN; that's why Sun also turned off UDP
checksuming by default --- it made the benchmarks look better, and
*usually* packets wouldn't get corrupted when they went across a local
ethernet....)
However, if the packets had to traverse the backbone of the network,
even if only 1 of the 8 fragments got dropped, all 8 packets had to be
retransmitted. The kernel on the NFS server would also waste kernel
memory holding the 7 fragments, and wait in vain for the 8th fragment to
appear, until finally the kernel gave up on the packet and threw away
the uncompleted packet. So under heavy load, UDP fragmentation makes
the loading problem catastrophically worse, both for the network router
and the NFS server.
Hence, setting the rsize and wsize to 1024 bytes was something which
*improved* the performance, not degraded it. Now that being said, it's
possible that our routers have improved enough since when we made this
decision that we can't get away with fragmenting the UDP packets. But I
would be very, very, carefully before making this conclusion ---
especially without doing lots of tests.
I then attached the 'arcinfo' NFS locker, and launched ArcView.
Since this executable is nearly 11 MB, it is a reasonable test of
NFS performance. On this modified workstation, ArcView launched
in about 42 seconds. On a similar workstation with the default
parameters, ArcView launched in about 99 seconds. Hence, with a
simple configuration change I decreased the program's launch time
by about 58%.
Did you do this test with the clients and the servers on the same
network, or on different networks? If you only did the test when the
client and the server were on the same subnet, it probably wasn't a a
good test of whether or not this will work in general. What works in
one scenario may make things worse in others.
For the curious, my experiment was conducted in the Electronic
Classroom 1-115 using two of the Sun Ultra 10 workstations
located there. The room was not being heavily used at the time.
While I tested only on Suns, the same attach.conf parameters are
used on other platforms.
This probably means that more tests need to be done before concluding
that something like this would be a wise thing to do.
- Ted
