[610] in Info-AFS_Redistribution
Re: NFS vs. AFS builds
daemon@ATHENA.MIT.EDU (Craig Anderson)
Thu Feb 13 17:58:35 1992
Date: Thu, 13 Feb 92 14:14:59
From: craig@aixwiz.austin.ibm.com (Craig Anderson)
To: Lyle_Seaman@transarc.com, dstokes@sunlight.austin.ibm.com
Cc: info-afs@transarc.com
Ok, like i thought i'd see about this stat performance everyone was talking
about.... Looks like it takes 15 times longer to stat a file on the AFS
source tree than on the NFS tree...
LOCAL:
% timex ./stat /usr/include/stdio.h 10000
real 4.60
user 0.40
sys 3.67
NFS:
# mount speedo:/SOURCE/2007/bos/prod/src /bos/3.1.2/2007
% timex ./stat /bos/3.1.2/2007/com/cmd/s/auth/tsmlogin.c 10000
real 8.40
user 0.40
sys 6.74
AFS: (no options to afsd)
%timex ./stat /afs/austin.ibm.com/aix/312/3005/bos/prod\
/src/com/cmd/s/auth/tsmlogin.c 10000
real 1:58.51
user 0.52
sys 1:45.48
stat:
% cat stat.c
#include <sys/types.h>
#include <sys/stat.h>
main(argc, argv)
int argc;
char *argv[];
{
struct stat buf;
int i = atoi(argv[2]);
while (i--)
if (stat(argv[1], &buf) < 0)
perror(argv[1]), exit(2);
exit(0);
}
WOW! I don't understand this... The first tests i ran were just against
some other files weren't so bad...
% timex ./stat /afs/austin/aix 10000
real 18.90
user 0.18
sys 18.31
What's happening here? /afs/austin.ibm.com/aix is in another building (i
don't even have a clue where it is, the stuff in .../312/3005/... is on
the same machine in the lab that the NFS code gets mounted from). Ok,
i can understand that i have to talk to a couple of servers for each stat,
but why does that consume so much CPU time?
I've been mulling around the idea that the (*x->vn_lookup)() path is
inefficient - i don't know if that's consistent with the improved times
using -memcache... i wish we had the source... so i try this:
AFS:
% timex ./stat /afs/austin/u4/craig/a 10000
real 26.49
user 0.46
sys 24.76
and then this:
% timex ./stat /afs/austin/u4/craig/a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z 10000
real 4:30.20
user 0.38
sys 4:00.72
WOW! ok but to be fair this isn't just a problem for AFS... i run it locally...
LOCAL:
% timex ./stat /tmp/austin/u4/craig/a 10000
real 5.55
user 0.32
sys 5.09
% timex ./stat /tmp/austin/u4/craig/a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z 10000
real 22.66
user 0.26
sys 20.98
Ok JFS is about 4 times slower for the long path. AFS degrades even
faster - it was more like 10 times slower for the long path.
OK, next i test open() performance... i expect this to be similar... it is.
It gives the same drop off in performance relative to path name length as
stat(). It takes very similar times. Looks more and more like work needs
to be done in (*x->vn_lookup)() all the time...
What do you think? Any ideas on how we can speed up our builds now?
Maybe we shouldn't let the directories get too deep?
-Craig
What follows was written before i ran the above tests...
------------------------------------------------------------------------------
>>>what is lm? I assume it's some kind of make variant, but without
>lm stands for libmake. You are correct in assuming it is a variant of make.
It preprocesses the source to determine what include files it depends on
and then builds a complete dependency tree. Then it does the usual make
stuff. The NFS guys assure me that this involves billions and billions
of stats. Oh yeah, it also reads a file which contains a list of
directories that also might contain the source. lm looks in each of
these directories and finds the newest version of the source. Typically
this is about 10 directories. Hence lm does more stats than you can
possibly imagine since it has to look for each file in 10 places...
>>>4. What settings are you using for the afsd parameters, especially
>>> -files, -dcache and -stat?
>The default afsd parameters are being used.
Actually i've moved -files around from 64 to 7000 (slight performance
degredation with the high numbers.) I've played with -stat from -64
to -4096 with apparently no effect. I've used -memcache and -dcache
from 512 to 8192 (888's with 8192 - probably just ran out of paging space.)
Both 512 or 1024 improved system CPU time usage by about 35%.
COMPLETE BUILD TIMES for rsdd (timex lm FORCE=FORCE list=list rsdd).
Real User Sys Comments
-------------------------------------------------------------------------------
NFS 1st time 4:15.01 2:15.38 25.71 ~64% CPU utilizaton
2nd time 4:04.51 2:15.29 26.31 ~66% CPU utilizaton
With /usr/vice/etc/afsd
AFS 1st time 7:54.13 2:42.61 3:49.02 ~82% CPU utilizaton
2nd time 7:54.49 2:42.77 3:48.24 ~82% CPU utilizaton
With /usr/vice/etc/afsd -memcache -dcache 512
AFS 6:37.68 2:41.44 2:33.57
With /usr/vice/etc/afsd -files 7000 -stat 3000
AFS 17:18.22 3:16.88 4:04.23
AFS 7:52.80 2:43.67 3:48.75
With /usr/vice/etc/afsd -files 64 -stat 4096
AFS 12:35.58 2:42.41 4:15.37
AFS 9:46.00 2:43.91 3:54.16
With /usr/vice/etc/afsd -memcache -dcache 1024 -files 64 -stat 128
AFS 6:25.41 2:42.54 2:30.59
AFS 6:16.71 2:41.70 2:32.04
-------------------------------------------------------------------------------
Note that for AFS the System time is being burned by the 'lm'
dependency determining process (~ 95% CPU utilization). This is what
concerns me. AFS is definately doing something inefficient. My first
guess would be down the (*vn_lookup)() path - i don't know if that's
consistent with the improved times using -memcache... i wish we had
the source...
I'll ask Dawn for the tools so i can get some debugging info. Are there
any trace hooks in this code? It would be easy to profile the code by
using trace if there were...
-Craig
P.S.: It sure would be nice if AFS wouldn't sleep with signals ignored.
This was fixed in NFS (so you can ^C out even when the hard mount option
is used - it's really nice...)
