[119] in Info-AFS_Redistribution
Re: Large AFS cache
daemon@ATHENA.MIT.EDU (Tony_Mason@transarc.com)
Mon May 13 09:07:59 1991
Date: Mon, 13 May 1991 07:39:56 -0400 (EDT)
From: Tony_Mason@transarc.com
To: Info-AFS@transarc.com, Keith Gorlen <Keith_Gorlen@alw.nih.gov>
Cc: Pierette_Maniago@transarc.com
In-Reply-To: <Yc_jtti0ts0t029Lxg@alw.nih.gov>
Here's a general description of how cache manager tuning works in
general; I'll use what you've told me later to demonstrate what seems
reasonable. Upon initialization the cache manager allocates disk &
memory resources based on six variables: stat cache size, number of
cache files, number of cache blocks, number of disk cache entries,
chunk size, and whether or not the cache manager is running a
memory-based cache.
When figuring out what values to use here it's important to note that
these variables aren't orthogonal. It is possible to set one variable
very high but have the resource still be under-utilized because the
other variables are very low. The default values for these variables
are:
# of stat cache entries = 300
# of cache files = 1000
# of cache blocks = from /usr/vice/cacheinfo
# of disk cache entries = 100
# of volume cache entries = 50
chunk size = 64K
memory cache flag = false
The # of cache blocks is typically the most well-known factor when
setting up the cache; the others affect performance significantly,
however. The minimum number of cache files should be:
[1] ceiling (# of cache blocks / chunk size in K)
to even be able to potentially use all of the available cache space.
The default is to set this to:
[2] floor (# of cache blocks / 8 )
which is betting that you are using a 64K chunk size, but that the
"average" size will be 8K. This is an attempt to factor in small files
(e.g. symbolic links which only take a few dozen bytes at worst) and
obtain better cache utilization. However, if you set this manually you
shouldn't set it to less than the value computed in [1].
The parameters which may affect your performance most significantly,
however, are those which control memory utilization, not disk space
utilization. Anytime an AFS user refers to a file, the cache manager
must search its stat cache to find if it has current stat information
about the particular file. If the stat information is not in memory,
the cache manager must call to the file server to obtain stat
information; the tradeoff here is that if this pool is too large, the
file server must constantly break callbacks to the cache manager (but
this only happens for files which are changing. Binaries and other
static or slowly changing files aren't generally subject to frequent
callback breaking.) Since the size of this cache is 300 entries, all
users have to do to drag down cache manager performance is do "ls -l" or
"ls -F" frequently - it takes little time then to throw all the good
cache entries away - forcing frequent reloading of this cache.
Each stat cache entry takes approximately 384 bytes; thus, the default
cache requires 113K. In general, then the number of stat cache entries
should be:
[3] ceil ( # of bytes of memory to allocate to stat cache entries / 384)
By avoiding calls to the server to stat a file, it can save loading a
file server and improve response time.
The disk cache entries are the actual catalog of chunk contents; they
list what is in the disk cache and where. Ideally, this number would be
the same as the max # of files. When it is less than the # of files,
the cache manager juggles these entries to and from disk. For a very
active cache, this can require multiple disk I/Os per fetch, since only
the most frequently used 100 cache entries are kept in memory. These
entries cost approximately 92 bytes per entry. The default (100) uses
~9K. The "ideal" is to have one dcache entry per cache file entry;
any less than that is simply:
[4] ceil ( # of bytes of memory to allocate to dcache entries/ 92 )
By removing the call to the on-disk dcache entry you remove a
serialized synchronous operation which must be completed before the
actual cache chunk can be given to the original requester of the data.
Afsd claims to take a "-volumes" switch to set the # of in-memory volume
entries but the cache manager currently ignores this value; once the
in-core volume table is exhausted, volume information is kept on-disk in
the VolumeItems file (in the cache directory).
Chunk size offers some control over how big the actual cache files are
(and what the data transfer size is between the cache manager and the
file server.) This value is a power-of-two value between 1K and 1024K.
The default is 64K for the disk based cache and 8K for the memory based
cache.
For your particular example (700MB cache) I'm going to assume you are
willing to allocate 4MB of memory to AFS:
# of stat cache entries = 3MB/384 = 8192
# of cache files files = 700000/8 (I'm using the default of the AFS
cache manager) = 87,500
# of dcache entries = 1MB/92 = 11398
I'm not going to claim these are the best values to have but they are
reasonable values to start with; tune from here depending upon what
kind of performance you actually get to see.
Good Luck!
Tony Mason
mason+@transarc.com
