[15096] in Athena Bugs
8.0K AFS cache corruption
daemon@ATHENA.MIT.EDU (John Hawkinson)
Tue Apr 22 01:06:43 1997
Date: Tue, 22 Apr 1997 01:06:38 -0400
To: bugs@MIT.EDU
From: John Hawkinson <jhawk@MIT.EDU>
I just encountered what appears to be AFS cache corruption on
portnoy, an 8.0K Solaris machine.
I observed that ls -l on /usr/lib gave anomalous results,
and I attempted to start a "script" to collect the data,
and succeeded in killing my xterm, after which the problem
went away :-(
During that initial occurance, fs flushv of the affected volume
did not help.
It resurfaced shortly thereafter: here was the output of ls -l on
/usr/bin:
./ctcom: No such file or directory
./b: No such file or directory
./dbib: No such file or directory
./mintool: No such file or directory
./iasadm: No such file or directory
./ropos: No such file or directory
./hena: No such file or directory
./q: No such file or directory
./dioconvert: No such file or directory
./dioplay: No such file or directory
./diorecord: No such file or directory
./nner: No such file or directory
./tch: No such file or directory
./iff: No such file or directory
./l: No such file or directory
./lendar: No such file or directory
./ncel: No such file or directory
./tman: No such file or directory
./eckeq: No such file or directory
./ecknr: No such file or directory
./key: No such file or directory
./rtbl: No such file or directory
./keywd: No such file or directory
./range: No such file or directory
./yorn: No such file or directory
./mm: No such file or directory
./io: No such file or directory
./mpress: No such file or directory
./ypt: No such file or directory
./h: No such file or directory
./h.sun: No such file or directory
./plit: No such file or directory
./ags: No such file or directory
./roff: No such file or directory
./ff: No such file or directory
./ff3: No such file or directory
./ffmk: No such file or directory
./rname: No such file or directory
./sable: No such file or directory
./spgid: No such file or directory
./spuid: No such file or directory
./esg: No such file or directory
./s2unix: No such file or directory
./mpcs: No such file or directory
./mpkeys: No such file or directory
./able: No such file or directory
./pand: No such file or directory
./format: No such file or directory
./nger: No such file or directory
./li: No such file or directory
./ld: No such file or directory
./p: No such file or directory
./sage: No such file or directory
./ncat: No such file or directory
./tdev: No such file or directory
./tdgrp: No such file or directory
./tent: No such file or directory
./topt: No such file or directory
./ttext: No such file or directory
./ttxt: No such file or directory
./tvol: No such file or directory
./aph: No such file or directory
./ad: No such file or directory
./stname: No such file or directory
./86: No such file or directory
./86: No such file or directory
./86: No such file or directory
./60: No such file or directory
./PX286: No such file or directory
./dxbib: No such file or directory
./focmp: No such file or directory
./crm: No such file or directory
./cs: No such file or directory
./in: No such file or directory
./waliases: No such file or directory
./ylogin: No such file or directory
./ylogout: No such file or directory
./gin.orig: No such file or directory
./gin: No such file or directory
./h: No such file or directory
./stcomm: No such file or directory
./stusers: No such file or directory
./ok: No such file or directory
./okbib: No such file or directory
./stat: No such file or directory
./8k: No such file or directory
./il: No such file or directory
./ilq: No such file or directory
./ilstats: No such file or directory
./ilx: No such file or directory
./n: No such file or directory
./68000: No such file or directory
./68010: No such file or directory
./68020: No such file or directory
./68030: No such file or directory
./68040: No such file or directory
./onnect: No such file or directory
./tstat: No such file or directory
./wform: No such file or directory
./wgrp: No such file or directory
./ws: No such file or directory
./sstat: No such file or directory
./ce: No such file or directory
./saddcred: No such file or directory
./scat: No such file or directory
./schgrp: No such file or directory
./schmod: No such file or directory
./schown: No such file or directory
./schttl: No such file or directory
./sdefaults: No such file or directory
./serror: No such file or directory
./sgrep: No such file or directory
./sgrpadm: No such file or directory
./sln: No such file or directory
./sls: No such file or directory
./smatch: No such file or directory
./smkdir: No such file or directory
./spasswd: No such file or directory
./spath: No such file or directory
./srm: No such file or directory
./srmdir: No such file or directory
./stbladm: No such file or directory
./stest: No such file or directory
./off: No such file or directory
./ge: No such file or directory
./ssmgmt: No such file or directory
./ste: No such file or directory
./ginfo: No such file or directory
./gmk: No such file or directory
./gparam: No such file or directory
./gproto: No such file or directory
./gtrans: No such file or directory
./conv: No such file or directory
./p: No such file or directory
./ist: No such file or directory
./ogin: No such file or directory
./h: No such file or directory
./ail: No such file or directory
./ffbib: No such file or directory
./p: No such file or directory
./ptime: No such file or directory
./sers: No such file or directory
./ho: No such file or directory
./g: No such file or directory
./r: No such file or directory
./rt: No such file or directory
./rtbib: No such file or directory
./arc: No such file or directory
./ell: No such file or directory
./lit: No such file or directory
./race: No such file or directory
./rclean: No such file or directory
./rerr: No such file or directory
./n: No such file or directory
./n2: No such file or directory
./n3: No such file or directory
./n3x: No such file or directory
./n4: No such file or directory
./n4c: No such file or directory
./n4d: No such file or directory
./n4e: No such file or directory
./n4m: No such file or directory
./nc: No such file or directory
./bs: No such file or directory
./lk: No such file or directory
./r: No such file or directory
./l: No such file or directory
./opy: No such file or directory
./e: No such file or directory
./c: No such file or directory
./lnet: No such file or directory
./tp: No such file or directory
./me: No such file or directory
./mex: No such file or directory
./p: No such file or directory
./lot: No such file or directory
./off: No such file or directory
./uss: No such file or directory
./yhstmgr: No such file or directory
./70: No such file or directory
./b: No such file or directory
./b15: No such file or directory
./b2: No such file or directory
./b5: No such file or directory
./its: No such file or directory
./ix2dos: No such file or directory
./decode: No such file or directory
./encode: No such file or directory
./glist: No such file or directory
./log: No such file or directory
./name: No such file or directory
./pick: No such file or directory
./stat: No such file or directory
./to: No such file or directory
./x: No such file or directory
./x: No such file or directory
./hrtbl: No such file or directory
./ois: No such file or directory
./match: No such file or directory
./sename: No such file or directory
./m: No such file or directory
./k: No such file or directory
./ptoinfo: No such file or directory
./s: No such file or directory
./grp: No such file or directory
./t: No such file or directory
./vreserv: No such file or directory
./mod: No such file or directory
./own: No such file or directory
./gid: No such file or directory
./int: No such file or directory
./item: No such file or directory
./path: No such file or directory
./str: No such file or directory
./uid: No such file or directory
./ear: No such file or directory
./p: No such file or directory
./l: No such file or directory
./lltbl: No such file or directory
./ontab: No such file or directory
./t: No such file or directory
./te: No such file or directory
./vattr: No such file or directory
./vfree: No such file or directory
./mainname: No such file or directory
./rcmp: No such file or directory
./ho: No such file or directory
./it: No such file or directory
./h: No such file or directory
./stdgrp: No such file or directory
./rep: No such file or directory
./ect: No such file or directory
./v: No such file or directory
./n: No such file or directory
./str: No such file or directory
./ctor: No such file or directory
./lse: No such file or directory
./etach: No such file or directory
./rep: No such file or directory
./le: No such file or directory
./nd: No such file or directory
./t: No such file or directory
./tmsg: No such file or directory
./ore: No such file or directory
./ep: No such file or directory
./oups: No such file or directory
./onv: No such file or directory
./eal: No such file or directory
./stat: No such file or directory
./d: No such file or directory
./dcomp: No such file or directory
./mon: No such file or directory
./ll: No such file or directory
./st: No such file or directory
./d: No such file or directory
./ne: No such file or directory
./adkeys: No such file or directory
./ilcompat: No such file or directory
./gins: No such file or directory
./gname: No such file or directory
./iocntl: No such file or directory
./kedev: No such file or directory
./sg: No such file or directory
./dir: No such file or directory
./fifo: No such file or directory
./msgs: No such file or directory
./ntbl: No such file or directory
./re: No such file or directory
./stat: No such file or directory
./gfmt: No such file or directory
./wk: No such file or directory
./qn: No such file or directory
./hup: No such file or directory
./wk: No such file or directory
./ck: No such file or directory
./sswd: No such file or directory
./intf: No such file or directory
./tdev: No such file or directory
./vecore: No such file or directory
./tdgrp: No such file or directory
./d: No such file or directory
./ate: No such file or directory
./d: No such file or directory
./fer: No such file or directory
./dir: No such file or directory
./cgen: No such file or directory
./cinfo: No such file or directory
./tuname: No such file or directory
./m: No such file or directory
./ript: No such file or directory
./iff: No such file or directory
./d: No such file or directory
./tpgrp: No such file or directory
./ttime: No such file or directory
./cation: No such file or directory
./owrev: No such file or directory
./eep: No such file or directory
./elim: No such file or directory
./chtxt: No such file or directory
./rchg: No such file or directory
./rconf: No such file or directory
./rings: No such file or directory
./ty: No such file or directory
./il: No such file or directory
./uch: No such file or directory
./ut: No such file or directory
./ue: No such file or directory
./y: No such file or directory
./ame: No such file or directory
./iq: No such file or directory
./lcheck: No such file or directory
./dit: No such file or directory
./rind: No such file or directory
./ew: No such file or directory
./stat: No such file or directory
./ettext: No such file or directory
./ig: No such file or directory
./ig: No such file or directory
./atis: No such file or directory
./ich: No such file or directory
./o: No such file or directory
./ite: No such file or directory
./rgs: No such file or directory
./6pc: No such file or directory
./ist: No such file or directory
./estroy: No such file or directory
./nit: No such file or directory
./rvtgt: No such file or directory
./msh: No such file or directory
./s: No such file or directory
total 686
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
drwxrwxr-x 3 root other 6656 Mar 9 12:29
lrwxrwxrwx 1 root root 14 Jun 19 1996 at -> /os/usr/bin/at
lrwxrwxrwx 1 root root 14 Jun 19 1996 at -> /os/usr/bin/at
-r-xr-xr-x 1 bin bin 9040 Jul 15 1994 cat
lrwxrwxrwx 1 root root 20 Jun 19 1996 compress -> /os/usr/bin/compress
-r-xr-xr-x 1 bin bin 13656 Jul 16 1994 cp
-r-xr-xr-x 1 bin bin 6968 Jul 15 1994 date
lrwxrwxrwx 1 root root 18 Jun 19 1996 expand -> /os/usr/bin/expand
-r-xr-xr-x 1 bin bin 2952 Jul 16 1994 line
lrwxrwxrwx 1 root root 16 Jun 19 1996 pack -> /os/usr/bin/pack
-r-sr-sr-x 1 root sys 11680 Jul 16 1994 passwd
-r-xr-xr-x 1 bin bin 21256 Jul 16 1994 pr
-r-sr-xr-x 1 root other 17812 Jul 28 1995 ps
-r-xr-xr-x 1 bin bin 7240 Jul 16 1994 rm
lrwxrwxrwx 1 root root 17 Jun 19 1996 sh -> ../../usr/bin/jsh
lrwxrwxrwx 1 root root 16 Jun 19 1996 time -> /os/usr/bin/time
lrwxrwxrwx 1 root root 16 Jun 19 1996 time -> /os/usr/bin/time
-r-xr-xr-x 1 bin bin 9496 Jul 16 1994 tr
-r-xr-xr-x 1 bin bin 1155 Jul 16 1994 which
That is very similar to the lossage I saw before. Also:
[portnoy!jhawk] ~> stty sane
stty: Command not found.
[portnoy!jhawk] ~> st^U
[portnoy!jhawk] ~> where stty
[portnoy!jhawk] ~> cd /usr/lib
[portnoy!jhawk] /usr/lib> echo *
echo: No match.
I tried to fs setcachsize to 1, and that did not help the problem
(even after perturbing the cache with an ls -lR of
/afs/athena/user/j).
At this point I crashed the machine with STOP-A 0 set-pc go.
It dumped as follows:
BAD TRAP: type=1 rp=f03e2b54 addr=0 mmu_fsr=164 rw=3
sched: Text fault
kernel read fault at addr=0x0, pme=0x0
MMU sfsr=164: Invalid Address on supv instr fetch at level 1
pid=0, pc=0x0, sp=0xf03e2ba0, psr=0x4400cc1, context=0
g1-g7: f00575e0, 40000e2, 2810, 3010, 3810, 1, f03e2ec0
Begin traceback... sp = f03e2ba0
Called from f0049370, fp=f03e2c00, args=44000e3 44000e3 f0152b78
44000e3 4 10
Called from fc1ca668, fp=f03e2c70, args=0 fc0fd72c fc0fd72c 0 21
f0152400
Called from fc1ca360, fp=f03e2ce0, args=fc01d018 4d 4d 0 1 fc01d
040
Called from f0069794, fp=f03e2d40, args=fc1c8b38 fc1c8af0 fc1c8a
f0 fc01d018 fc1c8b00 fc18bcc0
Called from f007efec, fp=f03e2da0, args=0 40000000 ffffffff fc1c
8af0 0 10
Called from f007f16c, fp=f03e2e00, args=f01744c0 f0175c14 18f31e
f 80000000 fc147918 904ac1ef
Called from f0058858, fp=f03e2e60, args=0 0 f01878c8 f01878c8 f0
1744d4 f01744c0
Called from f007f148, fp=0, args=0 0 1017296b 252e8f2c dcbdc1d5
5080809b
End traceback...
panic: Text fault
syncing file systems... [16] 18 [16] 18 [16] 18 [16] 18 [16] 18
[16] 18 [16] 18 [16] 18 [16] 18 [16] 18 [16] 18 [16] 18 [16] 18
[16] 18 [16] 18 [16] 18 [16] 18 [16] 18 [16] 18 [16] 18 done
2749 static and sysmap kernel pages
64 dynamic kernel data pages
102 kernel-pageable pages
0 segkmap kernel pages
0 segvn kernel pages
0 current user process pages
2915 total pages (2915 chunks)
dumping to vp fc1e0e1c, offset 75216
Which decodes as:
Called from [f0049370=debug_enter+d8], fp=f03e2c00, args=44
Called from [fc1ca668=kbdinput+2cc], fp=f03e2c70, args=0 [f
Called from [fc1ca360=kbdrput+130], fp=f03e2ce0, args=[fc01
Called from [f0069794=putnext+70], fp=f03e2d40, args=fc1c8b
Called from [f007efec=callout_execute+90], fp=f03e2da0, arg
Called from [f007f16c=callout_thread+24], fp=f03e2e00, args
Called from [f0058858=thread_start+4], fp=f03e2e60, args=0
Called from [f007f148=callout_thread], fp=0, args=0 0 10172
Needless to say, this is not very interesting :-)
Anyhow, the crash dump is available if anyone would care to look at
it. At boot time, the ws resized the afs cache and all problems
appeared to go away.
Thanks.
Any tips or hints for diagnosing this would be helpful.
In hindsight I should have gotten cmdebug prior to crashing.
--jhawk
