[3241] in Release_Engineering
URGENT: Status of Eros *READ BEFORE WORKING IN THE REL-ENG CELL*
daemon@ATHENA.MIT.EDU (Jeffrey I. Schiller)
Sat Jun 18 22:54:09 1994
Date: Sat, 18 Jun 94 22:53:43 -0400
From: Jeffrey I. Schiller <jis@MIT.EDU>
To: probe@MIT.EDU, vrt@MIT.EDU, cfields@MIT.EDU, rel-eng@MIT.EDU
Cc: op@MIT.EDU, tjm@MIT.EDU, tytso@MIT.EDU
Eros (a fileserver in the Rel-eng cell) crashed this evening because
first /vicepd filled up. Apparently the nature of the crash was such
that /vicepc got damaged as well (superblock and part of ilist trashed).
/vicepd was so full that volumes could not be moved off! (Moving volumes
requires a few pages of disk on the source partitions, but there was
none). Similarly the AFS Salvager could not run because it too requires
some disk space on the partition being salvaged!
I created some space by moving user.probe offline in an unorthodox
fashion. I fixed (with fsck) the damage on /vicepc and then ran the
the salvager on both /vicepc and /vicepd. Four volumes were damaged.
They are:
Eros:/vicepc
build.andrew.rt.74 <-- Ancient Volume, minor damage
build.andrew.rsaix.74 <-- Ancient Volume, minor damage
Eros:/vicepd
src.athena <-- Moderate damage someone who knows what this
volume is supposed to look like should check
it out. We should have good backup tapes for it.
project.aux <-- Possibly extensive damage. Old volume should
be on backup tape.
Before anyone goes and makes changes on src.athena, someone from rel-eng
should consult with Anne Salemme about what needs to be done vis a vis
restoring anything from tape. Someone from rel-eng should also resolve
the fate of the other damaged volumes.
Note: It is *very* *very* important to not overflow partitions. AFS
does not handle such overflow gracefully at all!
As of this message Eros is up but is fileserver is not running. I expect
to have it back in service within the hour. user.probe is still offline
should be back sometime this evening.
-Jeff
