[2059] in SIPB-AFS-requests
rosebud2 outage today
daemon@ATHENA.MIT.EDU (Greg Hudson)
Mon Jul 24 12:13:03 1995
Date: Mon, 24 Jul 1995 12:11:19 -0400
From: Greg Hudson <ghudson@MIT.EDU>
To: sipb-afsreq@MIT.EDU
At about 10:55, clients lost contact with rosebud2, but rosebud2's bos
server reported that everything was running. We had to power cycle
rosebud2 to reboot it.
While rosebud2 was salvaging, ronald-ann2 stopped responding to file
server queries from at least one client machine (nephthys). I asked
Marc to shut down the file server at that time, and nephthys unwedged.
Since rosebud2 was running db servers at that time, our best theory is
that ronald-ann2 became severely backlogged during rosebud2's reboot,
and restarting the fs instance would have been better.
This is the second outage of this nature we've had in as many months,
which is really unacceptable. According to jweiss, the symptoms we've
had are the same as what's known as "creeping rx failure," but we
don't know what causes that.failure mode.
kcr and I think a disk went catatonic, which is consistent with the
"communications failure (-1)" error message we got during the last
outage. jweiss suggested that our 2GB disk was not susceptible to
problems ops had with tartarus (because it is "rev 14"). We've had
previous problems with Micropolis 4GB disks.
That said, it would be reasonable to suspect that our Micropolis 4GB
disk is causing us problems, and we should swap it out for a Seagate
4GB disk (at a cost of about $200). I don't think we really have
enough information to justify this, but neither do we know any way of
getting any additional information (jhawk suggested resetting the
external disks the next time we have this kind of failure; we can try
that) or any other solution. So if the problem happens again, I'm
likely to take that approach.
