[2055] in SIPB-AFS-requests
Summary of 15 July SIPB server outage
daemon@ATHENA.MIT.EDU (John Hawkinson)
Sat Jul 15 12:32:58 1995
Date: Sat, 15 Jul 1995 12:32:48 -0400
Cc: sipb-staff@MIT.EDU
From: John Hawkinson <jhawk@MIT.EDU>
Somewhat contrary to the mail to athena-outage, this morning's
outage was planned to have two visible components:
Switching the machine room onto emergency power
Switching the machine room back to regular power
Between those two events, Physical Plant was working to repeat what
they tried to due half a month ago, namely finish PM and checkups on
W20 electrical equipment. The outage was expected to last about 3
hours.
As scheduled, at about 00:05, the fifth floor lost power, and the
cutover of the machine room to emergency power began (in general we
expect that takes ten to fifteen minutes). Due to power configuration
(being on the same UPS as the AFS servers), both news servers were
shut down prior to the outage cutover, as was bloom-picayune. Charon
was also shutdown, at about 00:09. The cutover finished at about
00:19, with rtfm returning to service around 00:30 (takes time to fsck
its disks). Seantor-bedfellow and bloom-beacon both experienced an
interesting failure mode. While changing the power configuration, we'd
accidently left their keyboards unplugged. When they were powered up,
their monitors failed to display anything, and the eventual solution
was to reboot them; this is presumably some sparcserver 20 failure
mode.
At 01:50 it became apparent that PhysPlant had finished in record
time, and expected to cut us back at about 02:00. Picayune was
shutdown at about 01:56, and beacon at 02:01. To conserve power for
senator-bedfellow, we shut down charon and anxiety-closet at
02:04. The cutover began at 02:06. About 10 minutes into the cutover,
PhysPlant encountered some sort of problem, extending the cutover. At
02:18 we elected to shutdown ronald-ann2 (rather than rosebud2, to
preserve quorum in the sipb cell) to increase the lifetime of the sipb
afs cell. At 02:32, w20-rtr's UPS failed and all of W20 went off-net
(In retrospect, perhaps that would have been an appropriate time to
bos shutdown rosebud, rather than waiting). At 02:41, the cutover
finished, and the w20-rtr regained power. The start of most SIPB
servers was delayed by waiting for ronald-ann2, which was salvaging
(apparently it is necessary to wait for the bosserver to die before
shutting down the machine, even after the fileserver has exitted,
otherwise a salvage is forced). Picayune came back at 02:51, and I
believe ra2 finished salvaging somewhere around then.
--jhawk
