[2034] in SIPB-AFS-requests
power is restored (finally)
daemon@ATHENA.MIT.EDU (Tom Yu)
Sun Jun 25 10:49:03 1995
Date: Sun, 25 Jun 1995 10:46:59 -0400
To: star-maintainers@MIT.EDU
Cc: sipb-staff@MIT.EDU
From: Tom Yu <tlyu@MIT.EDU>
Shortly before 02:00, I was called into the machine room to help shut
down machines. All the sipb server machines were brought down. At
about 04:20, I powered up the AFS servers, along with charon, rtfm,
and penguin-lust. The news servers and www were left powered off
because no one with their root passwords was around.
At about 09:00, I shut down the afs servers, along with charon,
penguin-lust, and rtfm. I restarted the nightly-jobs on charon; I
don't know what the status of the rtfm nightly-jobs is; there's a
/etc/nightly_jobs.pid that's dated at 00:51, so it may have bombed.
Someone who is more familiar than I am with rtfm's current state
should deal.
At about 09:25, the power came back on. AFS came up cleanly, as did
the other servers. Power was also restored to the 5th floor lights at
about this time. During the cutover back to building power, jhawk and
I noticed that the UPS 1 (using ghudson's labeling) that was supposed
to feed charon, anxiety-closet, penguin-lust, beacon, and a few
repeaters was not generating power. We should probably get this
replaced or otherwise dealt with.
It turns out the reason for the outage was that a 600 amp (!) breaker
was installed in a frame that was intended for a smaller breaker, and
the tabs on the bus bar were too small to fit into the jaws of the 600
amp breaker. The resulting arcing left a mess on the contacts of the
breaker, which stayed that way for many years. This was the reason
for another outage around 02:00; the cutover back to the building
power ran into problems, losing one of the phases to the circuit. The
machine room went back to generator power.
Some time before 09:00, the electricians had shuffled the machine room
circuit to another breaker, where it is connected now. There will be
another outage, possibly on July 7, to correct the problem with the
switchgear that started this whole mess. More details to come, I
guess.
---Tom
