[8704] in Hotline Meeting
The Physical Plant Caper:
daemon@ATHENA.MIT.EDU (larugsi@Athena.MIT.EDU)
Wed Apr 22 00:14:34 1992
From: larugsi@Athena.MIT.EDU
To: cec@Athena.MIT.EDU, jis@Athena.MIT.EDU, joanne@Athena.MIT.EDU
Cc: burke@Athena.MIT.EDU, kim@Athena.MIT.EDU, larugsi@Athena.MIT.EDU,
Date: Wed, 22 Apr 92 00:14:02 EDT
Hi all,
I will try to sum up the events that have taken place this evening
regarding heat problems, A/C unit failures etc..etc... Here I go!
At approx. 9:05 p.m. I have received a page from the Physical Plant Operations
Center, regarding an alarm from the 11-135 machine room. I have acted upon
this call immediately. The 11-135 machine room was like a blast furnace.
Both A/C units were not running. The A/C alarm box showed no red error alarm
lights, just the green status light was on. I quickly tried to manualy power
up the primary A/C unit (the A/C unit at the normal machine room door
entrance), and it worked! I then opened up both machine room doors and the
outside stairwell doors to get as much cooling as possible into 11-135. It
worked out well. At that moment I had received a call from Ron Hoffmann,
regarding the Cisco router, which I had power cycled at Ron's request. That
came up o.k. I had also talked to Stan at the 8400 number, who helped assist
me, at my request, in paging the operations emergency pager, and I believe
Jeff as well, regarding the Cisco router etc..
At this point in time, Heat and Vent had arrived. I had found out from him
that this alarm had gone off at approx. 7:30 p.m. and he was dispatched by the
Operations Center to 11-135 to reset it. This to me was a crusial mistake.
The mistake being that the Operations Center should have notified someone on
the emergency call list at this time. This was not done.
At this point, I had requested the presence of the 2nd shift HVAC supervisor,
Richard Sirois, to the 11-135 machine room. He stated that he would meet me
in about 30-40 minutes. Just prior to this, Jeff, and Ted showed up at the
machine room, and we began a plan of attack. As I was speaking to Richard
Sirois, Jeff was on the phone trying to contact Dave Millay. Dave was going to
return Jeff's call, when a power hit took place (at approx. 21:58) in the E40
machine room. Jeff and Ted headed over to E40, and I received Dave Millay's
call and had spoken to him at length. Dave stated that he was going to have to
re-write his notification policy on these alarm procedures etc.. and had
assured me that Plant will do everything in their power to keep us online
with, within the hour checks (at my request) through the rest of 2nd shift, and
through the entire 3rd shift hours, until the A/C alarm box people and his
people, attack this problem in the a.m. It was also understood that 1410 would
be notified before entering 11-135. Just in case, JJ has been sent mail to be
there first thing in the a.m.
At approx. 22:50, I was once again paged by Plant to meet Warren Scott, and
Richard Sirois in the 11-135 machine room. I was once again assured that
my request of within the hour checks, every hour, will be performed until the
problem is attacked in the a.m.
As far as 11-135 goes, everything has cooled down and will be monitored
throughout the evening. No hardware has been lost, but it must of been very
close to that point.
Another problem that happened today was the heat problem in the 37 facility.
After looking into why the temperature was hovering between 80-85 degrees, it
was finally determined, after calling Plant twice, that the chilled water
from the fan coil units for this area had been drained. I stated that it was
mandatory, that this problem be rectified. A call had to be placed to the
Supervisor, from the mechanic to get things rolling. Richard Sirois gave the
mechanic the o.k. to refill the fan coil units with chilled water. All now
seems alot cooler at this facility.
At approx. 23:00, I have received another page from Plant. This time Mary
Mullen had paged me to let me know that W20-575 was freezing, and that a HVAC
man was on his way over. I said I am on my way to meet him. This was an easy
problem. Someone had lowered the thermostat to around 60 degrees or so in the
new RT room. The thermostat was reset at around 66 degrees or so. Room temp.
has now returned to normal.
Well folks, I think thats about it. I hope this summary gives you the breakdown
of the flow of events as they happened. Any questions, feel free to fire away.
--Lou
