[249] in iswork
train.mit.edu flaked out, but a temporary solution is in place
daemon@ATHENA.MIT.EDU (Kevin M. Cunningham)
Tue Sep 4 22:48:59 2001
Mime-Version: 1.0
Message-Id: <p05010400b7bb34b85632@[18.152.1.50]>
Date: Tue, 4 Sep 2001 22:48:44 -0400
To: ssharari@mit.edu, shonool@mit.edu, liverman@mit.edu, pmaguire@mit.edu
From: "Kevin M. Cunningham" <kcunning@MIT.EDU>
Cc: cavan@mit.edu, istrain-reg@mit.edu, pctrain-reg@mit.edu,
fsstrain-reg@mit.edu, llucas@mit.edu, llewis@mit.edu,
ocstrain-reg@mit.edu, frc@mit.edu, progress@mit.edu, laurief@mit.edu,
is-stock@mit.edu, is-home@mit.edu, swrt@mit.edu, iswork@mit.edu,
support-tp@mit.edu, ehamp@mit.edu, pconant@mit.edu
Content-Type: multipart/alternative; boundary="============_-1212464367==_ma============"
--============_-1212464367==_ma============
Content-Type: text/plain; charset="us-ascii" ; format="flowed"
Hello database registrars and other interested parties,
I am sending this message to everyone who has data served by the
FileMaker systems we maintain here in N42, mostly to keep you
informed about an outage we had today and to request your
understanding as I reorganize my schedule to cope with an unexpected
problem in one of the servers. Currently, everything is running as
usual. Only some of OCSPA's data may have been endangered.
The problem we encountered today probably explains the server's
flakeyness over the past several weeks. So I'm hoping that, with the
resolution of this issue, the occassional problems reported over the
last few weeks will also have ended.
WHAT HAPPENED TODAY
Summary: train.mit.edu flaked out by 4pm, 5 hours later a temporary
fix was in place, the original machine needs immediate repairs, and
some OCSPA-related data *may* have been lost.
4pm - I got a call from Pat Maguire in OCSPA saying the PreMed
database was acting weird. (She uses FMPro on her desktop to access
the PreMed db directly from the FMServer machine train.mit.edu.) I
tried to bring up file on my own machine (again using FMPro to access
the file from train.mit.edu) and it basically froze. I went to the
server closet to restart it (I sent a message to the users online
saying "Down for 5 minutes" since I expected it just to be a reboot),
but from then on the system could not be brought up again properly.
Since the web services are based on the FMServer, they were also down
for the next 5 hours as well.
The server was making a noticeable whirrrrr-clicking noise -- a
regular pattern of long grinding then brief clicking, maybe 10 of
these sequences in a row then stopping. Upon extensive examination,
and with some help from Eric Hamp from the Help Desk, I concluded
that the machine was not freezing permanently, but rather was
attempting to read certain disk files and failing. The whirrr-click
was the hard drive trying to get to these files and failing, finally
giving up. Eventually, I was able to isolate the files: they were
about 10 databases related to OCSPA (the Activities and
Activities2000 files from the OCSPA were NOT among the problem
files). Everything else was fine (i.e., did not cause the whirr-click
when accessed). I could not access the bad files in any way, nor fix
the disk through any of the normal methods (Norton, Disk First Aid,
etc.)
I moved the suspect files to a separate new directory ("EVILEVIL") to
isolate them, then burned a CD with everything I could recover
(which included all the other database files as of 4pm, plus ALL the
databases as of last night's 11:45 backup). I then pulled the G4
machine out to go to PC service.
By 9pm, I finally got the old train.mit.edu machine (a G3, much
slower) up and running, using files saved from 4pm, plus OCSPA files
from last night's backup. That's what's running now.
WHAT'S NEXT
Unfortunately, I suspect a good deal of data was entered by OCSPA
into the PreMed dbs today. I hope we can recover the lost files in
the course of PC Service's work -- enough, at any rate, to get the
data. The database structure is of course easily reproducible as I
have many copies with the most recent field definitions etc. -- it's
only the recently-entered data that is possibly in danger.
I still need to assess any damage to other files. From my preliminary
forays, the problem was something to do with imperfections to the
hard disk itself, limited to the region where the OCSPA files were
written, so I don't have a reason to think the other files' data is
at all corrupted. Nevertheless, it makes sense to check all those
files and put the recovered data into cleanly generated new copies of
the files.
I will get the damaged server to PC Service first thing Wednesday,
and wait with bated breath to see whether any of those files on the
damaged part of the hard drive can be recovered.
This is an unexpected result, since the server is only several months
old. I strongly suspect a defective chip, but will of course be
interested in getting the real story.
I will keep you up to date as this progresses.
--kcunning, 09/04/2001 10:45pm
--============_-1212464367==_ma============
Content-Type: text/html; charset="us-ascii"
<!doctype html public "-//W3C//DTD W3 HTML//EN">
<html><head><style type="text/css"><!--
blockquote, dl, ul, ol, li { margin-top: 0 ; margin-bottom: 0 }
--></style><title>train.mit.edu flaked out, but a temporary solution
is</title></head><body>
<div><font face="Helvetica" color="#000000">Hello database registrars
and other interested parties,</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">I am sending this message
to everyone who has data served by the FileMaker systems we maintain
here in N42, mostly to keep you informed about an outage we had today
and to request your understanding as I reorganize my schedule to cope
with an unexpected problem in one of the servers. Currently,
everything is running as usual. Only some of OCSPA's data may have
been endangered.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">The problem we encountered
today probably explains the server's flakeyness over the past several
weeks. So I'm hoping that, with the resolution of this issue, the
occassional problems reported over the last few weeks will also have
ended.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">WHAT HAPPENED
TODAY</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica">Summary: train.mit.edu flaked out by 4pm,
5 hours later a<font color="#000000"> temporary fix was in place, the
original machine needs immediate repairs, and some OCSPA-related data
*may* have been lost.</font></font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">4pm - I got a call from
Pat Maguire in OCSPA saying the PreMed database was acting weird. (She
uses FMPro on her desktop to access the PreMed db directly from the
FMServer machine train.mit.edu.) I tried to bring up file on my own
machine (again using FMPro to access the file from train.mit.edu) and
it basically froze. I went to the server closet to restart it (I sent
a message to the users online saying "Down for 5 minutes"
since I expected it just to be a reboot), but from then on the system
could not be brought up again properly. Since the web services are
based on the FMServer, they were also down for the next 5 hours as
well.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">The server was making a
noticeable whirrrrr-clicking noise -- a regular pattern of long
grinding then brief clicking, maybe 10 of these sequences in a row
then stopping. Upon extensive examination, and with some help from
Eric Hamp from the Help Desk, I concluded that the machine was not
freezing permanently, but rather was attempting to read certain disk
files and failing. The whirrr-click was the hard drive trying to get
to these files and failing, finally giving up. Eventually, I was able
to isolate the files: they were about 10 databases related to OCSPA
(the Activities and Activities2000 files from the OCSPA were NOT among
the problem files). Everything else was fine (i.e., did not cause the
whirr-click when accessed). I could not access the bad files in any
way, nor fix the disk through any of the normal methods (Norton, Disk
First Aid, etc.)</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">I moved the suspect files
to a separate new directory ("EVILEVIL") to isolate them,
then burned a CD with everything I could recover (which included
all the other database files as of 4pm, plus ALL the databases as of
last night's 11:45 backup). I then pulled the G4 machine out to
go to PC service.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">By 9pm, I finally got the
old train.mit.edu machine (a G3, much slower) up and running, using
files saved from 4pm, plus OCSPA files from last night's backup.
That's what's running now.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">WHAT'S NEXT</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">Unfortunately, I suspect a
good deal of data was entered by OCSPA into the PreMed dbs today. I
hope we can recover the lost files in the course of PC Service's work
-- enough, at any rate, to get the data. The database structure is of
course easily reproducible as I have many copies with the most recent
field definitions etc. -- it's only the recently-entered data that is
possibly in danger.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">I still need to assess any
damage to other files. From my preliminary forays, the problem was
something to do with imperfections to the hard disk itself, limited to
the region where the OCSPA files were written, so I don't have a
reason to think the other files' data is at all corrupted.
Nevertheless, it makes sense to check all those files and put the
recovered data into cleanly generated new copies of the
files.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">I will get the damaged
server to PC Service first thing Wednesday, and wait with bated breath
to see whether any of those files on the damaged part of the hard
drive can be recovered.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">This is an unexpected
result, since the server is only several months old. I strongly
suspect a defective chip, but will of course be interested in getting
the real story.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">I will keep you up to date
as this progresses.</font></div>
<div><font face="Helvetica" color="#000000"><br></font></div>
<div><font face="Helvetica" color="#000000">--kcunning, 09/04/2001
10:45pm</font></div>
</body>
</html>
--============_-1212464367==_ma============--
