[21718] in Hotline Meeting
Moira is back!!!
daemon@ATHENA.MIT.EDU (Theodore Ts'o)
Wed Mar 16 11:00:30 1994
Date: Wed, 16 Mar 94 11:00:09 EST
From: tytso@MIT.EDU (Theodore Ts'o)
To: athena-outage@MIT.EDU
[ This was sent out late last night, but to the wrong email address
(athena-outages instead of athena-outage). ]
Date: Tue, 15 Mar 94 23:41:40 EST
From: tytso@ATHENA.MIT.EDU (Theodore Ts'o)
To: athena-outages@ATHENA.MIT.EDU
Subject: Moira is back!!!
As some of you no doubt noticed, Moira service was restored at 4:50pm
today. Fortunately, it did not become necessary to restore from
backups, and no data was lost.
The cause of the service disruption was an Ingres bug which caused the
moira daemons to deadlock on Sunday night, and caused a corruption in
the moira database which caused any attempt to modify lists in Moira to
fail. The corruption was relatively subtle, so it took quite a while to
figure out what was actually wrong. This was compouned by another bug
in Ingres which casued the program which checks database corruptions to
crash. Quite a long time was spent trying to debug this, before I was
informed by Mark that this was actually normal(!).
After I brought Moira back up, I did a careful comparison of the state
of the database while it was down, and the state of the database from
the last known good backup. All of the changes could be correlated to
the journal log. Thus, I am relatively confident that we have not lost
any data, and that the Moira database is consistent.
Again, my apologies for the long service disruption that we experienced
these past two days.
- Ted
