[836] in athena10
Auto-updating clusters
daemon@ATHENA.MIT.EDU (Evan Broder)
Tue Jan 13 18:06:54 2009
Message-ID: <496D1E47.1030002@mit.edu>
Date: Tue, 13 Jan 2009 18:05:43 -0500
From: Evan Broder <broder@MIT.EDU>
MIME-Version: 1.0
To: debathena@mit.edu
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Jon and I decided that we thought this was reasonable, but I wanted to
bring it up here just in case others had input.
Currently debathena-auto-update runs twice an hour, using desync with a
range of 0-1000 seconds (about 15 minutes). This means that if there's a
large update (a new version of OOo or something unfortunate like that),
it's very conceivable that you could end up with an entire cluster
downed by the update process (since you can't login when updates are
running).
I think that we should change the cron job to run every 2 hours instead
of twice an hour, and adjust the argument to desync to space updates
across that full 2 hour period.
I figure that (ignoring upgrades from one Ubuntu release to another) our
worst case scenario update is bounded by how many changes show up in the
apt repos in a 2 hour period. Given that, the worst case is probably an
update that takes about 1/2 an hour to install (since Ubuntu doesn't do
point-releases like Debian does). A 1/2 hour install period desynced
over 2 hours results in about 3/4 of the heads in a cluster being usable
at any given time during this worst-case update.
Given that such a worst-case update is relatively unlikely to happen
normally, meaning that the average percentage of heads downed by an
update would be much smaller, I think this is a reasonable period of time.
Do people object to me making that change?
- Evan
