[850] in athena10
Re: Auto-updating clusters
daemon@ATHENA.MIT.EDU (Tim Abbott)
Thu Jan 15 16:38:15 2009
Date: Thu, 15 Jan 2009 16:37:13 -0500 (EST)
From: Tim Abbott <tabbott@MIT.EDU>
To: Evan Broder <broder@mit.edu>
cc: debathena@mit.edu
In-Reply-To: <496D1E47.1030002@mit.edu>
Message-ID: <alpine.DEB.2.00.0901131818410.20675@vinegar-pot.mit.edu>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
Sounds fine to me.
-Tim Abbott
On Tue, 13 Jan 2009, Evan Broder wrote:
> Jon and I decided that we thought this was reasonable, but I wanted to
> bring it up here just in case others had input.
>
> Currently debathena-auto-update runs twice an hour, using desync with a
> range of 0-1000 seconds (about 15 minutes). This means that if there's a
> large update (a new version of OOo or something unfortunate like that),
> it's very conceivable that you could end up with an entire cluster
> downed by the update process (since you can't login when updates are
> running).
>
> I think that we should change the cron job to run every 2 hours instead
> of twice an hour, and adjust the argument to desync to space updates
> across that full 2 hour period.
>
> I figure that (ignoring upgrades from one Ubuntu release to another) our
> worst case scenario update is bounded by how many changes show up in the
> apt repos in a 2 hour period. Given that, the worst case is probably an
> update that takes about 1/2 an hour to install (since Ubuntu doesn't do
> point-releases like Debian does). A 1/2 hour install period desynced
> over 2 hours results in about 3/4 of the heads in a cluster being usable
> at any given time during this worst-case update.
>
> Given that such a worst-case update is relatively unlikely to happen
> normally, meaning that the average percentage of heads downed by an
> update would be much smaller, I think this is a reasonable period of time.
>
> Do people object to me making that change?
