[2028] in Moira
Re: Some new lockers creating with quota of 1
daemon@ATHENA.MIT.EDU (Garry Zacheiss)
Tue Sep 16 18:16:18 2003
Message-Id: <200309162211.h8GMBsKt024231@brad-majors.mit.edu>
To: Laura Baldwin <boojum@mit.edu>
cc: bug-moira@mit.edu, accounts@mit.edu
In-Reply-To: Your message of "Tue, 16 Sep 2003 09:06:16 EDT."
<200309161306.h8GD6G6Q001377@geskekelud.mit.edu>
Date: Tue, 16 Sep 2003 18:11:54 -0400
From: Garry Zacheiss <zacheiss@MIT.EDU>
The explanation for this is straightforward; a permanent solution that's
not a hack will be work.
The process by which moira creates a new filesystem consists of two afs
incrementals, one to create the volume (with a quota of 1) and one to
set the volume quota to what the user specified. There's nqothing that
formally specifies that the latter depends on the former. Moira
normally finesses this issue by running one incremental to completion
before starting the next one. However, if an incremental takes too
long, moira will timeout on it and move on to the next one.
In the two cases you specified, the sequence of events was this:
- moira fires off creation incremental, which does everything fine until
it gets to the point where it has to release the parent volume of the
filesystem it just created. This is very slow for reasons I speculate
about below.
- moira times out on the creation incremental and launches the next one,
to set the quota. Since the creation incremental is almost done, this
succeeds and moves on with life.
- Meanwhile, the creation incremental finally fails to release the
volume and as a result cleans up the volume it created; it then
retries the entire creation process, which this time succeeds.
However, the entire process was completed some time ago as far as the
quota setting incremental was concerned, so we're left with the quota
of 1.
This seems to happen fairly rarely. I only found one case you didn't
mention in your mail, looking at logs going back to June, and a scan of
the nightly volume list didn't turn up anything else with a quota of 1
not accounted for by the moira logs.
I suspect the volume releases are occasionally failing because of AFS
volserver problems other sites have long complained about. There's work
underway to change the way the volserver handles breaking callbacks and
(more long term) to make it threaded. Fixing this the right way
involves getting some of that work, which I'll look into.
A short term fix involves making the initial afs create retry the vos
release before declaring failure, which should be fairly straightforward
to implement.
Garry
