[122060] in North American Network Operators' Group
Re: Mitigating human error in the SP
daemon@ATHENA.MIT.EDU (Larry Sheldon)
Thu Feb 4 18:15:18 2010
Date: Thu, 04 Feb 2010 17:13:30 -0600
From: Larry Sheldon <LarrySheldon@cox.net>
To: nanog@nanog.org
In-Reply-To: <20100204133037.968F9638@resin09.mta.everyone.net>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
On 2/4/2010 3:30 PM, Scott Weeks wrote:
>
> A recent organizational change at my company has put someone in charge
> who is determined to make things perfect. We are a service provider,
>
> isn't a common occurrence, and the engineer in question has a pristine
> track record.
>
> This outage, of a high profile customer, triggered upper management to
> react by calling a meeting just days after. Put bluntly, we've been
> told "Human errors are unacceptable, and they will be completely
> eliminated. One is too many."
> ----------------------------------------------------------------
>
>
>
>> From experience...
>
> At one point this will become overwhelming. You'll wake up every morning dreading going to
> work instead of looking forward to it. Chain shot will be put in the 'blame cannon' and
> blasted regularly and at everyone. Update your resume and get everything in place just in
> case it gets to the point you can't take it anymore sooner than you expect. ;-)
This is a golden opportunity.
Prepare a pan for building the lab necessary to pre-test EVERYTHING.
Cost it out.
Present the cost and the plan in a public forum or widely distributed
memorandum (including as a minimum everybody that was at the meeting and
everybody in the chain(s) of command between you and the edict giver.
--
"Government big enough to supply everything you need is big enough to
take everything you have."
Remember: The Ark was built by amateurs, the Titanic by professionals.
Requiescas in pace o email
Ex turpi causa non oritur actio
Eppure si rinfresca
ICBM Targeting Information: http://tinyurl.com/4sqczs
http://tinyurl.com/7tp8ml
