[44839] in North American Network Operators' Group
Re: Network Operations Luminaries?
daemon@ATHENA.MIT.EDU (Jeff Rizzo)
Tue Dec 18 13:15:06 2001
Date: Tue, 18 Dec 2001 10:14:27 -0800
From: Jeff Rizzo <riz@boogers.sf.ca.us>
To: Clayton Fiske <clay@bloomcounty.org>
Cc: nanog@merit.edu
Message-ID: <20011218101427.A28826@hubba.boogers.sf.ca.us>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20011210111525.O870@bloomcounty.org>; from clay@bloomcounty.org on Mon, Dec 10, 2001 at 11:15:25AM -0800
Errors-To: owner-nanog-outgoing@merit.edu
On Mon, Dec 10, 2001 at 11:15:25AM -0800, Clayton Fiske wrote:
>
> On Mon, Dec 10, 2001 at 10:18:48AM -0700, Joel Baker wrote:
> >
> > I don't have a timeline to know which happened first; 2551 was down, or at
> > least the majority of it was, for something on the order of 48-72 hours.
> > The rendition I heard assigned the moniker because one of the major news
> > outlets said that the mistake which triggered it was "like a misplaced
> > ampersand". It was certainly the one on the wall beside his legacy Chevy's
> > sombrero.
>
> My oh my, how the versions differ.
>
> As was recounted to me, the outage was about 19 hours, and was due to
> the semantics of Cisco config mode. Something like:
>
> router ospf 1234
> redistribute bgp subnets route-map blah
>
> (everything fine, now let's turn it off...)
>
> no redistribute bgp subnets route-map blah
>
> So tell me, does this turn off the redistribution, or just remove the
> route-map... :)
>
> And this is certainly worth remembering. Had I not known of this, I
> could likely have made the same mistake at some point.
>
> -c
Ye gods. Guess I'd better keep a closer eye on NANOG.
For the record: This version (Clay's) is more or less correct. :)
The "oops" was pretty much *exactly* that... removing a route-map
when intending to remove a redistribution.
In my defense, however, we discovered the problem fairly quickly, but
OSPF bug extant in IOS 10.whatever that we were running prevented us
from cleaning it up in a timely manner. We basically wound up
partitioning the network and rebooting *every* ospf-speaking device
on it, because they were getting poisoned with gobs of LSA data that was
never getting cleared, and was propagating the bogus information.
Also for the record: I have *no idea* where Bob Metcalfe got that
"ampersand" thing. It did make for an amusing engraving on
a going-away gift I was given some years later:
!!!!!!!!!!!!!!!!!!!!&................!!!!!!!!!!!!!
:)
+j
--
Jeff Rizzo http://boogers.sf.ca.us/~riz
