[91063] in North American Network Operators' Group
Re: Nationwide Routing issues with Wiltel
daemon@ATHENA.MIT.EDU (david raistrick)
Wed Jun 28 16:39:42 2006
Date: Wed, 28 Jun 2006 16:39:12 -0400 (EDT)
From: david raistrick <drais@atlasta.net>
To: Jeremy Chadwick <nanog@jdc.parodius.com>
Cc: nanog@merit.edu
In-Reply-To: <20060627154418.GA18007@icarus.home.lan>
Errors-To: owner-nanog@merit.edu
On Tue, 27 Jun 2006, Jeremy Chadwick wrote:
> Was anyone able to get an RFO or post-mortem for this?
"An inaccurate set of BGP policies were distributed to routers connected
to AS791 1 when an automated update script ran at 1100 MDT. The update
script regularly ran every two hours to update the network with current
BGP information. Due to the scheduled shutdown of the legacy BGP policy
server and subsequent con version to the Level3 route registry engine, the
old server policy server was sh utdown. In addition, the scripts used to
update routes on the network were to be disabled. One of these scripts
wasn t disabled as intended. As a result, the script ran as scheduled at
1300MDT and consequently pushed partial configurations to production
routers because the script was unable to communicate with decommissioned
policy server. Incorrect policies were exchanged between AS7911 s
customers and peers resulted in increased latency; as large route blocks
attem pted to traverse individual customer connections.
Repair
Updated configurations were pushed to all the routers, individual
connections were cleaned up and BGP sessions were restored. In addition,
the automated BGP script has been shut-off. Maximum pre-fix list limits
have been established across the network as a risk mitigation step.
"
---
david raistrick http://www.netmeister.org/news/learn2quote.html
drais@atlasta.net http://www.expita.com/nomime.html
