[147002] in North American Network Operators' Group
Re: ATT GigE issue on 11/19 in Kansas City
daemon@ATHENA.MIT.EDU (Brad Fleming)
Wed Nov 30 12:38:13 2011
From: Brad Fleming <bdflemin@gmail.com>
In-Reply-To: <4ED650E4.3050705@ispn.net>
Date: Wed, 30 Nov 2011 11:37:18 -0600
To: Blake Hudson <blake@ispn.net>
Cc: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
On Nov 30, 2011, at 9:51 AM, Blake Hudson wrote:
> Stefan wrote the following on 11/30/2011 8:53 AM:
>> On Wed, Nov 30, 2011 at 8:21 AM, Brad Fleming<bdflemin@gmail.com> =
wrote:
>>> On Nov 29, 2011, at 8:17 PM,<comptech@kc.rr.com> wrote:
>>>=20
>>>> We lost several of our GigE links to AT&T for 6 hours on 11/19, =
anyone else see this and get a root cause from AT&T? All I can get is =
that they believe a change caused the issue.
>>>>=20
>>> We lost several (but not all) of our Optiman circuits on 11/19 at =
about 10:20am. We were told the root issue was that all VLANs in one of =
their switches had been accidentally deleted / removed. We were never =
able to get any additional detail (like "how") but services were =
restored about 16:45.
>> +1 to the above - we received the following RFO, from the their NOC:
>>=20
>> "All impacted VLANS were rebuilt to restore service. It is believed
>> there were some configuration changes that caused the VLAN troubles. =
A
>> case has been opened with Cisco to further investigate the root
>> cause."
>>=20
>=20
> Sounds like a VTP mishap.
>=20
That was my first thought as well.. it would just surprise me if a huge =
provider like AT&T was using VTP instead of using a provisioning tool =
that automates the manual pruning process to avoid issues like this. In =
either case I'm a customer and will likely never be told what went =
wrong. I'm OK with that so long as it doesn't happen again!=
