[146990] in North American Network Operators' Group
Re: ATT GigE issue on 11/19 in Kansas City
daemon@ATHENA.MIT.EDU (Blake Hudson)
Wed Nov 30 10:51:55 2011
Date: Wed, 30 Nov 2011 09:51:00 -0600
From: Blake Hudson <blake@ispn.net>
To: nanog@nanog.org
In-Reply-To: <CACk08n9KYyOLbOG==X+8n4pByy46kMVyRkmRw4-752__cr=xpQ@mail.gmail.com>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
Stefan wrote the following on 11/30/2011 8:53 AM:
> On Wed, Nov 30, 2011 at 8:21 AM, Brad Fleming<bdflemin@gmail.com> wrote:
>> On Nov 29, 2011, at 8:17 PM,<comptech@kc.rr.com> wrote:
>>
>>> We lost several of our GigE links to AT&T for 6 hours on 11/19, anyone else see this and get a root cause from AT&T? All I can get is that they believe a change caused the issue.
>>>
>> We lost several (but not all) of our Optiman circuits on 11/19 at about 10:20am. We were told the root issue was that all VLANs in one of their switches had been accidentally deleted / removed. We were never able to get any additional detail (like "how") but services were restored about 16:45.
> +1 to the above - we received the following RFO, from the their NOC:
>
> "All impacted VLANS were rebuilt to restore service. It is believed
> there were some configuration changes that caused the VLAN troubles. A
> case has been opened with Cisco to further investigate the root
> cause."
>
Sounds like a VTP mishap.
