[51508] in North American Network Operators' Group
RE: AT&T NYC
daemon@ATHENA.MIT.EDU (Frank Scalzo)
Wed Aug 28 23:23:36 2002
Date: Wed, 28 Aug 2002 23:21:04 -0400
From: "Frank Scalzo" <frank.scalzo@amerinex.net>
To: "Matt Levine" <matt@deliver3.com>,
"Mike Tancsa" <mike@sentex.net>
Cc: "Wes Bachman" <wbachman@leepfrog.com>, <nanog@merit.edu>
Errors-To: owner-nanog-outgoing@merit.edu
Whoops! 2 hours to find routers w/o an IGP tsk tsk.
Dear AT&T IP Services Customer,
Please be advised of the following:
Date: 8/28/02
Customer Care TT#: 1070909
Start of Impairment: 14:56 ET
End of Impairment: 16:52 ET
On behalf of AT&T, we would like to extend our apologies for any =
inconveniences to your business caused by the
service impairment and delay in restoring your service.
AT&T IP Services customers with traffic routed through our Chicago =
backbone routers may have been affected by
this impairment.
Our Network Engineers found that OSPF* network statements were missing =
from two backbone routers, causing the
routing issues that customers were experiencing. AT&T Network =
Engineers manually reloaded the OSPF network
statements and then proceeded to reboot the routers in order for the =
changes to take effect.=20
The network statements were mistakenly deleted during a routine =
configuration update to our backbone routers.=20
Management has been made aware of these findings and appropriate =
actions will be taken to ensure this situation
does not re-occur.
Prior reported information was leading the AT&T Network Engineers to =
believe that there were issues with the route
reflectors being out of synch. However, this was only symptomatic of =
the problem stated above. In other words, due
to the missing network statements, the route reflectors were not =
advertising certain networks; therefore traffic was
not properly routed.
It is AT&T=92s goal to provide the highest level of service to our =
customers.=20
We appreciate your business and look forward to continuing our =
relationship in the future.
*Routing protocol (Open Shortest Path First)
Depending on the particular IP access services to which you are =
subscribed you may also find additional information
as it becomes available at:
AT&T MIS:
https://mis-att.bus.att.com/
AT&T VPNS:
http://www.vpn.att.net
If you require further information, please feel free to contact AT&T =
at:
AT&T MIS: 1-888-613-6330
AT&T VPNS: 1-888-613-6501
Thank you for using AT&T.
Sincerely,
The AT&T Customer Care Team
-----Original Message-----
From: Matt Levine [mailto:matt@deliver3.com]
Sent: Wed 8/28/2002 6:21 PM
To: Mike Tancsa
Cc: Wes Bachman; nanog@merit.edu
Subject: Re: AT&T NYC
On Wednesday, August 28, 2002, at 04:17 PM, Mike Tancsa wrote:
>
>
> route-server.ip.att.net is not currently reachable, but AS15290's=20
> router server is for those who want a view on things...
Interestingly enough, ATT is announcing 12.0.0.0/23 to BBN (and nobody=20
else, including AS7018 internal)..
>
> route-server.east.attcanada.com.
> and
> route-server.west.attcanada.com.
>
> which come in handy :-)
>
> ---Mike
>
> At 04:11 PM 28/08/2002 -0400, Mike Tancsa wrote:
>
>
>> I am seeing this as well. One of my upstreams (AT&T Canada- 15290)=20
>> has connections with AT&T US (7018) in Chicago and Vancouver. =20
>> Chicago seems to have disappeared for me and all traffic bound via=20
>> that path is going via Vancouver now.
>>
>> ---Mike
>>
>> At 02:52 PM 28/08/2002 -0500, Wes Bachman wrote:
>>
>>> Bryan,
>>>
>>> There is a known AT&T outage in Chicago currently. Could this be
>>> effecting you in some way?
>>>
>>> -Wes
>>>
>>> On Wed, 2002-08-28 at 14:44, Bryan Heitman wrote:
>>> >
>>> > Anyone seeing any problems with ATT in new york?
>>> >
>>> >
>>> > Best regards,
>>> >
>>> >
>>> > Bryan Heitman
>>> > Interland, Inc.
>>> --
>>> Wes Bachman
>>> System & Network Administration, Software Development
>>> Leepfrog Technologies, Inc.
>>> wbachman@leepfrog.com
>
>
--
Matt Levine
@Home: matt@deliver3.com
@Work: matt@eldosales.com
ICQ : 17080004
AIM : exile
GPG : http://pgp.mit.edu:11371/pks/lookup?op=3Dget&search=3D0x6C0D04CF
"The Trouble with doing anything right the first time is that nobody
appreciates how difficult it was." -BIX
