[107323] in North American Network Operators' Group
Re: GNi/365main Above.net peering problems?
daemon@ATHENA.MIT.EDU (Rusty Hodge)
Sat Aug 30 16:55:35 2008
From: Rusty Hodge <rusty@hodge.com>
To: nanog@nanog.org
In-Reply-To: <76091495-11B2-4B98-A4B2-8FC78ABCFA32@hodge.com>
Date: Sat, 30 Aug 2008 13:55:28 -0700
Errors-To: nanog-bounces@nanog.org
Received this update for GNi:
> At this time we believe we have found a Cisco day 0 network
> vulnerability.
>
> We have 20+ routers in our core network - 6 of the 20 have the
> identical route
> processor and IOS version. These 6 have been affected in 3 separate
> geographical locations in the past several days. The network issues
> range
> from simple SNMP failure to loss of BGP or OSPF communications.
> These are
> creating black holes intermittently across our network. We are
> experiencing as
> much as 20% of networks not available at this time.
>
> We have been coordinating with Cisco for the past two days and have
> deployed
> patches to address the problem. Overall, we have determined that
> the fastest,
> surest path to restoring 100% network normality is to replace these
> 6 routers.
> We will continue this process to replace these 6 routers.
>
> Two of the six affected routers have already been replaced and
> clients have
> been moved. We are now replacing the remaining 4 affected routers.
> We know that this week has been very painful and frustrating for
> you. We will
> provide a detailed and open RFO Reason For Outage post mortem
> document as soon
> as we have the replaced the affected routers.
>
> Also, I will be updating all customers who have opened tickets at
> least every
> two hours from this time to ensure that all customers have the
> latest status
> information. If you would prefer a phone call status report, please
> note it
> on the ticket and I will also give you a call to answer any additional
> questions you may have. In addition, you may call the NOC at your
> convenience
> at (415) 979-9786.
>
> Thank you for your patience.
