[162905] in North American Network Operators' Group
Re: CenturyLink Outage?
daemon@ATHENA.MIT.EDU (Joshua Goldbard)
Tue May 7 13:18:31 2013
From: Joshua Goldbard <j@2600hz.com>
To: Jason Lester <jlester@wcs.k12.va.us>
Date: Tue, 7 May 2013 17:17:36 +0000
In-Reply-To: <CANoO+=9HOVyyqb5-mGJvnO-64Dt0OCfTATVGnnniXh8P=3s9KQ@mail.gmail.com>
Cc: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
Outages list is going bananas right now:
Same thing happened in Monroe LA that took down all of north Louisiana. It =
was an update that went bad and the switch had to be manually checked line =
by line. No backup was done before the maintenance. 16 hours of downtime ab=
out 45 days ago.
On Tuesday, May 7, 2013, Frank Bulk wrote:
Sounds like it was scheduled maintenance gone bad:
"Perrine said he spoke with CenturyLink at 6 a.m.<x-apple-data-dete=
ctors://1>
where they advised him the company was doing scheduled
maintenance. At 7:35 a.m.<x-apple-data-detectors://2> they told him=
something
had gone wrong during the maintenance and it was
affecting customers in 13 states."
http://www.tallahassee.com/article/20130507/NEWS/130507010/Centurylink-outa=
g
e-affecting-local-internet-provider?nclick_check=3D1
I bite my tongue in regards to Network Tallahasee's singlehomedness and
Perrine's comment on his questioning of the timing of the update.
I encourage all my competitors to stay single homed and to do their
maintenance Sunday morning<x-apple-data-detectors://4>, while their vendors=
are at their lowest staffing
levels.
Frank
-----Original Message-----
From: Outages [mailto:outages-bounces@outages.org<x-apple-msg-load://2CF090=
1A-5EB4-4331-8B21-8DE35C0C12D0/>] On Behalf Of Frank Bulk
Sent: Tuesday, May 07, 2013 8:29 AM
To: 'Marco Prechel'; outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4=
331-8B21-8DE35C0C12D0/>
Subject: Re: [outages] Centurylink nationwide outage
I first saw timeouts with www.qwest.com<http://www.qwest.com/> over IPv6 at=
3:08 am Central<x-apple-data-detectors://9> -- I
was wondering why, now I know. =3D)
Frank
-----Original Message-----
From: Outages [mailto:outages-bounces@outages.org<x-apple-msg-load://2CF090=
1A-5EB4-4331-8B21-8DE35C0C12D0/>] On Behalf Of Marco
Prechel
Sent: Tuesday, May 07, 2013 7:33 AM
To: outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C=
12D0/>
Subject: [outages] Centurylink nationwide outage
Saw our Centurylink Ethernet circuit lose L3 connectivity in SWFL around
0400 EDT.
ANS/ABS states it's a nationwide routing issue. No ETR.
_______________________________________________
Outages mailing list
Outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0=
/>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0=
/>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org<x-apple-msg-load://2CF0901A-5EB4-4331-8B21-8DE35C0C12D0=
/>
https://puck.nether.net/mailman/listinfo/outages
--
Jeremie Chism
Triton Communications
_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
Sent from my iPad
On May 7, 2013, at 10:16 AM, "Jason Lester" <jlester@wcs.k12.va.us<mailto:j=
lester@wcs.k12.va.us>> wrote:
Does anyone know what is going on with the nationwide CenturyLink outage?
Their NOC recording says it is a BGP routing issue with their upstream
peers affecting Internet traffic and traffic between regions. Our outside
connectivity with them has basically been down since about 4:00AM (EDT)
this morning. The prefixes we were receiving from them were fluctuating
between a few hundred and a few thousand all morning. We are getting the
full BGP table from them now (for about the last hour), but still not
seeing any incoming traffic. Seems like a major issue since it has been
almost 9 hours now.
Thanks,
Jason
--
Jason Lester
Administrator for Instructional Technology
Washington County Public Schools
Tel: 276-739-3060
Fax: 276-628-1893
http://www.wcs.k12.va.us
