[174182] in North American Network Operators' Group
RE: Time Warner outage?
daemon@ATHENA.MIT.EDU (Warren Bailey)
Thu Aug 28 10:23:29 2014
X-Original-To: nanog@nanog.org
From: Warren Bailey <wbailey@satelliteintelligencegroup.com>
To: "Naslund, Steve" <SNaslund@medline.com>, "nanog@nanog.org"
<nanog@nanog.org>
Date: Thu, 28 Aug 2014 14:19:00 +0000
In-Reply-To: <9578293AE169674F9A048B2BC9A081B40124F7B435@MUNPRDMBXA1.medline.com>
Reply-To: Warren Bailey <wbailey@satelliteintelligencegroup.com>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
As others have said, it's difficult to rationalize a situation where dhcp b=
reaks and your existing routing breaks as well. Especially one that kills t=
he Internet for a couple 100k subs. I'd shoot flames, but all too often I'v=
e been the guy working something like this after some maintenance became "s=
ervice affecting".. Lol
If you've ever been responsible for a large Internet connection with a lot =
of subs, have a beer for the guy answering emails from some no name middle =
manager about his new found PEP right now. ;)
Sent from my T-Mobile 4G LTE Device
-------- Original message --------
From: "Naslund, Steve" <SNaslund@medline.com>
Date: 08/28/2014 7:06 AM (GMT-08:00)
To: nanog@nanog.org
Subject: RE: Time Warner outage?
I don=92t buy that excuse either. If Level 3 is doing fiber maintenance on=
any route and takes down your entire network, then you have a pretty poor =
backbone design. It is not Level 3s fault if you design your network such =
that a single route loss causes a huge outage.
Steven Naslund
Chicago IL
From: Chris Lane [mailto:clane1875@gmail.com]
Sent: Thursday, August 28, 2014 8:51 AM
To: Warren Bailey
Cc: Chris Garrett; Naslund, Steve; nanog@nanog.org
Subject: Re: Time Warner outage?
Agreed on DHCP, just passing along something i had heard about..... With =
that said, why wouldn't the TW guys just post something oh right they did, =
blame the other guy "Level3"
On Thu, Aug 28, 2014 at 9:42 AM, Warren Bailey <wbailey@satelliteintelligen=
cegroup.com<mailto:wbailey@satelliteintelligencegroup.com>> wrote:
Sounds more likely than "we broke the dhcp server".
Sent from my T-Mobile 4G LTE Device
-------- Original message --------
From: Chris Garrett <chris@aperturefiber.com<mailto:chris@aperturefiber.com=
>>
Date: 08/28/2014 6:39 AM (GMT-08:00)
To: "Naslund, Steve" <SNaslund@medline.com<mailto:SNaslund@medline.com>>
Cc: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Time Warner outage?
I have never seen BGP drop peers because of a DHCP failure.
I would love to see the breakdown on that.
TW NOC had said it was a Level 3 fiber maintenance gone bad that took down =
their peering connections when we called in at the time of the event.
Can anyone confirm one way or the other?
On Aug 28, 2014, at 8:29 AM, Naslund, Steve <SNaslund@medline.com<mailto:SN=
aslund@medline.com>> wrote:
> Something sounds really unlikely about that. Lack of DHCP would not caus=
e reachability problems except for the clients. The trace below looks like=
a transit connection that should be unaffected by DHCP. Looks more like a=
routing issue. Also sounds unlikely that one DHCP server would be coverin=
g this large a scope.
>
> Steven Naslund
> Chicago IL
>
> -----Original Message-----
> From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.or=
g>] On Behalf Of Chris Lane
> Sent: Thursday, August 28, 2014 8:23 AM
> To: Stephen Satchell
> Cc: nanog@nanog.org<mailto:nanog@nanog.org>
> Subject: Re: Time Warner outage?
>
> I heard the following,
>
> It was actually an engineer doing maintenance on the dhpc server that sto=
pped customers from getting an IP address when the connected to the network=
between 5:30 and 7:00. The funny part is upper Managment heard about it o=
n the today show
>
>
> On Thu, Aug 28, 2014 at 8:46 AM, Stephen Satchell <list@satchell.net<mail=
to:list@satchell.net>> wrote:
>
>> This just keeps getting better and better:
>>
>> "Yahoo Logo
>> Will be right back...
>>
>> Thank you for your patience.
>>
>> Our engineers are working quickly to resolve the issue."
>>
>> My upstream is Charter Business...
>>
>> On 08/28/2014 04:05 AM, Chris Garrett wrote:
>>> I believe this is what is commonly referred to in our industry as a
>> =93resume generating event=94.
>>>
>>>
>>> On Aug 27, 2014, at 8:24 AM, Alexandru Suciu
>>> <alex.suciu.sum@gmail.com<mailto:alex.suciu.sum@gmail.com>>
>> wrote:
>>>
>>>> Can't wait to see the postmortem report on this one.
>>>>
>>>> On 8/27/2014 3:05 PM, Adam Greene wrote:
>>>>> Came back for us, too, at about 7am.
>>>>>
>>>>> Outage was a biggie this time:
>>>>>
>>>>>
>> https://news.yahoo.com/time-warner-cable-suffering-massive-nationwide-
>> outage
>>>>> -110549606.html
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nano=
g.org>] On Behalf Of David
>> Coulson
>>>>> Sent: Wednesday, August 27, 2014 6:59 AM
>>>>> To: Rick Coloccia
>>>>> Cc: <nanog@nanog.org<mailto:nanog@nanog.org>>
>>>>> Subject: Re: Time Warner outage?
>>>>>
>>>>> Just came back up for me.
>>>>>
>>>>> Sent from my iPad
>>>>>
>>>>>> On Aug 27, 2014, at 6:48 AM, Rick Coloccia <coloccia@geneseo.edu<mai=
lto:coloccia@geneseo.edu>>
>> wrote:
>>>>>>
>>>>>> BGPMON shows my routes falling off the net at around 5:49am.
>>>>>>
>>>>>> We now sit at their mercy....
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sent from my iPhone
>>>>>>
>>>>>>> On Aug 27, 2014, at 6:46 AM, "Adam Greene"
>>>>>>> <maillist@webjogger.net<mailto:maillist@webjogger.net>>
>>>>> wrote:
>>>>>>> Same here. Seems like no traffic is exiting TWC:
>>>>>>>
>>>>>>> Tracing route to ns03.savvis.net<http://ns03.savvis.net> [204.70.25=
.234<tel:%5B204.70.25.234>] over a maximum
>>>>>>> of
>> 30
>>>>>>> hops:
>>>>>>>
>>>>>>> 1 1 ms 1 ms 1 ms 10049.webjogger.net<http://10049.webj=
ogger.net> [204.8.80.49]
>>>>>>> 2 27 ms 6 ms 5 ms cpe-24-29-112-25.nyc.res.rr.com<http:=
//cpe-24-29-112-25.nyc.res.rr.com>
>>>>>>> [24.29.112.25]
>>>>>>> 3 12 ms 8 ms 7 ms rdc-69-193-225-74.nyc.bc.twcable.com<=
http://rdc-69-193-225-74.nyc.bc.twcable.com>
>>>>>>> [69.193.225.74]
>>>>>>> 4 9 ms 9 ms 12 ms rdc-69-193-225-137.nyc.bc.twcable.com=
<http://rdc-69-193-225-137.nyc.bc.twcable.com>
>>>>>>> [69.193.225.137]
>>>>>>> 5 9 ms 10 ms 8 ms rdc-69-193-225-222.nyc.bc.twcable.com=
<http://rdc-69-193-225-222.nyc.bc.twcable.com>
>>>>>>> [69.193.225.222]
>>>>>>> 6 * * * Request timed out.
>>>>>>> 7 * * * Request timed out.
>>>>>>> 8 * * * Request timed out.
>>>>>>> 9 * * * Request timed out.
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: NANOG [mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@na=
nog.org>] On Behalf Of Rick
>>>>>>> Coloccia
>>>>>>> Sent: Wednesday, August 27, 2014 6:43 AM
>>>>>>> To: nanog@nanog.org<mailto:nanog@nanog.org>
>>>>>>> Subject: Re: Time Warner outage?
>>>>>>>
>>>>>>> My whole campus (~10000 users) is down... Since roughly 6am.
>>>>>>> TWC is our upstream.
>>>>>>>
>>>>>>> --
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>>> On Aug 27, 2014, at 6:28 AM, Rob Barbeau
>>>>>>>> <rob.barbeau@gmail.com<mailto:rob.barbeau@gmail.com>>
>> wrote:
>>>>>>>>
>>>>>>>> David,
>>>>>>>>
>>>>>>>> I have a branch office in Syracuse,NY that appears to be down
>>>>>>>> at the moment that uses a time warner business connection for
>>>>>>>> internet
>> access.
>>>>>>>>
>>>>>>>> -rob
>>>>>>>> On Aug 27, 2014 5:20 AM, "David Hubbard"
>>>>>>>> <dhubbard@dino.hostasaurus.com<mailto:dhubbard@dino.hostasaurus.co=
m>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hey all, anyone else having issues with Time Warner
>>>>>>>>> residential or business connections? One of our offices is
>>>>>>>>> down and the route is not currently in bgp.
>>>>>>>>> http://downdetector.com/status/time-warner-cable
>>>>>>>>> shows thousands of reports of outages on the consumer side
>>>>>>>>> starting an hour or so ago so I figure it's a larger issue
>>>>>>>>> than just my one office; couldn't reach anyone by phone.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> David
>>>>
>>>
>>
>>
>
>
> --
> //CL
--
//CL
