[191975] in North American Network Operators' Group
RE: Level 3 voice outage
daemon@ATHENA.MIT.EDU (Gareth Tupper)
Wed Oct 5 13:11:07 2016
X-Original-To: nanog@nanog.org
From: Gareth Tupper <Gareth.Tupper@warnerpacific.com>
To: NANOG list <nanog@nanog.org>
Date: Wed, 5 Oct 2016 17:10:25 +0000
In-Reply-To: <CCFB6D5D-8816-48A3-9C3D-273F4A8C1B16@beckman.org>
Errors-To: nanog-bounces@nanog.org
Looks like a fat finger event...
From Level 3:
"On October 4, our voice network experienced a service disruption affecting=
some of our customers in North America due to a configuration error. We kn=
ow how important these services are to our customers. As an organization, w=
e're putting processes in place to prevent issues like this from recurring =
in the future. We were able to restore all services by 9:31am Mountain time=
."
http://www.theregister.co.uk/2016/10/05/level3_voip_blackout_cause/
-----Original Message-----
From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mel Beckman
Sent: Tuesday, October 4, 2016 1:09 PM
To: Marco Teixeira <admin@marcoteixeira.com>
Cc: Shawn Ritchie <shawnritchie@gmail.com>; NANOG list <nanog@nanog.org>
Subject: Re: Level 3 voice outage
Possibly somebody YANGed when they should have yinged :)
-mel beckman
On Oct 4, 2016, at 1:06 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:=
admin@marcoteixeira.com>> wrote:
Yeap, i know, it was what i understood, as it is my opinion that a zero day=
would fit better... in the pure speculation world :) At the end of the day=
... maybe some undocumented fault int some obscure functionality that was a=
ctivated/deployed a long time ago, and just revealed it self now... There a=
re so many things that can go wrong on complex networks even with all the c=
ontrols imposed on changes...
On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie <shawnritchie@gmail.com<mailt=
o:shawnritchie@gmail.com>> wrote:
Well, Level3 has by no means said that this was the result of a DDoS, that'=
s just speculation on behalf of folks who do not work at Level3 so far.
On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira <admin@marcoteixeira.com<mail=
to:admin@marcoteixeira.com>> wrote:
I won't believe a company like Level3 would not deploy backplane protection=
/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network didn'=
t pause or rebooted routers. And i guess both companies have had their shar=
e of (D)DoS in the past, so they had the time to get up to the challenge. N=
ow... there where times where one malformed IP packet would cause a memory =
leak leading to a router reboot... :)?
On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman <mel@beckman.org<mailto:mel@bec=
kman.org>> wrote:
> 765 Gbps per second directed at a router's interface IP might give the
> router pause, so to speak :)
>
> -mel
>
> On Oct 4, 2016, at 12:10 PM, Marco Teixeira
> <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>>
> wrote:
>
> Multiple reboots across several markets... Does not seem something
> that full pipes would trigger. Had it been an approved chance it would
> have been rolled back i guess... On the other hand, a zero day could appl=
y...
>
> Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org<mailto:mel@beckman.or=
g>> escreveu:
>
>> Sure. The recent release of the IoT DDoS attack code in the wild.
>>
>> -mel
>>
>> > On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu<mailto:Valdis.Kle=
tnieks@vt.edu> wrote:
>> >
>> > On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
>> >
>> >> This could be DoS attack.
>> >
>> > Or a missing comma in a code update.
>> >
>> > Or a fumble-fingered NOC monkey.
>> >
>> > Or....
>> >
>> > You have any reason to suspect a DoS attack rather than all the
>> > other possibilities?
>>
>>
>
--
--
Shawn
This electronic mail transmission contains information from Warner Pacific =
Insurance Services that may be confidential or privileged. Such information=
is solely for the intended recipient, and use by any other party is not au=
thorized. If you are not the intended recipient, be aware that any disclosu=
re, copying, distribution or use of this message, its contents or any attac=
hments is prohibited. Any wrongful interception of this message is punishab=
le as a Federal Crime. If you have received this message in error, please n=
otify the sender immediately by telephone (800) 801-2300 or by electronic m=
ail at postmaster@warnerpacific.com.
