[160393] in North American Network Operators' Group
Re: L3 East cost maint / fiber 05FEB2012 maintenance
daemon@ATHENA.MIT.EDU (Jason Lixfeld)
Tue Feb 5 11:02:01 2013
From: Jason Lixfeld <jason@lixfeld.ca>
In-Reply-To: <CALKMwxkH4RsR-JtnnJFMMNXmb3Ld+N9914_YNMRYPKe2z8kcCg@mail.gmail.com>
Date: Tue, 5 Feb 2013 11:00:56 -0500
To: esseph@gmail.com
Cc: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
I got notification of their maintenance window, albeit with < 24 hours =
notice. Notice came in at 11:00GMT-5 yesterday, maintenance was =
scheduled for 00:00GMT-5 this morning.
That said, the notice said that the maintenance was in Phoenix but I got =
a notice about my IPT circuit at 60 Hudson which I found confusing.
Based on my logs, our BGP session with them went down at 03:06GMT-5 and =
back up at 03:15GMT-5. Down again at 03:37GMT-5 until 04:20GMT-5. A =
third time at 06:41GMT-5 and back at 06:45GMT-5.
Traffic graphs tell a bit of a different story. Just before 05:00GMT-5, =
our outbound traffic to Level 3 dropped substantially. About that time, =
I started getting reports about issues to Level 3 destinations. Traces =
seemed to indicate a black hole condition within Level 3's network in =
NYC, seemingly at, or just past csw3.NewYork1.Level3.net. Stuff seemed =
to correct itself by about 06:45GMT-5, but due to Level 3 sending only =
about 180k routes. About 20 minutes later, the table was back to ~431K =
and all's been fine since.
On 2013-02-05, at 10:39 AM, Josh Reynolds <esseph@gmail.com> wrote:
> I know a lot of you are out of the office right now, but does anybody =
have
> any info on what happened with L3 this morning? They went into a 5 =
hour
> maintenance window with expected downtime of about 30 minutes while =
they
> upgraded something like *40* of their "core routers" (their words), =
but
> also did this during some fiber work and completely cut off several of
> their east coast peers for the entirety of the 5 hour window.
>=20
> If anybody has any more info on this, on a NOC contact for them on the =
East
> Coast for future issues, you can hit me off off-list if you don't feel
> comfortable replying with that info here.
>=20
> Thanks, and I hope hope you guys are enjoying Orlando.
>=20
> --=20
> *Josh Reynolds*
> esseph@gmail.com - (270) 302-3552
