[188461] in North American Network Operators' Group
Re: NTT communications horrible routing, unresponsive NOC
daemon@ATHENA.MIT.EDU (Jared Mauch)
Thu Mar 24 08:12:50 2016
X-Original-To: nanog@nanog.org
From: Jared Mauch <jared@puck.nether.net>
In-Reply-To: <CAJayEpF3=NYtZoJ0tZ_1D3z_ZGP1tL0fye4-EiRf1AmdDc8nbA@mail.gmail.com>
Date: Thu, 24 Mar 2016 08:12:10 -0400
To: Paras Jha <paras@protrafsolutions.com>
Cc: NANOG list <nanog@nanog.org>
Errors-To: nanog-bounces@nanog.org
Hello,
Could you send me some more details in private about who you spoke with incl=
uding email addresses and ticket numbers with our NOC if any?
I'd like to understand what happened here as this is an uncharacteristic out=
come. I know there was planned work in Seattle last night for a software upg=
rade combined with hardware work, but an operational issue like this should h=
ave been resolved.=20
I'd like to understand the timeline and what broke down if anything.=20
Thanks,
Jared Mauch
> On Mar 23, 2016, at 7:39 PM, Paras Jha <paras@protrafsolutions.com> wrote:=
>=20
> Hi all,
>=20
> I've been trying to get this issue resolved for the entire day now, but NT=
T
> has been pretty unreceptive here.
>=20
> We're announcing a large prefix for a client across our network, and we
> discovered some insanely high latency.
>=20
> After tracking down the issue, we determined it to be something wrong with=
> NTT at their Seattle location. We anycast this prefix, but no matter where=
> in the world traffic is originating from, it's going to Seattle and then t=
o
> Atlanta. Example: Rotterdam in the Netherlands routes from Europe -> east
> coast -> west coast Seattle -> los angeles -> atlanta. The gist of it is
> that something is seriously messed up at NTT in Seattle.
>=20
> We contacted our transit provider to try and carry the issue upstream, and=
> what they told us was
>=20
> Sorry for delay, I've asked NTT to clear the more specific for this one as=
> well.
> The problem seems to be a bug on the NTT side which keeps stale routes in
> the routing table for more specifics ( at random ).
> If you have more routes affected please notify us of the routes and I will=
> ask them to clear the routing table for these routes.
> NTT is working on this with their vendor to get this resolved as soon as
> possible.
>=20
>=20
> I had spoken to a sales rep for NTT a few weeks prior, and they assured me=
> that the NOC was top notch, and that all routes were redundant, and they
> guaranteed less than 50ms in the US, and all kinds of marketing. However,
> it looks like it's all marketing - for this entire day this router has bee=
n
> causing tons of issues for our clients.
>=20
> No-exporting it to NTT does not even solve the problem, as NTT's router in=
> Seattle apparently just decides to keep random small prefixes in it,
> causing traffic to go there.
>=20
> At a loss as to what to do now, since their NOC isn't receptive. Anyone
> have someone I can contact off-list to get this issue resolved? It's
> especially frustrating because the problem absolutely cannot be resolved o=
n
> our end, even with a no-export since NTT is keeping the routes in their
> router.
