[170866] in North American Network Operators' Group
RE: Chronic Abnormal Traceroutes Traversing Level 3
daemon@ATHENA.MIT.EDU (Jack Stonebraker)
Fri Apr 11 12:26:14 2014
From: Jack Stonebraker <Jack.Stonebraker@mygrande.com>
To: Trent Farrell <tfarrell@riotgames.com>
Date: Fri, 11 Apr 2014 16:25:25 +0000
In-Reply-To: <CAKDtZnnAA9vRtSL-5JF6oPateA2G_tPSP=jiLpGi3RLr=d6j-Q@mail.gmail.com>
Cc: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
Ah Ha! Thanks Nick and Trent! Well that explains the path being even at t=
he MPLS cloud.
JJ Stonebraker
IP Network Engineering
Grande Communications
512.878.5627
________________________________
From: Trent Farrell [mailto:tfarrell@riotgames.com]
Sent: Friday, April 11, 2014 11:19 AM
To: Jack Stonebraker
Cc: nanog@nanog.org
Subject: Re: Chronic Abnormal Traceroutes Traversing Level 3
RTT gets calculated before the packet enters a MPLS network.
On Fri, Apr 11, 2014 at 5:13 PM, Jack Stonebraker <Jack.Stonebraker@mygrand=
e.com<mailto:Jack.Stonebraker@mygrande.com>> wrote:
If there's anybody from Level 3 Transport available, I'd like to discuss so=
me bizarre results when traversing through your network, namely in Dallas, =
TX over the past few months? I'm working this through your NOC as well, bu=
t figured I would cover all avenues as this issue is pretty chronic.
Taken from your Looking Glass in Dallas to a destination of 195.110.36.136
1 vlan70.csw2.Dallas1.Level3.net<http://vlan70.csw2.Dallas1.Level3.net> (4.=
69.145.126) 120 msec
vlan60.csw1.Dallas1.Level3.net<http://vlan60.csw1.Dallas1.Level3.net> (=
4.69.145.62) 120 msec
vlan80.csw3.Dallas1.Level3.net<http://vlan80.csw3.Dallas1.Level3.net> (=
4.69.145.190) 120 msec
2 ae-93-93.ebr3.Dallas1.Level3.net<http://ae-93-93.ebr3.Dallas1.Level3.ne=
t> (4.69.151.170) 120 msec
ae-73-73.ebr3.Dallas1.Level3.net<http://ae-73-73.ebr3.Dallas1.Level3.ne=
t> (4.69.151.146) 120 msec
ae-63-63.ebr3.Dallas1.Level3.net<http://ae-63-63.ebr3.Dallas1.Level3.ne=
t> (4.69.151.134) 120 msec
3 ae-7-7.ebr4.Atlanta2.Level3.net<http://ae-7-7.ebr4.Atlanta2.Level3.net>=
(4.69.134.22) 36 msec 20 msec 20 msec
4 ae-2-2.ebr1.Washington1.Level3.net<http://ae-2-2.ebr1.Washington1.Level=
3.net> (4.69.132.86) 120 msec 116 msec 120 msec
5 ae-71-71.csw2.Washington1.Level3.net<http://ae-71-71.csw2.Washington1.L=
evel3.net> (4.69.134.134) 120 msec
ae-81-81.csw3.Washington1.Level3.net<http://ae-81-81.csw3.Washington1.L=
evel3.net> (4.69.134.138) 120 msec
ae-71-71.csw2.Washington1.Level3.net<http://ae-71-71.csw2.Washington1.L=
evel3.net> (4.69.134.134) 120 msec
6 ae-92-92.ebr2.Washington1.Level3.net<http://ae-92-92.ebr2.Washington1.L=
evel3.net> (4.69.134.157) 120 msec 120 msec
ae-72-72.ebr2.Washington1.Level3.net<http://ae-72-72.ebr2.Washington1.L=
evel3.net> (4.69.134.149) 124 msec
7 ae-44-44.ebr2.Paris1.Level3.net<http://ae-44-44.ebr2.Paris1.Level3.net>=
(4.69.137.61) 120 msec
ae-42-42.ebr2.Paris1.Level3.net<http://ae-42-42.ebr2.Paris1.Level3.net>=
(4.69.137.53) 116 msec 116 msec
8 ae-62-62.csw1.Paris1.Level3.net<http://ae-62-62.csw1.Paris1.Level3.net>=
(4.69.161.94) 124 msec
ae-72-72.csw2.Paris1.Level3.net<http://ae-72-72.csw2.Paris1.Level3.net>=
(4.69.161.98) 120 msec 120 msec
9 ae-81-81.ebr1.Paris1.Level3.net<http://ae-81-81.ebr1.Paris1.Level3.net>=
(4.69.161.85) 120 msec
ae-91-91.ebr1.Paris1.Level3.net<http://ae-91-91.ebr1.Paris1.Level3.net>=
(4.69.161.89) 124 msec
ae-71-71.ebr1.Paris1.Level3.net<http://ae-71-71.ebr1.Paris1.Level3.net>=
(4.69.161.81) 120 msec
10 ae-41-41.ebr2.London2.Level3.net<http://ae-41-41.ebr2.London2.Level3.ne=
t> (4.69.159.81) 120 msec 120 msec
ae-43-43.ebr2.London2.Level3.net<http://ae-43-43.ebr2.London2.Level3.ne=
t> (4.69.159.89) 120 msec
11 ae-12-3202.edge4.London2.Level3.net<http://ae-12-3202.edge4.London2.Lev=
el3.net> (4.69.202.182) 116 msec 120 msec 120 msec
12 ae-26-26.car2.London2.Level3.net<http://ae-26-26.car2.London2.Level3.ne=
t> (4.69.200.98) 120 msec 120 msec 120 msec
13 BACKBONE-CO.car2.London2.Level3.net<http://BACKBONE-CO.car2.London2.Lev=
el3.net> (195.50.121.50) 120 msec 120 msec 120 msec
14 * * *
15 * * *
16 * * *
I'm confused how the first and last hops are showing equal latency, while b=
eing half way around the world. I'm also confused why it's 120ms to the fi=
rst hop from a route server local to that market. While I wouldn't rely on=
a traceroute as a true measurement of end to end latency, I'm having probl=
ems explaining to customers experiencing tangible issues when their tracero=
ute looks like this:
traceroute source 24.155.144.226 195.110.36.136
traceroute to 195.110.36.136 (195.110.36.136) from 24.155.144.226, 30 hops =
max, 40 byte packets
1 lag-8-868.ear1.Dallas1.Level3.net<http://lag-8-868.ear1.Dallas1.Level3.=
net> (4.30.74.53) 924.705 ms 545.117 ms 512.992 ms
2 4.69.146.5 (4.69.146.5) 124.812 ms 4.69.146.21 (4.69.146.21) 125.686 =
ms 4.69.146.9 (4.69.146.9) 124.018 ms
MPLS Label=3D1965 CoS=3D0 TTL=3D1 S=3D1
3 ae-73-73.ebr3.Dallas1.Level3.net<http://ae-73-73.ebr3.Dallas1.Level3.ne=
t> (4.69.151.146) 125.012 ms ae-83-83.ebr3.Dallas1.Level3.net<http://ae-83=
-83.ebr3.Dallas1.Level3.net> (4.69.151.158) 141.585 ms ae-63-63.ebr3.Dalla=
s1.Level3.net<http://ae-63-63.ebr3.Dallas1.Level3.net> (4.69.151.134) 125.=
005 ms
MPLS Label=3D1810 CoS=3D0 TTL=3D1 S=3D1
4 * * *
5 ae-2-2.ebr1.Washington1.Level3.net<http://ae-2-2.ebr1.Washington1.Level=
3.net> (4.69.132.86) 126.085 ms 125.994 ms 127.148 ms
MPLS Label=3D1692 CoS=3D0 TTL=3D1 S=3D1
6 ae-61-61.csw1.Washington1.Level3.net<http://ae-61-61.csw1.Washington1.L=
evel3.net> (4.69.134.130) 124.500 ms ae-81-81.csw3.Washington1.Level3.net<=
http://ae-81-81.csw3.Washington1.Level3.net> (4.69.134.138) 125.369 ms ae-=
61-61.csw1.Washington1.Level3.net<http://ae-61-61.csw1.Washington1.Level3.n=
et> (4.69.134.130) 124.306 ms
MPLS Label=3D1555 CoS=3D0 TTL=3D1 S=3D1
7 ae-92-92.ebr2.Washington1.Level3.net<http://ae-92-92.ebr2.Washington1.L=
evel3.net> (4.69.134.157) 131.126 ms 128.607 ms ae-72-72.ebr2.Washington1=
.Level3.net<http://ae-72-72.ebr2.Washington1.Level3.net> (4.69.134.149) 12=
3.955 ms
MPLS Label=3D1636 CoS=3D0 TTL=3D1 S=3D1
8 ae-42-42.ebr2.Paris1.Level3.net<http://ae-42-42.ebr2.Paris1.Level3.net>=
(4.69.137.53) 127.156 ms ae-41-41.ebr2.Paris1.Level3.net<http://ae-41-41.=
ebr2.Paris1.Level3.net> (4.69.137.49) 125.736 ms ae-43-43.ebr2.Paris1.Leve=
l3.net<http://ae-43-43.ebr2.Paris1.Level3.net> (4.69.137.57) 143.070 ms
MPLS Label=3D1801 CoS=3D0 TTL=3D1 S=3D1
9 ae-82-82.csw3.Paris1.Level3.net<http://ae-82-82.csw3.Paris1.Level3.net>=
(4.69.161.102) 122.998 ms 122.245 ms ae-72-72.csw2.Paris1.Level3.net<htt=
p://ae-72-72.csw2.Paris1.Level3.net> (4.69.161.98) 124.499 ms
MPLS Label=3D1564 CoS=3D0 TTL=3D1 S=3D1
10 ae-61-61.ebr1.Paris1.Level3.net<http://ae-61-61.ebr1.Paris1.Level3.net>=
(4.69.161.77) 129.705 ms ae-81-81.ebr1.Paris1.Level3.net<http://ae-81-81.=
ebr1.Paris1.Level3.net> (4.69.161.85) 122.705 ms ae-91-91.ebr1.Paris1.Leve=
l3.net<http://ae-91-91.ebr1.Paris1.Level3.net> (4.69.161.89) 126.454 ms
MPLS Label=3D1322 CoS=3D0 TTL=3D1 S=3D1
11 ae-42-42.ebr2.London2.Level3.net<http://ae-42-42.ebr2.London2.Level3.ne=
t> (4.69.159.85) 126.019 ms 125.672 ms ae-44-44.ebr2.London2.Level3.net<h=
ttp://ae-44-44.ebr2.London2.Level3.net> (4.69.159.93) 127.026 ms
MPLS Label=3D1911 CoS=3D0 TTL=3D1 S=3D1
12 ae-12-3202.edge4.London2.Level3.net<http://ae-12-3202.edge4.London2.Lev=
el3.net> (4.69.202.182) 122.058 ms 124.301 ms 123.397 ms
MPLS Label=3D300112 CoS=3D0 TTL=3D1 S=3D1
13 ae-26-26.car2.London2.Level3.net<http://ae-26-26.car2.London2.Level3.ne=
t> (4.69.200.98) 124.003 ms 124.567 ms 130.346 ms
14 BACKBONE-CO.car2.London2.Level3.net<http://BACKBONE-CO.car2.London2.Lev=
el3.net> (195.50.121.50) 125.316 ms 125.991 ms 124.968 ms
15 * * *
This also doesn't seem to be circuit specific (Eliminating possible physica=
l circuit errors, etc) as I have a secondary 10-Gig sourcing out of San Ant=
onio, TX (Which ultimately terminates on your network in Dallas, same as th=
e Austin Circuit) which demonstrates similar results (120+ ms at the second=
hop in Dallas when entering your MPLS cloud). Killing the BGP adjacency t=
o both the Austin and San Antonio peers clears the issue up, but obviously =
that's not a long term viable option. Any help would be appreciated.
JJ Stonebraker
IP Network Engineering
Grande Communications
512.878.5627
--
Trent Farrell
Riot Games
IP Network Engineer
E: tfarrell@riotgames.com<mailto:tfarrell@riotgames.com> | IE: +353 86 845=
7230 | US: +1 424 285 9825
Summoner name: Foro
