[193704] in North American Network Operators' Group
Re: is something weird going on with cox, level3, and/or cogent
daemon@ATHENA.MIT.EDU (Mel Beckman)
Sun Feb 12 22:58:29 2017
X-Original-To: nanog@nanog.org
From: Mel Beckman <mel@beckman.org>
To: Miles Fidelman <mfidelman@meetinghouse.net>
Date: Mon, 13 Feb 2017 03:57:22 +0000
In-Reply-To: <C2193A5F-E9AC-4B92-B52C-7C306C3806EC@beckman.org>
Cc: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
Miles,
Have you tried trace routing through cellular data connections? The results=
you're seeing could be explained by congestion at the point of your modem,=
which I think is with the cox techs are implying.
-mel via cell
> On Feb 12, 2017, at 7:46 PM, Mel Beckman <mel@beckman.org> wrote:
>=20
> It looks like one or more circuits are down, so you're seeing asymmetrica=
l routing over congested paths in one direction.=20
>=20
> -mel via cell
>=20
>> On Feb 12, 2017, at 7:14 PM, Miles Fidelman <mfidelman@meetinghouse.net>=
wrote:
>>=20
>> Hi Folks,
>>=20
>> I'm visiting AZ, and seeing some really really poor performance accessin=
g some of our servers via Cox broadband. The folks at Cox technical suppor=
t are useless - all they say is "well you're on a DOCSIS 2 modem." Meanwhi=
le, everything I'm seeing is several hops upstream of the local segment - a=
nd all that Cox level2 tech support will say is "if there was a backbone pr=
oblem, our backbone people would have dealt with it."
>>=20
>> I'm having problems reaching both our own server, and sites like google,=
facebook, windows update.
>>=20
>> Traceroutes to and from our server are illustrative - note that for most=
of the past week, the average ping time was 85msec. Now we're seeing this:
>>=20
>> From 98.177.135.186 - the public IP address on Cox's local broadband ser=
vice.
>> To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierp=
oint data center, near Boston)
>> traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets
>> 1 10.128.128.1 (10.128.128.1) 199.893 ms 75.319 ms 27.295 ms
>> 2 100.127.69.178 (100.127.69.178) 38.710 ms 40.075 ms 43.598 ms
>> 3 72.215.229.22 (72.215.229.22) 39.674 ms 201.368 ms *
>> 4 lag-157.bear2.phoenix1.level3.net (4.28.82.53) 686.499 ms 1837.141 m=
s 16.273 ms
>> 5 ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 35.498 ms 964.377=
ms *
>> 6 ae-3-80.edge1.losangeles6.level3.net (4.69.144.144) 551.760 ms 525.=
014 ms
>> ae-1-60.edge1.losangeles6.level3.net (4.69.144.16) 2061.191 ms
>> 7 be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129) 847.778 ms 87=
.601 ms 71.504 ms
>> 8 be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1) 79.060 ms 225.6=
47 ms
>> be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77) 60.306 ms
>> 9 * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms 1=
85.180 ms
>> 10 be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66) 61.208 ms
>> be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78) 386.149 ms 1278=
.868 ms
>> 11 be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161) 384.136 ms
>> be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms 615=
.415 ms
>> 12 be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 233.061 ms
>> be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69) 87.902 ms
>> be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129) 861.159 ms
>> 13 be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221) 998.858 ms
>> be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157) 249.930 ms *
>> 14 be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 768.461 ms
>> be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105) 136.772 ms
>> be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109) 288.225 ms
>> 15 be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 271.736 ms 16=
6.224 ms
>> be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42) 565.015 ms
>> 16 te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms
>> te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms
>> te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms
>> 17 te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82) 107=
.207 ms
>> te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78) 295.8=
81 ms 185.453 ms
>> 18 38.122.127.18 (38.122.127.18) 115.652 ms 461.168 ms 615.526 ms
>> 19 207.154.0.57 (207.154.0.57) 1871.023 ms 1987.832 ms 2165.248 ms
>> 20 server1.ntcorp.com (207.154.13.58) 587.560 ms 263.328 ms 333.542 m=
s
>>=20
>>=20
>> Traceroute in the reverse direction:
>> 1 207.154.13.47 (207.154.13.47) 0.000 ms 0.000 ms 0.000 ms
>> 2 * * *
>> 3 h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms =
0.000 ms 0.000 ms
>> 4 xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms=
12.000 ms 12.000 ms
>> 5 et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms=
20.000 ms 20.000 ms
>> 6 10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37) 72.001 ms 2=
0.000 ms 16.000 ms
>> 7 chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193) 20.000 ms 20.000 m=
s 20.000 ms
>> 8 chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211) 72.001 ms 72.001 ms 72=
.001 ms
>> 9 * * *
>> 10 * * *
>> 11 * * *
>> <snip>
>> 30 * * *
>> <looks like a lot of the later hops don't respond to pings>
>>=20
>> Two things jump out at me:
>> 1. The rather large number of hops from cox to ntcorp - with high delays=
from several nodes in both the level3 and cogent networks.
>> 2. That there's a rather more direct path from the datacenter to cox, th=
at shows up in the reverse direction.
>>=20
>> Some kind of routing or peering issue, perhaps? (And I also note the ea=
rlier string of messages regarding youtube streaming problems - that also s=
eemed to involve cox and level3.
>>=20
>> Thanks for any insight (and better, for any fixes!).
>>=20
>> Miles Fidelman
>>=20
>>=20
>>=20
>>=20
>> --=20
>> In theory, there is no difference between theory and practice.
>> In practice, there is. .... Yogi Berra
>>=20
