[193707] in North American Network Operators' Group
Re: is something weird going on with cox, level3, and/or cogent
daemon@ATHENA.MIT.EDU (Mel Beckman)
Mon Feb 13 03:14:23 2017
X-Original-To: nanog@nanog.org
From: Mel Beckman <mel@beckman.org>
To: Miles Fidelman <mfidelman@meetinghouse.net>
Date: Mon, 13 Feb 2017 06:36:40 +0000
In-Reply-To: <e753450f-3ba9-404c-479d-2b81540e7740@meetinghouse.net>
Cc: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: nanog-bounces@nanog.org
It helps to follow some general guidelines to traceroute interpretation:
1. Intermediate hop time is not necessarily a measure of performance, as tr=
aceroute TTL expiration processing has low priority.
2. Time measurements areround-trip, not latency, and the bulk of the time m=
ay be incurred on the return trip. The only way to know for sure is bidirec=
tional traceroutes.
3. Seeing reported latency in the first few hops indicates a probable issue=
on the local network level.
In your original traceroute, latency jumps from 40ms to 200ms at 72.215.229=
.22, which is a Cox address. Since this is so close to you (third hop), I w=
ould apply guideline #3. All the hops after that have high latencies too, b=
ut if most of that is being injected in hop 3 then that means the problem s=
tarted there.
Verify that you don=92t have high traffic volume on your individual Cox cir=
cuit (e.g. a backup running or something). If that=92s not the case, then y=
ou=92re likely seeing a Cox problem with their own internal traffic enginee=
ring. If Cox is doing weekend maintenance, some circuits may be out of serv=
ice for a while. Cox is not good about passing that info down to residentia=
l customers, so you may never know what happened. A DIA business customer c=
an usually ask about global tickets, but Cox is loathe to give that informa=
tion out even then.
-mel
On Feb 12, 2017, at 8:27 PM, Miles Fidelman <mfidelman@meetinghouse.net<mai=
lto:mfidelman@meetinghouse.net>> wrote:
Now isn't that interesting.
1. Verizon mobile seems to also route through level3 and cogent.
2. But.. the performance seems to be a lot better.
3. That's really odd.
1 192.168.43.1 (192.168.43.1) 2.063 ms 2.382 ms 1.913 ms
2 7.sub-66-174-33.myvzw.com<http://7.sub-66-174-33.myvzw.com> (66.174.33.=
7) 50.676 ms 36.720 ms 41.142 ms
3 164.sub-69-83-172.myvzw.com<http://164.sub-69-83-172.myvzw.com> (69.83.=
172.164) 39.831 ms 40.240 ms 40.190 ms
4 178.sub-69-83-173.myvzw.com<http://178.sub-69-83-173.myvzw.com> (69.83.=
173.178) 39.851 ms 48.650 ms 30.921 ms
5 194.sub-69-83-173.myvzw.com<http://194.sub-69-83-173.myvzw.com> (69.83.=
173.194) 49.134 ms 31.131 ms 39.946 ms
6 8.sub-69-83-162.myvzw.com<http://8.sub-69-83-162.myvzw.com> (69.83.162.=
8) 42.841 ms 39.532 ms 40.024 ms
7 73.sub-66-174-29.myvzw.com<http://73.sub-66-174-29.myvzw.com> (66.174.2=
9.73) 44.074 ms 31.248 ms 42.199 ms
8 5-1-1.bear1.phoenix1.level3.net<http://5-1-1.bear1.phoenix1.level3.net>=
(4.16.142.89) 38.474 ms 39.555 ms 57.860 ms
9 ae-2-70.edge1.losangeles6.level3.net<http://ae-2-70.edge1.losangeles6.l=
evel3.net> (4.69.144.80) 86.101 ms 78.856 ms
ae-3-80.edge1.losangeles6.level3.net<http://ae-3-80.edge1.losangeles6.l=
evel3.net> (4.69.144.144) 70.001 ms
10 ae-4-90.edge1.losangeles6.level3.net<http://ae-4-90.edge1.losangeles6.l=
evel3.net> (4.69.144.208) 48.840 ms
ae-2-70.edge1.losangeles6.level3.net<http://ae-2-70.edge1.losangeles6.l=
evel3.net> (4.69.144.80) 71.847 ms
ae-3-80.edge1.losangeles6.level3.net<http://ae-3-80.edge1.losangeles6.l=
evel3.net> (4.69.144.144) 81.841 ms
11 be3036.ccr41.lax04.atlas.cogentco.com<http://ccr41.lax04.atlas.cogentco=
.com> (154.54.14.129) 70.538 ms 39.175 ms 45.213 ms
12 be2964.ccr21.lax01.atlas.cogentco.com<http://ccr21.lax01.atlas.cogentco=
.com> (154.54.44.77) 89.463 ms 78.656 ms 72.936 ms
13 be2932.ccr22.phx02.atlas.cogentco.com<http://ccr22.phx02.atlas.cogentco=
.com> (154.54.45.161) 106.896 ms 82.389 ms 87.310 ms
14 be2929.ccr21.elp01.atlas.cogentco.com<http://ccr21.elp01.atlas.cogentco=
.com> (154.54.42.66) 72.782 ms 76.034 ms 114.967 ms
15 be2928.ccr22.iah01.atlas.cogentco.com<http://ccr22.iah01.atlas.cogentco=
.com> (154.54.30.161) 87.883 ms 100.641 ms
be2927.ccr41.iah01.atlas.cogentco.com<http://ccr41.iah01.atlas.cogentco=
.com> (154.54.29.221) 117.904 ms
16 be2687.ccr41.atl01.atlas.cogentco.com<http://ccr41.atl01.atlas.cogentco=
.com> (154.54.28.69) 144.881 ms
be2690.ccr42.atl01.atlas.cogentco.com<http://ccr42.atl01.atlas.cogentco=
.com> (154.54.28.129) 135.465 ms 110.129 ms
17 be2112.ccr41.dca01.atlas.cogentco.com<http://ccr41.dca01.atlas.cogentco=
.com> (154.54.7.157) 93.713 ms
be2113.ccr42.dca01.atlas.cogentco.com<http://ccr42.dca01.atlas.cogentco=
.com> (154.54.24.221) 128.171 ms
be2112.ccr41.dca01.atlas.cogentco.com<http://ccr41.dca01.atlas.cogentco=
.com> (154.54.7.157) 109.207 ms
18 be2806.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco=
.com> (154.54.40.105) 136.620 ms
be2807.ccr42.jfk02.atlas.cogentco.com<http://ccr42.jfk02.atlas.cogentco=
.com> (154.54.40.109) 113.346 ms
be2806.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco=
.com> (154.54.40.105) 118.301 ms
19 be2096.ccr22.bos01.atlas.cogentco.com<http://ccr22.bos01.atlas.cogentco=
.com> (154.54.30.42) 148.778 ms 166.763 ms
be2094.ccr21.bos01.atlas.cogentco.com<http://ccr21.bos01.atlas.cogentco=
.com> (154.54.30.14) 153.433 ms
20 te0-4-1-7.agr21.bos01.atlas.cogentco.com<http://agr21.bos01.atlas.cogen=
tco.com> (154.54.47.254) 143.449 ms
te0-4-1-6.agr21.bos01.atlas.cogentco.com<http://agr21.bos01.atlas.cogen=
tco.com> (154.54.47.230) 242.964 ms
te0-4-1-6.agr22.bos01.atlas.cogentco.com<http://agr22.bos01.atlas.cogen=
tco.com> (154.54.80.10) 198.678 ms
21 te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com<http://b000254-0.bos0=
1.atlas.cogentco.com> (154.24.9.78) 139.454 ms
te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com<http://b000254-0.bos0=
1.atlas.cogentco.com> (154.24.9.82) 132.726 ms 107.254 ms
22 38.122.127.18 (38.122.127.18) 157.142 ms 158.882 ms 152.458 ms
23 207.154.0.57 (207.154.0.57) 144.913 ms 112.629 ms 112.519 ms
24 server1.ntcorp.com<http://server1.ntcorp.com> (207.154.13.58) 116.986 =
ms 120.276 ms 128.333 ms
Miles
On 2/12/17 8:57 PM, Mel Beckman wrote:
Miles,
Have you tried trace routing through cellular data connections? The results=
you're seeing could be explained by congestion at the point of your modem,=
which I think is with the cox techs are implying.
-mel via cell
On Feb 12, 2017, at 7:46 PM, Mel Beckman <mel@beckman.org><mailto:mel@beckm=
an.org> wrote:
It looks like one or more circuits are down, so you're seeing asymmetrical =
routing over congested paths in one direction.
-mel via cell
On Feb 12, 2017, at 7:14 PM, Miles Fidelman <mfidelman@meetinghouse.net><ma=
ilto:mfidelman@meetinghouse.net> wrote:
Hi Folks,
I'm visiting AZ, and seeing some really really poor performance accessing s=
ome of our servers via Cox broadband. The folks at Cox technical support a=
re useless - all they say is "well you're on a DOCSIS 2 modem." Meanwhile,=
everything I'm seeing is several hops upstream of the local segment - and =
all that Cox level2 tech support will say is "if there was a backbone probl=
em, our backbone people would have dealt with it."
I'm having problems reaching both our own server, and sites like google, fa=
cebook, windows update.
Traceroutes to and from our server are illustrative - note that for most of=
the past week, the average ping time was 85msec. Now we're seeing this:
From 98.177.135.186 - the public IP address on Cox's local broadband servic=
e.
To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoin=
t data center, near Boston)
traceroute to ntcorp.com<http://ntcorp.com> (207.154.13.58), 64 hops max, 5=
2 byte packets
1 10.128.128.1 (10.128.128.1) 199.893 ms 75.319 ms 27.295 ms
2 100.127.69.178 (100.127.69.178) 38.710 ms 40.075 ms 43.598 ms
3 72.215.229.22 (72.215.229.22) 39.674 ms 201.368 ms *
4 lag-157.bear2.phoenix1.level3.net<http://lag-157.bear2.phoenix1.level3.n=
et> (4.28.82.53) 686.499 ms 1837.141 ms 16.273 ms
5 ae-1-60.edge1.losangeles6.level3.net<http://ae-1-60.edge1.losangeles6.le=
vel3.net> (4.69.144.16) 35.498 ms 964.377 ms *
6 ae-3-80.edge1.losangeles6.level3.net<http://ae-3-80.edge1.losangeles6.le=
vel3.net> (4.69.144.144) 551.760 ms 525.014 ms
ae-1-60.edge1.losangeles6.level3.net<http://ae-1-60.edge1.losangeles6.lev=
el3.net> (4.69.144.16) 2061.191 ms
7 be3036.ccr41.lax04.atlas.cogentco.com<http://ccr41.lax04.atlas.cogentco.=
com> (154.54.14.129) 847.778 ms 87.601 ms 71.504 ms
8 be2965.ccr22.lax01.atlas.cogentco.com<http://ccr22.lax01.atlas.cogentco.=
com> (154.54.45.1) 79.060 ms 225.647 ms
be2964.ccr21.lax01.atlas.cogentco.com<http://ccr21.lax01.atlas.cogentco.c=
om> (154.54.44.77) 60.306 ms
9 * be2931.ccr21.phx02.atlas.cogentco.com<http://ccr21.phx02.atlas.cogentc=
o.com> (154.54.44.85) 2264.071 ms 185.180 ms
10 be2929.ccr21.elp01.atlas.cogentco.com<http://ccr21.elp01.atlas.cogentco=
.com> (154.54.42.66) 61.208 ms
be2930.ccr21.elp01.atlas.cogentco.com<http://ccr21.elp01.atlas.cogentco.c=
om> (154.54.42.78) 386.149 ms 1278.868 ms
11 be2928.ccr22.iah01.atlas.cogentco.com<http://ccr22.iah01.atlas.cogentco=
.com> (154.54.30.161) 384.136 ms
be2927.ccr41.iah01.atlas.cogentco.com<http://ccr41.iah01.atlas.cogentco.c=
om> (154.54.29.221) 2339.833 ms 615.415 ms
12 be2690.ccr42.atl01.atlas.cogentco.com<http://ccr42.atl01.atlas.cogentco=
.com> (154.54.28.129) 233.061 ms
be2687.ccr41.atl01.atlas.cogentco.com<http://ccr41.atl01.atlas.cogentco.c=
om> (154.54.28.69) 87.902 ms
be2690.ccr42.atl01.atlas.cogentco.com<http://ccr42.atl01.atlas.cogentco.c=
om> (154.54.28.129) 861.159 ms
13 be2113.ccr42.dca01.atlas.cogentco.com<http://ccr42.dca01.atlas.cogentco=
.com> (154.54.24.221) 998.858 ms
be2112.ccr41.dca01.atlas.cogentco.com<http://ccr41.dca01.atlas.cogentco.c=
om> (154.54.7.157) 249.930 ms *
14 be2807.ccr42.jfk02.atlas.cogentco.com<http://ccr42.jfk02.atlas.cogentco=
.com> (154.54.40.109) 768.461 ms
be2806.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco.c=
om> (154.54.40.105) 136.772 ms
be2807.ccr42.jfk02.atlas.cogentco.com<http://ccr42.jfk02.atlas.cogentco.c=
om> (154.54.40.109) 288.225 ms
15 be2094.ccr21.bos01.atlas.cogentco.com<http://ccr21.bos01.atlas.cogentco=
.com> (154.54.30.14) 271.736 ms 166.224 ms
be2096.ccr22.bos01.atlas.cogentco.com<http://ccr22.bos01.atlas.cogentco.c=
om> (154.54.30.42) 565.015 ms
16 te0-4-1-7.agr22.bos01.atlas.cogentco.com<http://agr22.bos01.atlas.cogen=
tco.com> (154.54.80.34) 1944.479 ms
te0-4-1-6.agr22.bos01.atlas.cogentco.com<http://agr22.bos01.atlas.cogentc=
o.com> (154.54.80.10) 149.803 ms
te0-4-1-6.agr21.bos01.atlas.cogentco.com<http://agr21.bos01.atlas.cogentc=
o.com> (154.54.47.230) 897.115 ms
17 te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com<http://b000254-0.bos0=
1.atlas.cogentco.com> (154.24.9.82) 107.207 ms
te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com<http://b000254-0.bos01.=
atlas.cogentco.com> (154.24.9.78) 295.881 ms 185.453 ms
18 38.122.127.18 (38.122.127.18) 115.652 ms 461.168 ms 615.526 ms
19 207.154.0.57 (207.154.0.57) 1871.023 ms 1987.832 ms 2165.248 ms
20 server1.ntcorp.com<http://server1.ntcorp.com> (207.154.13.58) 587.560 =
ms 263.328 ms 333.542 ms
Traceroute in the reverse direction:
1 207.154.13.47 (207.154.13.47) 0.000 ms 0.000 ms 0.000 ms
2 * * *
3 h130.207.190.173.static.ip.windstream.net<http://static.ip.windstream.ne=
t> (173.190.207.130) 0.000 ms 0.000 ms 0.000 ms
4 xe1-2-0-0.cr01.cley01-oh.us.windstream.net<http://cr01.cley01-oh.us.wind=
stream.net> (40.128.250.166) 12.000 ms 12.000 ms 12.000 ms
5 et11-0-0-0.cr01.chcg01-il.us.windstream.net<http://cr01.chcg01-il.us.win=
dstream.net> (40.128.248.71) 20.000 ms 20.000 ms 20.000 ms
6 10gigabitethernet4-1.core1.chi1.he.NET<http://10gigabitethernet4-1.core1=
.chi1.he.net> (206.223.119.37) 72.001 ms 20.000 ms 16.000 ms
7 chgobbrj01pos010100.r2.ch.cox.net<http://chgobbrj01pos010100.r2.ch.cox.n=
et> (68.105.30.193) 20.000 ms 20.000 ms 20.000 ms
8 chnddsrj01-ae1.0.rd.ph.cox.net<http://rd.ph.cox.net> (68.1.5.211) 72.00=
1 ms 72.001 ms 72.001 ms
9 * * *
10 * * *
11 * * *
<snip>
30 * * *
<looks like a lot of the later hops don't respond to pings>
Two things jump out at me:
1. The rather large number of hops from cox to ntcorp - with high delays f=
rom several nodes in both the level3 and cogent networks.
2. That there's a rather more direct path from the datacenter to cox, that =
shows up in the reverse direction.
Some kind of routing or peering issue, perhaps? (And I also note the earli=
er string of messages regarding youtube streaming problems - that also seem=
ed to involve cox and level3.
Thanks for any insight (and better, for any fixes!).
Miles Fidelman
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
