[175962] in North American Network Operators' Group
Re: v6 cdn problems
daemon@ATHENA.MIT.EDU (joel jaeggli)
Sun Nov 9 16:54:03 2014
X-Original-To: nanog@nanog.org
Date: Sun, 09 Nov 2014 11:53:42 -1000
From: joel jaeggli <joelja@bogus.com>
To: Frank Bulk <frnkblk@iname.com>, "'Pete Carah'" <pete@altadena.net>,
nanog@nanog.org
In-Reply-To: <000001cffba8$0e87ebe0$2b97c3a0$@iname.com>
Errors-To: nanog-bounces@nanog.org
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--69USCrE1gPPrIdjFXCuhgchMsop2Vpu5B
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
On 11/8/14 1:02 PM, Frank Bulk wrote:
> The Google angle is also being discussed on outages. Initial suspicion=
s are PTB packets not flowing through tunneled connections.
you can also have problems in the other direction e.g. if your tunnel
ingress sends a ptb towards a load balanced service it may not arrive.
https://tools.ietf.org/html/draft-v6ops-pmtud-ecmp-problem-00
if you're tunneled it does help a lot if your mss reflects the cost of
the tunnel you know exists.
> Frank
>=20
> -----Original Message-----
> From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Pete Carah
> Sent: Saturday, November 08, 2014 4:56 PM
> To: nanog@nanog.org
> Subject: v6 cdn problems
>=20
> Prefix this - I'm on fios in the Baltimore area, using a HE tunnel
> terminating in ashburn.
> (*still* no native v6 on fios :-( Speedtest shows little or no
> congestion, and didn't change significantly when I reduced mtu by 8.=20
> (interestingly, speedtest.net usually reads faster than verizon's
> internal speedtest, and rarely averages less than my billed speed.)
>=20
> I've recently had problems (started a few weeks ago with www.att.com,
> 4-5 days ago with *.google.com)
> which unfortunately happy eyeballs doesn't help.
> att.com uses akamai, google uses their own cdn (per dns; I don't know i=
f
> there are any connections
> behind the scenes.) This occurs on several google sites, all of which
> resolve to the same netblocks from here (maps.google.com,
> www.google.com, maps.gstatic.com, and at least one of the ad servers).
>=20
> Symptom with akamai is that it connects immediately then data transfer
> times out.
> With google, symptom involves both slow connection, and data transfer
> timing out. I don't know if chrome's happy eyeballs is working since i=
f
> it was, and absent address caching, I shouldn't see the slow connection=
=2E
>=20
> v6 connections to my hosts in Los Angeles (not on HE address space, but=
> we do peer with them on
> any2) work fine transferring graphics and large database files both
> ways, so I'm pretty sure I don't have an mtu problem nor some other fio=
s
> or HE problem. Just to be sure, I dropped the 1500 to 1492 on the fios=
> link and did the same adjustment to the mtu on my tunnel (becomes
> 1472). No change on my hosts. att.com appears a little better, though=
> still very slow. Google shows no change at all.
>=20
> I saw some of the same problem yesterday from Frederick on comcast (onl=
y
> to google, didn't look at att), but couldn't take the time to do
> traceroutes. If desired, I'm likely to go out there tomorrow and can d=
o
> that too. (we use a freebsd+pf router there).
>=20
> Is this a provisioning problem where v6 eyeballs are outstripping cdn
> provisioning (since win7 and 8 both default to v6)? Or is something
> else going on. Since this seems to affect more than one cdn, it seems =
odd.
>=20
> I run my own resolver locally instead of using verizon's. (and my own
> (vyatta) router instead of theirs. Actually I'm still using theirs as =
a
> bridge to talk to the set-top box (I don't know if Motorola still makes=
the
> coax terminal that would bridge it directly...)
>=20
> Resolve and traceroutes of relevant sites:
>=20
> [pete@port5 ~]$ host www.att.com
> www.att.com is an alias for prod-www.zr-att.com.akadns.net.
> prod-www.zr-att.com.akadns.net is an alias for www.att.com.edgekey.net.=
> www.att.com.edgekey.net is an alias for e2318.dscb.akamaiedge.net.
> e2318.dscb.akamaiedge.net has address 23.76.217.145
> e2318.dscb.akamaiedge.net has IPv6 address 2600:807:320:202:9200::90e
> e2318.dscb.akamaiedge.net has IPv6 address 2600:807:320:202:8600::90e
>=20
> Traceroute (v4) to this shows it is in Newark or environs:
> [pete@port5 ~]$ traceroute e2318.dscb.akamaiedge.net
> traceroute to e2318.dscb.akamaiedge.net (23.76.217.145), 30 hops max, 6=
0 byte packets
> 1 rtr.east.altadena.net (192.168.170.1) 2.008 ms 2.450 ms 3.091 ms=
> 2 L300.BLTMMD-VFTTP-64.verizon-gni.net (108.3.150.1) 9.021 ms 9.054=
ms 9.045 ms
> 3 G0-7-4-5.BLTMMD-LCR-21.verizon-gni.net (100.41.195.94) 10.670 ms =
10.683 ms 10.677 ms
> 4 ae4-0.RES-BB-RTR2.verizon-gni.net (130.81.209.230) 9.002 ms ae20-0=
=2ERES-BB-RTR1.verizon-gni.net (130.81.151.112) 8.995 ms so-1-1-0-0.RES-=
BB-RTR1.verizon-gni.net (130.81.199.2) 8.953 ms
> 5 * * *
> 6 * * *
> 7 0.xe-5-0-4.XL3.EWR6.ALTER.NET (140.222.225.73) 51.202 ms 41.102 m=
s 40.345 ms
> 8 0.ae1.XL4.EWR6.ALTER.NET (140.222.228.41) 33.065 ms TenGigE0-6-0-3=
=2EGW8.EWR6.ALTER.NET (152.63.19.158) 11.550 ms TenGigE0-6-0-6.GW8.EWR6.=
ALTER.NET (152.63.25.10) 11.659 ms
> 9 TenGigE0-7-0-1.GW8.EWR6.ALTER.NET (152.63.19.166) 19.854 ms akamai=
-gw.customer.alter.net (152.179.185.126) 1766.402 ms TenGigE0-7-0-7.GW8.=
EWR6.ALTER.NET (152.63.25.30) 18.227 ms
> 10 akamai-gw.customer.alter.net (152.179.185.126) 1747.269 ms a23-76-=
217-145.deploy.static.akamaitechnologies.com (23.76.217.145) 10.672 ms =
11.263 ms
>=20
> Traceroute6 to it appears to be local (but is hard to tell. Next-to-la=
st hop looks like either a router or=20
> load-balancer is overloaded. Same for the v4 traceroute...
>=20
> [pete@port5 ~]$ traceroute6 www.att.com
> traceroute to www.att.com (2600:807:320:202:9200::90e), 30 hops max, 80=
byte packets
> 1 rtr.east.altadena.net (2001:470:e160:1::1) 5.182 ms 5.274 ms 5.2=
54 ms
> 2 altadenamd-1.tunnel.tserv13.ash1.ipv6.he.net (2001:470:7:126::1) 1=
1.452 ms 15.040 ms 18.592 ms
> 3 ge4-12.core1.ash1.he.net (2001:470:0:90::1) 20.273 ms 20.574 ms =
20.567 ms
> 4 eqx.br6.iad8.verizonbusiness.com (2001:504:0:2::701:1) 20.522 ms =
20.232 ms 20.475 ms
> 5 * * *
> 6 2600:802:44f::2 (2600:802:44f::2) 1283.113 ms 1296.115 ms 1296.1=
81 ms
> 7 2600:807:320:200::1743:f397 (2600:807:320:200::1743:f397) 20.181 m=
s 16.169 ms 14.073 ms
>=20
>=20
> [pete@port5 ~]$ host www.google.com
> www.google.com has address 74.125.228.16
> www.google.com has address 74.125.228.20
> www.google.com has address 74.125.228.17
> www.google.com has address 74.125.228.19
> www.google.com has address 74.125.228.18
> www.google.com has IPv6 address 2607:f8b0:4004:800::1012
>=20
> Traceroute (v4) to this shows something odd, but I don't know where "bu=
rl" is for verizon. Also I appear to
> hit two nodes for the terminal. At least one google node appears to be =
ashburn (or environs)
> :
> [pete@port5 ~]$ traceroute www.google.com
> traceroute to www.google.com (74.125.228.20), 30 hops max, 60 byte pack=
ets
> 1 rtr.east.altadena.net (192.168.170.1) 2.646 ms 2.816 ms 3.536 ms=
> 2 L300.BLTMMD-VFTTP-64.verizon-gni.net (108.3.150.1) 4.109 ms 4.194=
ms 4.186 ms
> 3 G0-7-4-4.BLTMMD-LCR-22.verizon-gni.net (130.81.170.84) 7.928 ms 8=
=2E096 ms 8.088 ms
> 4 ae20-0.PHIL-BB-RTR1.verizon-gni.net (130.81.151.120) 10.881 ms ae2=
0-0.PHIL-BB-RTR2.verizon-gni.net (130.81.151.124) 11.074 ms so-6-1-0-0.P=
HIL-BB-RTR2.verizon-gni.net (130.81.199.4) 11.047 ms
> 5 0.xe-7-0-2.XL2.IAD8.ALTER.NET (152.63.4.93) 14.872 ms 0.xe-3-0-1.X=
L3.IAD8.ALTER.NET (152.63.3.61) 37.703 ms 12.268 ms
> 6 0.xe-9-2-0.GW9.IAD8.ALTER.NET (152.63.36.30) 12.866 ms 0.xe-11-2-1=
=2EGW9.IAD8.ALTER.NET (152.63.42.2) 14.442 ms 0.xe-9-2-0.GW9.IAD8.ALTER.=
NET (152.63.36.30) 11.918 ms
> 7 * 0.xe-10-1-1.GW9.IAD8.ALTER.NET (152.63.35.113) 16.901 ms pool-96=
-236-104-66.burl.east.verizon.net (96.236.104.66) 136.110 ms
> 8 pool-96-236-104-66.burl.east.verizon.net (96.236.104.66) 137.977 m=
s 216.239.46.248 (216.239.46.248) 13.875 ms pool-96-236-104-66.burl.east=
=2Everizon.net (96.236.104.66) 134.602 ms
> 9 216.239.46.248 (216.239.46.248) 15.918 ms 10.708 ms 10.162 ms
> 10 72.14.238.173 (72.14.238.173) 11.347 ms 12.111 ms iad23s05-in-f20=
=2E1e100.net (74.125.228.20) 12.769 ms
>=20
> Corresponding traceroute6 (shows lack of reverse on most hits...):
> [pete@port5 ~]$ traceroute6 www.google.com
> traceroute to www.google.com (2607:f8b0:4004:800::1011), 30 hops max, 8=
0 byte packets
> 1 rtr.east.altadena.net (2001:470:e160:1::1) 1.640 ms 1.811 ms 1.8=
01 ms
> 2 altadenamd-1.tunnel.tserv13.ash1.ipv6.he.net (2001:470:7:126::1) 1=
1.977 ms 15.279 ms 19.265 ms
> 3 ge4-12.core1.ash1.he.net (2001:470:0:90::1) 19.779 ms 20.776 ms =
22.303 ms
> 4 2001:4860:1:1:0:1b1b:0:d (2001:4860:1:1:0:1b1b:0:d) 22.267 ms 22.=
514 ms 22.507 ms
> 5 2001:4860::1:0:9ff (2001:4860::1:0:9ff) 22.508 ms 22.471 ms 22.4=
55 ms
> 6 2001:4860:0:1::551 (2001:4860:0:1::551) 22.467 ms 19.139 ms 19.1=
16 ms
> 7 2607:f8b0:8000:18::c (2607:f8b0:8000:18::c) 19.054 ms 2607:f8b0:80=
00:18::f (2607:f8b0:8000:18::f) 7.716 ms 2607:f8b0:4004:800::1b (2607:f8=
b0:4004:800::1b) 8.379 ms
>=20
> Again shows multiple terminals for the given address.
>=20
>=20
> Ping works fine to all of the addresses, both v4 and v6, and the att on=
e
> always connects immediately. The google one doesn't always.
>=20
> When I disable the HE tunnel, (and restart the browser - apparently
> happy-eyeballs caches), everything works just fine, so the problem does=
> appear to relate to v6.
>=20
> For reference, I mostly use chrome in linux. My daughter sees the same=
> problem with google, mostly using chrome in win 7. I see the problem
> with firefox (in linux) also (to both sites).
>=20
> -- Pete
>=20
>=20
>=20
>=20
--69USCrE1gPPrIdjFXCuhgchMsop2Vpu5B
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org
iEYEARECAAYFAlRf4mcACgkQ8AA1q7Z/VrLRYACfW7Ldc8cc2C4E4XfrT0QWnUxy
bGEAnjtBli0wIKNkEGl7yaYz1JYl5cFh
=lDox
-----END PGP SIGNATURE-----
--69USCrE1gPPrIdjFXCuhgchMsop2Vpu5B--
