[162513] in North American Network Operators' Group
RE: Comcast NOC - issues to/from AS13331 (Seattle)
daemon@ATHENA.MIT.EDU (Vinod K)
Wed Apr 24 04:24:56 2013
From: Vinod K <vinod408@hotmail.com>
To: Michael Loftis <mloftis@wgops.com>, nanog <nanog@nanog.org>
Date: Wed, 24 Apr 2013 00:41:17 +0000
In-Reply-To: <CAHDg04tGvXUpKXGesc4RhC6dfd+XD3miSUERP1uswJf5z8FyUA@mail.gmail.com>
Cc: "ren_provo@cable.comcast.com" <ren_provo@cable.comcast.com>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
We have same problem. Cogent noc says use another provider=2C Comcast will=
not upgrade unless they pay for the traffic b/c they send more then they t=
ake in. I attach Ren Provo for official statement.
Vinod
> Date: Mon=2C 22 Apr 2013 22:35:27 -0700
> Subject: Comcast NOC - issues to/from AS13331 (Seattle)
> From: mloftis@wgops.com
> To: nanog@nanog.org
>=20
> Comcast doesn't appear to have any usable NOC contacts via whois=2C and
> this issue is apparently very widespread. Comcast obviously has
> multiple saturated paths out in this area=2C so if you're seeing issues
> getting to your customers on Comcast...well=2C it's probably Comcast.
> Sort of an ongoing/me too on last months thread about same...
>=20
>=20
> Outbound traffic to Comcast via basically any of our (AS13331)
> upstreams except Spectrum (AS11404) is experiencing very high packet
> loss once inside Comcast's network=2C we suspect a router or link very
> near to us in Seattle is failing. I've marked out the destination
> customer IPs but they're available to Comcast engineers if they get in
> touch with me directly or as noc@metapeer.com -- TIA -- and again=2C
> sorry to the list.
>=20
> Below are three failing/high loss traces=2C and then same three traces
> from our other DC which is routing out via spectrum. The loss is
> present on Cogent and L3 outgoing paths at the least=2C maybe others.
> As of now I've routed around the problem point for via the good path.
>=20
> fission:~# mtr -s 900 --report --report-cycles 30 1.2.3.4
> fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrs=
t StDev
> ve75-gw1-xmr.metapeer.com 0.0% 0.4 1.4 0.3 11.=
8 2.9
> te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.9 0.6 0.5 0.=
9 0.1
> te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 43.7 42.3 38.8 43.=
7 1.4
> be-13-cr01.seattle.wa.ibone.comcast.net 3.3% 47.7 44.6 41.3 47.=
7 1.8
> pos-0-7-0-0-cr01.denver.co.ibone.comcast.net 3.3% 69.4 68.7 65.0 71.=
3 1.5
> he-5-15-0-0-cr01.350ecermak.il.ibone.comcast 6.7% 92.6 97.3 91.4 102.=
0 3.8
> he-3-15-0-0-ar01.woburn.ma.boston.comcast.ne 10.0% 120.3 119.8 116.3 122.=
0 1.8
> pos-0-1-0-0-ar01.needham.ma.boston.comcast.n 0.0% 122.0 120.5 117.1 123.=
9 1.8
> po-80-ur01.deering.nh.boston.comcast.net 0.0% 121.4 121.3 118.3 133.=
8 2.8
> po-21-ur01.concord.nh.boston.comcast.net 6.7% 119.6 130.1 119.3 272.=
8 31.8
> te-1-0-0-ten01.concord.nh.boston.comcast.net 6.7% 118.7 120.0 117.6 121.=
4 1.3
> ??? 100.0 0.0 0.0 0.0 0.=
0 0.0
> You have new mail in /var/spool/mail/root
> fission:~# mtr -s 900 --report --report-cycles 30 5.6.7.8
> fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrs=
t StDev
> ve75-gw1-xmr.metapeer.com 0.0% 0.3 0.6 0.3 5.=
2 0.9
> te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.8 0.6 0.5 0.=
8 0.1
> te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 0.0% 41.4 42.6 38.5 43.=
8 1.3
> be-13-cr01.seattle.wa.ibone.comcast.net 0.0% 45.6 44.1 40.4 46.=
7 1.8
> pos-0-8-0-0-cr01.denver.co.ibone.comcast.net 6.7% 71.2 69.0 65.0 72.=
1 1.9
> he-5-12-0-0-cr01.350ecermak.il.ibone.comcast 3.3% 98.1 96.3 86.4 101.=
2 3.9
> he-0-15-0-0-ar01.pontiac.mi.michigan.comcast 10.0% 93.5 97.3 93.5 100.=
6 2.3
> xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.c 0.0% 97.2 99.1 92.9 141.=
8 10.4
> te-17-10-cdn05.rochestrhlls.mi.michigan.comc 3.3% 118.9 113.2 104.6 118.=
9 3.5
> ??? 100.0 0.0 0.0 0.0 0.=
0 0.0
> fission:~# mtr -s 900 --report --report-cycles 30 a.b.c.d
> fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrs=
t StDev
> ve75-gw1-xmr.metapeer.com 0.0% 0.3 2.1 0.3 47.=
1 8.6
> te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.7 0.6 0.5 0.=
8 0.1
> te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 40.6 42.4 39.7 45.=
2 1.3
> be-15-cr01.seattle.wa.ibone.comcast.net 0.0% 47.3 48.2 41.3 54.=
8 4.3
> 68.86.92.34 3.3% 43.7 45.5 39.5 101.=
7 11.6
> be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 43.8 43.0 40.2 44.=
3 1.4
> te-3-0-0-ten15.bellevue.wa.seattle.comcast.n 6.7% 42.7 44.1 42.1 45.=
7 1.1
> ??? 100.0 0.0 0.0 0.0 0.=
0 0.0
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
> mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 1.=
2.3.4
> HOST: phobos Loss% Snt
> Last Avg Best Wrst StDev
> 1. 207.229.74.1 0.0% 30
> 0.4 5.1 0.3 132.4 24.1
> 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30
> 1.5 14.0 1.5 206.4 39.8
> 3. 23.30.206.33 0.0% 30
> 1.8 2.0 1.7 3.5 0.3
> 4. be-17-cr01.seattle.wa.ibone.comcast.net 0.0% 30
> 12.6 5.3 2.1 12.6 2.7
> 5. pos-0-4-0-0-cr01.denver.co.ibone.comcast.net 0.0% 30
> 28.5 29.2 27.2 31.4 1.3
> 6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 30
> 49.9 52.2 49.9 56.7 2.8
> 7. he-3-5-0-0-ar01.woburn.ma.boston.comcast.net 0.0% 30
> 78.5 79.5 78.3 81.0 1.0
> 8. pos-1-12-0-0-ar01.needham.ma.boston.comcast.net 0.0% 30
> 80.2 80.1 78.2 81.7 1.1
> 9. po-80-ur01.deering.nh.boston.comcast.net 0.0% 30
> 80.0 80.1 79.9 80.4 0.1
> 10. po-21-ur01.concord.nh.boston.comcast.net 0.0% 30
> 80.7 85.1 80.7 199.3 21.6
> 11. te-1-0-0-ten01.concord.nh.boston.comcast.net 0.0% 30
> 80.5 80.5 80.5 80.7 0.0
> 12. ??? 100.0 30
> 0.0 0.0 0.0 0.0 0.0
> mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 5.=
6.7.8
> HOST: phobos Loss%
> Snt Last Avg Best Wrst StDev
> 1. 207.229.74.1 0.0%
> 30 0.3 3.6 0.3 96.3 17.5
> 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0%
> 30 10.2 10.4 1.5 58.5 15.1
> 3. 23.30.206.33 0.0%
> 30 2.4 2.0 1.8 2.4 0.1
> 4. be-14-cr01.seattle.wa.ibone.comcast.net 0.0%
> 30 3.6 4.4 1.7 10.6 2.4
> 5. pos-0-3-0-0-cr01.denver.co.ibone.comcast.net 0.0%
> 30 31.1 29.5 27.2 31.6 1.3
> 6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0%
> 30 52.3 55.4 50.3 61.5 3.6
> 7. he-0-14-0-0-ar01.pontiac.mi.michigan.comcast.net 0.0%
> 30 58.9 58.0 56.1 60.1 1.2
> 8. xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.comcast.net 0.0%
> 30 71.2 58.9 56.6 83.6 6.4
> 9. te-17-10-cdn05.rochestrhlls.mi.michigan.comcast.net 0.0%
> 30 71.2 72.6 67.6 78.0 3.2
> 10. ??? 100.0
> 30 0.0 0.0 0.0 0.0 0.0
> mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 a.=
b.c.d
> HOST: phobos Loss% Snt
> Last Avg Best Wrst StDev
> 1. 207.229.74.1 0.0% 30 0.7
> 5.1 0.3 143.4 26.1
> 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 1.5
> 3.1 1.5 30.1 5.6
> 3. 23.30.206.33 0.0% 30 1.9
> 2.0 1.9 2.2 0.1
> 4. be-12-cr01.seattle.wa.ibone.comcast.net 0.0% 30 6.9
> 5.1 1.7 12.3 3.1
> 5. 68.86.94.70 0.0% 30 1.8
> 2.4 1.8 17.3 2.8
> 6. be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 30 3.5
> 3.4 3.3 3.5 0.1
> 7. te-3-0-0-ten15.bellevue.wa.seattle.comcast.net 0.0% 30 3.3
> 3.2 3.2 3.4 0.0
> 8. ??? 100.0 30 0.0
> 0.0 0.0 0.0 0.0
> mloftis@phobos:~$
>=20
> --
>=20
> "Genius might be described as a supreme capacity for getting its possesso=
rs
> into trouble of all kinds."
> -- Samuel Butler
>=20
=
