[166061] in North American Network Operators' Group
It worked! Huge Thanks Re: verizon trouble ticket NJ DQ04PWR9 -- is
daemon@ATHENA.MIT.EDU (Gordon Cook)
Fri Oct 4 17:33:33 2013
From: Gordon Cook <cook@cookreport.com>
In-reply-to: <20131004195042.5632144.95858.2585@one.verizon.com>
Date: Fri, 04 Oct 2013 16:48:46 -0400
To: "Moore, Matthew S" <mscott.moore@verizon.com>,
"nanog@nanog.org list" <nanog@nanog.org>
Cc: "Young, David E" <david.e.young@verizon.com>, "Sieg,
Eric W" <eric.sieg@verizon.com>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
Really glad i posted take a boqand thank you again!
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
The COOK Report on Internet Protocol, (PSTN) 609 882-2572=20
Back Issues: =
http://www.cookreport.com/index.php?option=3Dcom_docman&task=3Dcat_view&gi=
d=3D37&Itemid=3D61 =20
Cook's Collaborative Edge Blog=20
http://www.cookreport.com/wp/
Subscription info: =
http://www.cookreport.com/index.php?option=3Dcom_content&view=3Darticle&id=
=3D54&Itemid=3D65
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
On Oct 4, 2013, at 3:50 PM, "Moore, Matthew S" =
<mscott.moore@verizon.com> wrote:
> Thanks eric...scott
>=20
> From: Sieg, Eric W
> Sent: Friday, October 4, 2013 3:14 PM
> To: Christopher Morrow
> Cc: Moore, Matthew S; Gordon Cook; Young, David E
> Subject: RE: verizon trouble ticket NJ DQ04PWR9 -- is verizon blocking =
FLOKsociety.org by accident or on purpose?
>=20
> Everything is complete, all associated prefix-lists for this customer =
have been updated.=20
> =20
> Thank you for choosing Verizon,
> =20
> Eric Sieg
> IP Non Managed Operations Support | IP/DNS/DNE Administration
> Tel: 248 728 5294 | Vnet: 443-5294
> =20
> <image001.gif>
> <image002.jpg> <image003.jpg> <image004.jpg> <image005.jpg>
> =20
> From: christopher.morrow@gmail.com =
[mailto:christopher.morrow@gmail.com] On Behalf Of Christopher Morrow
> Sent: Friday, October 04, 2013 3:01 PM
> To: Sieg, Eric W
> Cc: Moore, Matthew S; Gordon Cook; Young, David E
> Subject: Re: verizon trouble ticket NJ DQ04PWR9 -- is verizon blocking =
FLOKsociety.org by accident or on purpose?
> =20
> =20
> =20
>=20
> On Fri, Oct 4, 2013 at 2:59 PM, Christopher Morrow =
<morrowc.lists@gmail.com> wrote:
> =20
> =20
>=20
> On Fri, Oct 4, 2013 at 2:54 PM, Sieg, Eric W <eric.sieg@verizon.com> =
wrote:
> I=92ll get it added to their other sites and follow up with them to =
let them know. Give me about 20.
> =20
> =20
> sweet, thanks!
> =20
> forgot, from a fios customer perspective:
> 2 G0-9-4-6.WASHDC-LCR-22.verizon-gni.net (130.81.183.118) 10.434 ms =
6.308 ms 3.699 ms
> 3 ae6-0.RES-BB-RTR1.verizon-gni.net (130.81.199.146) 5.45 ms =
ae5-0.RES-BB-RTR1.verizon-gni.net (130.81.209.222) 3.375 ms =
ae2-0.RES-BB-RTR1.verizon-gni.net (130.81.199.138) 4.592 ms
> 4 0.ae4.XL2.IAD8.ALTER.NET (152.63.8.125) 8.32 ms 3.457 ms 5.700 =
ms
> 5 0.xe-9-3-0.GW9.IAD8.ALTER.NET (152.63.36.34) 5.820 ms =
0.xe-9-0-0.GW9.IAD8.ALTER.NET (152.63.36.18) 7.311 ms =
0.xe-11-2-1.GW9.IAD8.ALTER.NET (152.63.42.2) 6.568 ms
> 6 telefonica-gw.customer.alter.net (152.179.50.114) 3.767 ms 4.74 =
ms 5.569 ms
> 7 Te0-7-0-5-grtmiana4.red.telefonica-wholesale.net (94.142.126.182) =
48.345 ms 46.997 ms =
Xe4-1-6-0-grtmiana2.red.telefonica-wholesale.net(94.142.123.145) 46.154 =
ms
> 8 176.52.251.197 (176.52.251.197) 44.747 ms 176.52.251.189 =
(176.52.251.189) 74.746 ms 176.52.249.241 (176.52.249.241) 49.805 ms
> 9 176.52.252.66 (176.52.252.66) 99.787 ms 98.471 ms 97.103 ms
> ^C
> =20
> looks like it works now.
> =20
> =20
> Thank you for choosing Verizon,
> =20
> Eric Sieg
> IP Non Managed Operations Support | IP/DNS/DNE Administration
> Tel: 248 728 5294 | Vnet: 443-5294
> =20
> <image001.gif>
> <image002.jpg> <image003.jpg> <image004.jpg> <image005.jpg>
> =20
> From: Moore, Matthew S=20
> Sent: Friday, October 04, 2013 2:43 PM
> To: Christopher Morrow; Gordon Cook
> Cc: Young, David E; Sieg, Eric W
> Subject: RE: verizon trouble ticket NJ DQ04PWR9 -- is verizon blocking =
FLOKsociety.org by accident or on purpose?
> =20
> Hi Chris,
>=20
> =20
>=20
> Uncloaking..... ;-)
>=20
> =20
>=20
> It does look as though it's a matter of an un-updated prefix-filter =
for telefonica. The existing filter allows a bunch of /24's
>=20
> close to the address in question but 200.10.150.0/24 is not among =
them, whereas telefonica is announcing it to us. (shown, hidden, below)
>=20
> =20
>=20
> scottm@GW9.IAD8> show route receive-protocol bgp 152.179.50.114 hidden =
| match 200.10.150
>=20
> 200.10.150.0/24 152.179.50.114 0 =
12956 12956 12956 12956 12956 12956 12956 19169 27947 28027 I
>=20
> =20
>=20
> I added the filter, and I'm cc'ng Eric in Customer Support to let him =
know. It will make sense to update it across all 12956 sessions, as I =
only did this one=85 no longer hidden=85=20
> =20
>=20
> =20
>=20
> scottm@GW9.IAD8> show route receive-protocol bgp 152.179.50.114 | =
match 200.10.150
>=20
> * 200.10.150.0/24 152.179.50.114 0 =
12956 12956 12956 12956 12956 12956 12956 19169 27947 28027 I
>=20
> =20
>=20
> =20
>=20
> Regards...Scott
>=20
> =20
>=20
> -----Original Message-----
> From: christopher.morrow@gmail.com =
[mailto:christopher.morrow@gmail.com] On Behalf Of Christopher Morrow
> Sent: Friday, October 04, 2013 1:39 PM
> To: Gordon Cook
> Cc: Young, David E
> Subject: Re: verizon trouble ticket NJ DQ04PWR9 -- is verizon blocking =
FLOKsociety.org by accident or on purpose?
>=20
> =20
>=20
> I always go with my standby: scott moore ... he can uncloak from bcc =
land if he's able/willing/etc to help in this case.
>=20
> =20
>=20
> -chris
>=20
> =20
>=20
> On Fri, Oct 4, 2013 at 1:37 PM, Gordon Cook <cook@cookreport.com> =
wrote:
>=20
> > again thank you very very much. I have to leave routes huit for =
root canal but i do expect to be back can anyone nominate someone at =
verizon who would do to checking with telefonca?
>=20
> >=20
>=20
> > david young is the only guy i know.
>=20
> >=20
>=20
> > But question now is how to rattle a cage at verizon and possible
>=20
> > telefonica - I will also send word of this trouble shooting to the
>=20
> > folk in ecuador
>=20
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> > The COOK Report on Internet Protocol, (PSTN) 609 882-2572 Back =
Issues:
>=20
> > =
http://www.cookreport.com/index.php?option=3Dcom_docman&task=3Dcat_view&gi=
>=20
> > d=3D37&Itemid=3D61
>=20
> > Cook's Collaborative Edge Blog
>=20
> > http://www.cookreport.com/wp/
>=20
> > Subscription info:
>=20
> > =
http://www.cookreport.com/index.php?option=3Dcom_content&view=3Darticle&id=
>=20
> > =3D54&Itemid=3D65
>=20
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> >=20
>=20
> >=20
>=20
> >=20
>=20
> >=20
>=20
> >=20
>=20
> >=20
>=20
> >=20
>=20
> >=20
>=20
> >=20
>=20
> >=20
>=20
> > On Oct 4, 2013, at 1:22 PM, Christopher Morrow =
<morrowc.lists@gmail.com> wrote:
>=20
> >=20
>=20
> >> On Fri, Oct 4, 2013 at 1:15 PM, Gordon Cook <cook@cookreport.com> =
wrote:
>=20
> >>> Hi chris
>=20
> >>>=20
>=20
> >>> really appreciate the help from ALL you guys
>=20
> >>=20
>=20
> >> sure thing.
>=20
> >>=20
>=20
> >>> does what you just said mean that non reachability for version =
customer may mean a config problem for a small bloc and not something =
intentional??
>=20
> >>=20
>=20
> >> that's probably hard to say... I do know that:
>=20
> >>=20
>=20
> >> 5 0.xe-9-2-0.GW9.IAD8.ALTER.NET (152.63.36.30) 6.457 ms 6.821 ms
>=20
> >> 0.xe-11-2-1.GW9.IAD8.ALTER.NET (152.63.42.2) 5.932 ms
>=20
> >> 6 telefonica-gw.customer.alter.net (152.179.50.114) 7.182 ms =
5.433
>=20
> >> ms 5.431 ms
>=20
> >> 7 Xe4-1-6-0-grtmiana2.red.telefonica-wholesale.net =
(94.142.123.145)
>=20
> >> 47.271 ms 48.381 ms =
Te0-7-0-5-grtmiana4.red.telefonica-wholesale.net
>=20
> >> (94.142.126.182) 50.290 ms
>=20
> >>=20
>=20
> >>=20
>=20
> >> telefonica is a 'customer' not a 'peer' of 701, based on their
>=20
> >> connectivity. This means that telefonica has to tell 701: "Yo, I =
need
>=20
> >> you to accept routes for x, y, z, ktnxbi!'
>=20
> >>=20
>=20
> >> knowing the normal (well, 5yrs ago) config for customers of 701, I
>=20
> >> don't expect there'd be special handling of this prefix either... =
so
>=20
> >> I suspect either no one told 701 to accept this, or telefonica
>=20
> >> pouched it at the handoff :(
>=20
> >>=20
>=20
> >> If they didn't do that, these routes wouldn't be accepted.
>=20
> >> If telefonica botched some filter on their side (see the comment
>=20
> >> about prepending).
>=20
> >>=20
>=20
> >> It's fairly sure though that the prefix isn't 'blackholed'... since
>=20
> >> it's everything in the /22 not just the local /32 or /24. Bill
>=20
> >> Herrin's on target as well, it's really hard to say from here :)
>=20
> >>=20
>=20
> >> -chris
>=20
> >>=20
>=20
> >>>=20
>=20
> >>> On Oct 4, 2013, at 12:54 PM, Christopher Morrow =
<morrowc.lists@gmail.com> wrote:
>=20
> >>>=20
>=20
> >>>> err.. nothing in the /24 is reachable from 701's perspective (so =
it
>=20
> >>>> seems)... so I'd suspect that there's a routing problem with the
>=20
> >>>> /24, in fact the surrounding /24's also seem to be having the =
same problem.
>=20
> >>>>=20
>=20
> >>>> On Fri, Oct 4, 2013 at 12:42 PM, Miles Fidelman
>=20
> >>>> <mfidelman@meetinghouse.net> wrote:
>=20
> >>>>> Also inaccessible from FIOS Boston:
>=20
> >>>>> new-host-2:~ mfidelman$ traceroute floksociety.org traceroute to
>=20
> >>>>> floksociety.org (200.10.150.169), 64 hops max, 52 byte packets
>=20
> >>>>> 1 wireless_broadband_router (192.168.1.1) 1.534 ms 0.853 ms
>=20
> >>>>> 0.724 ms
>=20
> >>>>> 2 l100.bstnma-vfttp-84.verizon-gni.net (96.252.37.1) 7.619 ms
>=20
> >>>>> 6.855 ms
>=20
> >>>>> 7.304 ms
>=20
> >>>>> 3 200.10.150.169 (200.10.150.169) 10.482 ms !N *^C
>=20
> >>>>>=20
>=20
> >>>>> But just fine from our datacenter via xo.net. And the web =
server
>=20
> >>>>> is up - at least to a text browser (Lynx).
>=20
> >>>>>=20
>=20
> >>>>> Also via Verizion cell network (Boston area).
>=20
> >>>>>=20
>=20
> >>>>> Some kind of routing table glitch or peering issue, perhaps?
>=20
> >>>>>=20
>=20
> >>>>>=20
>=20
> >>>>>=20
>=20
> >>>>> William Herrin wrote:
>=20
> >>>>>>=20
>=20
> >>>>>> On Fri, Oct 4, 2013 at 12:09 PM, Gordon Cook =
<cook@cookreport.com> wrote:
>=20
> >>>>>>>>=20
>=20
> >>>>>>>> traceroute to floksociety.org (200.10.150.169), 64 hops max, =
72
>=20
> >>>>>>>> byte packets
>=20
> >>>>>>>> 1 192.168.1.1 (192.168.1.1) 0.759 ms 0.309 ms 0.357 ms
>=20
> >>>>>>>> 2 l100.cmdnnj-vfttp-26.verizon-gni.net (98.110.50.1) 36.778
>=20
> >>>>>>>> ms
>=20
> >>>>>>>> 17.508 ms 7.316 ms
>=20
> >>>>>>>> 3 * g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119)=20=
>=20
> >>>>>>>> 6.482 ms !N *
>=20
> >>>>>>>> 4 * * g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net =
(130.81.184.119)=20
>=20
> >>>>>>>> 7.101 ms !N
>=20
> >>>>>>>> 5 * g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119)=20=
>=20
> >>>>>>>> 9.239 ms !N *
>=20
> >>>>>>>> 6 g0-3-4-5.cmdnnj-lcr-21.verizon-gni.net (130.81.184.119)=20
>=20
> >>>>>>>> 6.823 ms !N * 8.846 ms !N
>=20
> >>>>>>=20
>=20
> >>>>>> Inaccessible via FIOS Washington DC too:
>=20
> >>>>>>=20
>=20
> >>>>>> traceroute -T -p 80 200.10.150.169 traceroute to 200.10.150.169
>=20
> >>>>>> (200.10.150.169), 30 hops max, 40 byte packets
>=20
> >>>>>> 1 L300.WASHDC-VFTTP-91.verizon-gni.net (173.73.47.1) 1.804 ms
>=20
> >>>>>> 1.595 ms 1.562 ms
>=20
> >>>>>> 2 G0-6-4-7.WASHDC-LCR-22.verizon-gni.net (130.81.216.250) =
5.321
>=20
> >>>>>> ms !N
>=20
> >>>>>> * *
>=20
> >>>>>>=20
>=20
> >>>>>> Correctly accessible via Cox, Qwest, Sprint and others, but the
>=20
> >>>>>> network path is really slow and really long.
>=20
> >>>>>>=20
>=20
> >>>>>> The border is consistently with telefonica-wholesale.net and =
then
>=20
> >>>>>> telconet.net. Beyond the border there are badly behaving =
routers,
>=20
> >>>>>> including ones configured with RFC 1918 addresses. The
>=20
> >>>>>> addressable routers are reachable via Verizon, just not the =
last hop.
>=20
> >>>>>>=20
>=20
> >>>>>> traceroute -T -p 80 200.10.150.169 traceroute to 200.10.150.169
>=20
> >>>>>> (200.10.150.169), 30 hops max, 60 byte packets
>=20
> >>>>>> 1 sark.dirtside.com (70.182.189.216) 0.708 ms 0.689 ms =
0.569
>=20
> >>>>>> ms
>=20
> >>>>>> 2 10.1.192.1 (10.1.192.1) 9.957 ms 9.874 ms 9.725 ms
>=20
> >>>>>> 3 ip68-100-3-49.dc.dc.cox.net (68.100.3.49) 9.631 ms 9.507 =
ms=20
>=20
> >>>>>> 9.424 ms
>=20
> >>>>>> 4 ip68-100-3-113.dc.dc.cox.net (68.100.3.113) 9.310 ms 9.226
>=20
> >>>>>> ms
>=20
> >>>>>> 9.140 ms
>=20
> >>>>>> 5 mrfddsrj02gex070002.rd.dc.cox.net (68.100.0.145) 9.111 ms=20=
>=20
> >>>>>> 9.019 ms 8.929 ms
>=20
> >>>>>> 6 68.1.4.139 (68.1.4.139) 8.791 ms * 5.981 ms
>=20
> >>>>>> 7 209.48.42.61 (209.48.42.61) 5.748 ms 11.361 ms 10.948 ms
>=20
> >>>>>> 8 vb2000d2.rar3.washington-dc.us.xo.net (207.88.13.66) 58.454
>=20
> >>>>>> ms
>=20
> >>>>>> 52.415 ms 52.421 ms
>=20
> >>>>>> 9 te-3-0-0.rar3.atlanta-ga.us.xo.net (207.88.12.9) 60.543 ms
>=20
> >>>>>> 60.397 ms 60.378 ms
>=20
> >>>>>> 10 te-3-0-0.rar3.dallas-tx.us.xo.net (207.88.12.2) 58.211 ms=20=
>=20
> >>>>>> 58.407 ms 58.392 ms
>=20
> >>>>>> 11 * * *
>=20
> >>>>>> 12 206.111.5.226.ptr.us.xo.net (206.111.5.226) 53.378 ms=20
>=20
> >>>>>> 49.080 ms
>=20
> >>>>>> 47.435 ms
>=20
> >>>>>> 13 Xe-8-1-0-0-grtmiabr3.red.telefonica-wholesale.net
>=20
> >>>>>> (94.142.125.54)
>=20
> >>>>>> 76.006 ms Xe8-0-2-0-grtmiabr4.red.telefonica-wholesale.net
>=20
> >>>>>> (94.142.119.38) 60.181 ms
>=20
> >>>>>> Xe13-1-4-0-grtmiabr4.red.telefonica-wholesale.net
>=20
> >>>>>> (213.140.43.109)
>=20
> >>>>>> 125.888 ms
>=20
> >>>>>> 14 Te-0-2-0-0-grtmiana4.red.telefonica-wholesale.net
>=20
> >>>>>> (94.142.119.233)
>=20
> >>>>>> 67.105 ms Te0-1-0-0-grtmiana4.red.telefonica-wholesale.net
>=20
> >>>>>> (213.140.37.77) 63.435 ms
>=20
> >>>>>> Xe5-1-8-0-grtmiana2.red.telefonica-wholesale.net =
(213.140.36.89)
>=20
> >>>>>> 141.873 ms
>=20
> >>>>>> 15 Xe9-3-0-0-gramiana4.red.telefonica-wholesale.net
>=20
> >>>>>> (94.142.126.197)
>=20
> >>>>>> 62.450 ms 176.52.249.245 (176.52.249.245) 66.665 ms
>=20
> >>>>>> 176.52.249.241
>=20
> >>>>>> (176.52.249.241) 64.668 ms
>=20
> >>>>>> 16 176.52.252.66 (176.52.252.66) 118.619 ms 118.057 ms=20
>=20
> >>>>>> 117.934 ms
>=20
> >>>>>> 17 * * *
>=20
> >>>>>> 18 * * *
>=20
> >>>>>> 19 * * *
>=20
> >>>>>> 20 host-186-5-116-193.telconet.net (186.5.116.193) 122.586 ms
>=20
> >>>>>> 120.967 ms 115.040 ms
>=20
> >>>>>> 21 host-186-101-89-42.telconet.net (186.101.89.42) 122.801 ms
>=20
> >>>>>> 125.164 ms 119.520 ms
>=20
> >>>>>> 22 * * *
>=20
> >>>>>> 23 200.10.150.169 (200.10.150.169) 253.710 ms 246.684 ms=20
>=20
> >>>>>> 244.845 ms
>=20
> >>>>>>=20
>=20
> >>>>>>=20
>=20
> >>>>>>=20
>=20
> >>>>>>=20
>=20
> >>>>>>=20
>=20
> >>>>>=20
>=20
> >>>>>=20
>=20
> >>>>> --
>=20
> >>>>> In theory, there is no difference between theory and practice.
>=20
> >>>>> In practice, there is. .... Yogi Berra
>=20
> >>>>>=20
>=20
> >>>>>=20
>=20
> >>>>=20
>=20
> >>>=20
>=20
> >=20
>=20
> =20
> =20
