[144629] in North American Network Operators' Group
Re: routing issue for verizon dsl customers in western massachusetts
daemon@ATHENA.MIT.EDU (Christopher Morrow)
Thu Sep 15 16:53:18 2011
In-Reply-To: <73C3F4CB-C541-438F-BC68-6074BD8B8045@simons-rock.edu>
Date: Thu, 15 Sep 2011 16:52:26 -0400
From: Christopher Morrow <morrowc.lists@gmail.com>
To: Steve Bohrer <skbohrer@simons-rock.edu>
Cc: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
On Thu, Sep 15, 2011 at 4:13 PM, Steve Bohrer <skbohrer@simons-rock.edu> wr=
ote:
> On Sep 15, 2011, at 3:39 PM, Christopher Morrow wrote:
>
>> On Thu, Sep 15, 2011 at 3:34 PM, Brian Gold <bgold@simons-rock.edu> wrot=
e:
>>>
>>> Over the past week, we've discovered that there is an issue with the wa=
y
>>> some Verizon DSL customers are being routed in Western Massachusetts th=
at
>>> is
>>> preventing them from reaching my employers public IPs. The problem is
>>> only
>>> limited to Verizon DSL customers, everyone else can reach these IP
>>> addresses
>>> just fine. After many hours on the phone with Verizon tech support, I
>>> finally managed to get myself and one of my coworker's home dsl
>>> connections
>>> switched from a "redback router" to a "juniper router" which resolved t=
he
>>> issue, but only for us.
>
> [...]
>>
>> If you buy verizon services at your day job you can probably make
>> noise through your sales droids better than here (sadly)... verizon
>> likes to jump when customers have problems, if the customer is a large
>> corporation or other 'important' customer.
>
>
> That is just the problem! The college does not buy any Verizon network st=
uff
> directly, so we don't really have any access to their support. (We have a
> few cell phones, but not enough to be "important".)
>
> Brian Gold (who first posted) happens to have their DSL to his house, and=
he
> was one of five who have reported the problem, so that gave him a slight =
in.
> But the only techs he could reach as an "end user" were not high enough u=
p
> to fix this problem in a general way. After pressing them for literally
> hours, he was able to get transfered to their NOC, and get the problem
> resolved for his one address. But, they would not give him the NOC contac=
t,
> and he had to repeat this multi-hour process to get it fixed for an other
> user. Verizon's DSL support suggested that we get our bandwith provider
> involved, and so they tried to pitch in, but they don't have any Verizon =
NOC
> contact either, especially since this issue is purely within a small corn=
er
> of Verizon's DSL network, not on any of Verizon's links to our provider.
>
> This issue hits only a few Verizon DSL users in NW Mass. It does not real=
ly
> seem like a routing problem, because the affected users can reach many of
> the servers in our AS, but not some addresses. Unfortunately, the "blocke=
d"
> addresses include our web server and our mail server, so our staff who li=
ve
> out there noticed the issue pretty quickly. Traceroutes from Brian's hous=
e
> show that for our blocked hosts, the users don't get beyond Verizon's NAT=
.
I wasn't aware verizon implemented CGN already... way to be a 'first
mover' in this field verizon!
>
> The Verizon tech's "fix" of re-patching Brian's DSL line in to a differen=
t
> router feels to me like there is a config problem in the other router, bu=
t
> the tech we got is not authorized to alter the config. It would be nice i=
f
> we could reach someone who could actually edit the broken config and make=
it
> right. Anyone from Verzion's NOC for Western Mass reading this? Or, does
> anyone else have useful contact info for them?
you probably want someone in the NOC which is (I think) stil in
Reston, va... I don't think they have separate noc's per region. The
first-line tech folks you chat with on the phone really arent' able
(even to login really) to fix devices in the field :(
anyways, this looks crappy :( but yeah for CGN being all it's cracked up to=
be!
-chris
>
> FWIW, Simon's Rock is 208.81.88.0/21, AS 19345. Here are a failed and a g=
ood
> trace from Brian's house, to different servers on our campus :
>
> FAILS:
> Tracing route to wilbur.simons-rock.edu [208.81.88.15]
> over a maximum of 30 hops:
>
> =A01 =A0 =A0<1 ms =A0 =A0<1 ms =A0 =A0<1 ms =A0192.168.10.1
> =A02 =A0 =A0 1 ms =A0 =A0 1 ms =A0 =A0 1 ms =A0192.168.1.1
> =A03 =A0 =A053 ms =A0 104 ms =A0 116 ms =A010.14.1.1
> =A04 =A0 =A0 * =A0 =A0 =A0 =A0* =A0 =A0 =A0 =A0* =A0 =A0 Request timed ou=
t.
> =A05 =A0 =A0 * =A0 =A0 =A0 =A0* =A0 =A0 =A0 =A0* =A0 =A0 Request timed ou=
t.
> =A06 =A0 =A0 * =A0 =A0 =A0 =A0* =A0 =A0 =A0 =A0* =A0 =A0 Request timed ou=
t.
> =A07 =A0 =A0 * =A0 =A0 =A0 =A0* =A0 =A0 =A0 =A0* =A0 =A0 Request timed ou=
t.
>
> WORKS:
> Tracing route to dev.simons-rock.edu [208.81.88.25]
> over a maximum of 30 hops:
>
> 1 =A0 =A0<1 ms =A0 =A0<1 ms =A0 =A0<1 ms =A0192.168.10.1
> 2 =A0 =A0 1 ms =A0 =A0 1 ms =A0 =A0 1 ms =A0192.168.1.1
> 3 =A0 =A087 ms =A0 =A054 ms =A0 =A054 ms =A010.14.1.1
> 4 =A0 =A099 ms =A0 109 ms =A0 103 ms at-0-3-0-1711.WMA-CORE-RTR2.verizon-=
gni.net
> [130.81.10.77]
> 5 =A0 =A016 ms =A0 =A018 ms =A0 =A016 ms =A0so-7-3-1-0.NY5030-BB-RTR2.ver=
izon-gni.net
> [130.81.20.6]
> 6 =A0 =A019 ms =A0 =A017 ms =A0 =A017 ms =A00.xe-3-1-0.BR3.NYC4.ALTER.NET=
[152.63.2.81]
> 7 =A0 =A018 ms =A0 =A021 ms =A0 =A018 ms =A0204.255.168.194
> 8 =A0 108 ms =A0 188 ms =A0 116 ms =A0pos5-0-2488M.cr1.BOS1.gblx.net [67.=
17.94.57]
> 9 =A0 =A024 ms =A0 =A028 ms =A0 =A023 ms =A0pos0-0-0-155M.ar1.BOS1.gblx.n=
et [67.17.70.162]
> 10 =A0 121 ms =A0 160 ms =A0 127 ms =A064.213.79.250
> 11 =A0 =A077 ms =A0 =A077 ms =A0 =A078 ms =A0208.81.88.25
>
> Trace complete.
>
> Anyways, thanks for any suggestions you can offer.
>
> Steve Bohrer
> Network Administrator
> ITS, Bard College at Simon's Rock
> 413-528-7645
>
>
>
>
