[27022] in North American Network Operators' Group
Re: RR.COM has backbone problems
daemon@ATHENA.MIT.EDU (Neil J. McRae)
Mon Feb 7 12:41:57 2000
Message-Id: <200002071726.RAA08882@genesis.domino.org>
To: "Kevin McElearney" <kmcelearney@va.rr.com>
Cc: "Vadim Antonov" <avg@kotovnik.com>, nanog@merit.edu,
neil@domino.org
In-reply-to: Your message of "Mon, 07 Feb 2000 11:51:12 EST."
<NABBLHFCFPMEHCGJGKJPEEHBFLAA.kmcelearney@va.rr.com>
From: "Neil J. McRae" <neil@domino.org>
Reply-To: "Neil J. McRae" <neil@domino.org>
Date: Mon, 07 Feb 2000 17:26:08 +0000
Errors-To: owner-nanog-outgoing@merit.edu
On Mon, 7 Feb 2000 11:51:12 -0500
"Kevin McElearney" <kmcelearney@va.rr.com> wrote:
Vadim, well spotted, this has been causing one of our customers
lots of issues.
>
> Vadim, thank you for pointing this out. Or Network Engineers believe this
> is an issue with an optical repeater from our carrier. We will be doing
> intrusive testing to confirm.
>
> The second issue was a graveyard staff training issue and will be corrected
> as well.
>
> - Kevin
>
> -----Original Message-----
> From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of
> Vadim Antonov
> Sent: Monday, February 07, 2000 4:07 AM
> To: nanog@merit.edu
> Subject: RR.COM has backbone problems
>
>
>
>
> I tried to contact their NOC, but the bozo on call didn't
> even want to hear what the problem is, advising me to contact
> my cable provider (how helpful).
>
> Please if there's anyone with a clue from RR.COM / MEDIAONE.NET
> or any other Time Warner holding - wake up, you've got a big
> problem. It doesn't show on your network monitors, but it is sure
> to screw your customers.
>
> And, please track down who was on the NOC telephone ((703) 345-3416)
> at this time (see message headers) and kick him hard for being
> an idiot and not taking a real trouble report.
>
>
> It appears that somehow long packets between boxes
> 24.218.189.197 and 24.218.188.118 get corrupted:
>
> kitty 67: ping -s1500 24.218.188.118
> PING 24.218.188.118 (24.218.188.118): 1500 data bytes
> 1508 bytes from 24.218.188.118: icmp_seq=0 ttl=246 time=164.0 ms
> wrong data byte #1404 should be 0x7c but was 0xaa
> 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a
> 2b 2c 2d 2e 2f 30 31 32 33
> <skipped>
> 1508 bytes from 24.218.188.118: icmp_seq=1 ttl=246 time=164.6 ms
> 1508 bytes from 24.218.188.118: icmp_seq=2 ttl=246 time=166.1 ms
> 1508 bytes from 24.218.188.118: icmp_seq=3 ttl=246 time=164.7 ms
> 1508 bytes from 24.218.188.118: icmp_seq=5 ttl=246 time=164.3 ms
> 1508 bytes from 24.218.188.118: icmp_seq=6 ttl=246 time=168.6 ms
> wrong data byte #1404 should be 0x7c but was 0xaa
> 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b
2c
> 2d 2e 2f 30 31 32 33
> <skipped>
> 1508 bytes from 24.218.188.118: icmp_seq=7 ttl=246 time=163.9 ms
> wrong data byte #924 should be 0x9c but was 0xaa
> 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a
> 2b 2c 2d 2e 2f 30 31 32 33
> <skipped>
>
> kitty 70: ping -s1500 24.218.189.197
> PING 24.218.189.197 (24.218.189.197): 1500 data bytes
> 1508 bytes from 24.218.189.197: icmp_seq=0 ttl=248 time=144.3 ms
> 1508 bytes from 24.218.189.197: icmp_seq=1 ttl=248 time=83.2 ms
> 1508 bytes from 24.218.189.197: icmp_seq=2 ttl=248 time=82.4 ms
> 1508 bytes from 24.218.189.197: icmp_seq=3 ttl=248 time=80.8 ms
> 1508 bytes from 24.218.189.197: icmp_seq=4 ttl=248 time=80.9 ms
>
>
> kitty 72: /usr/sbin/traceroute 24.218.189.170
> traceroute: Warning: Multiple interfaces found; using 205.158.141.97 @ eth0
> traceroute to 24.218.188.118 (24.218.188.118), 30 hops max, 40 byte packets
> 1 192.168.203.1 (192.168.203.1) 19.680 ms 19.540 ms 19.521 ms
> 2 mpk.core.rtr.wavepath.net (205.158.140.129) 19.825 ms 19.699 ms
> 21.328 ms
> 3 pop.access.rtr.wavepath.net (205.158.140.249) 21.097 ms 21.252 ms
> 21.030 ms
> 4 pop.border.rtr.wavepath.net (205.158.140.1) 21.204 ms 21.146 ms
> 23.449 ms
> 5 us-ca-scl-web1-e8.rtr.concentric.net (205.158.7.97) 22.883 ms 21.793
> ms 22.879 ms
> 6 us-ca-scl-border2-fe4-0d0.rtr.concentric.net (207.88.95.9) 24.269 ms
> 101.210 ms 23.812 ms
> 7 us-ca-scl-core2-g5-0-0.rtr.concentric.net (207.88.1.3) 23.506 ms
> 24.663 ms 22.793 ms
> 8 PacBellrr.com (198.32.128.50) 23.761 ms 25.667 ms 26.280 ms
> 9 pabrt02-pabrt03.rr.com (24.218.189.197) 26.108 ms 57.919 ms 24.778 ms
> 10 24.218.188.118 (24.218.188.118) 110.427 ms * 116.969 ms
> 11 bsgsr01-nygsr01.rr.com (24.218.188.21) 116.396 ms 115.738 ms 116.053
> ms
> 12 cambrt01-bsgsr01.rr.com (24.128.191.58) 113.668 ms 114.166 ms 115.836
> ms
> 13 cmrtr01-srp3.ne.mediaone.net (24.218.189.170) 113.589 ms 114.720 ms
> 113.175 ms
>
>
> (Note that the problem affects all _transit_ packets):
>
> kitty 78: ping -s1500 24.218.188.21
> PING 24.218.188.21 (24.218.188.21): 1500 data bytes
> 1508 bytes from 24.218.188.21: icmp_seq=1 ttl=245 time=525.0 ms
> 1508 bytes from 24.218.188.21: icmp_seq=2 ttl=245 time=171.2 ms
> 1508 bytes from 24.218.188.21: icmp_seq=3 ttl=245 time=173.8 ms
> 1508 bytes from 24.218.188.21: icmp_seq=4 ttl=245 time=174.3 ms
> wrong data byte #1404 should be 0x7c but was 0xaa
> 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b
2c
> 2d 2e 2f 30 31 32 33
> <skipped>
>
>
> Regards,
>
> --vadim
>
>
--
Neil J. McRae - Alive and Kicking
neil@DOMINO.ORG
