[182166] in North American Network Operators' Group
Re: AW: Level3 routing issue US west coast
daemon@ATHENA.MIT.EDU (joel jaeggli)
Sat Jul 11 16:11:51 2015
X-Original-To: nanog@nanog.org
To: =?UTF-8?Q?J=c3=bcrgen_Jaritsch?= <jj@anexia.at>,
Ca By <cb.list6@gmail.com>, Joseph Jenkins <joe@breathe-underwater.com>
From: joel jaeggli <joelja@bogus.com>
Date: Sat, 11 Jul 2015 13:11:29 -0700
In-Reply-To: <11cd4154ae384babbdd11b5df252ee1f@anx-i-dag02.anx.local>
Cc: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: nanog-bounces@nanog.org
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--x2MVSFFXMAWuGwVGugJF8mATtmxXClUaU
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
On 7/10/15 7:54 AM, J=C3=BCrgen Jaritsch wrote:
> Wow .... Level3 responded to me that they had an issue last night .... =
but they simply did nothing ... for at least 10 hours they did nothing to=
fix the issue:
It's more likely that there's a certain amount of nonsense and a lot of
loose ends in the ticketing system; than that the issue resolved itself
without intervention.
You might reasonably conclude that the corrective action and the ticket
are unrelated at least as far as the noc is concerned.
> ###
>=20
> Event Case ID: 9446216
> Location: Los Angeles, CA
> Impacted For: 10 hours 52 minutes
> ETR: Unknown
> Bridge: N/A
>=20
>=20
> 08:52 GMT - Event Conclusion Summary=20
>=20
> Start Time: 07/09 19:48 GMT=20
> Stop Time: 07/10 06:40 GMT=20
>=20
> Root Cause: A packet loss issue in Los Angeles, CA.=20
>=20
> Fix Action: The packet loss issue resolved before any action was taken.=
=20
> Summary: The IP NOC is currently investigated a packet loss issue in Lo=
s Angeles, CA that was impacting IP services. The packet loss issue resol=
ved before any action was taken and the IP NOC deemed services are stable=
after monitoring was concluded.
>=20
> 07:53 GMT - The IP NOC is currently monitoring services after the packe=
t loss issue resolved before any action was taken. An update will be sent=
when traffic is deemed stable or if status changes.
>=20
> 06:43 GMT - The IP NOC is currently investigating a packet loss issue i=
n Los Angeles, CA that is impacting IP services. Troubleshooting efforts =
are ongoing with no estimated time to restore services available at this =
point. Please be advised that updates for this event will be relayed hour=
ly unless otherwise noted.
>=20
> ###
>=20
>=20
> best regards
>=20
> J=C3=BCrgen Jaritsch
> Head of Network & Infrastructure
>=20
> ANEXIA Internetdienstleistungs GmbH
>=20
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
>=20
> E-Mail: jj@anexia.at=20
> Web: http://www.anexia.at
>=20
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=C3=9Fe 140, 9020 Klage=
nfurt
> Gesch=C3=A4ftsf=C3=BChrer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63=
216601
>=20
>=20
> -----Urspr=C3=BCngliche Nachricht-----
> Von: J=C3=BCrgen Jaritsch=20
> Gesendet: Freitag, 10. Juli 2015 15:47
> An: 'Ca By'; Joseph Jenkins
> Cc: nanog@nanog.org
> Betreff: AW: Level3 routing issue US west coast
>=20
> Hi,
>=20
> sitting here and watching the packet loss coming and going :(.
>=20
> It changes every 10-25min. Looks like an massive issue in San Jose - ro=
uters out there sometimes have an latency from 5-6 SECONDS ...
>=20
> best regards
>=20
> J=C3=BCrgen Jaritsch
> Head of Network & Infrastructure
>=20
> ANEXIA Internetdienstleistungs GmbH
>=20
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
>=20
> E-Mail: jj@anexia.at=20
> Web: http://www.anexia.at
>=20
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=C3=9Fe 140, 9020 Klage=
nfurt
> Gesch=C3=A4ftsf=C3=BChrer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63=
216601
>=20
>=20
> -----Urspr=C3=BCngliche Nachricht-----
> Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Ca By
> Gesendet: Freitag, 10. Juli 2015 15:42
> An: Joseph Jenkins
> Cc: nanog@nanog.org
> Betreff: Re: Level3 routing issue US west coast
>=20
> On Friday, July 10, 2015, Joseph Jenkins <joe@breathe-underwater.com> w=
rote:
>=20
>> Level3 had an issue with one of their core routers in Los Angeles last=
>> night(7pm Pacific) and early this morning(1am Pacific). Last update to=
my
>> trouble ticket had the issue still being reviewed by engineering, but =
that
>> a core router was dropping packets.
>>
>>
> I have seen this several times with level3. They confirm packets are
> dropping and sevice is degraded yet they refuse to take tactical correc=
tive
> action for hours and hours.
>=20
> Makes me furious.
>=20
> CB
>=20
>=20
>>> On Jul 10, 2015, at 3:59 AM, J=C3=BCrgen Jaritsch <jj@anexia.at
>> <javascript:;>> wrote:
>>>
>>> Hi,
>>>
>>> does anyone else experience issues with the Level3 network at the US
>> west coast? We see lots of broken paths like this:
>>>
>>> Packets
>> Pings
>>> Host Loss% Snt Last=
>> Avg Best Wrst StDev
>>> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6=
>> 0.5 0.2 18.1 1.2
>>> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5=
>> 9.9 0.3 361.1 40.1
>>> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5=
>> 7.7 6.3 49.7 5.3
>>> 4. win-b4-link.telia.net 0.0% 230 6.6=
>> 6.8 6.4 20.2 1.5
>>> 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6=
>> 9.3 6.3 69.1 9.6
>>> 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8=
>> 165.0 164.5 194.9 2.6
>>> 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7=
>> 164.8 164.5 174.1 0.9
>>> 8. 4.53.208.102 34.3% 230 634.9=
>> 310.7 168.5 680.1 199.7
>>> 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0=
>> 455.2 304.9 954.6 203.4
>>> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.=
4
>> 441.4 323.1 822.0 182.1
>>> 11. 211.115.201.92 38.4% 230 289.=
8
>> 412.9 289.6 846.7 185.4
>>> 12. 61.250.89.2 35.8% 230 290.=
6
>> 439.4 290.2 804.3 205.1
>>> 13. ???
>>>
>>> Trace from NYC is also broken:
>>>
>>> Packets
>> Pings
>>> Host Loss% Snt
>> Last Avg Best Wrst StDev
>>> 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30
>> 0.4 4.3 0.4 57.8 13.3
>>> 2. nyk-b5-link.telia.net 0.0% 30
>> 0.3 0.4 0.3 0.9 0.1
>>> 3. ???
>>> 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30
>> 71.7 73.2 71.7 98.7 5.5
>>> 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30
>> 71.8 71.8 71.7 72.0 0.1
>>> 6. 4.53.208.102 31.0% 30
>> 569.6 250.7 70.5 579.3 231.2
>>> 7. 63.218.250.73 31.0% 30
>> 672.6 355.5 178.0 672.6 232.1
>>>
>>>
>>> At 10:24 UTC+2 it was even more broken:
>>>
>>> Packets
>> Pings
>>> Host Loss% Snt Last=
>> Avg Best Wrst StDev
>>> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4=
>> 0.5 0.3 39.7 2.2
>>> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5=
>> 6.7 0.3 198.1 26.3
>>> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6=
>> 7.6 6.4 43.6 4.5
>>> 4. win-b4-link.telia.net 0.0% 326 6.7=
>> 7.4 6.3 43.1 3.2
>>> 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9=
>> 9.2 6.3 73.2 10.1
>>> 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7=
>> 165.5 164.5 176.7 1.5
>>> ae-2-70.edge1.SanJose3.Level3.net
>>> 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8=
>> 165.8 164.6 204.2 3.9
>>> ae-2-70.edge1.SanJose3.Level3.net
>>> 8. 205.129.5.70 74.2% 326 799.9=
>> 487.2 169.0 799.9 305.7
>>> 4.53.208.102
>>> 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359.=
>> 701.0 308.7 3716. 510.6
>>> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.=
4
>> 643.0 323.4 960.4 307.6
>>> 11. 211.115.201.92 68.9% 326 925.=
3
>> 674.2 289.8 932.3 296.6
>>> 12. 61.250.89.2 72.9% 326 928.=
5
>> 637.2 291.9 928.5 304.3
>>> 13. ???
>>>
>>>
>>> best regards
>>>
>>> J=C3=BCrgen Jaritsch
>>> Head of Network & Infrastructure
>>>
>>> ANEXIA Internetdienstleistungs GmbH
>>>
>>> Telefon: +43-5-0556-300
>>> Telefax: +43-5-0556-500
>>>
>>> E-Mail: jj@anexia.at <javascript:;><mailto:jj@anexia.at <javascript:;=
>>
>>> Web: http://www.anexia.at<http://www.anexia.at/>
>>>
>>> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=C3=9Fe 140, 9020 Kla=
genfurt
>>> Gesch=C3=A4ftsf=C3=BChrer: Alexander Windbichler
>>> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT
>> U63216601
>>>
>>
>>
>=20
--x2MVSFFXMAWuGwVGugJF8mATtmxXClUaU
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2
Comment: GPGTools - http://gpgtools.org
iEYEARECAAYFAlWheHEACgkQ8AA1q7Z/VrL2lACePtOzU30LUsWR93c/kI9aZIQL
ndEAnRVFD6JAGmtwDyWW3YNKJAiNjYgc
=H6+e
-----END PGP SIGNATURE-----
--x2MVSFFXMAWuGwVGugJF8mATtmxXClUaU--
