[182103] in North American Network Operators' Group
AW: Level3 routing issue US west coast
daemon@ATHENA.MIT.EDU (=?iso-8859-1?Q?J=FCrgen_Jaritsch?=)
Fri Jul 10 09:06:49 2015
X-Original-To: nanog@nanog.org
From: =?iso-8859-1?Q?J=FCrgen_Jaritsch?= <jj@anexia.at>
To: Joseph Jenkins <joe@breathe-underwater.com>, "nanog@nanog.org"
<nanog@nanog.org>
Date: Fri, 10 Jul 2015 13:06:45 +0000
In-Reply-To: <3F419656-849B-4D64-A5F1-5E081C91E2C5@breathe-underwater.com>
Errors-To: nanog-bounces@nanog.org
Hi Joseph,
in the meantime I have ~20 verified paths which are affected and Level3 is =
simply not competent enough to reroute/drop the affected path ...
FYI: my private ticket # is 9446435
###
There is also a new global ticket available:
Network Event Detail
Network Event Summary: Multiple devices were unreachable in Europe impacti=
ng IP services. =20
Event Ticket ID: 9446797
Market Area Affected: Multiple Markets in Europe =20
Ticket Create Date: 7/10/15 11:15:39 AM GMT =20
Impacted For: 12 minutes =20
Event Status: Active =20
Time Since Last Update: 1 hour 39 minutes =20
7/10/15 11:26:39 AM GMT The IP NOC reported multiple devices were simultan=
eously unreachable in Europe impacting IP services. The devices recovered w=
ithout any intervention from Level 3 after an interruption lasting twelve m=
inutes. Due to the possibility of additional impacted, the IP NOC will moni=
tor services until the completion of multiple ongoing events in the region.=
A final update will be se
###
best regards
J=FCrgen Jaritsch
Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300
Telefax: +43-5-0556-500
E-Mail: jj@anexia.at=20
Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=DFe 140, 9020 Klagenfurt
Gesch=E4ftsf=FChrer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U632166=
01
-----Urspr=FCngliche Nachricht-----
Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Joseph Jenkins
Gesendet: Freitag, 10. Juli 2015 14:55
An: nanog@nanog.org
Betreff: Re: Level3 routing issue US west coast
Level3 had an issue with one of their core routers in Los Angeles last nigh=
t(7pm Pacific) and early this morning(1am Pacific). Last update to my troub=
le ticket had the issue still being reviewed by engineering, but that a cor=
e router was dropping packets.
> On Jul 10, 2015, at 3:59 AM, J=FCrgen Jaritsch <jj@anexia.at> wrote:
>=20
> Hi,
>=20
> does anyone else experience issues with the Level3 network at the US west=
coast? We see lots of broken paths like this:
>=20
> Packets =
Pings
> Host Loss% Snt Last A=
vg Best Wrst StDev
> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0=
.5 0.2 18.1 1.2
> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9=
.9 0.3 361.1 40.1
> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7=
.7 6.3 49.7 5.3
> 4. win-b4-link.telia.net 0.0% 230 6.6 6=
.8 6.4 20.2 1.5
> 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9=
.3 6.3 69.1 9.6
> 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165=
.0 164.5 194.9 2.6
> 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164=
.8 164.5 174.1 0.9
> 8. 4.53.208.102 34.3% 230 634.9 310=
.7 168.5 680.1 199.7
> 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455=
.2 304.9 954.6 203.4
> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 44=
1.4 323.1 822.0 182.1
> 11. 211.115.201.92 38.4% 230 289.8 41=
2.9 289.6 846.7 185.4
> 12. 61.250.89.2 35.8% 230 290.6 43=
9.4 290.2 804.3 205.1
> 13. ???
>=20
> Trace from NYC is also broken:
>=20
> Packets =
Pings
> Host Loss% Snt Last=
Avg Best Wrst StDev
> 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4=
4.3 0.4 57.8 13.3
> 2. nyk-b5-link.telia.net 0.0% 30 0.3=
0.4 0.3 0.9 0.1
> 3. ???
> 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7=
73.2 71.7 98.7 5.5
> 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8=
71.8 71.7 72.0 0.1
> 6. 4.53.208.102 31.0% 30 569.6=
250.7 70.5 579.3 231.2
> 7. 63.218.250.73 31.0% 30 672.6=
355.5 178.0 672.6 232.1
>=20
>=20
> At 10:24 UTC+2 it was even more broken:
>=20
> Packets =
Pings
> Host Loss% Snt Last A=
vg Best Wrst StDev
> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0=
.5 0.3 39.7 2.2
> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6=
.7 0.3 198.1 26.3
> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7=
.6 6.4 43.6 4.5
> 4. win-b4-link.telia.net 0.0% 326 6.7 7=
.4 6.3 43.1 3.2
> 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9=
.2 6.3 73.2 10.1
> 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165=
.5 164.5 176.7 1.5
> ae-2-70.edge1.SanJose3.Level3.net
> 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165=
.8 164.6 204.2 3.9
> ae-2-70.edge1.SanJose3.Level3.net
> 8. 205.129.5.70 74.2% 326 799.9 487=
.2 169.0 799.9 305.7
> 4.53.208.102
> 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701=
.0 308.7 3716. 510.6
> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 64=
3.0 323.4 960.4 307.6
> 11. 211.115.201.92 68.9% 326 925.3 67=
4.2 289.8 932.3 296.6
> 12. 61.250.89.2 72.9% 326 928.5 63=
7.2 291.9 928.5 304.3
> 13. ???
>=20
>=20
> best regards
>=20
> J=FCrgen Jaritsch
> Head of Network & Infrastructure
>=20
> ANEXIA Internetdienstleistungs GmbH
>=20
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
>=20
> E-Mail: jj@anexia.at<mailto:jj@anexia.at>
> Web: http://www.anexia.at<http://www.anexia.at/>
>=20
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=DFe 140, 9020 Klagenfurt
> Gesch=E4ftsf=FChrer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U6321=
6601
>=20
