[112346] in North American Network Operators' Group
RE: Illegal header length in BGP error
daemon@ATHENA.MIT.EDU (Matthew Huff)
Tue Feb 24 12:30:41 2009
From: Matthew Huff <mhuff@ox.com>
To: 'Paul Cosgrove' <paul.cosgrove@heanet.ie>, "'Mills, Charles'"
<cmills@accessdc.com>
Date: Tue, 24 Feb 2009 12:29:29 -0500
In-Reply-To: <49A42DB5.9050003@heanet.ie>
Cc: "'nanog@nanog.org'" <nanog@nanog.org>
Errors-To: nanog-bounces@nanog.org
------=_NextPart_000_0078_01C9967B.89A3FFA0
Content-Type: multipart/mixed;
boundary="----=_NextPart_001_0079_01C9967B.89A3FFA0"
------=_NextPart_001_0079_01C9967B.89A3FFA0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
We were using PMTUD. However:
1) The link was iBGP and was done via crossever with both having default =
MTU
2) I tried disabling PMTUD with no difference
3) Cisco admitted it was a known bug, and downreving it to 12.4(15)T
resolved the issue.
----
Matthew Huff=A0=A0=A0=A0=A0=A0 | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
http://www.ox.com | Phone: 914-460-4039
aim: matthewbhuff=A0 | Fax:=A0=A0 914-460-4139
> -----Original Message-----
> From: Paul Cosgrove [mailto:paul.cosgrove@heanet.ie]
> Sent: Tuesday, February 24, 2009 12:26 PM
> To: Mills, Charles
> Cc: Renaud RAKOTOMALALA; Matthew Huff; nanog@nanog.org
> Subject: Re: Illegal header length in BGP error
>=20
> Are you using PMTUD?
>=20
> We saw this on a couple of our route reflectors and on one occasion
> picked it up in a capture. So I can say that the issue is due to bad
> packets being sent, rather than an inaccurate error. It can be
> reported
> differently according to where the corruption occurs (e.g. unsupported
> message type, update malformed etc.).
>=20
> Two production BGP sessions were affected at different times, and one
> showed errors every few days, the other weeks apart. Both sessions
> were
> from route reflectors to other routers receiving full tables, and both
> traversed multiple hops. All other sessions of these routers were =
fine.
> Whilst investigating we identified that different MTUs were being used
> on the device interfaces at each end of the sessions. The session on
> which we saw most errors also had lower MTUs on intervening links, so
> PMTUD was suspected to be a factor.
>=20
> I replaced one of the paths with a direct link, using identical MTUs,
> and that stopped the errors on that session (since PMTUD had nothing =
to
> do anymore). Just to be sure we recreated a multiple hop topology =
from
> our production route reflectors to isolated lab routers, with low
> intervening link MTUs and ACLs to keep out other unwanted traffic -
> which also produced the same error on those sessions (but only once
> each
> over three months).
>=20
> After correcting all the MTUs in the production network the errors
> ceased completely. Our test routers shared these links, but also used
> an additional link with a low mtu which we deliberately did not fix; =
as
> it turned out we not see it again there either so the trigger was not
> entirely clear.
>=20
> One other thing to note is that, at the time, we were seeing some =
other
> problems with these production routers, whichcisco believed may have
> been due to SNMP polling of BGP stats. If you have been changing that
> recently I would also consider it a possibility.
>=20
> Paul.
>=20
>=20
>=20
> Mills, Charles wrote:
> > I ran into exactly the same thing during a code upgrade a few weeks
> ago.
> >
> > I wrote it off as a bug in BGP and backed off the code until a new
> release was out. I was also running 12.4(22)T
> > On an NPE-G2.
> >
> > Chuck
> >
> > -----Original Message-----
> > From: Renaud RAKOTOMALALA [mailto:renaud@rakotomalala.com]
> > Sent: Tuesday, February 24, 2009 10:49 AM
> > To: Matthew Huff; 'nanog@nanog.org'
> > Subject: Re: Illegal header length in BGP error
> >
> > Hello Matthew,
> >
> > We changed the motherboard from cisco one of our from 7206VXR (NPE-
> G1)
> > to 7206VXR (NPE-G2).
> >
> > Due to incompability with the IOS 12.3(4r)T3 we upgraded this IOS to
> > 12.4(12.2r)T. At the end we've got the same problem as you between
> one
> > of our 7200 in 12.3 and the new one in 12.4 ....
> >
> > We solved the problem by upgrading the cisco withe the IOS from
> > 12.4(12.2r) to 12.4(4)XD10 and the BGP session came back alive ....
> >
> > So now everything work fine between our 7200 (IOS 12.3) and the =
other
> > 7200 in IOS 12.4(4)XD10
> >
> > I hope it could help you ...
> >
> > Cheers,
> > Renaud
> >
> >
> > Matthew Huff a =E9crit :
> >
> >> One of our upstream providers flapped this morning, and since then
> they are
> >> sending corrupted BPG data. I'm running 12.4(22)T on cisco 7200s.
> I'm
> >> getting no BGP errors from that providers and the number of routes
> and basic
> >> sanity check looks okay. However, when it tries to redistribute the
> bgp
> >> routes via iBGP to our other board routers, we get:
> >>
> >> 003372: Feb 24 09:17:13.963 EST: %BGP-5-ADJCHANGE: neighbor x.x.x.x
> Down BGP
> >> Notification sent
> >> 003373: Feb 24 09:17:13.963 EST: %BGP-3-NOTIFICATION: sent to
> neighbor
> >> x.x.x.x 1/2 (illegal header length) 2 bytes
> >>
> >>
> >> All routes have identical hardware and IOS versions. My google and
> cisco
> >> search fu leads me to the AS path length bug, but the interesting
> thing is
> >> that since we have "bgp maxas-limit 75" configured and a recent =
IOS,
> we
> >> haven't had the problem before when other people were reporting
> issues. I've
> >> also looked at the path mtu issue, and although we haven't had a
> problem
> >> before I disabled bgp mtu path discovery, but have the same issues.
> >>
> >> Anyone seeing something like this today, and or does anyone have a
> >> suggestion on finding out more specific info (which as path for
> example so I
> >> can filter it)?
> >>
> >>
> >
> >
> >
> > This e-mail message and any files transmitted with it contain
> confidential information intended only for the person(s) to whom this
> email message is addressed. If you have received this e-mail message =
in
> error, please notify the sender immediately by telephone or e-mail and
> destroy the original message without making a copy. Thank you.
> > Neither this information block, the typed name of the sender, nor
> anything else in this message is intended to constitute an electronic
> signature unless a specific statement to the contrary is included in
> this message.
> >
> >
> >
> >
> >
------=_NextPart_001_0079_01C9967B.89A3FFA0
Content-Type: application/octet-stream;
name="Matthew Huff.vcf"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="Matthew Huff.vcf"
BEGIN:VCARD
VERSION:2.1
X-MS-SIGNATURE:YES
N:Huff;Matthew
FN:Matthew Huff
ORG:OTA Management LLC
TITLE:Director of Operations
TEL;WORK;VOICE:(914) 460-4039
ADR;WORK;PREF:;;1 Manhattanville Road;Purchase;NY;10577;United States of =
America
LABEL;WORK;PREF;ENCODING=3DQUOTED-PRINTABLE:1 Manhattanville =
Road=3D0D=3D0A=3D
Purchase, NY 10577
X-MS-OL-DEFAULT-POSTAL-ADDRESS:2
URL;WORK:http://www.ox.com
EMAIL;PREF;INTERNET:mhuff@ox.com
X-MS-OL-DESIGN;CHARSET=3Dutf-8:<card =
xmlns=3D"http://schemas.microsoft.com/office/outlook/12/electronicbusines=
scards" ver=3D"1.0" layout=3D"left" bgcolor=3D"ffffff"><img xmlns=3D"" =
align=3D"fit" area=3D"16" use=3D"cardpicture"/><fld xmlns=3D"" =
prop=3D"name" align=3D"left" dir=3D"ltr" style=3D"b" color=3D"000000" =
size=3D"10"/><fld xmlns=3D"" prop=3D"org" align=3D"left" dir=3D"ltr" =
color=3D"000000" size=3D"8"/><fld xmlns=3D"" prop=3D"title" =
align=3D"left" dir=3D"ltr" color=3D"000000" size=3D"8"/><fld xmlns=3D"" =
prop=3D"blank" size=3D"8"/><fld xmlns=3D"" prop=3D"telwork" =
align=3D"left" dir=3D"ltr" color=3D"000000" size=3D"8"><label =
align=3D"right" color=3D"626262">Work</label></fld><fld xmlns=3D"" =
prop=3D"email" align=3D"left" dir=3D"ltr" color=3D"000000" =
size=3D"8"/><fld xmlns=3D"" prop=3D"addrwork" align=3D"left" dir=3D"ltr" =
color=3D"000000" size=3D"8"/><fld xmlns=3D"" prop=3D"webwork" =
align=3D"left" dir=3D"ltr" color=3D"000000" size=3D"8"/><fld xmlns=3D"" =
prop=3D"blank" size=3D"8"/><fld xmlns=3D"" prop=3D"blank" =
size=3D"8"/><fld xmlns=3D"" prop=3D"blank" size=3D"8"/><fld xmlns=3D"" =
prop=3D"blank" size=3D"8"/><fld xmlns=3D"" prop=3D"blank" =
size=3D"8"/><fld xmlns=3D"" prop=3D"blank" size=3D"8"/><fld xmlns=3D"" =
prop=3D"blank" size=3D"8"/><fld xmlns=3D"" prop=3D"blank" =
size=3D"8"/></card>
REV:20090109T152814Z
END:VCARD
------=_NextPart_001_0079_01C9967B.89A3FFA0--
------=_NextPart_000_0078_01C9967B.89A3FFA0
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIL1DCCAj0w
ggGmAhEAzbp/VvDf5LxU/iKss3KqVTANBgkqhkiG9w0BAQIFADBfMQswCQYDVQQGEwJVUzEXMBUG
A1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVibGljIFByaW1hcnkgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNOTYwMTI5MDAwMDAwWhcNMjgwODAxMjM1OTU5WjBfMQsw
CQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVi
bGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwgZ8wDQYJKoZIhvcNAQEBBQADgY0A
MIGJAoGBAOUZv22jVmEtmUhx9mfeuY3rt56GgAqRDvo4Ja9GiILlc6igmyRdDR/MZW4MsNBWhBiH
mgabEKFz37RYOWtuwfYV1aioP6oSBo0xrH+wNNePNGeICc0UEeJORVZpH3gCgNrcR5EpuzbJY1zF
4Ncth3uhtzKwezC6Ki8xqu6jZ9rbAgMBAAEwDQYJKoZIhvcNAQECBQADgYEATD+4i8Zo3+5DMw5d
6abLB4RNejP/khv0Nq3YlSI2aBFsfELM85wuxAc/FLAPT/+Qknb54rxK6Y/NoIAK98Up8YIiXbix
3YEjo3slFUYweRb46gVLlH8dwhzI47f0EEA8E8NfH1PoSOSGtHuhNbB7Jbq4046rPzidADQAmPPR
cZQwggS/MIIDp6ADAgECAhBslbQv0S59MM/oRC2B0lgrMA0GCSqGSIb3DQEBBQUAMIHdMQswCQYD
VQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0
IE5ldHdvcmsxOzA5BgNVBAsTMlRlcm1zIG9mIHVzZSBhdCBodHRwczovL3d3dy52ZXJpc2lnbi5j
b20vcnBhIChjKTA1MR4wHAYDVQQLExVQZXJzb25hIE5vdCBWYWxpZGF0ZWQxNzA1BgNVBAMTLlZl
cmlTaWduIENsYXNzIDEgSW5kaXZpZHVhbCBTdWJzY3JpYmVyIENBIC0gRzIwHhcNMDkwMTA5MDAw
MDAwWhcNMTAwMTA5MjM1OTU5WjCCAQwxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMUYwRAYDVQQLEz13d3cudmVyaXNpZ24uY29tL3JlcG9z
aXRvcnkvUlBBIEluY29ycC4gYnkgUmVmLixMSUFCLkxURChjKTk4MR4wHAYDVQQLExVQZXJzb25h
IE5vdCBWYWxpZGF0ZWQxNDAyBgNVBAsTK0RpZ2l0YWwgSUQgQ2xhc3MgMSAtIE1pY3Jvc29mdCBG
dWxsIFNlcnZpY2UxFTATBgNVBAMUDE1hdHRod2UgSHVmZjEbMBkGCSqGSIb3DQEJARYMbWh1ZmZA
b3guY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC1wlVAitEiaVcyZo3NXIUzup2tf83U
lg8pIvzsD5TIYLcmDoS42zyF3x/orVC9V9saVsbIjA83n+Oc8KaIS9nBBf9SbKl01MCcxqtLbL0j
PGmdUviVeKHNsMCItlyNhfa2kM2lEfzKVXPzKJx0RdcJjeXcseeQRx1jM7xsNIGXYwIDAQABo4HM
MIHJMAkGA1UdEwQCMAAwRAYDVR0gBD0wOzA5BgtghkgBhvhFAQcXATAqMCgGCCsGAQUFBwIBFhxo
dHRwczovL3d3dy52ZXJpc2lnbi5jb20vcnBhMAsGA1UdDwQEAwIFoDAdBgNVHSUEFjAUBggrBgEF
BQcDBAYIKwYBBQUHAwIwSgYDVR0fBEMwQTA/oD2gO4Y5aHR0cDovL0luZEMxRGlnaXRhbElELWNy
bC52ZXJpc2lnbi5jb20vSW5kQzFEaWdpdGFsSUQuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQA7m0dM
g4SQgbazEj3wwqH4AFg+ldFd9KAwLAz41tITOEznwBFMt9Zz7t1JwtWv9vCL4PRl5p8J52g6/Cy3
0Uk4R2bLV6W0YD080P8euP3WgpOaQ5yfvqnPhezPHHYUa3Ali4fkmvQtjED+4oVUX5jqitdHTaO/
2mEBHaIuRPmGMdpHRH1xiwuNwinWyona3OyXQUvhL1AHntQFE7D0278yvAn6rbc0scqevy29l4hf
lwc1t5//mUiJuK6VxZvwpC406H0TP3dXXYdQNpPlmu+F3IRCZ7QOh+v08bCnJdd/OLF8LqIv7bsX
LRsS5iwkj5HOA8QdWp1V5jpdM3UjbGsmMIIEzDCCBDWgAwIBAgIQHK6da5r05i8iiqPadGFsHjAN
BgkqhkiG9w0BAQUFADBfMQswCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1
BgNVBAsTLkNsYXNzIDEgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcN
MDUxMDI4MDAwMDAwWhcNMTUxMDI3MjM1OTU5WjCB3TELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZl
cmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJU
ZXJtcyBvZiB1c2UgYXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykwNTEeMBwGA1UE
CxMVUGVyc29uYSBOb3QgVmFsaWRhdGVkMTcwNQYDVQQDEy5WZXJpU2lnbiBDbGFzcyAxIEluZGl2
aWR1YWwgU3Vic2NyaWJlciBDQSAtIEcyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
yd+s5+r4+AMUxACS1cF+NsI873xyFcvAq4w9HJXObx4QLD8A7Zcm5rbH5q1DHT+kh0dHTD5U+Gz4
x/yxnr0wcLyXsQMF6pXxrUDFRHpLBaLyYPzXOmVi7/8Qe6JWu8VOcC3Woh887bBC6F6NVyGsppnZ
EenSGgfAdEdCC/zFNOr95rok0R0IFTei13PPAUEvY7I6P76lGm70yUpbPZWmFbs1Ahn51O+8jw5x
dlm7S7Y+1vxaFvTWDonySf5sDO0V6dmIdZx5zmAn3bmtdc4vc5V6QDqFdUmwuN9ovKvNE4KFEVCj
4DwLrsAKU83XMG+FMkYb5EkQwmzirx95/9u0tQIDAQABo4IBhDCCAYAwEgYDVR0TAQH/BAgwBgEB
/wIBADBEBgNVHSAEPTA7MDkGC2CGSAGG+EUBBxcBMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3
LnZlcmlzaWduLmNvbS9ycGEwCwYDVR0PBAQDAgEGMBEGCWCGSAGG+EIBAQQEAwIBBjAuBgNVHREE
JzAlpCMwITEfMB0GA1UEAxMWUHJpdmF0ZUxhYmVsMy0yMDQ4LTE1NTAdBgNVHQ4EFgQUEX1eGX08
BN9qbNaiiho/Mdg7lFIwMQYDVR0fBCowKDAmoCSgIoYgaHR0cDovL2NybC52ZXJpc2lnbi5jb20v
cGNhMS5jcmwwgYEGA1UdIwR6MHihY6RhMF8xCzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJpU2ln
biwgSW5jLjE3MDUGA1UECxMuQ2xhc3MgMSBQdWJsaWMgUHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1
dGhvcml0eYIRAM26f1bw3+S8VP4irLNyqlUwDQYJKoZIhvcNAQEFBQADgYEAsS/ZluGSou6BYOXI
KiD74Wcs1gCYU6MCG+mQS/gYRJ8PRvf6oP7THRij0r8c7NYZn0pNQ/jKu74TgEkF3SFzM1fCQlq+
+gCTsuYEMZFOXTzwcwU3Y+u/g1mY/Wbe6YYympIpPDquVNqmElGxj8jK00d45tulHocG49EUwMIh
9roxggRzMIIEbwIBATCB8jCB3TELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMu
MR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug
YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykwNTEeMBwGA1UECxMVUGVyc29uYSBO
b3QgVmFsaWRhdGVkMTcwNQYDVQQDEy5WZXJpU2lnbiBDbGFzcyAxIEluZGl2aWR1YWwgU3Vic2Ny
aWJlciBDQSAtIEcyAhBslbQv0S59MM/oRC2B0lgrMAkGBSsOAwIaBQCgggLWMBgGCSqGSIb3DQEJ
AzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA5MDIyNDE3MjkyOVowIwYJKoZIhvcNAQkE
MRYEFJzfZb4PMTFRsrSHKRhdjzHuRpC5MGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYI
KoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsO
AwIaMAoGCCqGSIb3DQIFMIIBAwYJKwYBBAGCNxAEMYH1MIHyMIHdMQswCQYDVQQGEwJVUzEXMBUG
A1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0IE5ldHdvcmsxOzA5
BgNVBAsTMlRlcm1zIG9mIHVzZSBhdCBodHRwczovL3d3dy52ZXJpc2lnbi5jb20vcnBhIChjKTA1
MR4wHAYDVQQLExVQZXJzb25hIE5vdCBWYWxpZGF0ZWQxNzA1BgNVBAMTLlZlcmlTaWduIENsYXNz
IDEgSW5kaXZpZHVhbCBTdWJzY3JpYmVyIENBIC0gRzICEGyVtC/RLn0wz+hELYHSWCswggEFBgsq
hkiG9w0BCRACCzGB9aCB8jCB3TELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMu
MR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug
YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykwNTEeMBwGA1UECxMVUGVyc29uYSBO
b3QgVmFsaWRhdGVkMTcwNQYDVQQDEy5WZXJpU2lnbiBDbGFzcyAxIEluZGl2aWR1YWwgU3Vic2Ny
aWJlciBDQSAtIEcyAhBslbQv0S59MM/oRC2B0lgrMA0GCSqGSIb3DQEBAQUABIGAhvmRLxe/tnOM
A54XveKT7D6nZQ/L8s2YZiaJiunOibN21XU1UOVW4AZNr1R0S4FnePKAzEmJHyQEv/44piOvECCz
92rhg3sKRvqKzW6TaW60uqSncedbI9tr6tpJDqf/ngbmN2QekSxBqClRU6B9AKpwP672N/2v/cAk
KLBDMqgAAAAAAAA=
------=_NextPart_000_0078_01C9967B.89A3FFA0--
