[184420] in North American Network Operators' Group
Re: AW: AW: /27 the new /24
daemon@ATHENA.MIT.EDU (Mel Beckman)
Fri Oct 2 20:00:45 2015
X-Original-To: nanog@nanog.org
From: Mel Beckman <mel@beckman.org>
To: =?Windows-1252?Q?J=FCrgen_Jaritsch?= <jj@anexia.at>
Date: Sat, 3 Oct 2015 00:00:39 +0000
In-Reply-To: <133ff0fed3394045add071349c00fcef@anx-i-dag02.anx.local>
Cc: NANOG <nanog@nanog.org>
Errors-To: nanog-bounces@nanog.org
Well said, J=FCrgen!
-mel via cell
> On Oct 2, 2015, at 4:13 PM, J=FCrgen Jaritsch <jj@anexia.at> wrote:
>=20
> Hi Mike,
>=20
> sorry, this was probably sent to quick ... let me please explain my POV o=
f your statement:
>=20
> I want to concentrate my detailed answer only to the backbone situation w=
hich is often handled by the 6500/7600 - I guess all of us know that the 65=
00/7600 has a ton of additional features ...
>=20
>=20
> 6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone c=
apacities built with platforms that only had n*100M interfaces another 3-5 =
years before. Their only invest in these 3-5 years was to add the Gig line =
cards, install some software updates and add new fibre optics (GBICs). Chas=
sis, cabling, management interfaces etc could be remain in the cabinet - th=
ey only had to replace ONE line card (let's say for a few thousand bucks) a=
nd with this invest they were able to scale up their capacities. Of course:=
at some point they also had to replace the SUPs, PSUs, FANs, etc. But the =
invest in the surrounding stuff is nothing compared with completely new mac=
hines.
>=20
> So what all these companies did was buying a machine with an basic config=
uration and since 10(!) years they are able to expand this machines with (m=
ore or less) small and cheap upgrades.=20
>=20
> In backbone situations the 6500/7600 are definitely at the end of the res=
ources the platform can provide. Most of the carriers (and of course also t=
he bigger ISPs) had a real chance to evaluate a new model/vendor to ran fut=
ure networks (with possibly also a very good scale-up path and scaling- and=
upgrade-options). Most of the before mentioned are already in an migration=
process (let's take a look at Seabone ... they are migration from Cisco to=
a mix of Juniper and Huawei).
>=20
> Summary: there are strict limitations within the Cisco 6500/7600 platform=
and these limitations forces the big players to move this boxes out (or mo=
ve them into other parts of their network). The limitation with 1Mio routes=
is not a secret and the admins of these boxes decide what they want to use=
(e.g. 768k routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If =
the global routing table reaches the 768k mark (I guess this will happen in=
the next 12-18months) most of the boxes will crash again (as it happened i=
n Aug 2014).=20
>=20
>=20
> Regarding the words "I have a small router which handles multiple full ta=
bles ...": push and pull a few full tables at the same time and you'll see =
what's happening: the CCRs are SLOW. And why? Because the software is not a=
s good as it could be: the BGP daemon uses only one core of a 36(?) core CP=
U. Same problem in the past with the EoIP daemon (not sure if they fixed it=
on the CCRs - they fixed it on x86).
>=20
> Routerboards are nice and cool and to be honest: I'm a big fan of this st=
uff (also Ubiquiti). But with this boxes you're not able to ran a stable en=
terprise class carrier network with >99,5% uptime. And that=92s thei MAIN r=
eason why "the old shit" is still online :).
>=20
> Hopefully my words explained my hard "you know nothing" blabla ?
>=20
> Best regards
>=20
>=20
> J=FCrgen Jaritsch
> Head of Network & Infrastructure
>=20
> ANEXIA Internetdienstleistungs GmbH
>=20
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
>=20
> E-Mail: JJaritsch@anexia-it.com=20
> Web: http://www.anexia-it.com=20
>=20
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=DFe 140, 9020 Klagenfurt
> Gesch=E4ftsf=FChrer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U6321=
6601
>=20
> -----Urspr=FCngliche Nachricht-----
> Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett
> Gesendet: Freitag, 02. Oktober 2015 21:33
> Cc: NANOG <nanog@nanog.org>
> Betreff: Re: AW: /27 the new /24
>=20
> Hrm.=20
>=20
>=20
>=20
>=20
> -----=20
> Mike Hammett=20
> Intelligent Computing Solutions=20
> http://www.ics-il.com=20
>=20
>=20
>=20
> Midwest Internet Exchange=20
> http://www.midwest-ix.com=20
>=20
>=20
> ----- Original Message -----
>=20
> From: "J=FCrgen Jaritsch" <jj@anexia.at>=20
> To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org>=20
> Sent: Friday, October 2, 2015 2:25:10 PM=20
> Subject: AW: /27 the new /24=20
>=20
>> Stop using old shit.
>=20
> Sorry, but the truth is: you have no idea about how earning revenue works=
and you obviously also have no idea about carrier grade networks.=20
>=20
>=20
>=20
>=20
> J=FCrgen Jaritsch=20
> Head of Network & Infrastructure=20
>=20
> ANEXIA Internetdienstleistungs GmbH=20
>=20
> Telefon: +43-5-0556-300=20
> Telefax: +43-5-0556-500=20
>=20
> E-Mail: JJaritsch@anexia-it.com=20
> Web: http://www.anexia-it.com=20
>=20
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=DFe 140, 9020 Klagenfurt=
=20
> Gesch=E4ftsf=FChrer: Alexander Windbichler=20
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U6321=
6601=20
>=20
> -----Urspr=FCngliche Nachricht-----=20
> Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett=20
> Gesendet: Freitag, 02. Oktober 2015 20:38=20
> An: NANOG <nanog@nanog.org>=20
> Betreff: Re: /27 the new /24=20
>=20
> Chances are the revenue passing scales to some degree as well. Small busi=
ness with small bandwidth needs buys small and has small revenue. Big busin=
ess with big bandwidth needs buys big and has big revenue to support big ro=
uter.=20
>=20
> I can think of no reason why ten years goes by and you haven't had a need=
to throw out the old network for new. If your business hasn't scaled with =
the times, then you need to get rid of your Cat 6500 and get something more=
power, space, heat, etc. efficient.=20
>=20
>=20
> I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco r=
outers. I don't know what they were at the moment, but they had GBICs, so t=
hey weren't exactly new. Each router had two 2500w power supplies. They'll =
be worse in every way (other than *possibly* BGP convergence). The old setu=
p consumed at most 300 watts. The new setup requires $500/month in power...=
and is worse.=20
>=20
> Stop using old shit.=20
>=20
>=20
>=20
>=20
> -----=20
> Mike Hammett=20
> Intelligent Computing Solutions=20
> http://www.ics-il.com=20
>=20
>=20
>=20
> Midwest Internet Exchange=20
> http://www.midwest-ix.com=20
>=20
>=20
> ----- Original Message -----=20
>=20
> From: "William Herrin" <bill@herrin.us>=20
> To: "Mike Hammett" <nanog@ics-il.net>=20
> Cc: "NANOG" <nanog@nanog.org>=20
> Sent: Friday, October 2, 2015 1:09:16 PM=20
> Subject: Re: /27 the new /24=20
>=20
>> On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote:=
=20
>> How many routers out there have this limitation? A $100 router=20
>> I bought ten years ago could manage many full tables. If=20
>> someone's network can't match that today, should I really have=20
>> any pity for them?
>=20
> Hi Mike,=20
>=20
> The technology doesn't work the way you think it does. Or more=20
> precisely, it only works the way you think it does on small (cheap)=20
> end-user routers. Those routers do everything in software on a=20
> general-purpose CPU using radix tries for the forwarding table (FIB).=20
> They don't have to (and can't) handle both high data rates and large=20
> routing tables at the same time.=20
>=20
> For a better understanding how the big iron works, check out=20
> https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see=20
> folks here talk about TCAM. This stands for Ternary Content=20
> Addressable Memory. It's a special circuit, different from DRAM and=20
> SRAM, used by most (but not all) big iron routers. The TCAM permits an=20
> O(1) route lookup instead of an O(log n) lookup. The architectural=20
> differences which balloon from there move the router cost from your=20
> $100 router into the hundreds of thousands of dollars.=20
>=20
> Your BGP advertisement doesn't just have to be carried on your $100=20
> router. It also has to be carried on the half-million-dollar routers.=20
> That makes it expensive.=20
>=20
> Though out of date, this paper should help you better understand the=20
> systemic cost of a BGP route advertisement:=20
> http://bill.herrin.us/network/bgpcost.html=20
>=20
> Regards,=20
> Bill Herrin=20
>=20
>=20
>=20
>=20
> --=20
> William Herrin ................ herrin@dirtside.com bill@herrin.us=20
> Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>=20
>=20
>=20
