[184422] in North American Network Operators' Group
Re: AW: AW: /27 the new /24
daemon@ATHENA.MIT.EDU (Mike Hammett)
Fri Oct 2 20:52:23 2015
X-Original-To: nanog@nanog.org
Date: Fri, 2 Oct 2015 19:52:17 -0500 (CDT)
From: Mike Hammett <nanog@ics-il.net>
Cc: NANOG <nanog@nanog.org>
In-Reply-To: <133ff0fed3394045add071349c00fcef@anx-i-dag02.anx.local>
Errors-To: nanog-bounces@nanog.org
I don't expect carriers to be running UBNT\Mikrotik, but the boxes that hav=
e been there for 10 years have more than paid for themselves (unless they'r=
e a shitty business). It's time to rip and replace with whatever is appropr=
iate for that site. No, I obviously don't think I'm going to change anyone'=
s opinion on the matter (at least not anyone that matters in one of these n=
etworks). What I was saying is that my little business with meager means (a=
nd revenues) can afford a box to do it. They can too.=20
I don't doubt their situation sucks... but either you fix it or you don't. =
Time and the rest of the Internet won't wait for them.=20
If their business hasn't boomed, maybe it's time to replace that old 6500 w=
ith a 4500x or a QFX-5100 or an x670 or whatever. Your decreased power bill=
alone will pay it off. If it has boomed, then ten years of revenues should=
get you whatever the bigger Ciscos are or an MX or whatever the bigger Ext=
remes are.=20
Don't whine about my choices in gear I mentioned. I was just throwing thing=
s out there. Old big, new small if no money or old big new big if money.=20
BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle f=
ull table imports in a significantly reduced time. Oh, and I'm not sure tha=
t you couldn't do at least three nines with MT\UBNT. Well, no experience wi=
th the EdgeRouters yet.=20
-----=20
Mike Hammett=20
Intelligent Computing Solutions=20
http://www.ics-il.com=20
Midwest Internet Exchange=20
http://www.midwest-ix.com=20
----- Original Message -----
From: "J=C3=BCrgen Jaritsch" <jj@anexia.at>=20
To: "Mike Hammett" <nanog@ics-il.net>=20
Cc: "NANOG" <nanog@nanog.org>=20
Sent: Friday, October 2, 2015 6:11:47 PM=20
Subject: AW: AW: /27 the new /24=20
Hi Mike,=20
sorry, this was probably sent to quick ... let me please explain my POV of =
your statement:=20
I want to concentrate my detailed answer only to the backbone situation whi=
ch is often handled by the 6500/7600 - I guess all of us know that the 6500=
/7600 has a ton of additional features ...=20
6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone cap=
acities built with platforms that only had n*100M interfaces another 3-5 ye=
ars before. Their only invest in these 3-5 years was to add the Gig line ca=
rds, install some software updates and add new fibre optics (GBICs). Chassi=
s, cabling, management interfaces etc could be remain in the cabinet - they=
only had to replace ONE line card (let's say for a few thousand bucks) and=
with this invest they were able to scale up their capacities. Of course: a=
t some point they also had to replace the SUPs, PSUs, FANs, etc. But the in=
vest in the surrounding stuff is nothing compared with completely new machi=
nes.=20
So what all these companies did was buying a machine with an basic configur=
ation and since 10(!) years they are able to expand this machines with (mor=
e or less) small and cheap upgrades.=20
In backbone situations the 6500/7600 are definitely at the end of the resou=
rces the platform can provide. Most of the carriers (and of course also the=
bigger ISPs) had a real chance to evaluate a new model/vendor to ran futur=
e networks (with possibly also a very good scale-up path and scaling- and u=
pgrade-options). Most of the before mentioned are already in an migration p=
rocess (let's take a look at Seabone ... they are migration from Cisco to a=
mix of Juniper and Huawei).=20
Summary: there are strict limitations within the Cisco 6500/7600 platform a=
nd these limitations forces the big players to move this boxes out (or move=
them into other parts of their network). The limitation with 1Mio routes i=
s not a secret and the admins of these boxes decide what they want to use (=
e.g. 768k routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If th=
e global routing table reaches the 768k mark (I guess this will happen in t=
he next 12-18months) most of the boxes will crash again (as it happened in =
Aug 2014).=20
Regarding the words "I have a small router which handles multiple full tabl=
es ...": push and pull a few full tables at the same time and you'll see wh=
at's happening: the CCRs are SLOW. And why? Because the software is not as =
good as it could be: the BGP daemon uses only one core of a 36(?) core CPU.=
Same problem in the past with the EoIP daemon (not sure if they fixed it o=
n the CCRs - they fixed it on x86).=20
Routerboards are nice and cool and to be honest: I'm a big fan of this stuf=
f (also Ubiquiti). But with this boxes you're not able to ran a stable ente=
rprise class carrier network with >99,5% uptime. And that=E2=80=99s thei MA=
IN reason why "the old shit" is still online :).=20
Hopefully my words explained my hard "you know nothing" blabla ?=20
Best regards=20
J=C3=BCrgen Jaritsch=20
Head of Network & Infrastructure=20
ANEXIA Internetdienstleistungs GmbH=20
Telefon: +43-5-0556-300=20
Telefax: +43-5-0556-500=20
E-Mail: JJaritsch@anexia-it.com=20
Web: http://www.anexia-it.com=20
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=C3=9Fe 140, 9020 Klagenfur=
t=20
Gesch=C3=A4ftsf=C3=BChrer: Alexander Windbichler=20
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U632166=
01=20
-----Urspr=C3=BCngliche Nachricht-----=20
Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett=20
Gesendet: Freitag, 02. Oktober 2015 21:33=20
Cc: NANOG <nanog@nanog.org>=20
Betreff: Re: AW: /27 the new /24=20
Hrm.=20
-----=20
Mike Hammett=20
Intelligent Computing Solutions=20
http://www.ics-il.com=20
Midwest Internet Exchange=20
http://www.midwest-ix.com=20
----- Original Message -----=20
From: "J=C3=BCrgen Jaritsch" <jj@anexia.at>=20
To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org>=20
Sent: Friday, October 2, 2015 2:25:10 PM=20
Subject: AW: /27 the new /24=20
> Stop using old shit.=20
Sorry, but the truth is: you have no idea about how earning revenue works a=
nd you obviously also have no idea about carrier grade networks.=20
J=C3=BCrgen Jaritsch=20
Head of Network & Infrastructure=20
ANEXIA Internetdienstleistungs GmbH=20
Telefon: +43-5-0556-300=20
Telefax: +43-5-0556-500=20
E-Mail: JJaritsch@anexia-it.com=20
Web: http://www.anexia-it.com=20
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=C3=9Fe 140, 9020 Klagenfur=
t=20
Gesch=C3=A4ftsf=C3=BChrer: Alexander Windbichler=20
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U632166=
01=20
-----Urspr=C3=BCngliche Nachricht-----=20
Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett=20
Gesendet: Freitag, 02. Oktober 2015 20:38=20
An: NANOG <nanog@nanog.org>=20
Betreff: Re: /27 the new /24=20
Chances are the revenue passing scales to some degree as well. Small busine=
ss with small bandwidth needs buys small and has small revenue. Big busines=
s with big bandwidth needs buys big and has big revenue to support big rout=
er.=20
I can think of no reason why ten years goes by and you haven't had a need t=
o throw out the old network for new. If your business hasn't scaled with th=
e times, then you need to get rid of your Cat 6500 and get something more p=
ower, space, heat, etc. efficient.=20
I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco rou=
ters. I don't know what they were at the moment, but they had GBICs, so the=
y weren't exactly new. Each router had two 2500w power supplies. They'll be=
worse in every way (other than *possibly* BGP convergence). The old setup =
consumed at most 300 watts. The new setup requires $500/month in power... a=
nd is worse.=20
Stop using old shit.=20
-----=20
Mike Hammett=20
Intelligent Computing Solutions=20
http://www.ics-il.com=20
Midwest Internet Exchange=20
http://www.midwest-ix.com=20
----- Original Message -----=20
From: "William Herrin" <bill@herrin.us>=20
To: "Mike Hammett" <nanog@ics-il.net>=20
Cc: "NANOG" <nanog@nanog.org>=20
Sent: Friday, October 2, 2015 1:09:16 PM=20
Subject: Re: /27 the new /24=20
On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote:=20
> How many routers out there have this limitation? A $100 router=20
> I bought ten years ago could manage many full tables. If=20
> someone's network can't match that today, should I really have=20
> any pity for them?=20
Hi Mike,=20
The technology doesn't work the way you think it does. Or more=20
precisely, it only works the way you think it does on small (cheap)=20
end-user routers. Those routers do everything in software on a=20
general-purpose CPU using radix tries for the forwarding table (FIB).=20
They don't have to (and can't) handle both high data rates and large=20
routing tables at the same time.=20
For a better understanding how the big iron works, check out=20
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see=20
folks here talk about TCAM. This stands for Ternary Content=20
Addressable Memory. It's a special circuit, different from DRAM and=20
SRAM, used by most (but not all) big iron routers. The TCAM permits an=20
O(1) route lookup instead of an O(log n) lookup. The architectural=20
differences which balloon from there move the router cost from your=20
$100 router into the hundreds of thousands of dollars.=20
Your BGP advertisement doesn't just have to be carried on your $100=20
router. It also has to be carried on the half-million-dollar routers.=20
That makes it expensive.=20
Though out of date, this paper should help you better understand the=20
systemic cost of a BGP route advertisement:=20
http://bill.herrin.us/network/bgpcost.html=20
Regards,=20
Bill Herrin=20
--=20
William Herrin ................ herrin@dirtside.com bill@herrin.us=20
Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>=20
