[184441] in North American Network Operators' Group
Re: AW: AW: AW: /27 the new /24
daemon@ATHENA.MIT.EDU (Mike Hammett)
Sat Oct 3 09:10:43 2015
X-Original-To: nanog@nanog.org
Date: Sat, 3 Oct 2015 08:10:36 -0500 (CDT)
From: Mike Hammett <nanog@ics-il.net>
Cc: NANOG <nanog@nanog.org>
In-Reply-To: <a2a21ecceb5641999c9b728af616f703@anx-i-dag02.anx.local>
Errors-To: nanog-bounces@nanog.org
I don't think we are talking different things, though I think we are talkin=
g in circles and thus the thread probably needs to die.=20
People keep thinking I want Level 3 to replace a loaded 6500 with a CCR and=
that's simply not what I'm saying at all. The point of rattling off the ne=
wer\smaller hardware was to say that if the site doesn't require 40G\100G, =
doesn't have the revenue to support an MX480, etc. you should put in a smal=
ler\cheaper box. Cost is a non-issue at that point because the smaller gear=
that's all you need will have far less operational cost. Someone thought a=
particular POP was going to be a big hit... and wasn't. On the flip side, =
if there are 200 ports of customers chances are you need the big interfaces=
that aren't on the old boxes. You have the bigger revenue. Heck, the new b=
ig boxes probably still use less power than the old big boxes anyway.=20
What I learned from this thread: Once you mention MT\UBNT routers, people a=
ssume you're using a MT\UBNT hammer everywhere.=20
-----=20
Mike Hammett=20
Intelligent Computing Solutions=20
http://www.ics-il.com=20
Midwest Internet Exchange=20
http://www.midwest-ix.com=20
----- Original Message -----
From: "J=C3=BCrgen Jaritsch" <jj@anexia.at>=20
To: "Mike Hammett" <nanog@ics-il.net>=20
Cc: "NANOG" <nanog@nanog.org>=20
Sent: Saturday, October 3, 2015 6:06:59 AM=20
Subject: AW: AW: AW: /27 the new /24=20
Hi Mike,=20
> but the boxes that have been there for 10 years have more than paid for t=
hemselves (unless they're a shitty business).=20
No question about that! But why should they throw them away if they can sti=
ll print $$$ with these boxes? They have to change nothing till the global =
routing table reaches at least 768k ... so let's say this will happen in 12=
-18 months. They have enough time to prepare, migrate, etc ... and while al=
l the side stories are happening they are still able to print $$$ with the =
"old shit".=20
> What I was saying is that my little business with meager means (and reven=
ues) can afford a box to do it.=20
This is definitely a question about sizing. Replacing a box with ~200 conne=
cted customers (only at this box!) is way more complex and this is nothing =
unrealistic.=20
> If their business hasn't boomed, maybe it's time to replace that old 6500=
with a 4500x or a QFX-5100 or an x670 or whatever.=20
4500x =3D> no MPLS features=20
QFX-5100 =3D> very nice box (I'm a big fan) but complicate (and expensive!)=
licensing.=20
Extreme x670 =3D> nice box too - we also use this. But it's simply too smal=
l and the BGP configuration on these boxes is horrible. It's also not possi=
ble to provide Ethernet over MPLS with LACP BPDU forwarding ... too less fe=
atures. Nice for aggregation and POP interconnect.=20
All three models are new and shiny but they can't replace a 6500/7600. Too =
less port density and too less features (people are still using SDH. You ne=
ed SDH in an 6500/7600? Simply install the required line card ...). If you =
really plan to replace a 6509 or even a 6513 you have to go with something =
like Juniper MX480/960 (I'm in love ... :D) or Cisco Nexus 7k/9k.=20
One thing that will more and more happen: physical separation. There will b=
e boxes with 10G/40G/100G only and boxes with 100M/1G only. Why? It's easie=
r for vendors to remove old compatibility requirements (like electrical int=
erfaces). So what we did in the past 3 years (replacing old boxes with new =
boxes with 1G/10G interfaces) was useless - we'll get our "old shit" back i=
n place and bring them up and running. Of course: the "old shit" will be re=
duced to do aggregation layer or to something like "multihop instance" to t=
ransport the customers access port to the "real big and powerful router". S=
olving this with Layer2 extensions (like VLANs) is not practicable because =
you'll ran into other problems (like STP instances, etc). Probably it makes=
sense to solve it with Layer2VPN (Ethernet over MPLS, etc) to transport th=
e physical interface to a virtual interface.=20
Lots of things to think about :(.=20
> Your decreased power bill alone will pay it off. If it has boomed, then t=
en years of revenues should get you whatever the bigger Ciscos are or an MX=
or whatever the bigger Extremes are.=20
Power is no argument. You get power starting at 0,10 Eur /kWh. Another 0,10=
Eur / kWh for cooling and we talk about 0,20 Eur / kWh =3D> Cisco 6513 (co=
nfigured with 11 line cards + 2x SUP) with 2x 6kW PSU uses 3,8kW. 3,8kW * 2=
4 hours * 30 days =3D 2.736 kWh per month. 2.736 * 0,20 Eur =3D 547,2 Eur p=
er month for power consumption + cooling. If you have a good sales engineer=
you earn the revenue for this "side cost" with 1 customer :). Realistic ca=
lculation is: 10 customers are required to earn the money for the footprint=
.=20
> Don't whine about my choices in gear I mentioned. I was just throwing thi=
ngs out there. Old big, new small if no money or old big new big if money.=
=20
Think the other way around: companies are earning Mio (or even Bil??) with =
the old equipment and everything is up and running. Only sometimes there is=
a small hick up because (of course!) also the "old shit" gets stuck from t=
ime to time and crashes. They did everything the right way (especially Leve=
l3 ...) from the commercial POV.=20
> BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle=
full table imports in a significantly reduced time. Oh, and I'm not sure t=
hat you couldn't do at least three nines with MT\UBNT. Well, no experience =
with the EdgeRouters yet.=20
Never tried the earlier versions - my last tests happened in the end of 201=
4. I think we're talking a little bit about different sizes: you're talking=
about the CCRs and EdgeRouters (which are nice of course - no question abo=
ut that!) and I'm talking about customer access devices (not CEP!) at carri=
er grade networks. Boxes I'm talking about have at least a few hundred port=
s.=20
I think it's very important what UBNT and MT does: they bring fresh wind at=
the customer/semi-pro market and they show up that you (as a vendor) could=
get in touch with customers and optimize your equipment with customers fee=
dback.=20
best regards=20
J=C3=BCrgen Jaritsch=20
Head of Network & Infrastructure=20
ANEXIA Internetdienstleistungs GmbH=20
Telefon: +43-5-0556-300=20
Telefax: +43-5-0556-500=20
E-Mail: JJaritsch@anexia-it.com=20
Web: http://www.anexia-it.com=20
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=C3=9Fe 140, 9020 Klagenfur=
t=20
Gesch=C3=A4ftsf=C3=BChrer: Alexander Windbichler=20
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U632166=
01=20
-----Urspr=C3=BCngliche Nachricht-----=20
Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett=20
Gesendet: Samstag, 03. Oktober 2015 02:52=20
Cc: NANOG <nanog@nanog.org>=20
Betreff: Re: AW: AW: /27 the new /24=20
I don't expect carriers to be running UBNT\Mikrotik, but the boxes that hav=
e been there for 10 years have more than paid for themselves (unless they'r=
e a shitty business). It's time to rip and replace with whatever is appropr=
iate for that site. No, I obviously don't think I'm going to change anyone'=
s opinion on the matter (at least not anyone that matters in one of these n=
etworks). What I was saying is that my little business with meager means (a=
nd revenues) can afford a box to do it. They can too.=20
I don't doubt their situation sucks... but either you fix it or you don't. =
Time and the rest of the Internet won't wait for them.=20
If their business hasn't boomed, maybe it's time to replace that old 6500 w=
ith a 4500x or a QFX-5100 or an x670 or whatever. Your decreased power bill=
alone will pay it off. If it has boomed, then ten years of revenues should=
get you whatever the bigger Ciscos are or an MX or whatever the bigger Ext=
remes are.=20
Don't whine about my choices in gear I mentioned. I was just throwing thing=
s out there. Old big, new small if no money or old big new big if money.=20
BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle f=
ull table imports in a significantly reduced time. Oh, and I'm not sure tha=
t you couldn't do at least three nines with MT\UBNT. Well, no experience wi=
th the EdgeRouters yet.=20
-----=20
Mike Hammett=20
Intelligent Computing Solutions=20
http://www.ics-il.com=20
Midwest Internet Exchange=20
http://www.midwest-ix.com=20
----- Original Message -----=20
From: "J=C3=BCrgen Jaritsch" <jj@anexia.at>=20
To: "Mike Hammett" <nanog@ics-il.net>=20
Cc: "NANOG" <nanog@nanog.org>=20
Sent: Friday, October 2, 2015 6:11:47 PM=20
Subject: AW: AW: /27 the new /24=20
Hi Mike,=20
sorry, this was probably sent to quick ... let me please explain my POV of =
your statement:=20
I want to concentrate my detailed answer only to the backbone situation whi=
ch is often handled by the 6500/7600 - I guess all of us know that the 6500=
/7600 has a ton of additional features ...=20
6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone cap=
acities built with platforms that only had n*100M interfaces another 3-5 ye=
ars before. Their only invest in these 3-5 years was to add the Gig line ca=
rds, install some software updates and add new fibre optics (GBICs). Chassi=
s, cabling, management interfaces etc could be remain in the cabinet - they=
only had to replace ONE line card (let's say for a few thousand bucks) and=
with this invest they were able to scale up their capacities. Of course: a=
t some point they also had to replace the SUPs, PSUs, FANs, etc. But the in=
vest in the surrounding stuff is nothing compared with completely new machi=
nes.=20
So what all these companies did was buying a machine with an basic configur=
ation and since 10(!) years they are able to expand this machines with (mor=
e or less) small and cheap upgrades.=20
In backbone situations the 6500/7600 are definitely at the end of the resou=
rces the platform can provide. Most of the carriers (and of course also the=
bigger ISPs) had a real chance to evaluate a new model/vendor to ran futur=
e networks (with possibly also a very good scale-up path and scaling- and u=
pgrade-options). Most of the before mentioned are already in an migration p=
rocess (let's take a look at Seabone ... they are migration from Cisco to a=
mix of Juniper and Huawei).=20
Summary: there are strict limitations within the Cisco 6500/7600 platform a=
nd these limitations forces the big players to move this boxes out (or move=
them into other parts of their network). The limitation with 1Mio routes i=
s not a secret and the admins of these boxes decide what they want to use (=
e.g. 768k routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If th=
e global routing table reaches the 768k mark (I guess this will happen in t=
he next 12-18months) most of the boxes will crash again (as it happened in =
Aug 2014).=20
Regarding the words "I have a small router which handles multiple full tabl=
es ...": push and pull a few full tables at the same time and you'll see wh=
at's happening: the CCRs are SLOW. And why? Because the software is not as =
good as it could be: the BGP daemon uses only one core of a 36(?) core CPU.=
Same problem in the past with the EoIP daemon (not sure if they fixed it o=
n the CCRs - they fixed it on x86).=20
Routerboards are nice and cool and to be honest: I'm a big fan of this stuf=
f (also Ubiquiti). But with this boxes you're not able to ran a stable ente=
rprise class carrier network with >99,5% uptime. And that=E2=80=99s thei MA=
IN reason why "the old shit" is still online :).=20
Hopefully my words explained my hard "you know nothing" blabla ?=20
Best regards=20
J=C3=BCrgen Jaritsch=20
Head of Network & Infrastructure=20
ANEXIA Internetdienstleistungs GmbH=20
Telefon: +43-5-0556-300=20
Telefax: +43-5-0556-500=20
E-Mail: JJaritsch@anexia-it.com=20
Web: http://www.anexia-it.com=20
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=C3=9Fe 140, 9020 Klagenfur=
t=20
Gesch=C3=A4ftsf=C3=BChrer: Alexander Windbichler=20
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U632166=
01=20
-----Urspr=C3=BCngliche Nachricht-----=20
Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett=20
Gesendet: Freitag, 02. Oktober 2015 21:33=20
Cc: NANOG <nanog@nanog.org>=20
Betreff: Re: AW: /27 the new /24=20
Hrm.=20
-----=20
Mike Hammett=20
Intelligent Computing Solutions=20
http://www.ics-il.com=20
Midwest Internet Exchange=20
http://www.midwest-ix.com=20
----- Original Message -----=20
From: "J=C3=BCrgen Jaritsch" <jj@anexia.at>=20
To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org>=20
Sent: Friday, October 2, 2015 2:25:10 PM=20
Subject: AW: /27 the new /24=20
> Stop using old shit.=20
Sorry, but the truth is: you have no idea about how earning revenue works a=
nd you obviously also have no idea about carrier grade networks.=20
J=C3=BCrgen Jaritsch=20
Head of Network & Infrastructure=20
ANEXIA Internetdienstleistungs GmbH=20
Telefon: +43-5-0556-300=20
Telefax: +43-5-0556-500=20
E-Mail: JJaritsch@anexia-it.com=20
Web: http://www.anexia-it.com=20
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstra=C3=9Fe 140, 9020 Klagenfur=
t=20
Gesch=C3=A4ftsf=C3=BChrer: Alexander Windbichler=20
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U632166=
01=20
-----Urspr=C3=BCngliche Nachricht-----=20
Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett=20
Gesendet: Freitag, 02. Oktober 2015 20:38=20
An: NANOG <nanog@nanog.org>=20
Betreff: Re: /27 the new /24=20
Chances are the revenue passing scales to some degree as well. Small busine=
ss with small bandwidth needs buys small and has small revenue. Big busines=
s with big bandwidth needs buys big and has big revenue to support big rout=
er.=20
I can think of no reason why ten years goes by and you haven't had a need t=
o throw out the old network for new. If your business hasn't scaled with th=
e times, then you need to get rid of your Cat 6500 and get something more p=
ower, space, heat, etc. efficient.=20
I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco rou=
ters. I don't know what they were at the moment, but they had GBICs, so the=
y weren't exactly new. Each router had two 2500w power supplies. They'll be=
worse in every way (other than *possibly* BGP convergence). The old setup =
consumed at most 300 watts. The new setup requires $500/month in power... a=
nd is worse.=20
Stop using old shit.=20
-----=20
Mike Hammett=20
Intelligent Computing Solutions=20
http://www.ics-il.com=20
Midwest Internet Exchange=20
http://www.midwest-ix.com=20
----- Original Message -----=20
From: "William Herrin" <bill@herrin.us>=20
To: "Mike Hammett" <nanog@ics-il.net>=20
Cc: "NANOG" <nanog@nanog.org>=20
Sent: Friday, October 2, 2015 1:09:16 PM=20
Subject: Re: /27 the new /24=20
On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote:=20
> How many routers out there have this limitation? A $100 router=20
> I bought ten years ago could manage many full tables. If=20
> someone's network can't match that today, should I really have=20
> any pity for them?=20
Hi Mike,=20
The technology doesn't work the way you think it does. Or more=20
precisely, it only works the way you think it does on small (cheap)=20
end-user routers. Those routers do everything in software on a=20
general-purpose CPU using radix tries for the forwarding table (FIB).=20
They don't have to (and can't) handle both high data rates and large=20
routing tables at the same time.=20
For a better understanding how the big iron works, check out=20
https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see=20
folks here talk about TCAM. This stands for Ternary Content=20
Addressable Memory. It's a special circuit, different from DRAM and=20
SRAM, used by most (but not all) big iron routers. The TCAM permits an=20
O(1) route lookup instead of an O(log n) lookup. The architectural=20
differences which balloon from there move the router cost from your=20
$100 router into the hundreds of thousands of dollars.=20
Your BGP advertisement doesn't just have to be carried on your $100=20
router. It also has to be carried on the half-million-dollar routers.=20
That makes it expensive.=20
Though out of date, this paper should help you better understand the=20
systemic cost of a BGP route advertisement:=20
http://bill.herrin.us/network/bgpcost.html=20
Regards,=20
Bill Herrin=20
--=20
William Herrin ................ herrin@dirtside.com bill@herrin.us=20
Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>=20
