[184443] in North American Network Operators' Group
Re: Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24)
daemon@ATHENA.MIT.EDU (Mike Hammett)
Sat Oct 3 09:29:52 2015
X-Original-To: nanog@nanog.org
Date: Sat, 3 Oct 2015 08:29:44 -0500 (CDT)
From: Mike Hammett <nanog@ics-il.net>
Cc: nanog@nanog.org
In-Reply-To: <20151003.092349.2236946013734960508.wwaites@tardis.ed.ac.uk>
Errors-To: nanog-bounces@nanog.org
Sure MT has issues, but so does everyone. As someone that has used them for=
10+ years, the past six months has seen a bit of a re-awakening over there=
. You can see this in the time to completion of many feature requests, bug =
fixes, new features, etc. I'm not sure they're going to do everything every=
one is after, but they certainly have shown a huge increase in willingness =
to go the right direction.=20
Of course it's easy for someone running big iron to scoff at the lack of fe=
ature X or feature Y. To that I say, what are the capabilities of your $200=
router? Your $2k router? I haven't priced out new low-end gear from the bi=
g iron vendors, but I can't imagine at what price point you need to be at t=
o have a multi-gig capable VPLS router. For Mikrotik you're in the $200 - $=
1k range, depending on what you mean by "multi-gig". One thing I miss as I =
start to use more non-Mikrotik hardware... Torch. I wish everything had Tor=
ch. Put Packet Sniffer in the list of things I'd like to see everywhere. I =
don't want port mirror as who's to say I have something to mirror to everyw=
here that can also capture? Put a few basic filters and drop the PCAP right=
on the damn box. Now obviously with something running BSD you could code u=
p whatever you'd like or have an array of open-source packages to work with=
, but that wouldn't have the nice feature integration of a router OS.=20
I have no problem running Mikrotik in the DFZ. Mine pull down full tables i=
n 30 - 35 seconds, can handle somewhere in the 30 - 60 gb range when firewa=
ll rules are applied and so on. They'd cost under $1,500 new, but I got min=
e put together for a fraction of that. They're so cheap you can run two. Ru=
n two and now you have the advantage of being able to do maintenance withou=
t downtime. It's a little kludgey, but can get get the job done at a price =
point the others can't. Maybe with newer CCRs and ROS7 I could drop the nee=
d for the x86 boxes. We'll see.=20
-----=20
Mike Hammett=20
Intelligent Computing Solutions=20
http://www.ics-il.com=20
Midwest Internet Exchange=20
http://www.midwest-ix.com=20
----- Original Message -----
From: "William Waites" <wwaites@tardis.ed.ac.uk>=20
To: jj@anexia.at=20
Cc: nanog@ics-il.net, nanog@nanog.org=20
Sent: Saturday, October 3, 2015 3:23:49 AM=20
Subject: Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24)=20
On Fri, 2 Oct 2015 23:11:47 +0000, J=C3=BCrgen Jaritsch <jj@anexia.at> said=
:=20
> Regarding the words "I have a small router which handles=20
> multiple full tables ...": push and pull a few full tables at=20
> the same time and you'll see what's happening: the CCRs are=20
> SLOW. And why? Because the software is not as good as it could=20
> be: the BGP daemon uses only one core of a 36(?) core CPU.=20
To expand on this, the problem is worse than being single-threaded. I=20
had one of these in the lab and fed it 2x full tables. Sure it wasn't=20
the fastest at accepting them but then I noticed that even in steady=20
state one of the CPUs was pegged. What was happening -- and this was=20
confirmed by Mikrotik -- was that it was recalculating the *entire*=20
FIB for each update. The general background noise of announce /=20
withdraw messages means it is doing this all the time. Any churn and=20
it would have a very hard time.=20
There are other serious bugs such as not doing recursive next hop=20
lookup for IPv6 (it does for IPv4). This makes them unuseable as BGP=20
routers even for partial tables with most non-trivial iBGP=20
topologies. All of which may be fixed one day in version 7 of their=20
operating system, which will inevitably have many bugs as any software=20
project .0 release will, so we'll have to wait for 7.x for it to be=20
reasonably safe to use.=20
That said, we use a lot of Mikrotik kit for our rural=20
networks. They're weird and quirky but you can't beat them on price,=20
port density and power consumption. With 16 ports and 36 cores surely=20
they should be capable of pushing several Gbps of traffic with a few=20
full tables.=20
I wish it were possible today to run different software on their=20
larger boxes. If some like-minded small providers wanted to get=20
together with us to fund a FreeBSD port to the CCR routers that would=20
be great. Please contact me off-list if you are interested in this,=20
I'll coordinate.=20
As it is we don't let them anywhere near the DFZ, that's done with PCs=20
running FreeBSD and BIRD which can easily do the job but is still an=20
order of magnitude more expensive (and an order of magnitude less=20
expensive than what you need if you want 10s of Gbps).=20
-w=20
--=20
William Waites <wwaites@tardis.ed.ac.uk> | School of Informatics=20
http://tardis.ed.ac.uk/~wwaites/ | University of Edinburgh=20
https://hubs.net.uk/ | HUBS AS60241=20
The University of Edinburgh is a charitable body, registered in=20
Scotland, with registration number SC005336.=20
