[179885] in North American Network Operators' Group
RE: Thousands of hosts on a gigabit LAN, maybe not
daemon@ATHENA.MIT.EDU (c b)
Sun May 10 04:15:10 2015
X-Original-To: nanog@nanog.org
From: c b <bz_siege_01@hotmail.com>
To: John Levine <johnl@iecc.com>, "nanog@nanog.org" <nanog@nanog.org>
Date: Sun, 10 May 2015 01:15:05 -0700
In-Reply-To: <20150510011707.64948.qmail@ary.lan>
Errors-To: nanog-bounces@nanog.org
If you need that kind of density=2C I recommend a Clos fabric. Arista=2C Ju=
niper=2C Brocade=2C Big Switch BCF and Cisco all have solutions that would =
allow you to build a high-density leaf/spine. You can build the Cisco solut=
ion with NXOS or ACI=2C depending which models you choose. The prices on th=
ese solutions are all somewhat in the same ballpark based on list pricing I=
've seen... even Cisco (the Nexus 9k is surprisingly in the same range as b=
randed whitebox). There is also Pluribus which offers a fabric=2C but their=
niche is having server procs on board the switches and it seems like your =
project involves physical rather than virtual servers. Still=2C the Pluribu=
s could be used without taking advantage of the on board server compute I s=
uppose.
I also recommend looking into a solution that supports VXLAN (or GENEVE=2C =
or whatever overlay works for your needs) simply because MAC is carried in =
Layer-3 so you won't have to deal with spanning tree or monstrous mac table=
s. But you don't need to do an overlay if you just segment with traditional=
VLANs.
I'm guessing you don't need HA (A/B uplinks utilizing LACP) for these serve=
rs?
Also=2C do you need line rate forwarding? Having 1=2C000 devices with 1Gb u=
plinks doesn't necessarily mean that full throughput is required... the clu=
stering and the applications may be sporadic and bursty? I have seen load-t=
esting clusters=2C hadoop and data warehousing pushing high volumes but the=
individual NICs in the clusters never actually hit capacity... If you need=
line-rate=2C then you need to do a deep dive with several of the vendors b=
ecause there are significant differences in buffers on some models.
And... what support do you need? Just one spare on the shelf or full vendor=
support on every switch? That will impact which vendor you choose.
I'd like to hear more about this effort once you get it going. Which vendor=
you went with=2C how you tuned it=2C and why you selected who you did. Als=
o=2C how it works.
LFoD
> Date: Sun=2C 10 May 2015 01:17:07 +0000
> From: johnl@iecc.com
> To: nanog@nanog.org
> Subject: Re: Thousands of hosts on a gigabit LAN=2C maybe not
>=20
> In article <CAHf3uWyPQn1NS_umjZ-zNuk3i5uFcZBu9L39b-crovG6yUm2qA@mail.gmai=
l.com> you write:
> >Juniper OCX1100 have 72 ports in 1U.
>=20
> Yeah=2C too bad it costs $32=2C000. Other than that it'd be perfect.
>=20
> R's=2C
> John
=
