[149064] in North American Network Operators' Group
Re: 10GE TOR port buffers (was Re: 10G switch recommendaton)
daemon@ATHENA.MIT.EDU (Leo Bicknell)
Fri Jan 27 18:04:39 2012
Date: Fri, 27 Jan 2012 15:03:49 -0800
From: Leo Bicknell <bicknell@ufp.org>
To: bas <kilobit@gmail.com>
Mail-Followup-To: bas <kilobit@gmail.com>, Saku Ytti <saku@ytti.fi>,
nanog@nanog.org
In-Reply-To: <CAEs2Zi+L7COnhDr7qaZ9oguM3jk_+JrbqbAbp3ZEswA70ih5bg@mail.gmail.com>
Cc: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
--G4iJoqBmSsgzjUCe
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In a message written on Fri, Jan 27, 2012 at 11:30:14PM +0100, bas wrote:
> While your reasoning holds truth it does not explain why the expensive
> chassis solution (good) makes my customers happy, and the cheaper TOR
> solution makes my customers unhappy.....
>=20
> Bufferbloat does not matter to them as jitter and latency does not matter.
> As long as the TCP window size negotioation is not reset the total
> amount of bit/sec increases for them.
I obviously don't know your application. The bufferbloat problem
exists for 99.99% of the standard applications in the world. There
are, however, a few corner cases. For instance, if you want to
move a _single_ TCP stream at more than 1Gbps you need deep buffers.
Dropping a single packet slows throughput too much due to a slow-start
event. For most of the world with hundreds or thousands of TCP
streams across a single port, such problems never occur.
> If deep buffers are bad I would expect high-end chassis solutions not
> to offer them either.
> But the market seems to offer expensive deep buffer chassis solutions
> and cheap (per 10GE) TOR solutions.
The margin on a top-of-rack switch is very low. 48 port gige with
10GE uplinks are basically commodity boxes, with plenty of competition.
Saving $100 on the bill of materials by cutting out some buffer
makes the box more competitive when it's at a $2k price point.
In contrast, large, modular chasses have a much higher margin. They are
designed with great flexability, to take things like firewall modules
and SSL accelerator cards. There are configs where you want some (not
much) buffer due to these active appliances in the chassis, plus it is
easier to hide an extra $100 of RAM in a $100k box.
Also, as was pointed out to me privately, it is also important to loook
at adaptive queue management features. The most famous is WRED, but
there are other choices. Having a queue management solution on your
routers and switches that works in concert with the congestion control
mechanism used by the end stations always results in better goodput.
Many of the low end switches have limited or no AQM choices, while the
higher end switches with fancier ASICs can default to something like
WRED. Be sure it is the deeper buffers that are making the difference,
and not simply some queue management.
--=20
Leo Bicknell - bicknell@ufp.org - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
--G4iJoqBmSsgzjUCe
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (FreeBSD)
iQIVAwUBTyMtVbN3O8aJIdTMAQJtzBAAhxAUhwLOq9coU/nuWk2VV+m03W0mYWyM
9fme7fghwNDioXx1wwNen1e3Obsz3Sy4qIRffXqO4v0sUOxMi0PwERBq3TR/z3ws
mXu5gY0CzF+K52Ovd6q4P9S9yqCoW05GhlHlhq/Gj8CncZ8rzP4u9BT8o4cmP6lV
weMogwcmXpuiyWy+bhY3iR9DfZ8RCKmHKPi7r+K6PpNpj+MxeKuqwKv5FB13iQVJ
nrkMkoLaJEp71D7aM3brF4aFqp6BBVzh0NWPYU/oFGJ7nq0m97QQu0csNwaGQMqV
8KadPaafn/wOltNa9U/2epkjTkH+9EHqaOzMES/8a+2pwwEixZ6lVLdsXV4gUBHh
lqDIjTXA2XhwbCW40KXCo/0muOp8wZObc4ULVLHi1UeUI4uApjmd5bH6fpo5RNFO
4+dOkKpHQz8VZX7KEViIHnSf1V6LdTCc2BBsk37xyDnSUrfXdNJG9boTCaCwDNPO
gic2FTV9oWmwSXH4rc1ISQ9HjmpchfMFns1lnng/w/iK2K5Q3Kel00bd7SopLxNy
xZuCyaXUTM59FE0F781ojlFbdmSoUeopDGtE54GWi/MpygXn/KltUtobFu0uRRls
8OneGYxH5ul1HkYquqVgZLxsZnXvS8Qv6VsP0zuuhsuam7Hm0hl7aausVbTHeDTv
8KAta9F5WoI=
=KmfB
-----END PGP SIGNATURE-----
--G4iJoqBmSsgzjUCe--
