[195806] in North American Network Operators' Group
Re: 100G QSFP28 DAC cables - experience
daemon@ATHENA.MIT.EDU (Hugo Slabbert)
Sat Sep 16 04:43:27 2017
X-Original-To: nanog@nanog.org
Date: Thu, 14 Sep 2017 09:54:31 -0700
From: Hugo Slabbert <hugo@slabnet.com>
To: Jiri Prochazka <jiri@cdn77.com>
In-Reply-To: <c8680f47-f195-7c01-b75b-2a4ff4505e63@cdn77.com>
Cc: nanog@nanog.org
Errors-To: nanog-bounces@nanog.org
--gKMricLos+KVdGMg
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Wed 2017-Sep-06 09:17:39 +0200, Jiri Prochazka <jiri@cdn77.com> wrote:
>Hi folks,
>
>I'm wondering if anyone have (either positive or negative) experience=20
>with 100G QSFP28 DAC cables?
>
>Is there anyone who is using 100G DAC in large scale and would=20
>recommend it (which means there are no issues compared to SR4 links)?
>
>I'm thinking about cables with lenght up to 1m, not more.
>
>We have had quite bad experience with 10G DAC in the past - but I do=20
>not want to be slave of the past.
>
>Thank you for your thoughts!
>
>Jiri
We're deploying a decent chunk of 100G QSFP28 at the moment, but it's a mix=
=20
of:
- a handful of 100G QSFP28 copper DACs for some switch peerlinks
- a bit >100x 100G QSFP28 AOC for interswitch links
- a lot more 100G QSFP28 -> 4x25G SFP28 copper breakouts
We're only a few weeks in at this point, so mileage may vary in the long=20
run etc.
The copper peerlinks are mostly 1M with some 3M. We've had no issues with=
=20
them so far.
The AOC interswitch links vary more in length, but some of those are >10M=
=20
(hence AOC rather than copper). We've faced no issues with those. =20
Granted, there is BGP with BFD running across those, so those should help=
=20
in terms of liveness checks and such.
I mention that because where we _have_ had issues are on the 100G -> 4x25G=
=20
copper breakouts. Those are for 25G edge connectivity. It's a decent=20
sample size with a bit north of 600x 25G ports. The trouble we've had=20
there have been with some links showing link up on the switch and server=20
side but actually failing to pass any traffic, so we need to stuff some >L1=
=20
liveness checks on there to ensure those links are good while we sort out=
=20
the root issue. It is not yet clear if this is a cable fault, driver=20
issue, or something firmware-ish on the NICs.
Also, fun fact: 25G only made its way into the 802.3ad bonding mode driver=
=20
in the Linux kernel in March this year[1].
--=20
Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com
pgp key: B178313E | also on Signal
[1]https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi=
t/?id=3D19ddde1eeca1ee81f4add5e04da66055e09281ac
--gKMricLos+KVdGMg
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBCgAGBQJZurRHAAoJEFsnhBAb2KmAf0YP/1+tFyKY+tEsyKx6PgOtHeM1
a7ZDV15/qsL0U3kyIMlu5INS8dBM5M3GadGMmCLeU0F9Lz6BATTy1f0htkULf9AM
qJULoqu3dpBx+ZCDh/k2SbBt80Wn6g1dJnsEMaj54AAtVqdzmyePQplGodaxfizp
NOtffDDBxSrrVOME3ZthnZc2ZuL2y0NkxOZtgvvBafaoqFnsld3oOgJ72Khle6bv
GSYrnAZRVMj2MGUpN1MbEejlG0fgGRh3gx0Zs28n1Bw2lsdmG+jveAOp+I5fZMm5
0RUmO8DXh0FOr5sxBGFykQhDtmZSdxNDVItLfprwlcpdnhokMlpdGdLD/8Aiq06W
udTMPWDUjcRzG9L3h22EaEwLrgbccM9XiL2YFSJ0e7jMI9HmckAEw6nzoqfIpoVC
AS0btWQBrBkL4LwxjGoU0Kq4qI6hu+5rmuxTE6jJ7bqj3RMv1Ey0Qqq/tL31/uon
2l4fVbkn7o8KwhltxNrDrS/bS6gzfKju/aoTd3aMBmqCBsa5+3HgGnkJTSi7OYfS
vU1c6cYRYEHCRJHsoItOq5redc5EisVQkzO9kIx1/AWV3xhnjZu+U8425DfI0OXE
WAlEgJFrwWk7CcoCKdj+3sCITXHLJgeze3E5vKZKxd+oliFKHWVS45J/xDTKnk6M
6srJC/bzbvkrrtR12wqu
=u4c6
-----END PGP SIGNATURE-----
--gKMricLos+KVdGMg--
