[189814] in North American Network Operators' Group
Re: Monitoring system recommendation
daemon@ATHENA.MIT.EDU (Crier, Brent)
Tue Jun 7 08:32:59 2016
X-Original-To: nanog@nanog.org
X-Barracuda-Envelope-From: Brent.Crier@nsight.com
From: "Crier, Brent" <Brent.Crier@nsight.com>
To: Mikael Falkvidd <mikael.falkvidd@op5.com>
Date: Tue, 7 Jun 2016 12:32:50 +0000
In-Reply-To: <CAA=aWqX0Xw68_1kGqo15nZnedtPP1h0QdO8j3HPBo1BONbmxww@mail.gmail.com>
Cc: NANOG <nanog@nanog.org>
Errors-To: nanog-bounces@nanog.org
--Apple-Mail=_E6CFD3B8-0F22-4369-A55F-442E3B85AE5D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
We use Zabbix here pretty heavily. Monitoring roughly 10,000 hosts =
13,000 interfaces and a mirage of services.
-Brent
> On Jun 7, 2016, at 2:42 AM, Mikael Falkvidd <mikael.falkvidd@op5.com> =
wrote:
>=20
>>=20
>> On Monday, June 6, 2016, Manuel Mar=C3=ADn <mmg@transtelco.net> =
wrote:
>>=20
>>> Dear Nanog community
>>>=20
>>> We are currently planning to upgrade our monitoring system (Opsview) =
due
>> to
>>> scalability issues and I was wondering what do you recommend for
>> monitoring
>>> 5000 hosts and 35000 services. We would like to use a monitoring =
system
>>> that is compatible with the nagios plugin format, however we are not =
sure
>>> if systems like Icinga/Shinken/Op5 are the way to go.
>>>=20
>>> Is someone using systems like Op5 or Icinga2 for monitoring > 5000 =
hosts?
>>> Would you recommend commercial systems like Sevone, Zabbix, etc =
instead
>> of
>>> open source ones?
>>=20
>=20
> We (op5) have customers running > 50,000 hosts and > 300,000 services. =
So
> 5,000 hosts is generally not a problem.
>=20
> As mentioned by Jeff, the forking model *can* become a problem. Small
> binaries
> that don't load a lot of libraries fork pretty fast. A test we made =
some
> time ago
> showed a 15 minute load peak at 3.89 (on 24 cores/hyperthreads) when
> checking
> 100,000 services every 5 minutes. Check latencies were 0.8 seconds max =
and
> 0.002 seconds avg. Average cpu load was 15%.
>=20
> Specs for the machine used:
> Dell PowerEdge R620
> 2x Intel Xeon E5-2620
> 24 GB ram
> Dell PERC H710 hardware RAID card
> RAID10 on 4x300GB 15kRPM SAS drives
>=20
> So a single (now almost vintage) server can handle 300 plugin =
executions per
> second without breaking a sweat. Scaling up is definitely a =
possibility, but
> scaling out (using mod gearman, mk or merlin, all open source) is =
available
> as
> well.
>=20
> Complex plugins, for example check_vmware_api which loads the large =
VMware
> perl SDK can get you in trouble though. I suggest you run a test with =
the
> plugin
> mix you are planning to use.
>=20
> If scaling out is not an option, and you want to stay in the =
nagios/naemon
> world,
> a custom worker can be developed to get rid of the loading overhead.
> Documentation is available at
> http://www.naemon.org/documentation/developer/workers.html
>=20
> Full disclosure: I work as development team lead at op5
>=20
> best regards
> Mikael Falkvidd
--Apple-Mail=_E6CFD3B8-0F22-4369-A55F-442E3B85AE5D
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="signature.asc"
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Message signed with OpenPGP using GPGMail
-----BEGIN PGP SIGNATURE-----
iQIcBAEBCAAGBQJXVr7yAAoJEITjTrmUDRb2YuUQAJcqcm2zdy4DBDVKLba8QTCR
eIWLiCGj2p8RKOAJU19Mca//IOV1pYOAL/y3iMUIL69pm8ryc9ITWlnnKAbOj6RA
LqRJ/4VF5FiNmiDtqDMJNXdK5VuYHy5iszQNgY8szLAC6BCK4Oxtmr9tTaXfVv+l
CfCuDyTKP3pBYOTCrZ9eJELprHd3AAoHYjyp1P4YWoSZffotoQYbpYz/S2HDakrc
Xz1ape/Qq0LA6ZPx65sBKRLkHAwFPpl1kdrT0DVUjSquDCg7P9IaLV/Ucy6jqDuW
HCF9558A8zABsQq5kPx3seCyhkLVEAKoaK9TT7TYIdj4YnqsUa0B4SNP9sj7gWfw
1DaMp3zB+gPfpDKLKNRXkjZeTPkdL+bKccPaJrNwTrPYDKkdgzclkIyuHBlR0odP
OaxspL3AEbaFd85RoVgFmXgGb7JH18Fxqzx0nDDpvbncsH18gxhhO/bT1Ofn1htj
ABCjMe1AMDVuqzR11S7Q+8fzFHLJK8FcC3pmIDQsp/qGTdxGNPPd/nO7FYsNMOqV
MVHRpTApb30Tck8D5oJIa0YnlqDNjqU9CJKDqE5WZ2vV9bu9uB41fJJxdvrKIXqN
4NOJswGZGZFTZObLDDj9weebiyN7/pL6b+WZA71B2Lu2XhZXJ+BvT5XOhkMrpYTV
FufljLRrhUeMXHeYGFgZ
=ptBZ
-----END PGP SIGNATURE-----
--Apple-Mail=_E6CFD3B8-0F22-4369-A55F-442E3B85AE5D--
