[152147] in North American Network Operators' Group
Re: Network Storage
daemon@ATHENA.MIT.EDU (Leo Bicknell)
Sun Apr 15 09:39:22 2012
Date: Sun, 15 Apr 2012 06:38:28 -0700
From: Leo Bicknell <bicknell@ufp.org>
To: nanog@nanog.org
Mail-Followup-To: nanog@nanog.org
In-Reply-To: <CA+vWMo5NB4B1Vg5Pb-yUXndsYdP8zbSoYsu=K1gSkvu6xqGvmA@mail.gmail.com>
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
--ZGiS0Q5IWpPtfppv
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In a message written on Thu, Apr 12, 2012 at 05:16:27PM -0400, Maverick wro=
te:
> 1) My goal is to store the traffic may be fore ever, and analyze it in
> the future for security related incidents detected by ids/ips.
Let's just assume you have enough disk space that you can write out
every packet, or even just packet header. That's a hard problem,
but you've received plenty of suggestions on how to go down that
path.
Once you have that data, how are you going to process it?
Yes, disk reads are faster than disk writes, but not by that much.
If it takes you 24 hours to write a day of data to disk, it might
take you 12 hours just to read it all back off and process it.
Processing a weeks worth of back data could take days. I'm also
not even starting to count the CPU and memory necessary to build
state tables and statistical analysis tables to generate useful
data.
There's a reason why most network traffic tools summarize early,
as early as on the network device when using Netflow type collection.
It's not just to save storage space on disk, but it's to make the
processing of the data fast enough that it can be done in a short
enough time that the data is still relevant when the processing is
complete.
--=20
Leo Bicknell - bicknell@ufp.org - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/
--ZGiS0Q5IWpPtfppv
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (FreeBSD)
iQIVAwUBT4rPVLN3O8aJIdTMAQLLcxAApckEC3UU3r9IZ1gxTbjCorZUId9Cz7N5
5awQXSojr36ttR1dBLvYHomzyyfn23m7dNz7h8VCUZFg/jqa/UDdyixvth6mAg9N
3iPzHcPP4u21JG9B++UxoGw9FFv1ugvenekFaTa3hZIsZhlECd45gnxM1uuQldL0
1dDV4vuP37r7aQ33TjwIIMOditE993tLl/9VlZsX9ngIhV6F6qF/2VmqQyyZ3Sls
chfIGhwZmahD+krCHnpond1L0sx6SZpRXIXSWyU9wExMRGo5UNJsSxJZUtJgZ6gC
yTJdWqa13ygC9+nFz3xLL9XY7h9H9ccm5MfTcqNjuvXaQOMMUwVZZ2iJzWKfZe2r
Fe9i8J/1lGI9m3yFJr8Ik0Zsx6FmUbAOQGvZJyPC9hw33w2KgW3U+Yc2CxgAjmgq
5T7SU7s2c9wQywHRR5Bt0hoeVzAszjOKgPkf4+koL8bvhvHw9FtmBAAIagdEOW/B
5Ym4M9PnWgnNvei++iC1fTIgdYPqvBwUk5ApDCy0bUFyUJlxsJa5TPSpr+M3VIas
XdtrmgbYn8ymwSV6C4k5V6bk5M+GBq4pDPtVI0O9QwJ9triLGrxUAvjToPv6rI7d
zaMV2YS6immeMsgpbu2AO6sUS8nS8YqfBVINlU5D2a7GJWGiqOdX+oYt7vSu3uBf
CrJ66BWf8iY=
=z4yp
-----END PGP SIGNATURE-----
--ZGiS0Q5IWpPtfppv--
