[88639] in North American Network Operators' Group
NANOG36-NOTES 2006.02.14 talk 2 Netflow Visualization Tools
daemon@ATHENA.MIT.EDU (Matthew Petach)
Tue Feb 14 15:01:32 2006
Date: Tue, 14 Feb 2006 12:00:47 -0800
From: Matthew Petach <mpetach@netflight.com>
To: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: owner-nanog@merit.edu
------=_Part_394_21290216.1139947247681
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
2006.02.14 talk 2 Netflow tools
Bill Yurcik
byurcik at ncsa.uiuc.edu
NVisionIP and VisFlowConnect-IP
probably a dozen tools out there, this is just
two of them. Concenses is there's something to
this.
They're an edge network, comes into ISP domain,
their tools are used by entities with many
subnet blocks.
Overview
Project Motifivation
Netflows for Security
Two visualization tools
NVisionIP
VisFlowConnect-IP
Summary
Internet Security:
N-Dimensional Work Space
large--already lots of data to process
complex--combinatorics explode quickly
time dynamics--things can change quickly!
Visualizations can help!
in near-realtime
overview-browse-details on demand
People are wired to do near-realtime processing
of visual information, so that's a good way to
present information for humans.
HCI says use overview-browse-details paradigm.
Netflows for security
can identify connection-oriented stats to see
things like attacks, DoS, DDoS, etc.
Most people don't use the data portion of the
flow field, the first 64 bytes, they just look
at header info or aggregated flow records.
Can spot how many users are on your system at
a given time, to schedule upgrades.
Who are your top talkers?
How long do my users surf? What are people using
the network for?
Where do users go? Where did they come from?
Are users following the security policy?
What are the top N destination ports?
Is there traffic to vulnerable hosts?
Can you identify and block scanners/bad guys?
This doesn't replace other systems like syslog, etc.;
it integrates and works alongside them.
architecture slide for NCSA.
Can't really do sampled view for security, so probably
need distributed flow collector farm to get all the
raw data safely.
Two visualization tools:
NVisionIP, VisFlowConnect-IP
focus on quick overview of tools
security.ncsa.uiuc.edu/
3 level hierarchical tool;
galaxy view (small multiple view) ((machine view))
Galaxy is overview of the whole network.
color and shape of dots is each host in a network.
settable parameters for each dot.
Animated toolbar and clock show changes over time
in the galaxy.
Lets you get high-level content quickly and easily.
Domain view lets you drill in a bit more; small
multiple view looks at the traffic within the
block.
upper histogram is lower, well known ports; lower
histogram is ports over 1024
You can click on a given multiple view entry to
delve into one machine.
Many graphs for each machine in the most detailed
view.
well known ports first, then rest of ports (sorted)
then source and destination traffic broken out.
Designed for class Bs.
http://security.ncsa.uiuc.edu/distribution/VisFlowConnectDownload.html
3 vertical lines, comes from edge network perspective;
middle line is edge network to manage. You set range
of networks you care about. Outside lines are people
sourcing or sinking traffic to you, from outside
domains.
There's a time axis, traffic only shown for the slice
of time currently under consideration.
Uses VCR-like controls to move time forward/backward
Lets you see traffic/interactivity, drill into that
domain, see host level connectivity flows.
Shows MS Blaster virus traffic as an example.
Example 2, a scan example. Just because it looks
like one IP hitting many others doesn't mean it's
really a security incident, though; could be a
cluster getting traffic.
web crawlers hitting NCSA web servers make for
a very charateristic pattern over time.
Summary
Netflows analysis is non-trivial,
NVisionIP
VisFlowConnect-IP
lots of references listed in very fine blue font.
http://security.ncsa.uiuc.edu/distribution/NVisionIPDownload
Avi Freedman, Akamai, Argus was mentioned a lot; it
lets you grab symmetric netflows, but also does TCP
analysis, shows some performance data as well. not
sure if people are studying the impact of correlating
argus data with flow data.
Roland Douta? of Cisco; many people are using netflow
to track security issues. They now have ingress and
egress flow data on many of their platforms.
In reading paper describing it, there's data conversion
that needs to happen into an internal format that
nVision can understand. It reads log files at the
moment, takes about 5 minutes to process files. Lets
them take different file data sources, make the tool
for visualization independent of the input format.
They can read large files, but there is a performance
hit when doing it.
Are they planning on doing further work on the tool
to collect TCP flags, for frags, drop traffic, etc?
They've looked at it, but they leave it to IDS tools
for flag activity. Might be of interest to consider
for future versions of the tools.
Last question came up, echoed about argus.
Question about interactivity, they are working on
feedback through tools. Question about alarming
on patterns; but once you start alarming or putting
up visual indicators, it distracts from rest of
the overall pattern, you tend to miss other information.
------=_Part_394_21290216.1139947247681
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
<br>
2006.02.14 talk 2 Netflow tools<br>
<br>
Bill Yurcik<br>
byurcik at <a href=3D"http://ncsa.uiuc.edu">ncsa.uiuc.edu</a><br>
<br>
NVisionIP and VisFlowConnect-IP<br>
<br>
probably a dozen tools out there, this is just<br>
two of them. Concenses is there's something to<br>
this.<br>
<br>
They're an edge network, comes into ISP domain,<br>
their tools are used by entities with many<br>
subnet blocks.<br>
<br>
Overview<br>
Project Motifivation<br>
Netflows for Security<br>
Two visualization tools<br>
NVisionIP<br>
VisFlowConnect-IP<br>
Summary<br>
<br>
Internet Security:<br>
N-Dimensional Work Space<br>
<br>
large--already lots of data to process<br>
complex--combinatorics explode quickly<br>
time dynamics--things can change quickly!<br>
Visualizations can help!<br>
in near-realtime<br>
overview-browse-details on demand<br>
<br>
People are wired to do near-realtime processing<br>
of visual information, so that's a good way to<br>
present information for humans.<br>
HCI says use overview-browse-details paradigm.<br>
<br>
Netflows for security<br>
can identify connection-oriented stats to see<br>
things like attacks, DoS, DDoS, etc.<br>
Most people don't use the data portion of the<br>
flow field, the first 64 bytes, they just look<br>
at header info or aggregated flow records.<br>
<br>
Can spot how many users are on your system at<br>
a given time, to schedule upgrades.<br>
<br>
Who are your top talkers?<br>
<br>
How long do my users surf? What are people using<br>
the network for?<br>
<br>
Where do users go? Where did they come from?<br>
<br>
Are users following the security policy?<br>
<br>
What are the top N destination ports?<br>
Is there traffic to vulnerable hosts?<br>
<br>
Can you identify and block scanners/bad guys?<br>
<br>
This doesn't replace other systems like syslog, etc.;<br>
it integrates and works alongside them.<br>
<br>
architecture slide for NCSA.<br>
<br>
Can't really do sampled view for security, so probably<br>
need distributed flow collector farm to get all the<br>
raw data safely.<br>
<br>
Two visualization tools:<br>
NVisionIP, VisFlowConnect-IP<br>
<br>
focus on quick overview of tools<br>
<a href=3D"http://security.ncsa.uiuc.edu/">security.ncsa.uiuc.edu/</a><br>
<br>
3 level hierarchical tool;<br>
galaxy view (small multiple view) ((machine view))<br>
<br>
Galaxy is overview of the whole network.<br>
color and shape of dots is each host in a network.<br>
settable parameters for each dot.<br>
<br>
Animated toolbar and clock show changes over time<br>
in the galaxy.<br>
Lets you get high-level content quickly and easily.<br>
<br>
Domain view lets you drill in a bit more; small<br>
multiple view looks at the traffic within the<br>
block.<br>
upper histogram is lower, well known ports; lower<br>
histogram is ports over 1024<br>
<br>
You can click on a given multiple view entry to<br>
delve into one machine.<br>
Many graphs for each machine in the most detailed<br>
view.<br>
<br>
well known ports first, then rest of ports (sorted)<br>
then source and destination traffic broken out.<br>
<br>
Designed for class Bs.<br>
<br>
<a href=3D"http://security.ncsa.uiuc.edu/distribution/VisFlowConnectDownloa=
d.html">http://security.ncsa.uiuc.edu/distribution/VisFlowConnectDownload.h=
tml</a><br>
<br>
3 vertical lines, comes from edge network perspective; <br>
middle line is edge network to manage. You set range<br>
of networks you care about. Outside lines are people<br>
sourcing or sinking traffic to you, from outside<br>
domains.<br>
<br>
There's a time axis, traffic only shown for the slice<br>
of time currently under consideration.<br>
Uses VCR-like controls to move time forward/backward<br>
<br>
Lets you see traffic/interactivity, drill into that<br>
domain, see host level connectivity flows.<br>
<br>
Shows MS Blaster virus traffic as an example.<br>
<br>
Example 2, a scan example. Just because it looks<br>
like one IP hitting many others doesn't mean it's<br>
really a security incident, though; could be a<br>
cluster getting traffic.<br>
<br>
web crawlers hitting NCSA web servers make for<br>
a very charateristic pattern over time.<br>
<br>
Summary<br>
Netflows analysis is non-trivial, <br>
<br>
NVisionIP<br>
VisFlowConnect-IP<br>
<br>
lots of references listed in very fine blue font.<br>
<br>
<a href=3D"http://security.ncsa.uiuc.edu/distribution/NVisionIPDownload">ht=
tp://security.ncsa.uiuc.edu/distribution/NVisionIPDownload</a><br>
<br>
Avi Freedman, Akamai, Argus was mentioned a lot; it<br>
lets you grab symmetric netflows, but also does TCP<br>
analysis, shows some performance data as well. not<br>
sure if people are studying the impact of correlating<br>
argus data with flow data.<br>
<br>
Roland Douta? of Cisco; many people are using netflow<br>
to track security issues. They now have ingress and<br>
egress flow data on many of their platforms.<br>
In reading paper describing it, there's data conversion<br>
that needs to happen into an internal format that<br>
nVision can understand. It reads log files at the<br>
moment, takes about 5 minutes to process files. Lets<br>
them take different file data sources, make the tool<br>
for visualization independent of the input format.<br>
They can read large files, but there is a performance<br>
hit when doing it.<br>
Are they planning on doing further work on the tool<br>
to collect TCP flags, for frags, drop traffic, etc?<br>
They've looked at it, but they leave it to IDS tools<br>
for flag activity. Might be of interest to consider<br>
for future versions of the tools.<br>
<br>
Last question came up, echoed about argus.<br>
Question about interactivity, they are working on<br>
feedback through tools. Question about alarming<br>
on patterns; but once you start alarming or putting<br>
up visual indicators, it distracts from rest of<br>
the overall pattern, you tend to miss other information.<br>
<br>
<br>
<br>
<br>
------=_Part_394_21290216.1139947247681--
