[194668] in North American Network Operators' Group
Re: vFlow :: IPFIX, sFlow and Netflow collector
daemon@ATHENA.MIT.EDU (i mawsog via NANOG)
Wed May 17 12:44:57 2017
X-Original-To: nanog@nanog.org
Date: Wed, 17 May 2017 15:48:26 +0000 (UTC)
To: Mehrdad Arshad Rad <arshad.rad@gmail.com>,
Vitaly Nikolaev <nvitaly@gmail.com>
In-Reply-To: <CAPefw3cxDqdudCf0f-RC4KfuW5GUp5thJZhsFwShAaKm+d_fPg@mail.gmail.com>
From: i mawsog via NANOG <nanog@nanog.org>
Reply-To: i mawsog <imawsog@yahoo.com>
Cc: "nanog@nanog.org" <nanog@nanog.org>
Errors-To: nanog-bounces@nanog.org
A few questions and =C2=A0comments.=C2=A0
1. Is there any =C2=A0good =C2=A0open repository of netwflow data ? =C2=A0 =
=C2=A0=C2=A0
2. How about open repository of raw packet capture ?=C2=A03. There are many=
companies that help collect =C2=A0raw packet - =C2=A0Gigamon, BigSwitch, .=
.. . =C2=A0Do folks on this list have any experiences =C2=A0with these vend=
ors ?=C2=A03. xFLows are =C2=A0apparently the only =C2=A0 detailed metric c=
ollected on a wider scale. =C2=A0I heard even that is often considered a nu=
isance for the value it =C2=A0provides . =C2=A0What are the experiences of =
the the folks on this list =C2=A0? =C2=A0 Where and how netflow is usually =
collected ?=C2=A0
SG
From: Mehrdad Arshad Rad <arshad.rad@gmail.com>
To: Vitaly Nikolaev <nvitaly@gmail.com>=20
Cc: nanog@nanog.org
Sent: Wednesday, May 17, 2017 7:01 AM
Subject: Re: vFlow :: IPFIX, sFlow and Netflow collector
=20
I tried w/ standalone MemSQL w/ 100K IPFIX samples per second and it works.
if you pay MemSQL license you can have more than one node (cluster).
another solution is ClickHouse https://clickhouse.yandex/ but I'm gonna to
test it soon :-)
The MemSQL's nice feature is it has built in Kafka consumer w/ transform
feature.
On Tue, May 16, 2017 at 8:04 AM, Vitaly Nikolaev <nvitaly@gmail.com> wrote:
> Hello,
>
> Interesting, what receives and where do you keep flows at the other end o=
f
> messaging bus ?
>
>
> PS: in my case I am talking about hundreds of kilo flows/s that I would
> like to keep for at least few weeks, so MemSQL or any other SQLs are out =
of
> the picture.
>
> Thank you
>
>
> On Mon, May 15, 2017 at 2:31 PM, Mehrdad Arshad Rad <arshad.rad@gmail.com=
>
> wrote:
>
>> Hi all,
>>
>> I just wanted to share the vFlow - IPFIX, sFlow and Netflow collector,
>> it's
>> scalable and reliable, written by pure Golang!
>> It doesn't have any library dependency and works w/ Kafka and NSQ (you c=
an
>> write your own MQ plugin).
>>
>> https://github.com/VerizonDigital/vflow
>>
>> For more information
>> https://www.linkedin.com/pulse/high-performance-scalable-
>> reliable-ipfix-sflow-open-arshad-rad
>>
>> It can be able to integrate w/ MemSQL easy and you can have kind of belo=
w
>> SQL query:
>>
>> memsql> select * from samples order by bytes desc limit 20;
>> +----------------+-----------------+-----------------+------
>> --+--------+-------+---------+---------+----------+--------+
>> ---------------------+
>> | device=C2=A0 =C2=A0 =C2=A0 =C2=A0 | src=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 | dst=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | srcASN | dstASN
>> | proto | srcPort | dstPort | tcpFlags | bytes=C2=A0 | datetime
>> |
>> +----------------+-----------------+-----------------+------
>> --+--------+-------+---------+---------+----------+--------+
>> ---------------------+
>> | 192.129.230.0=C2=A0 | 87.11.81.121=C2=A0 =C2=A0 | 61.231.215.18=C2=A0 =
| 131780 |=C2=A0 21773
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 64670 | 0x10=C2=A0 =C2=
=A0 | 342000 | 2017-04-27 22:05:55
>> |
>> | 52.20.79.116=C2=A0 | 87.11.81.100=C2=A0 =C2=A0 | 216.38.140.154=C2=A0 =
|=C2=A0 41171 |=C2=A0 7994
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 443 |=C2=A0 26798 | 0x18=C2=A0 =C2=A0 | =
283364 | 2017-04-27 22:06:00
>> |
>> | 52.20.79.116=C2=A0 | 192.229.211.70=C2=A0 | 50.240.197.150=C2=A0 |=C2=
=A0 41171 |=C2=A0 33651
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 23397 | 0x10=C2=A0 =C2=
=A0 | 216000 | 2017-04-27 22:05:55
>> |
>> | 108.161.249.16 | 152.125.33.113=C2=A0 | 74.121.78.10=C2=A0 =C2=A0 |=C2=
=A0 13768 |=C2=A0 9551
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 49217 | 0x18=C2=A0 =C2=
=A0 | 196500 | 2017-04-27 22:05:59
>> |
>> | 192.229.130.0=C2=A0 | 87.21.81.254=C2=A0 =C2=A0 | 94.56.54.135=C2=A0 =
=C2=A0 | 132780 |=C2=A0 21773
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 52853 | 0x18=C2=A0 =C2=
=A0 | 165000 | 2017-04-27 22:05:55
>> |
>> | 108.161.229.96 | 93.184.215.169=C2=A0 | 152.157.32.200=C2=A0 |=C2=A0 1=
2768 |=C2=A0 11430
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 443 |=C2=A0 50488 | 0x18=C2=A0 =C2=A0 |=
=C2=A0 86400 | 2017-04-27 22:06:01
>> |
>> | 52.22.49.106=C2=A0 | 122.229.210.189 | 99.31.208.183=C2=A0 |=C2=A0 221=
71 |=C2=A0 8018
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 443 |=C2=A0 33059 | 0x18=C2=A0 =C2=A0 |=
=C2=A0 73500 | 2017-04-27 22:05:55
>> |
>> | 52.22.49.126=C2=A0 | 81.21.81.131=C2=A0 =C2=A0 | 66.215.169.120=C2=A0 =
|=C2=A0 22171 |=C2=A0 20115
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 57468 | 0x10=C2=A0 =C2=
=A0 |=C2=A0 66000 | 2017-04-27 22:05:59
>> |
>> | 108.160.149.96 | 94.184.215.151=C2=A0 | 123.90.233.120=C2=A0 |=C2=A0 1=
6768 |=C2=A0 14476
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 63905 | 0x18=C2=A0 =C2=
=A0 |=C2=A0 65540 | 2017-04-27 22:05:57
>> |
>> | 52.22.79.116=C2=A0 | 162.129.210.181 | 60.180.253.156=C2=A0 |=C2=A0 21=
271 |=C2=A0 31651
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 443 |=C2=A0 59652 | 0x18=C2=A0 =C2=A0 |=
=C2=A0 64805 | 2017-04-27 22:06:00
>> |
>> | 108.161.149.90 | 93.184.215.169=C2=A0 | 80.96.58.146=C2=A0 =C2=A0 |=C2=
=A0 13868 |=C2=A0 22394
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 443 |=C2=A0 =C2=A0 1151 | 0x18=C2=A0 =C2=
=A0 |=C2=A0 59976 | 2017-04-27 22:05:54
>> |
>> | 102.232.179.20 | 111.18.232.131=C2=A0 | 121.62.44.149=C2=A0 |=C2=A0 24=
658 |=C2=A0 4771
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 61076 | 0x10=C2=A0 =C2=
=A0 |=C2=A0 59532 | 2017-04-27 22:05:54
>> |
>> | 102.232.179.20 | 192.129.145.6=C2=A0 | 110.49.221.232=C2=A0 |=C2=A0 24=
658 |=C2=A0 4804
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 443 |=C2=A0 50002 | 0x10=C2=A0 =C2=A0 |=
=C2=A0 58500 | 2017-04-27 22:05:55
>> |
>> | 102.232.179.20 | 192.129.232.112 | 124.132.217.101 |=C2=A0 24658 |=C2=
=A0 43124
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 443 |=C2=A0 37686 | 0x10=C2=A0 =C2=A0 |=
=C2=A0 57000 | 2017-04-27 22:06:00
>> |
>> | 192.229.230.0=C2=A0 | 87.11.81.253=C2=A0 =C2=A0 | 219.147.144.22=C2=A0=
| 132380 |=C2=A0 2900
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 25202 | 0x18=C2=A0 =C2=
=A0 |=C2=A0 56120 | 2017-04-27 22:05:58
>> |
>> | 192.129.130.0=C2=A0 | 87.21.11.200=C2=A0 =C2=A0 | 180.239.187.151 | 13=
2380 |=C2=A0 8151
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 443 |=C2=A0 55062 | 0x18=C2=A0 =C2=A0 |=
=C2=A0 52220 | 2017-04-27 22:05:59
>> |
>> | 52.12.79.126=C2=A0 | 87.21.11.254=C2=A0 =C2=A0 | 64.30.125.221=C2=A0 |=
=C2=A0 21071 |=C2=A0 14051
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 57072 | 0x10=C2=A0 =C2=
=A0 |=C2=A0 51000 | 2017-04-27 22:05:54
>> |
>> | 192.229.110.1=C2=A0 | 150.195.33.40=C2=A0 | 98.171.170.51=C2=A0 | 1329=
80 |=C2=A0 28773
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 53270 | 0x18=C2=A0 =C2=
=A0 |=C2=A0 51000 | 2017-04-27 22:05:57
>> |
>> | 192.229.110.1=C2=A0 | 87.21.81.254=C2=A0 =C2=A0 | 68.96.162.21=C2=A0 =
=C2=A0 | 132980 |=C2=A0 28773
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 46727 | 0x18=C2=A0 =C2=
=A0 |=C2=A0 49500 | 2017-04-27 22:06:01
>> |
>> | 52.22.59.110=C2=A0 | 192.129.210.181 | 151.203.130.228 |=C2=A0 21271 |=
=C2=A0 12452
>> |=C2=A0 =C2=A0 6 |=C2=A0 =C2=A0 =C2=A0 80 |=C2=A0 43720 | 0x18=C2=A0 =C2=
=A0 |=C2=A0 49500 | 2017-04-27 22:05:55
>> |
>> +----------------+-----------------+-----------------+------
>> --+--------+-------+---------+---------+----------+--------+
>> ---------------------+
>> 20 rows in set (0.06 sec)
>>
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>> Mehrdad
>>
>> --
>> *M*ehrdad Arshad Rad
>> *P*rincipal Software Engineer
>> https://www.linkedin.com/in/mehrdadrad
>>
>
>
>
> --
> --
> Vitaly Nikolaev
>
--=20
*M*ehrdad Arshad Rad
*P*rincipal Software Engineer
https://www.linkedin.com/in/mehrdadrad
=20
