[netsa-tools-discuss] Store all IPFIX flows from NAT

Thu Jun 21 13:49:23 EDT 2018

 Well, thank you Chris, you got it!If that ever changes, let me know and we'll push it offline.
Here is the Q&A so far.Alexander - please continue answering inline.
Q: Are you talking about elements 225-228 and 323 as defined in https://www.iana.org/ assignments/ipfix/ipfix.xhtml?
A: Yes
Q: How is the flow data coming into the system - UDP, PCAP, other?A: UDP
Q: How is the data produced today?A: Today data stored in MySQL by nfacct but it is take too much disk space. And we try to find better solution to reduce disk space usage.

Q: With the right version of mysql, you can compress tables - did you try this?A: 

Q: How do you need the data out - CSV, sql database, other?A: 
Q: Do you have requirements as to which OS (Windows 7, Linux*, ...) this needs to run on?
A: 
Q: What architecture do you need to run on (32-bit, 64-bit)?A: 
Q: Is a tool like this OK?
     This parses packets coming in to localhost:2055 and dumps as CSV to stdout:         java -jar flowparser.jar -input udp://127.0.0.1:2055 -csv
     This dumps to a rotated file instead         java -jar flowparser.jar -input udp://127.0.0.1:2055 -output output.gzip -rotate 1000000 -maxage 3600
A: 

Sal.
    On Thursday, June 21, 2018, 9:52:32 AM PDT, Chris Inacio <inacio at cert.org> wrote:  

All,

I’m perfectly fine if you want to have that conversation off-list, but I’m also perfectly happy if you have the conversation on list. If you have needs that SiLK (or other NetSA tools) don’t do and you’ve found an alternative - that’s fine. It would be good for the community of users to know those alternatives and how people have used them.

Our tools are not all things to all people - and they never will be. My personal opinion is that this sharing benefits everyone, including the CERT NetSA tool developers and analysts.

Regards, 

-- 
Chris Inacio
inacio at cert.org

From: Sal Ingrilli <shortpasta at yahoo.com>
Date: June 19, 2018 at 1:07:02 PM
To: Alexander Khokhlov <hohlovap at gmail.com>
Cc: netsa-tools-discuss at cert.org <netsa-tools-discuss at cert.org>
Subject:  Re: [netsa-tools-discuss] Store all IPFIX flows from NAT

I have other questions unrelated to silk, so I will move this offline.Your English is fine.
On Tuesday, June 19, 2018, 1:13:49 AM PDT, Alexander Khokhlov <hohlovap at gmail.com> wrote:

Hi, thanks, yes, all according tohttps://www.iana.org/assignments/ipfix/ipfix.xhtml

Flow data coming by UDP

Today data stored in MySQL by nfacct but it is take too much disk space. And we try to find better solution to reduce disk space usage.

p.s. Sorry, for my english)

2018-06-19 0:11 GMT+03:00 Sal Ingrilli<shortpasta at yahoo.com>:

I have a tool that can collect all elements.
Are you talking about elements 225-228 and 323 as defined here?https://www.iana.org/ assignments/ipfix/ipfix.xhtml

How is the flow data coming into the system?UDP, PCAP, other?
How do you need the data out?CSV, sql database, other?

On Monday, June 18, 2018, 1:59:44 PM PDT, Mark Thomas <mthomas at cert.org> wrote:

No, SiLK does not have support for capturing those information
elements.

-Mark

-----Original Message-----
From: Alexander Khokhlov <hohlovap at gmail.com>
Date: Mon, 18 Jun 2018 17:10:59 +0300
To: <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Store all IPFIX flows from NAT

Hello, I need to collect and store IPFIX flows from NAT servers.
Is it possible to collect IE 225-228,323? Please help, cant handle it!

Jun 18 16:50:07 s078r rwflowpack[27567]:
IGNORED|10.203.9.160|46.173. 38.219|57099|41328|6|0|0|no forward/reverse
octets|
Jun 18 16:50:07 s078r rwflowpack[27567]: IPFIX Message out of sequence (in
domain 00000000, expected 19369e1b, got 469de6cb)
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Contains 11 Elements, Enabled by SILK_IPFIX_PRINT_TEMPLATES
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  0, Length    8, IE        323, Name observationTimeMilliseconds
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  1, Length    4, IE          8, Name sourceIPv4Address
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  2, Length    4, IE          12, Name destinationIPv4Address
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  3, Length    4, IE        225, Name postNATSourceIPv4Address
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  4, Length    4, IE        226, Name
postNATDestinationIPv4Address
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  5, Length    2, IE          7, Name sourceTransportPort
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  6, Length    2, IE          11, Name destinationTransportPort
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  7, Length    2, IE        227, Name postNAPTSourceTransportPort
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  8, Length    2, IE        228, Name
postNAPTDestinationTransportPo rt
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  9, Length    1, IE          4, Name protocolIdentifier
Jun 18 16:50:07 s078r rwflowpack[27567]: Domain 000000, TemplateID 0X0102,
Position  10, Length    1, IE        230, Name natEvent
Jun 18 16:50:07 s078r rwflowpack[27567]:
IGNORED|10.202.181.195|80.77. 168.44|37680|80|6|0|0|no forward/reverse
octets|
Jun 18 16:50:07 s078r rwflowpack[27567]:
IGNORED|10.202.180.217|149. 154.175.50|55263|443|6|0|0|no forward/reverse
octets|
Jun 18 16:50:07 s078r rwflowpack[27567]:
IGNORED|10.202.148.162|149. 154.167.91|55439|5222|6|0|0|no forward/reverse
octets|
Jun 18 16:50:07 s078r rwflowpack[27567]:
IGNORED|10.202.157.221|94.100. 180.26|54182|80|6|0|0|no forward/reverse
octets|
Jun 18 16:50:07 s078r rwflowpack[27567]:
IGNORED|10.201.6.97|149.154. 167.51|19207|443|6|0|0|no forward/reverse
octets|

-------------- next part --------------
HTML attachment scrubbed and removed