[netsa-tools-discuss] Accessing translated src/dst address fields?

[Richard Graham]

Hi Brian,

> In that case, I'm wondering what's the simplest way to store and query
> complete IPFIX data.

BLUF: I still have a lot of work to do, but so far I'm pretty
satisfied with PCAP->yaf->super_mediator->JSON->MongoDB.

I'm currently exploring/struggling with this issue too and hope that
some of my experiences might be of some interest.  My approach has
evolved over time as I've learned more and I'm sure there's much more
for me to learn.  Currently, I'm only dealing with large collections
of PCAPs, so real-time processing isn't significant to me yet,
although going fast is a definite plus.

I don't really know all the questions I want to answer from IPFIX data
yet, so I'm reluctant to throw away any of the elements yaf produces.
I even re-compiled yaf to enable almost all of the extra features,
e.g., forward/reverse payload entropy.  I can see all the IPFIX
records yaf produces with ipfixDump and it looks very complete.  SiLK
looks awesome, and it's fast, but I'm not sure all my flow questions
can be answered with SiLK processing - the most basic and important
ones can though.

When I was more naive than I am now, my plan was to use a pure Python
interface to the IPFIX files to retrieve elements.  Even with a Python
IPFIX library, this is more complicated than I want - and my code to
retrieve simple elements was *really* slow.  My next consideration was
to use super_mediator to populate a MySQL/MariaDB database from the
yaf IPFIX.  This works, but I'm no database wizard and having to fully
specify RDBM tables/records/types/etc. is not fun, and TBH I don't
even know how many iterations I'd need to get it right.

So, currently I'm using super_mediator to output JSON and then import
that into MongoDB (I'm no wizard with that either). [Sorry Mark, I
should submit a bug report since there's a small JSON formatting error
in super_mediator's output.  I easily "fixed" it with sed so I bet the
super_mediator fix will be easy too.]  MongoDB has a Python connector
package so making queries from Python isn't difficult and reasonably
fast.  I'm still in the beginning stages, but it looks like virtually
all the IPFIX data is in my MongoDB now.  MongoDB has a GUI tool
called "Compass" that's pretty handy for viewing the documents in the
database collections.  It can even execute simple to complex
aggregations that can then be exported to Python, and some other
languages.

I still have a lot of work to do, but so far I'm pretty satisfied with
PCAP->yaf->super_mediator->JSON->MongoDB.

When you pursue your path, I hope you will share it too.

Regards,
Richard