[netsa-tools-discuss] Extracting fields from Netflow V9 using SiLK

[Mark Thomas]

Sukrit-

If you want to do a "quick and dirty" hack to support MAC addresses,
you could repurpose the NextHop IP field to hold the MAC addresses
as long as you force all records to be stored as IPv6.

To do that:

The IPFIX/NetFlow v9 to SiLK translation occurs in the file
src/libflowsource/skipfix.c.  Modify the ski_rwrec_spec[] array in
that file so that the MAC addresses (plus some padding) appears in
the 16 bytes normally used for the "ipNextHopIPv6Address".

Ensure src/libflowsource/skipfix.c always writes its records as IPv6
so the the MAC addresses are visible when viewing a record.  This
maps the IPv4 source and destination addresses into the
::ffff:0:0/96 net-block.

Be certain to configure and compile SiLK with IPv6 support.

In the rwflowpack configuration, be sure to include the
--pack-interfaces switch so the NextHop IP addresses are stored (by
default, they and the SNMP interfaces are dropped).


To do things the "correct" way would take much more work.  SiLK Flow
records are defined by a C struct, and changes to this structure can
have affects throughout the tool suite.  In addition, knowledge of
the fields available in the structure is shared throughout the tool
suite.  Adding new fields requires changes in many parts of the
code.  That said, the task is not impossible and it tends to be more
tedious than difficult.

The in-core SiLK format is defined in src/libsilk/rwrec.h.  Storage
of the SiLK format is defined in several files in the src/libsilk
directory; for an IPv6 compile of SiLK, the on-disk format is given
in the rwipv6io.c and rwipv6routingio.c files.  Uses of the SiLK
fields in the source code can be found the the Perl regexp
'rwRec(Mem|)(Get|Set)'.  Where that regexp occurs, you probably want
to add similar code to handle the MAC addresses.


SiLK is released by CERT/CC, which is part of the Software
Engineering Institute, a Federally Funded Research and Development
Center (FFRDC).  As an FFRDC, code we write must pass through
release approval before it is made public.  While we would like to
get more feedback earlier, it is difficult given the release
approval requirement.

-Mark


On Tue, 18 Nov 2014 10:17:21 -0500, Sukrit Dasgupta wrote:

> (including the netsa mailing list to keep record if others have
> the same query).
>
> Thanks for your response Mark.
>
> This is sad (However I still have long term plans to continue to use it because it is very useful). Few questions:
>
> 1. Any chance I could get some pointers on where the code is if we
> needed to implement these changes ourselves (for the time being)?
>
> 2. Any way I could get a chance to work on an early alpha/pre-alpha release?
>
>
> Thanks!
> -- Sukrit
>
> On Nov 18, 2014, at 9:53 AM, Mark Thomas <mthomas at cert.org> wrote:
>
>> Sukrit-
>> 
>> Thank you for your email.  It is very nice to read that you find the
>> NetSA tools so useful.
>> 
>> Unfortunately, the answer to your question is that the current
>> version of SiLK supports a fixed set of fields, and adding new
>> fields to the core of SiLK is fairly involved.
>> 
>> The next major release of SiLK will support flexible file formats,
>> and adding additional fields to at that point should be
>> straightforward.  Currently there is no estimate as to when the next
>> release of SiLK will be available.
>> 
>> Thanks again for your question.
>> 
>> -Mark
>> 
>> 
>> On Mon, 17 Nov 2014 12:23:45 -0500, Sukrit Dasgupta wrote:
>> 
>>> Hi SiLK team,
>>> 
>>> I started using SiLK about 3 weeks ago for large scale NFv9 record
>>> analysis and it has proven to be invaluable!
>>> 
>>> Is it possible to extract other non 5-tuple fields that have been
>>> configured to be exported using NFv9 on a Cisco router?
>>> 
>>> For example, in a record configuration like this (Cisco IOS):
>>> 
>>> match ipv4 protocol  
>>> match ipv4 source address  
>>> match ipv4 destination address  
>>> match transport source-port  
>>> match transport destination-port  
>>> collect datalink mac source address input
>>> collect datalink mac destination address output
>>> 
>>> How could we extract the fields that have been specified as "collect":
>>> the MAC addresses in this case. As NFv9 sends out templates, I was
>>> hoping this was doable.
>>> 
>>> Any help will be great!
>>> 
>>> Thanks in advance,
>>> -- Sukrit