[netsa-tools-discuss] Extracting fields from Netflow V9 using SiLK

[Sukrit Dasgupta]

Mark thanks a lot for these pointers. I took some time to go through the code and figure out the moving parts before replying to you.

Will try to get this going.

Thanks,
-- Sukrit

On Nov 19, 2014, at 3:06 PM, Mark Thomas <mthomas at cert.org> wrote:

> Sukrit-
> 
> If you want to do a "quick and dirty" hack to support MAC addresses,
> you could repurpose the NextHop IP field to hold the MAC addresses
> as long as you force all records to be stored as IPv6.
> 
> To do that:
> 
> The IPFIX/NetFlow v9 to SiLK translation occurs in the file
> src/libflowsource/skipfix.c.  Modify the ski_rwrec_spec[] array in
> that file so that the MAC addresses (plus some padding) appears in
> the 16 bytes normally used for the "ipNextHopIPv6Address".
> 
> Ensure src/libflowsource/skipfix.c always writes its records as IPv6
> so the the MAC addresses are visible when viewing a record.  This
> maps the IPv4 source and destination addresses into the
> ::ffff:0:0/96 net-block.
> 
> Be certain to configure and compile SiLK with IPv6 support.
> 
> In the rwflowpack configuration, be sure to include the
> --pack-interfaces switch so the NextHop IP addresses are stored (by
> default, they and the SNMP interfaces are dropped).
> 
> 
> To do things the "correct" way would take much more work.  SiLK Flow
> records are defined by a C struct, and changes to this structure can
> have affects throughout the tool suite.  In addition, knowledge of
> the fields available in the structure is shared throughout the tool
> suite.  Adding new fields requires changes in many parts of the
> code.  That said, the task is not impossible and it tends to be more
> tedious than difficult.
> 
> The in-core SiLK format is defined in src/libsilk/rwrec.h.  Storage
> of the SiLK format is defined in several files in the src/libsilk
> directory; for an IPv6 compile of SiLK, the on-disk format is given
> in the rwipv6io.c and rwipv6routingio.c files.  Uses of the SiLK
> fields in the source code can be found the the Perl regexp
> 'rwRec(Mem|)(Get|Set)'.  Where that regexp occurs, you probably want
> to add similar code to handle the MAC addresses.
> 
> 
> SiLK is released by CERT/CC, which is part of the Software
> Engineering Institute, a Federally Funded Research and Development
> Center (FFRDC).  As an FFRDC, code we write must pass through
> release approval before it is made public.  While we would like to
> get more feedback earlier, it is difficult given the release
> approval requirement.
> 
> -Mark
> 
> 
> On Tue, 18 Nov 2014 10:17:21 -0500, Sukrit Dasgupta wrote:
> 
>> (including the netsa mailing list to keep record if others have
>> the same query).
>> 
>> Thanks for your response Mark.
>> 
>> This is sad (However I still have long term plans to continue to use it because it is very useful). Few questions:
>> 
>> 1. Any chance I could get some pointers on where the code is if we
>> needed to implement these changes ourselves (for the time being)?
>> 
>> 2. Any way I could get a chance to work on an early alpha/pre-alpha release?
>> 
>> 
>> Thanks!
>> -- Sukrit
>> 
>> On Nov 18, 2014, at 9:53 AM, Mark Thomas <mthomas at cert.org> wrote:
>> 
>>> Sukrit-
>>> 
>>> Thank you for your email.  It is very nice to read that you find the
>>> NetSA tools so useful.
>>> 
>>> Unfortunately, the answer to your question is that the current
>>> version of SiLK supports a fixed set of fields, and adding new
>>> fields to the core of SiLK is fairly involved.
>>> 
>>> The next major release of SiLK will support flexible file formats,
>>> and adding additional fields to at that point should be
>>> straightforward.  Currently there is no estimate as to when the next
>>> release of SiLK will be available.
>>> 
>>> Thanks again for your question.
>>> 
>>> -Mark
>>> 
>>> 
>>> On Mon, 17 Nov 2014 12:23:45 -0500, Sukrit Dasgupta wrote:
>>> 
>>>> Hi SiLK team,
>>>> 
>>>> I started using SiLK about 3 weeks ago for large scale NFv9 record
>>>> analysis and it has proven to be invaluable!
>>>> 
>>>> Is it possible to extract other non 5-tuple fields that have been
>>>> configured to be exported using NFv9 on a Cisco router?
>>>> 
>>>> For example, in a record configuration like this (Cisco IOS):
>>>> 
>>>> match ipv4 protocol  
>>>> match ipv4 source address  
>>>> match ipv4 destination address  
>>>> match transport source-port  
>>>> match transport destination-port  
>>>> collect datalink mac source address input
>>>> collect datalink mac destination address output
>>>> 
>>>> How could we extract the fields that have been specified as "collect":
>>>> the MAC addresses in this case. As NFv9 sends out templates, I was
>>>> hoping this was doable.
>>>> 
>>>> Any help will be great!
>>>> 
>>>> Thanks in advance,
>>>> -- Sukrit

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20141124/b3b71786/attachment.sig>