[netsa-tools-discuss] flowcap formats

[Mark Thomas]

John-

On Fri, 11 Sep 2015 14:28:07 +0000, John Green wrote:

> As a result I've found a number of flows with old etimes (all with
> duration of 65535 seconds).  Looking more closely this appears to be a
> result of flowcap using FT_FLOWCAP for v5 records which stores flow
> duration in seconds using 2 bytes.  Flowcap uses FT_RWIPV6ROUTING for
> non-v5 sources, in msec using 4 bytes (giving the 49 day maximum seen
> elsewhere) 
>
> I'm going to try changing flowcap.c to use FT_RWIPV6ROUTING regardless,
> at the expense of larger files, but I just wanted to check I wasn't
> missing something?

There is an assumption in SiLK that the flow generator (router or
YAF) splits long-lived flow records into multiple records using an
active-timeout of about an hour or less.

If you look at the repository storage formats used for IPv4 data
(such as FT_RWSPLIT), you will notice that the duration field is 22
bits wide, allowing for a maximum duration of 4194.303 seconds.

The very first formats supported an 11 bit field to hold the
duration in seconds resolution, requiring an active-timeout of 2048
seconds.

When we added IPv6 support, we decided not to continue using
bit-bashing to make the records as small as possible, since we
figured with the huge IPv6 addresses that battle was already lost.

Using the FT_RWGENERIC format in place of FT_FLOWCAP will give you
the 49 day maximum duration without requiring the extra space for
the IPv6 addresses.

> As FT_FLOWCAP uses 3 further bytes for stime/elapsed msec this seems
> like quite a compromise to save 1 byte!

Agreed.  I think that is just a result of how the code evolved over
time, and that the format was initially written local to the flowcap
binary and not part of libsilk.

> Thanks
> John

-Mark