[netsa-tools-discuss] Maximum duration

[Mark Thomas]

My reply is below.

On Wed, 28 Jan 2015 08:58:07 -0500, inetjunkmail at gmail.com wrote:

> I just got SiLK installed and am seeing 100's of thee per second
> in my logs:
>
> Jan 28 08:53:04 silk rwflowpack[13441]: Record's duration greater than that allowed in file '/data/silk/ext2ext/2588/07/15/ext2ext-S1_25880715.02': 1474356 > 4095
>
> This is data from several IPFIX Sensors.  Is the 4095 limit
> tunable or does this indicate a problem with the data that's being
> received?
>
> Thanks,
> E

Before we solve the long duration issue we should determine why SiLK
thinks your flow records have a start date of July 15, 2588.

When a flow exporter writes IPFIX data, there are several different
information elements that it may use to express the timestamps on a
record.  Some of these are absolute times, some are offsets from the
initialization time, and some are offsets from the packet's export
time that is contained in the IPFIX header.

The IPFIX reading code in SiLK has code to handle these many types
of time input.  SiLK makes a good-faith effort in the cases where
the IPFIX record is under-specified---for example, the record
specifies the timestamps as offsets from an initialization time but
that initialization time is not provided.

The odd times you see could be a result of

* SiLK's good-faith effort being incorrect,

* the flow record using a combination of information elements that
  SiLK does not expect, or

* a bug in one of the time-handling code-paths that has not been
  exercised often.

If you are using SiLK 3.10.0, you can tell SiLK to log how it is
determining the times by modifying the "probe" block in the
"sensor.conf" file to contain the line

  log-flags default record-timestamps


The maximum duration of 4095 is due to the file format that SiLK
uses to store IPv4 flow records.  There are a couple of ways to tell
SiLK to use a file format that does not have that limitation:

* you can add the following to the probe block

  quirks zero-packets

* you can reconfigure and rebuild SiLK to support IPv6 flow records

Either of those approaches cause SiLK to use a file format that
supports a duration of 49 days.

-Mark