[netsa-tools-discuss] Issue with date?

[Mark Thomas]

That seems odd, but I am glad to hear that the problem appears to
have resolved itself.

Please let us know if the problem resurfaces or if you need further
assistance.

-Mark


On Fri, 31 Oct 2014 08:50:10 +0000, Xander Maas wrote:

Hello Mark,

As it may seem weird, but I reinstalled the server (thus reinstalled SiLK) and the issue is gone (??).


With kind regards,

Xander Maas
Technisch Coordinator
Systeem beheer Nimeto Utrecht

E: x.maas at nimeto.nl<mailto:x.maas at nimeto.nl>
T: 030 275 30 51

On 30 Oct 2014, at 18:57, Mark Thomas <mthomas at cert.org<mailto:mthomas at cert.org>> wrote:

Xander-

Thank you for your question.  From our online search, it appears
that the SonicWall device supports exporting NetFlow v5 and v9, as
well as IPFIX.  Each of these formats has its own quirks and
pitfalls.

My first suggestion is to use wireshark to examine the data coming
from the SonicWall device to ensure that the problem is isolated to
SiLK.

If wireshark reports correct times, we need to determine why SiLK's
timestamps are incorrect.  Unfortunately, this is not
straightforward, and I wish SiLK provided a better way to debug
these issues when they arise.

First, some information on how times are represented: For NetFlow v5
and typically for NetFlow v9, the start-time and end-time of an
individual record are specified as offsets from the initialization
time of the card or the device (for example, the router's boot
time).  The header of the NetFlow record contains two values for the
current time, the first is an offset of the initialization time and
the second is a value in UNIX epoch seconds.  Given that
information, the timestamps of a record can be determined.

We have seen NetFlow v5 data where the offset time in the header is
the maximum of all the offsets (as if it was set just as the packet
was exported) and where it is the minimum of all the offsets (as if
the header was generated first).

In addition, these offsets are reported in milliseconds and they are
stored in a 32-bit value.  When a device is alive for over 49 days,
the offsets roll-over.

To handle these various situations, there are rules in SiLK about
when to assume roll-over has occurred.  If the data does not match
the code in SiLK, records can be stored with incorrect time stamps.


When writing IPFIX data, there are several different information
elements that the flow exporter may use to express the timestamps on
a record.  Some of these are absolute times, some are offsets from
the initialization time, and some are offsets from the packet's
export time that is contained in the IPFIX header.

The IPFIX reading code in SiLK has code to handle these many types
of time input.  SiLK makes a good-faith effort in the cases where
the IPFIX record is under-specified---for example, the record
specifies the timestamps as offsets from an initialization time but
that initialization time is not provided.  The odd timestamps could
be a result of this good-faith effort being incorrect.

Since there are several IPFIX time formats, you also may have
uncovered a bug in a little used code path.


If you are using NetFlow v9 or IPFIX, setting the
SILK_IPFIX_PRINT_TEMPLATES environment variable to 1 before you
start rwflowpack will cause it to record in the log file the
information elements in the templates it receives.  Having the
elements in the template may help us to determine what code path is
being taken in SiLK.

I am sorry I cannot provide a more definitive answer.

Cheers,

-Mark


On Wed, 29 Oct 2014 13:48:38 +0000, Xander Maas wrote:

Hi all,

We are testing a SiLK environment (with FlowViewer) with our
SonicWall NSA4500. It all seems fine, although some things are
still a bit fuzzy to me..)

We see in the logs (and in the corresponding directory
/data/nsa4500) that traffic is received for november 2014(?!)

/data/nsa4500/out/2014/11/10/out-nsa4500_20141110.05: 1 recs

This what is in the file (IP is from my iMac)
rwcut /data/nsa4500/out/2014/11/10/out-nsa4500_20141110.05 | head -200
                                   sIP|                                    dIP|sPort|dPort|pro|   packets|     bytes|   flags|                  sTime| duration|                  eTime| sensor|
                           xx.x.xx.xxx|                         xx.xxx.xxx.xxx|62474|  993|  6|        20|      1956|        |2014/11/10T06:01:11.148|99322.000|2014/11/11T09:36:33.148|nsa4500|

We installed everything on a Xen guest with CentOS 6.5 from the yum repo.

(This happens

With kind regards,

Xander Maas
Technisch Coordinator
Systeem beheer Nimeto Utrecht

E: x.maas at nimeto.nl<mailto:x.maas at nimeto.nl><mailto:x.maas at nimeto.nl>
T: 030 275 30 51