[netsa-tools-discuss] Date on SILK/rwflowpack directory is in the future by a good bit.

Mark Thomas mthomas at cert.org
Thu Jan 25 17:28:33 EST 2018


Jeff-

Thank you for using the NetSA tools.  Thanks also for your question.

In NetFlow v9, the starting and ending times for an individual
record are given as offsets from a timestamp in the message header.
(Details below.)  Sometimes SiLK may misinterpret these offsets and
produce odd timestamps.

You can tell rwflowpack to write the incoming record timestamps in
the log file.  To do this, you must modify the sensor.conf file.
Locate the probe block(s) in the file, and add the following
statement:

  log-flags default record-timestamps

(If you already have a log-flags statement for a probe, add the
record-timestamps value to it.)

Once you restart rwflowpack, it will print values it receives for
each flow record and how it interprets those values to create an
absolute timestamp.  Note that this produces a lot of output in the
log file.

Having the timestamp data in the log file should help in debugging
the issue.

If you find that the systemInitTimeMilliseconds is about 1000 times
less than the flowStartSysUpTime, the Gigamon may have the same
issue as we recently discovered on SonicWall appliances.  We added
a quirk to work-around the problem, which you can enable by adding

  quirks nf9-sysuptime-seconds

to the probe block(s) and restarting rwflowpack.

I hope this helps, and if you have further questions or would like
assistance in interpreting the log output, please let us know.

Cheers,

-Mark

NetFlow timestamp details:

The starting and ending times for an individual NetFlow v9 record
are given as offsets from the device's initialization time (where
the initialization time is often the router's boot time).  The
NetFlow message header contains the message's export time as both an
offset from the initialization time and in the traditional form used
by UNIX (seconds since January 1, 1970).  The offsets have
millisecond precision and are stored in 32-bit numbers; this causes
the values to roll-over every 49.7 days.

SiLK expects all the offsets to be fairly close to one another.  If
they are not, SiLK assumes a value has rolled over and attempts to
compensate for that.  This can cause SiLK to produce timestamps that
occur about 50 days in the future or past.

The record-timestamps log-flag causes rwflowpack to record the
offsets on each record (flowStartSysUpTime, flowEndSysUpTime), the
offset in the header (systemInitTimeMilliseconds), the current UNIX
time (exportTimeSeconds), the start time it computes, and whether it
assumed offsets had rolled-over.


-----Original Message-----
From: "Collyer, Jeffrey W. (jwc3f)" <jwc3f at virginia.edu>
Date: Tue, 23 Jan 2018 20:57:33 +0000
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Date on SILK/rwflowpack directory is in the
	future by a good bit.

So I have a Gigamon generating Netflow v9 and sending it to two collectors, one of which is Silk.
The Silk collector is suddenly writing files with dates in the future.
The second collector is seeing the same flow data and only sees time/date stamps for today, nothing in the future.

Much as I would love to know the future I can’t figure out what is causing this.

For example

/opt/silk/data/GIGAMON-NF-1/in/2018/02/12# ls -al
total 2998086
drwxr-xr-x 2 root root        14 Jan 23 15:27 .
drwxr-xr-x 5 root root         5 Jan 23 04:27 ..
-rw-r--r-- 1 root root 221513566 Jan 23 05:29 in-GIGAMON-NF-1_20180212.00
-rw-r--r-- 1 root root 220770422 Jan 23 06:29 in-GIGAMON-NF-1_20180212.01
-rw-r--r-- 1 root root 238481114 Jan 23 07:29 in-GIGAMON-NF-1_20180212.02
-rw-r--r-- 1 root root 248975819 Jan 23 08:29 in-GIGAMON-NF-1_20180212.03
-rw-r--r-- 1 root root 278772583 Jan 23 09:29 in-GIGAMON-NF-1_20180212.04
-rw-r--r-- 1 root root 288936702 Jan 23 10:29 in-GIGAMON-NF-1_20180212.05
-rw-r--r-- 1 root root 297647756 Jan 23 11:29 in-GIGAMON-NF-1_20180212.06
-rw-r--r-- 1 root root 301989843 Jan 23 12:29 in-GIGAMON-NF-1_20180212.07
-rw-r--r-- 1 root root 296719238 Jan 23 13:29 in-GIGAMON-NF-1_20180212.08
-rw-r--r-- 1 root root 300709672 Jan 23 14:29 in-GIGAMON-NF-1_20180212.09
-rw-r--r-- 1 root root 298666055 Jan 23 15:29 in-GIGAMON-NF-1_20180212.10
-rw-r--r-- 1 root root  77347926 Jan 23  2018 in-GIGAMON-NF-1_20180212.11
/opt/silk/data/GIGAMON-NF-1/in/2018/02/12# date
Tue Jan 23 15:43:12 EST 2018

Ive checked the date on the Gigamon, the VM that Silk is running on, and the Hypervisor.  All are correct and are NTP synced.
I restarted rwflowpack, rebooted the VM, and even motioned it off and bounced the hypervisor.
I was on silk-3.14.0, but I just upgraded to silk-3.16.0 in case that might help. It didn’t.

My conf files are pretty stock, and previously it had been working fine.

Any help/hints/idea about whats wrong and how I might fix this greatly appreciated.

Jeff

Jeffrey Collyer
Information Security Engineer
University of Virginia


More information about the netsa-tools-discuss mailing list