[netsa-tools-discuss] rwflowpack bug?

[Mark Thomas]

Vincent-

Short answer:

In the sensors.conf file, modify the probe block to contain
"record-timestamps" in the log-flags statement as shown here:

    probe pfSense netflow-v9
        listen-on-port 9900
        protocol udp
        accept-from-host 192.168.10.250
        log-flags default record-timestamps
    end probe

Restart rwflowpack and you should see in the rwflowpack log file (or
in syslog) statements that indicate how rwflowpack is setting the
time.

Longer answer:

That thread from 2014 prompted us to add ability to for rwflowpack
to log the values it is using to set the start time.  Since that can
produce a lot of output, it is not enabled by default.  My "Short
answer" above is asking you to enable this logging.

Since Oct 1 was about 49 days ago, I would not be surprised to find
that there is an issue in the way that SiLK is interpreting the 32
bit values used to represent "the number of milliseconds since the
router booted", since those values roll-over about every 49 days.

-Mark


-----Original Message-----
From: Vincent Ragosta <ragosta at plummerslade.com>
Date: Fri, 20 Nov 2015 16:54:59 +0000
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] rwflowpack bug?

Hello,

On my system rwflowpack is saving some of the SiLK records with an incorrect sTime value of 10/01/2015:

Nov 19 19:28:07 silk-01 rwflowpack[7606]: Flushing files after 120 seconds.
Nov 19 19:28:07 silk-01 rwflowpack[7606]: /data/pfSense/inweb/2015/10/01/iw-pfSense_20151001.07: 42 recs
Nov 19 19:28:07 silk-01 rwflowpack[7606]: /data/pfSense/outweb/2015/10/01/ow-pfSense_20151001.07: 42 recs
Nov 19 19:28:07 silk-01 rwflowpack[7606]: /data/pfSense/int2int/2015/11/20/int2int-pfSense_20151120.00: 48 recs
Nov 19 19:28:07 silk-01 rwflowpack[7606]: /data/pfSense/int2int/2015/10/01/int2int-pfSense_20151001.07: 88 recs
Nov 19 19:28:07 silk-01 rwflowpack[7606]: /data/pfSense/in/2015/10/01/in-pfSense_20151001.07: 4 recs
Nov 19 19:28:07 silk-01 rwflowpack[7606]: /data/pfSense/out/2015/10/01/out-pfSense_20151001.07: 4 recs
Nov 19 19:28:07 silk-01 rwflowpack[7606]: 'pfSense': forward 228, reverse 0, ignored 0, nf9: missing-pkts 108

This is similar behavior to that previously reported on the listserv -- https://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/2014-October/000032.html.

I performed a packet capture of the netflow data coming off of my router and could not detect any invalid time stamps.

Here is some supplemental data that may be of use:

psi at silk-01:/data$ rwcut /data/pfSense/out/2015/10/01/out-pfSense_20151001.07 | tail
                           192.168.1.97|                            224.0.0.252|61130| 5355| 17|         2|       100|        |2015/10/01T07:31:14.121|4294966.879|2015/11/20T00:34:01.000|pfSense|
                           192.168.1.97|                            224.0.0.252|51671| 5355| 17|         2|       100|        |2015/10/01T07:31:14.121|4294966.879|2015/11/20T00:34:01.000|pfSense|
                           192.168.1.97|                            224.0.0.252|61130| 5355| 17|         2|       100|        |2015/10/01T07:31:14.121|4294966.879|2015/11/20T00:34:01.000|pfSense|
                           192.168.1.97|                            224.0.0.252|51671| 5355| 17|         2|       100|        |2015/10/01T07:31:14.121|4294966.879|2015/11/20T00:34:01.000|pfSense|
                           192.168.1.11|                          74.125.21.125|56701| 5222|  6|         9|      1836|   PA   |2015/10/01T07:45:55.584|4294145.416|2015/11/20T00:35:01.000|pfSense|
                           192.168.1.11|                          74.125.21.125|56701| 5222|  6|         9|      1836|   PA   |2015/10/01T07:45:55.584|4294145.416|2015/11/20T00:35:01.000|pfSense|
                           192.168.1.79|                                8.8.8.8| 3530|   53| 17|         1|        58|        |2015/10/01T07:33:13.732|4294967.268|2015/11/20T00:36:01.000|pfSense|
                           192.168.1.79|                         128.182.58.100| 3528|  123| 17|         1|        76|        |2015/10/01T07:33:13.727|4294967.273|2015/11/20T00:36:01.000|pfSense|
                           192.168.1.79|                                8.8.8.8| 3530|   53| 17|         1|        58|        |2015/10/01T07:33:13.732|4294967.268|2015/11/20T00:36:01.000|pfSense|
                           192.168.1.79|                         128.182.58.100| 3528|  123| 17|         1|        76|        |2015/10/01T07:33:13.727|4294967.273|2015/11/20T00:36:01.000|pfSense|


psi at silk-01:/data$ cat sensors.conf
probe pfSense netflow-v9
    listen-on-port 9900
    protocol udp
    accept-from-host 192.168.10.250
end probe

group my-network
    ipblocks 192.168.10.0/24
    ipblocks 192.168.1.0/24
    ipblocks 192.168.5.0/24
end group

sensor pfSense
    netflow-v9-probes pfSense
    internal-ipblocks @my-network
    external-ipblocks remainder
end sensor


psi at silk-01:/data$ rwflowpack --version
rwflowpack: part of SiLK 3.11.0.1; configuration settings:
    * Root of packed data tree:         /data
    * Packing logic:                    Run-time plug-in
    * Timezone support:                 UTC
    * Available compression methods:    none [default], zlib
    * IPv6 network connections:         yes
    * IPv6 flow record support:         yes
    * IPFIX/NetFlow9/sFlow collection:  ipfix,netflow9,sflow
    * Transport encryption:             no
    * PySiLK support:                   /usr/lib/python2.7/dist-packages
    * Enable assert():                  no


psi at silk-01:/data$ cat /etc/issue
Ubuntu 14.04.3 LTS \n \l

Please let me know if any additional data is required.


Thanks,

Vincent