[netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES introduced in SiLK 3.16.0

[Bo Bayles]

I found I started having an issue with processing data from some NetFlow v9
exporters after upgrading SiLK.

Specifically, rwcut and rwuniq used to properly reflect the bidirectional
traffic being reported by these exporters, but they don't anymore.

After doing some digging, I found that things changed for the worse in version
3.16.0. I suspect the relevant Release Note is:

> Change processing of NetFlow v9 records so that, when SiLK is compiled
> against libfixbuf 1.8.0, the OUT_BYTES and OUT_PKTS values are used when the
> IN_BYTES and IN_PKTS values are 0.

The specific problem occurs when an exporter specifies both traffic directions
in its template. That is, both IN_BYTES and OUT_BYTES (RFC 3954 fields 1 and
23, respectively) / both IN_PKTS and OUT_PKTS (fields 2 and 24).

Given a flow like this, the SiLK tools used to report two rows - one for the
bytes and packets from 192.168.12.205 -> 15.73.97.78, and one for the bytes and
packets from 15.73.97.78 -> 192.168.12.205.

* IP_SRC_ADDR: 192.168.12.205
* IP_DST_ADDR: 15.73.97.78
* IN_PKTS: 3987
* OUT_PKTS: 1807
* IN_BYTES: 634500
* OUT_BYTES: 9580

Output when captured by flowcap 3.14 (3.15 is the same):

$ rwcut --fields 1,2,6,7 --no-columns "version_3.14.silk"
sIP|dIP|packets|bytes|
192.168.12.205|15.73.97.78|3987|634500|
15.73.97.78|192.168.12.205|1807|9580|

Output when captured by flowcap 3.16.1:

$ rwcut --fields 1,2,6,7 --no-columns "version_3.16.1.silk"
sIP|dIP|packets|bytes|
192.168.12.205|15.73.97.78|3987|634500|

Clearly useful information is being lots in the second version - I hope this
can be fixed in the next release?

I've attached a PCAP with the originating v9 packets, plus the two SiLK files.

Many thanks,
-Bo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: silk-diagnostic.zip
Type: application/zip
Size: 742 bytes
Desc: not available
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20180502/5e797143/attachment.zip>