[netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES introduced in SiLK 3.16.0

[Mark Thomas]

Bo-

We believe the new behavior to be correct.

The IN_BYTES and IN_PKTS counters are computed when packets enter
the router, and the OUT_BYTES and OUT_PKTS counters represent values
when packets leave the router.  One reason those values can be
different for packets traveling in one direction are WAN optimizers
that compress/uncompress the data as it leaves/enters the monitored
network.

Fixbuf 1.8.0 maps the OUT_BYTES and OUT_PKTS values to their IPFIX
equivalents, postOctetDeltaCount and postPacketDeltaCount.  (The
OUT_ values are labeled "post" in IPFIX to denote they are values
computed after the observation point.)

If all interfaces of a border router are monitored, then you can get
double counting of the data in SiLK:

  * Internet->Local traffic entering the router. incoming IN_BYTES
  * Internet->Local traffic leaving the router.  incoming OUT_BYTES
  * Local->Internet traffic entering the router. outgoing IN_BYTES
  * Local->Internet traffic entering the router. outgoing OUT_BYTES

The changes in SiLK 3.16.0 and Fixbuf 1.8.0 are meant to avoid the
double counting.

-Mark


-----Original Message-----
From: Bo Bayles <bbayles at obsrvbl.com>
Date: Wed, 2 May 2018 12:51:17 -0500
To: <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES
	introduced in SiLK 3.16.0

I found I started having an issue with processing data from some NetFlow v9
exporters after upgrading SiLK.

Specifically, rwcut and rwuniq used to properly reflect the bidirectional
traffic being reported by these exporters, but they don't anymore.

After doing some digging, I found that things changed for the worse in version
3.16.0. I suspect the relevant Release Note is:

> Change processing of NetFlow v9 records so that, when SiLK is compiled
> against libfixbuf 1.8.0, the OUT_BYTES and OUT_PKTS values are used when the
> IN_BYTES and IN_PKTS values are 0.

The specific problem occurs when an exporter specifies both traffic directions
in its template. That is, both IN_BYTES and OUT_BYTES (RFC 3954 fields 1 and
23, respectively) / both IN_PKTS and OUT_PKTS (fields 2 and 24).

Given a flow like this, the SiLK tools used to report two rows - one for the
bytes and packets from 192.168.12.205 -> 15.73.97.78, and one for the bytes and
packets from 15.73.97.78 -> 192.168.12.205.

* IP_SRC_ADDR: 192.168.12.205
* IP_DST_ADDR: 15.73.97.78
* IN_PKTS: 3987
* OUT_PKTS: 1807
* IN_BYTES: 634500
* OUT_BYTES: 9580

Output when captured by flowcap 3.14 (3.15 is the same):

$ rwcut --fields 1,2,6,7 --no-columns "version_3.14.silk"
sIP|dIP|packets|bytes|
192.168.12.205|15.73.97.78|3987|634500|
15.73.97.78|192.168.12.205|1807|9580|

Output when captured by flowcap 3.16.1:

$ rwcut --fields 1,2,6,7 --no-columns "version_3.16.1.silk"
sIP|dIP|packets|bytes|
192.168.12.205|15.73.97.78|3987|634500|

Clearly useful information is being lots in the second version - I hope this
can be fixed in the next release?

I've attached a PCAP with the originating v9 packets, plus the two SiLK files.

Many thanks,
-Bo