[netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES introduced in SiLK 3.16.0

[Bo Bayles]

Mark, thanks for writing.

I'm afraid I disagree - a literal reading RFC 3954 doesn't rule out your
interpretation, but it doesn't specify it either.
(This vagueness is, of course, one of the major flaws of that spec!)

The reason why this is a problem for me is that Meraki MX devices use the
interpretation that old fixbuf used.
If I'm to continue monitoring them, must I freeze at libfixbuf 1.7.1? I'd
hate to do that. Perhaps this behavior
could be configured as a "quirk?"

Appreciate your attention to this.

-Bo

On Thu, May 3, 2018 at 12:09 PM, Mark Thomas <mthomas at cert.org> wrote:

> Bo-
>
> We believe the new behavior to be correct.
>
> The IN_BYTES and IN_PKTS counters are computed when packets enter
> the router, and the OUT_BYTES and OUT_PKTS counters represent values
> when packets leave the router.  One reason those values can be
> different for packets traveling in one direction are WAN optimizers
> that compress/uncompress the data as it leaves/enters the monitored
> network.
>
> Fixbuf 1.8.0 maps the OUT_BYTES and OUT_PKTS values to their IPFIX
> equivalents, postOctetDeltaCount and postPacketDeltaCount.  (The
> OUT_ values are labeled "post" in IPFIX to denote they are values
> computed after the observation point.)
>
> If all interfaces of a border router are monitored, then you can get
> double counting of the data in SiLK:
>
>   * Internet->Local traffic entering the router. incoming IN_BYTES
>   * Internet->Local traffic leaving the router.  incoming OUT_BYTES
>   * Local->Internet traffic entering the router. outgoing IN_BYTES
>   * Local->Internet traffic entering the router. outgoing OUT_BYTES
>
> The changes in SiLK 3.16.0 and Fixbuf 1.8.0 are meant to avoid the
> double counting.
>
> -Mark
>
>
> -----Original Message-----
> From: Bo Bayles <bbayles at obsrvbl.com>
> Date: Wed, 2 May 2018 12:51:17 -0500
> To: <netsa-tools-discuss at cert.org>
> Subject: [netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES
>         introduced in SiLK 3.16.0
>
> I found I started having an issue with processing data from some NetFlow v9
> exporters after upgrading SiLK.
>
> Specifically, rwcut and rwuniq used to properly reflect the bidirectional
> traffic being reported by these exporters, but they don't anymore.
>
> After doing some digging, I found that things changed for the worse in
> version
> 3.16.0. I suspect the relevant Release Note is:
>
> > Change processing of NetFlow v9 records so that, when SiLK is compiled
> > against libfixbuf 1.8.0, the OUT_BYTES and OUT_PKTS values are used when
> the
> > IN_BYTES and IN_PKTS values are 0.
>
> The specific problem occurs when an exporter specifies both traffic
> directions
> in its template. That is, both IN_BYTES and OUT_BYTES (RFC 3954 fields 1
> and
> 23, respectively) / both IN_PKTS and OUT_PKTS (fields 2 and 24).
>
> Given a flow like this, the SiLK tools used to report two rows - one for
> the
> bytes and packets from 192.168.12.205 -> 15.73.97.78, and one for the
> bytes and
> packets from 15.73.97.78 -> 192.168.12.205.
>
> * IP_SRC_ADDR: 192.168.12.205
> * IP_DST_ADDR: 15.73.97.78
> * IN_PKTS: 3987
> * OUT_PKTS: 1807
> * IN_BYTES: 634500
> * OUT_BYTES: 9580
>
> Output when captured by flowcap 3.14 (3.15 is the same):
>
> $ rwcut --fields 1,2,6,7 --no-columns "version_3.14.silk"
> sIP|dIP|packets|bytes|
> 192.168.12.205|15.73.97.78|3987|634500|
> 15.73.97.78|192.168.12.205|1807|9580|
>
> Output when captured by flowcap 3.16.1:
>
> $ rwcut --fields 1,2,6,7 --no-columns "version_3.16.1.silk"
> sIP|dIP|packets|bytes|
> 192.168.12.205|15.73.97.78|3987|634500|
>
> Clearly useful information is being lots in the second version - I hope
> this
> can be fixed in the next release?
>
> I've attached a PCAP with the originating v9 packets, plus the two SiLK
> files.
>
> Many thanks,
> -Bo
>
-------------- next part --------------
HTML attachment scrubbed and removed