[netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES introduced in SiLK 3.16.0

Mark Thomas mthomas at cert.org
Thu May 10 17:40:08 EDT 2018


Bo-

I apologize for the delay in responding.

Yes, RFC 3954 is confusing.  Reading the "OUT_BYTES" and "OUT_PKTS"
labels to mean the "non-incoming traffic" is not unique to Meraki,
since the original author of the code in libfixbuf that supports
NetFlow V9 read them the same way.

Yes, I can add a nf9-post-is-reverse quirk that undoes the effects
of the recent changes in SiLK and libfixbuf.  It should be available
by mid-June at the latest.

-Mark


On Thu, 3 May 2018 16:38:20 -0500, Bo Bayles wrote:

> Mark, thanks for writing.
>
> I'm afraid I disagree - a literal reading RFC 3954 doesn't rule out your
> interpretation, but it doesn't specify it either.
> (This vagueness is, of course, one of the major flaws of that spec!)
>
> The reason why this is a problem for me is that Meraki MX devices use the
> interpretation that old fixbuf used.
> If I'm to continue monitoring them, must I freeze at libfixbuf 1.7.1? I'd
> hate to do that. Perhaps this behavior
> could be configured as a "quirk?"
>
> Appreciate your attention to this.
>
> -Bo
>
> On Thu, May 3, 2018 at 12:09 PM, Mark Thomas <mthomas at cert.org> wrote:
>
>> Bo-
>>
>> We believe the new behavior to be correct.
>>
>> The IN_BYTES and IN_PKTS counters are computed when packets enter
>> the router, and the OUT_BYTES and OUT_PKTS counters represent values
>> when packets leave the router.  One reason those values can be
>> different for packets traveling in one direction are WAN optimizers
>> that compress/uncompress the data as it leaves/enters the monitored
>> network.
>>
>> Fixbuf 1.8.0 maps the OUT_BYTES and OUT_PKTS values to their IPFIX
>> equivalents, postOctetDeltaCount and postPacketDeltaCount.  (The
>> OUT_ values are labeled "post" in IPFIX to denote they are values
>> computed after the observation point.)
>>
>> If all interfaces of a border router are monitored, then you can get
>> double counting of the data in SiLK:
>>
>>   * Internet->Local traffic entering the router. incoming IN_BYTES
>>   * Internet->Local traffic leaving the router.  incoming OUT_BYTES
>>   * Local->Internet traffic entering the router. outgoing IN_BYTES
>>   * Local->Internet traffic entering the router. outgoing OUT_BYTES
>>
>> The changes in SiLK 3.16.0 and Fixbuf 1.8.0 are meant to avoid the
>> double counting.
>>
>> -Mark
>>
>>
>> -----Original Message-----
>> From: Bo Bayles <bbayles at obsrvbl.com>
>> Date: Wed, 2 May 2018 12:51:17 -0500
>> To: <netsa-tools-discuss at cert.org>
>> Subject: [netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES
>>         introduced in SiLK 3.16.0
>>
>> I found I started having an issue with processing data from some NetFlow v9
>> exporters after upgrading SiLK.
>>
>> Specifically, rwcut and rwuniq used to properly reflect the bidirectional
>> traffic being reported by these exporters, but they don't anymore.
>>
>> After doing some digging, I found that things changed for the worse in
>> version
>> 3.16.0. I suspect the relevant Release Note is:
>>
>> > Change processing of NetFlow v9 records so that, when SiLK is compiled
>> > against libfixbuf 1.8.0, the OUT_BYTES and OUT_PKTS values are used when
>> the
>> > IN_BYTES and IN_PKTS values are 0.
>>
>> The specific problem occurs when an exporter specifies both traffic
>> directions
>> in its template. That is, both IN_BYTES and OUT_BYTES (RFC 3954 fields 1
>> and
>> 23, respectively) / both IN_PKTS and OUT_PKTS (fields 2 and 24).
>>
>> Given a flow like this, the SiLK tools used to report two rows - one for
>> the
>> bytes and packets from 192.168.12.205 -> 15.73.97.78, and one for the
>> bytes and
>> packets from 15.73.97.78 -> 192.168.12.205.
>>
>> * IP_SRC_ADDR: 192.168.12.205
>> * IP_DST_ADDR: 15.73.97.78
>> * IN_PKTS: 3987
>> * OUT_PKTS: 1807
>> * IN_BYTES: 634500
>> * OUT_BYTES: 9580
>>
>> Output when captured by flowcap 3.14 (3.15 is the same):
>>
>> $ rwcut --fields 1,2,6,7 --no-columns "version_3.14.silk"
>> sIP|dIP|packets|bytes|
>> 192.168.12.205|15.73.97.78|3987|634500|
>> 15.73.97.78|192.168.12.205|1807|9580|
>>
>> Output when captured by flowcap 3.16.1:
>>
>> $ rwcut --fields 1,2,6,7 --no-columns "version_3.16.1.silk"
>> sIP|dIP|packets|bytes|
>> 192.168.12.205|15.73.97.78|3987|634500|
>>
>> Clearly useful information is being lots in the second version - I hope
>> this
>> can be fixed in the next release?
>>
>> I've attached a PCAP with the originating v9 packets, plus the two SiLK
>> files.
>>
>> Many thanks,
>> -Bo
>>


More information about the netsa-tools-discuss mailing list