[netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES introduced in SiLK 3.16.0

Mark Thomas mthomas at cert.org
Thu Jun 28 17:18:23 EDT 2018


Bo-

A new version of SiLK with this feature has been released.
https://tools.netsa.cert.org/silk/download.html#release-3.17.2

The quirk is called nf9-out-is-reverse.  Please let me/us know if
you run into any issues with it.

Cheers,

-Mark


On Thu, 10 May 2018 17:40:08 -0400, Mark Thomas wrote:

> Bo-
>
> I apologize for the delay in responding.
>
> Yes, RFC 3954 is confusing.  Reading the "OUT_BYTES" and "OUT_PKTS"
> labels to mean the "non-incoming traffic" is not unique to Meraki,
> since the original author of the code in libfixbuf that supports
> NetFlow V9 read them the same way.
>
> Yes, I can add a nf9-post-is-reverse quirk that undoes the effects
> of the recent changes in SiLK and libfixbuf.  It should be available
> by mid-June at the latest.
>
> -Mark
>
>
> On Thu, 3 May 2018 16:38:20 -0500, Bo Bayles wrote:
>
>> Mark, thanks for writing.
>>
>> I'm afraid I disagree - a literal reading RFC 3954 doesn't rule out your
>> interpretation, but it doesn't specify it either.
>> (This vagueness is, of course, one of the major flaws of that spec!)
>>
>> The reason why this is a problem for me is that Meraki MX devices use the
>> interpretation that old fixbuf used.
>> If I'm to continue monitoring them, must I freeze at libfixbuf 1.7.1? I'd
>> hate to do that. Perhaps this behavior
>> could be configured as a "quirk?"
>>
>> Appreciate your attention to this.
>>
>> -Bo
>>
>> On Thu, May 3, 2018 at 12:09 PM, Mark Thomas <mthomas at cert.org> wrote:
>>
>>> Bo-
>>>
>>> We believe the new behavior to be correct.
>>>
>>> The IN_BYTES and IN_PKTS counters are computed when packets enter
>>> the router, and the OUT_BYTES and OUT_PKTS counters represent values
>>> when packets leave the router.  One reason those values can be
>>> different for packets traveling in one direction are WAN optimizers
>>> that compress/uncompress the data as it leaves/enters the monitored
>>> network.
>>>
>>> Fixbuf 1.8.0 maps the OUT_BYTES and OUT_PKTS values to their IPFIX
>>> equivalents, postOctetDeltaCount and postPacketDeltaCount.  (The
>>> OUT_ values are labeled "post" in IPFIX to denote they are values
>>> computed after the observation point.)
>>>
>>> If all interfaces of a border router are monitored, then you can get
>>> double counting of the data in SiLK:
>>>
>>>   * Internet->Local traffic entering the router. incoming IN_BYTES
>>>   * Internet->Local traffic leaving the router.  incoming OUT_BYTES
>>>   * Local->Internet traffic entering the router. outgoing IN_BYTES
>>>   * Local->Internet traffic entering the router. outgoing OUT_BYTES
>>>
>>> The changes in SiLK 3.16.0 and Fixbuf 1.8.0 are meant to avoid the
>>> double counting.
>>>
>>> -Mark
>>>
>>>
>>> -----Original Message-----
>>> From: Bo Bayles <bbayles at obsrvbl.com>
>>> Date: Wed, 2 May 2018 12:51:17 -0500
>>> To: <netsa-tools-discuss at cert.org>
>>> Subject: [netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES
>>>         introduced in SiLK 3.16.0
>>>
>>> I found I started having an issue with processing data from some NetFlow v9
>>> exporters after upgrading SiLK.
>>>
>>> Specifically, rwcut and rwuniq used to properly reflect the bidirectional
>>> traffic being reported by these exporters, but they don't anymore.
>>>
>>> After doing some digging, I found that things changed for the worse in
>>> version
>>> 3.16.0. I suspect the relevant Release Note is:
>>>
>>> > Change processing of NetFlow v9 records so that, when SiLK is compiled
>>> > against libfixbuf 1.8.0, the OUT_BYTES and OUT_PKTS values are used when
>>> the
>>> > IN_BYTES and IN_PKTS values are 0.
>>>
>>> The specific problem occurs when an exporter specifies both traffic
>>> directions
>>> in its template. That is, both IN_BYTES and OUT_BYTES (RFC 3954 fields 1
>>> and
>>> 23, respectively) / both IN_PKTS and OUT_PKTS (fields 2 and 24).
>>>
>>> Given a flow like this, the SiLK tools used to report two rows - one for
>>> the
>>> bytes and packets from 192.168.12.205 -> 15.73.97.78, and one for the
>>> bytes and
>>> packets from 15.73.97.78 -> 192.168.12.205.
>>>
>>> * IP_SRC_ADDR: 192.168.12.205
>>> * IP_DST_ADDR: 15.73.97.78
>>> * IN_PKTS: 3987
>>> * OUT_PKTS: 1807
>>> * IN_BYTES: 634500
>>> * OUT_BYTES: 9580
>>>
>>> Output when captured by flowcap 3.14 (3.15 is the same):
>>>
>>> $ rwcut --fields 1,2,6,7 --no-columns "version_3.14.silk"
>>> sIP|dIP|packets|bytes|
>>> 192.168.12.205|15.73.97.78|3987|634500|
>>> 15.73.97.78|192.168.12.205|1807|9580|
>>>
>>> Output when captured by flowcap 3.16.1:
>>>
>>> $ rwcut --fields 1,2,6,7 --no-columns "version_3.16.1.silk"
>>> sIP|dIP|packets|bytes|
>>> 192.168.12.205|15.73.97.78|3987|634500|
>>>
>>> Clearly useful information is being lots in the second version - I hope
>>> this
>>> can be fixed in the next release?
>>>
>>> I've attached a PCAP with the originating v9 packets, plus the two SiLK
>>> files.
>>>
>>> Many thanks,
>>> -Bo
>>>


More information about the netsa-tools-discuss mailing list