[netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES introduced in SiLK 3.16.0

[Bo Bayles]

Many thanks! I will test this out and report back if I find any issues.

-Bo

On Thu, Jun 28, 2018 at 4:18 PM Mark Thomas <mthomas at cert.org> wrote:

> Bo-
>
> A new version of SiLK with this feature has been released.
> https://tools.netsa.cert.org/silk/download.html#release-3.17.2
>
> The quirk is called nf9-out-is-reverse.  Please let me/us know if
> you run into any issues with it.
>
> Cheers,
>
> -Mark
>
>
> On Thu, 10 May 2018 17:40:08 -0400, Mark Thomas wrote:
>
> > Bo-
> >
> > I apologize for the delay in responding.
> >
> > Yes, RFC 3954 is confusing.  Reading the "OUT_BYTES" and "OUT_PKTS"
> > labels to mean the "non-incoming traffic" is not unique to Meraki,
> > since the original author of the code in libfixbuf that supports
> > NetFlow V9 read them the same way.
> >
> > Yes, I can add a nf9-post-is-reverse quirk that undoes the effects
> > of the recent changes in SiLK and libfixbuf.  It should be available
> > by mid-June at the latest.
> >
> > -Mark
> >
> >
> > On Thu, 3 May 2018 16:38:20 -0500, Bo Bayles wrote:
> >
> >> Mark, thanks for writing.
> >>
> >> I'm afraid I disagree - a literal reading RFC 3954 doesn't rule out your
> >> interpretation, but it doesn't specify it either.
> >> (This vagueness is, of course, one of the major flaws of that spec!)
> >>
> >> The reason why this is a problem for me is that Meraki MX devices use
> the
> >> interpretation that old fixbuf used.
> >> If I'm to continue monitoring them, must I freeze at libfixbuf 1.7.1?
> I'd
> >> hate to do that. Perhaps this behavior
> >> could be configured as a "quirk?"
> >>
> >> Appreciate your attention to this.
> >>
> >> -Bo
> >>
> >> On Thu, May 3, 2018 at 12:09 PM, Mark Thomas <mthomas at cert.org> wrote:
> >>
> >>> Bo-
> >>>
> >>> We believe the new behavior to be correct.
> >>>
> >>> The IN_BYTES and IN_PKTS counters are computed when packets enter
> >>> the router, and the OUT_BYTES and OUT_PKTS counters represent values
> >>> when packets leave the router.  One reason those values can be
> >>> different for packets traveling in one direction are WAN optimizers
> >>> that compress/uncompress the data as it leaves/enters the monitored
> >>> network.
> >>>
> >>> Fixbuf 1.8.0 maps the OUT_BYTES and OUT_PKTS values to their IPFIX
> >>> equivalents, postOctetDeltaCount and postPacketDeltaCount.  (The
> >>> OUT_ values are labeled "post" in IPFIX to denote they are values
> >>> computed after the observation point.)
> >>>
> >>> If all interfaces of a border router are monitored, then you can get
> >>> double counting of the data in SiLK:
> >>>
> >>>   * Internet->Local traffic entering the router. incoming IN_BYTES
> >>>   * Internet->Local traffic leaving the router.  incoming OUT_BYTES
> >>>   * Local->Internet traffic entering the router. outgoing IN_BYTES
> >>>   * Local->Internet traffic entering the router. outgoing OUT_BYTES
> >>>
> >>> The changes in SiLK 3.16.0 and Fixbuf 1.8.0 are meant to avoid the
> >>> double counting.
> >>>
> >>> -Mark
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Bo Bayles <bbayles at obsrvbl.com>
> >>> Date: Wed, 2 May 2018 12:51:17 -0500
> >>> To: <netsa-tools-discuss at cert.org>
> >>> Subject: [netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES
> >>>         introduced in SiLK 3.16.0
> >>>
> >>> I found I started having an issue with processing data from some
> NetFlow v9
> >>> exporters after upgrading SiLK.
> >>>
> >>> Specifically, rwcut and rwuniq used to properly reflect the
> bidirectional
> >>> traffic being reported by these exporters, but they don't anymore.
> >>>
> >>> After doing some digging, I found that things changed for the worse in
> >>> version
> >>> 3.16.0. I suspect the relevant Release Note is:
> >>>
> >>> > Change processing of NetFlow v9 records so that, when SiLK is
> compiled
> >>> > against libfixbuf 1.8.0, the OUT_BYTES and OUT_PKTS values are used
> when
> >>> the
> >>> > IN_BYTES and IN_PKTS values are 0.
> >>>
> >>> The specific problem occurs when an exporter specifies both traffic
> >>> directions
> >>> in its template. That is, both IN_BYTES and OUT_BYTES (RFC 3954 fields
> 1
> >>> and
> >>> 23, respectively) / both IN_PKTS and OUT_PKTS (fields 2 and 24).
> >>>
> >>> Given a flow like this, the SiLK tools used to report two rows - one
> for
> >>> the
> >>> bytes and packets from 192.168.12.205 -> 15.73.97.78, and one for the
> >>> bytes and
> >>> packets from 15.73.97.78 -> 192.168.12.205.
> >>>
> >>> * IP_SRC_ADDR: 192.168.12.205
> >>> * IP_DST_ADDR: 15.73.97.78
> >>> * IN_PKTS: 3987
> >>> * OUT_PKTS: 1807
> >>> * IN_BYTES: 634500
> >>> * OUT_BYTES: 9580
> >>>
> >>> Output when captured by flowcap 3.14 (3.15 is the same):
> >>>
> >>> $ rwcut --fields 1,2,6,7 --no-columns "version_3.14.silk"
> >>> sIP|dIP|packets|bytes|
> >>> 192.168.12.205|15.73.97.78|3987|634500|
> >>> 15.73.97.78|192.168.12.205|1807|9580|
> >>>
> >>> Output when captured by flowcap 3.16.1:
> >>>
> >>> $ rwcut --fields 1,2,6,7 --no-columns "version_3.16.1.silk"
> >>> sIP|dIP|packets|bytes|
> >>> 192.168.12.205|15.73.97.78|3987|634500|
> >>>
> >>> Clearly useful information is being lots in the second version - I hope
> >>> this
> >>> can be fixed in the next release?
> >>>
> >>> I've attached a PCAP with the originating v9 packets, plus the two SiLK
> >>> files.
> >>>
> >>> Many thanks,
> >>> -Bo
> >>>
>
-------------- next part --------------
HTML attachment scrubbed and removed