[netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES introduced in SiLK 3.16.0

[Bo Bayles]

Mark, many thanks for this! I'll look forward to testing the new version.

-Bo

On Thu, May 10, 2018 at 4:40 PM, Mark Thomas <mthomas at cert.org> wrote:

> Bo-
>
> I apologize for the delay in responding.
>
> Yes, RFC 3954 is confusing.  Reading the "OUT_BYTES" and "OUT_PKTS"
> labels to mean the "non-incoming traffic" is not unique to Meraki,
> since the original author of the code in libfixbuf that supports
> NetFlow V9 read them the same way.
>
> Yes, I can add a nf9-post-is-reverse quirk that undoes the effects
> of the recent changes in SiLK and libfixbuf.  It should be available
> by mid-June at the latest.
>
> -Mark
>
>
> On Thu, 3 May 2018 16:38:20 -0500, Bo Bayles wrote:
>
> > Mark, thanks for writing.
> >
> > I'm afraid I disagree - a literal reading RFC 3954 doesn't rule out your
> > interpretation, but it doesn't specify it either.
> > (This vagueness is, of course, one of the major flaws of that spec!)
> >
> > The reason why this is a problem for me is that Meraki MX devices use the
> > interpretation that old fixbuf used.
> > If I'm to continue monitoring them, must I freeze at libfixbuf 1.7.1? I'd
> > hate to do that. Perhaps this behavior
> > could be configured as a "quirk?"
> >
> > Appreciate your attention to this.
> >
> > -Bo
> >
> > On Thu, May 3, 2018 at 12:09 PM, Mark Thomas <mthomas at cert.org> wrote:
> >
> >> Bo-
> >>
> >> We believe the new behavior to be correct.
> >>
> >> The IN_BYTES and IN_PKTS counters are computed when packets enter
> >> the router, and the OUT_BYTES and OUT_PKTS counters represent values
> >> when packets leave the router.  One reason those values can be
> >> different for packets traveling in one direction are WAN optimizers
> >> that compress/uncompress the data as it leaves/enters the monitored
> >> network.
> >>
> >> Fixbuf 1.8.0 maps the OUT_BYTES and OUT_PKTS values to their IPFIX
> >> equivalents, postOctetDeltaCount and postPacketDeltaCount.  (The
> >> OUT_ values are labeled "post" in IPFIX to denote they are values
> >> computed after the observation point.)
> >>
> >> If all interfaces of a border router are monitored, then you can get
> >> double counting of the data in SiLK:
> >>
> >>   * Internet->Local traffic entering the router. incoming IN_BYTES
> >>   * Internet->Local traffic leaving the router.  incoming OUT_BYTES
> >>   * Local->Internet traffic entering the router. outgoing IN_BYTES
> >>   * Local->Internet traffic entering the router. outgoing OUT_BYTES
> >>
> >> The changes in SiLK 3.16.0 and Fixbuf 1.8.0 are meant to avoid the
> >> double counting.
> >>
> >> -Mark
> >>
> >>
> >> -----Original Message-----
> >> From: Bo Bayles <bbayles at obsrvbl.com>
> >> Date: Wed, 2 May 2018 12:51:17 -0500
> >> To: <netsa-tools-discuss at cert.org>
> >> Subject: [netsa-tools-discuss] Problem with IN_BYTES and OUT_BYTES
> >>         introduced in SiLK 3.16.0
> >>
> >> I found I started having an issue with processing data from some
> NetFlow v9
> >> exporters after upgrading SiLK.
> >>
> >> Specifically, rwcut and rwuniq used to properly reflect the
> bidirectional
> >> traffic being reported by these exporters, but they don't anymore.
> >>
> >> After doing some digging, I found that things changed for the worse in
> >> version
> >> 3.16.0. I suspect the relevant Release Note is:
> >>
> >> > Change processing of NetFlow v9 records so that, when SiLK is compiled
> >> > against libfixbuf 1.8.0, the OUT_BYTES and OUT_PKTS values are used
> when
> >> the
> >> > IN_BYTES and IN_PKTS values are 0.
> >>
> >> The specific problem occurs when an exporter specifies both traffic
> >> directions
> >> in its template. That is, both IN_BYTES and OUT_BYTES (RFC 3954 fields 1
> >> and
> >> 23, respectively) / both IN_PKTS and OUT_PKTS (fields 2 and 24).
> >>
> >> Given a flow like this, the SiLK tools used to report two rows - one for
> >> the
> >> bytes and packets from 192.168.12.205 -> 15.73.97.78, and one for the
> >> bytes and
> >> packets from 15.73.97.78 -> 192.168.12.205.
> >>
> >> * IP_SRC_ADDR: 192.168.12.205
> >> * IP_DST_ADDR: 15.73.97.78
> >> * IN_PKTS: 3987
> >> * OUT_PKTS: 1807
> >> * IN_BYTES: 634500
> >> * OUT_BYTES: 9580
> >>
> >> Output when captured by flowcap 3.14 (3.15 is the same):
> >>
> >> $ rwcut --fields 1,2,6,7 --no-columns "version_3.14.silk"
> >> sIP|dIP|packets|bytes|
> >> 192.168.12.205|15.73.97.78|3987|634500|
> >> 15.73.97.78|192.168.12.205|1807|9580|
> >>
> >> Output when captured by flowcap 3.16.1:
> >>
> >> $ rwcut --fields 1,2,6,7 --no-columns "version_3.16.1.silk"
> >> sIP|dIP|packets|bytes|
> >> 192.168.12.205|15.73.97.78|3987|634500|
> >>
> >> Clearly useful information is being lots in the second version - I hope
> >> this
> >> can be fixed in the next release?
> >>
> >> I've attached a PCAP with the originating v9 packets, plus the two SiLK
> >> files.
> >>
> >> Many thanks,
> >> -Bo
> >>
>
-------------- next part --------------
HTML attachment scrubbed and removed