[netsa-tools-discuss] Fwd: silk
Mark Thomas
mthomas at cert.org
Fri Dec 15 15:50:56 EST 2017
Daniel-
You have diagnosed the issue correctly. SiLK should be using the
NetFlow processing code instead of the YAF code for your data.
When determining whether the YAF code may be used, the library only
checks for information elements that prevent the YAF template from
being used. What the library needs to do in addition is to ensure
that a minimum set of elements is present.
I will make this change for an future SiLK release.
What you can do in the meantime is to edit the code so the YAF block
is never invoked. Change the if() statement that determines whether
to use YAF from
if (0 == (bmap & ~TMPL_MASK_YAFREC)
&& (bmap & TMPL_MASK_IPADDRESS)
&& ((bmap & TMPL_MASK_VOLUME_YAF) == TMPL_BIT_octetDeltaCount
|| (bmap & TMPL_MASK_VOLUME_YAF) == TMPL_BIT_octetTotalCount))
to
if (0)
That will make the library make the NetFlow v9 check, which should
allow your data to be processed.
Thank you for bringing this issue to our attention.
-Mark
-----Original Message-----
From: Daniel Hermans <daniel.hermans at gmail.com>
Date: Thu, 14 Dec 2017 08:26:11 +1100
To: Mark Thomas <mthomas at cert.org>
Cc: <netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] Fwd: silk
Also - am curious why the flow is being processed by ski_yafrec_next rather
than ski_nf9rec_next (which has the quirks logic)?
or is that the normal flow?
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101 [0x7f98ec0014c0] skiTemplateCallback()
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101, bmap 0x0000008000, IE sourceIPv4Address (0/8)
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101, bmap 0x0000008000, IE destinationIPv4Address (0/12)
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101, bmap 0x0000008000, IE ingressInterface (0/10)
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101, bmap 0x0000008000, IE egressInterface (0/14)
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101, bmap 0x0000008000, IE sourceTransportPort (0/7)
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101, bmap 0x0000008000, IE destinationTransportPort (0/11
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101, bmap 0x0000008000, IE protocolIdentifier (0/4)
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101, bmap 0x0100008000, IE octetDeltaCount (0/1)
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Will process template
0x0101 with the YAF template
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101 [0x7f98ec0014c0], bmap 0x0000712c, written
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: 'ME1A': conn =
0x7f98ec002a00, source = 0x2502080, fbuf = 0x2561f50
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101 [0x7f98ec0014c0], bmap 0x0000712c, read by ski_yafrec
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Found zero bytes or
packets; byte=344, pkt=0, rev_byte=0, rev_pkt=0
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]:
IGNORED|10.39.252.91|10.39.132.10|51702|25|6|0|344|byte or packet count is
zero|
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Domain 0X0001,
TemplateID 0X0101 [0x7f98ec0014c0], bmap 0x0000712c, read by ski_yafrec
Dec 14 08:17:08 ip-10-39-128-243 rwflowpack[9199]: Found zero bytes or
packets; byte=7173, pkt=0, rev_byte=0, rev_pkt=0
Thanks!
On Thu, Dec 14, 2017 at 8:14 AM, Daniel Hermans <daniel.hermans at gmail.com>
wrote:
> Hi,
> Thanks for prompt reply.
> Apologies for not including my config files but yes I am using the
> zero-packets quirk:
>
> probe ME1A netflow-v9
> listen-on-port 2055
> listen-as-host 10.39.32.50
> protocol udp
> quirks zero-packets
> end probe
>
> group local-networks
> ipblocks 10.0.0.0/8
> end group
>
> sensor ME1A
> netflow-v9-probes ME1A
> internal-ipblocks @local-networks
> external-ipblocks remainder
> end sensor
>
> i recompiled with TRACE active - i just get a new message:
> Found zero bytes or packets; byte=344, pkt=0, rev_byte=0, rev_pkt=0
>
> inexpertly reading the code - i see the quirk is processed in this if
> block:
> } else if (!(record->bmap & NF9REC_INITIATOR)) }
>
> which it never enters for my data
>
>
> So as you suggest i'm doing something wrong in my config for it not to
> process the quirk
> Any ideas appreciated!
>
> On Thu, Dec 14, 2017 at 6:50 AM, Mark Thomas <mthomas at cert.org> wrote:
>
>> Daniel-
>>
>> Thank you for using the NetSA tools.
>>
>> SiLK is ignoring your data because the template used by your router
>> does not include the number of packets in the flow. The template
>> include the octetDeltaCount but not a corresponding
>> packetDeltaCount.
>>
>> There is a work-around for this issue in SiLK, which requires that
>> you edit the sensor.conf file that rwflowpack is using. In the
>> probe block for your router, add the following:
>>
>> quirks zero-packets
>>
>> This causes rwflowpack to use a packet count of 1 for all incoming
>> flow records that do not have their own packet count, and that
>> should allow rwflowpack to store the records.
>>
>> I hope that solves your issue.
>>
>> -Mark
>>
>>
>> -----Original Message-----
>> From: Daniel Hermans <daniel.hermans at gmail.com>
>> Date: Wed, 13 Dec 2017 09:51:05 +1100
>> To: <netsa-tools-discuss at cert.org>
>> Subject: [netsa-tools-discuss] Fwd: silk
>>
>> Hi SiLK team,
>>
>> I'm trying to get SILK to ingest netflow v9 flows and having some issues
>> as
>> can be seen by the logs below (with the template shown):
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Will process template
>> 0x0101 with the YAF template
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Contains 8 Elements, Enabled by
>> SILK_IPFIX_PRINT_TEMPLATES
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Position 0, Length 4, IE 8, Name
>> sourceIPv4Address
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Position 1, Length 4, IE 12, Name
>> destinationIPv4Address
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Position 2, Length 4, IE 10, Name
>> ingressInterface
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Position 3, Length 4, IE 14, Name
>> egressInterface
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Position 4, Length 2, IE 7, Name
>> sourceTransportPort
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Position 5, Length 2, IE 11, Name
>> destinationTransportPort
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Position 6, Length 1, IE 4, Name
>> protocolIdentifier
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]: Domain 0X0001,
>> TemplateID 0X0101, Position 7, Length 4, IE 1, Name
>> octetDeltaCount
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.252.91|10.39.132.10|51702|25|6|0|344|byte or packet count
>> is
>> zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.39.221|10.11.40.143|8089|26286|6|0|7173|byte or packet
>> count
>> is zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.133.138|10.39.248.145|25|16976|6|0|566|byte or packet count
>> is zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.181.67|10.39.145.255|31055|443|6|0|2107|byte or packet
>> count
>> is zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.39.221|10.40.29.69|8089|62069|6|0|6970|byte or packet count
>> is zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.38.4|10.11.40.233|8089|29179|6|0|6021|byte or packet count
>> is
>> zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.39.46|10.108.10.84|58492|80|6|0|551|byte or packet count is
>> zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.39.120|10.40.28.248|9997|50238|6|0|12292|byte or packet
>> count
>> is zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.38.4|10.11.40.40|8089|33242|6|0|6125|byte or packet count
>> is
>> zero|
>>
>> Dec 12 14:47:27 ip-10-39-128-243 rwflowpack[4370]:
>> IGNORED|10.39.132.10|10.39.252.91|25|51702|6|0|566|byte or packet count
>> is
>> zero|
>>
>>
>>
>>
>>
>> So all messages are ignored. A typical flow record as seen in wireshark
>> looks like:
>>
>> Flow 1
>>
>> SrcAddr: 10.39.154.46
>>
>> DstAddr: 10.108.162.144
>>
>> InputInt: 133
>>
>> OutputInt: 0
>>
>> SrcPort: 443
>>
>> DstPort: 51769
>>
>> Protocol: TCP (6)
>>
>> Octets: 83026
>>
>>
>>
>>
>> Any thoughts as to what I'm doing wrong?
>>
>> I don't actually know the model number of Cisco device(s) sending the flow
>> ( the flows are sent from a service provider ) but i believe it's a Nexus
>> 9K
>>
>>
>> My sensor and silk configs are small and simple - 1 sensor, 1 v9 flow
>> etc..
>>
>>
>> I'm rusty on C but tried to enable TRACE mode based on a post I saw on the
>> archive list.
>>
>>
>> Didn't get too far with that either, as when I compile libfixbuf-1.8.0 it
>> installs the shared object /usr/local/lib/libfixbuf.so.3.2.0 rather than
>> /usr/local/lib/libfixbuf.so.1.8.0
>>
>>
>> The silk-3.16.0 configure then does not find the shared object and
>> doesn't
>> enable netflow support ( fails the version check )
>>
>>
>>
>>
>> Thanks for any help!
>>
>
>
More information about the netsa-tools-discuss
mailing list