[netsa-tools-discuss] SiLK: Extending rwRec stucture from extended template

[Mark Thomas]

Regarding the logging:  Unfortunately, the "Missing external
template" and "Message out of sequence" warnings from libfixbuf are
generated by g_warning() which is different than g_set_error().

Setting the SILK_LIBFIXBUF_SUPPRESS_WARNINGS environment variable to
1 before running rwflowpack and flowcap suppresses all
g_warning()-generated messages.  When that environment variable is
set, the SiLK code calls g_log_set_handler() with a handler that
does nothing.

-Mark


On Thu, 30 Apr 2015 16:10:27 -0500, B. Galliart wrote:

> Thanks for responding so quickly.
>
> The NextHop field does not fit this project as well as ASNs do.   However,
> it looks like it is already provided in the existing rwRec including API
> functions of rwRecGetNhIPv4, rwRecMemGetNhIPv4 and rwRecMemGetNhIPv6.
> Currently there doesn't seem to be anything similar to that for
> bgpSourceAsNumber and bgpDestionationAsNumber in rwRec.  I am just looking
> for the best way to add them in.
>
> It is great news that SiLK v4 will be based more on a template system
> rather than a fixed rwRec.  I will still need advice on extending SiLK v3.
> While this project can't wait for the SiLK v4 release, I would be
> interested in knowing where I can check it's status (pre-alpha, alpha,
> beta, etc.) and if there is a way to get to a pre-release copy?  Is there a
> public git yet despite it not being release ready?  I really look forward
> with the direction that SiLK v4 takes things.
>
> Also, thanks for the advice on catching the g_set_error() calls with
> g_log_set_handler(), I will look into it.
>
> Thanks
>
>
> On Thu, Apr 30, 2015 at 3:05 PM, Chris Inacio <inacio at cert.org> wrote:
>
>>
>>
>>
>> > On Apr 28, 2015, at 10:51 PM, B Galliart <bgallia at gmail.com> wrote:
>> >
>> > When I enable SILK_IPFIX_PRINT_TEMPLATES, it shows that my IPFIX
>> exporter is providing bgpSourceAsNumber and bgpDestinationAsNumber but it
>> appears these values are then discarded.
>> >
>> > Anyone have any tips, advice or guide of what needs to be done to extend
>> SiLK to support these?
>> > From what I can tell, I will need to extend the rwRec structure to
>> include these?  Based on how I read the code, at the very least, it looks
>> like I need to modify libflowsource/skipfix.c and libsilk/rwrec.[ch].
>> Then, it seems, I need to update the utilities that I want to act on the
>> new rwRec methods.  But before I get entrenched in doing all of that, I
>> want to make sure I wasn't missing a simpler solution.
>> >
>> >
>> > I also had a feature request: Could the libfixbuf API be updated to
>> provide options to suppress specific error messages without having to
>> recompile the library?  For example, if I expect to get an external
>> template from UDP at least once every 60 seconds then I would like to be
>> able to call an API function which tells libfixbuf to not generate "Missing
>> external template" warnings for the first 120 seconds of each new session.
>> It would also be nice to have the flexibility to suppress "IPFIX Message
>> out of sequence" message without recompiling the library.  Possibly with an
>> API function which provides a callback function for error message
>> handling?  If the callback is never provided, the library continues to
>> operate as it does today but otherwise the callback gets to decide how to
>> handle the warnings.  Ultimately, I would like to be able to leverage the
>> CERT provided RPM repo without have to modify the spec file and recompile
>> each time there is an update.
>> >
>> >
>> > By the way, thanks for the great IPFIX library and utility suite!
>> >
>>
>>
>>
>> Thanks for your interest.  I’m sure the primary developer will be happy to
>> hear that you appreciate her work.  I’m going to try to handle these
>> questions, but I’m sure other members of the team will jump in and correct
>> my mistakes.
>>
>> I’ll let Mark or Michael jump in to extending SiLK.  They might also be
>> able to advise on how to use the NextHop field too.  But I’ll also comment
>> that we’re working on getting SiLK v4 publicly released, which allows IPFIX
>> format to be used on disk.  It might still have a bit of ways to make full
>> use of that, but we’re working on that.
>>
>> The error reporting/logging in YAF is implemented using GLIB2’s message
>> logging mechanism.  As such, every message has a domain and a code which
>> could be filtered.  You can also register a function (see
>> g_log_set_handler()) to handle the messages and deal with them however you
>> see fit.  See
>> https://developer.gnome.org/glib/2.44/glib-Message-Logging.html#g-warning
>> for information about implementing the various function calls.  The catalog
>> of message codes is in the <fixbuf/public.h> header file.  The domain for
>> the error codes is held in the constant FB_ERROR_DOMAIN, to find fixbuf
>> events.  The message codes specify an area of functionality in fixbuf
>> (really, its used as a group, not a code, per se) and for the NetFlow
>> translation system those are in the constant FB_ERROR_NETFLOWV9.  In
>> retrospect, we should have been a lot smarter and assigned everything a
>> unique code; in general, the code base does string comparisons against the
>> messages to suppress things internally.
>>
>> Regards,
>> --
>> Chris Inacio
>> inacio at cert.org
>>