[netsa-tools-discuss] SiLK Plugins

[B Galliart]

Speaking of plugins, it would be nice if flowcap had a direct plugin system.

This would be something along the lines of a shared-object file specified
in the sensor.conf.  Flowcap would then use dlopen() to use the plugin.

The plugin (shared-object file) would then provide two functions:

(1) an initialization function called around startReaders() which returns a
pointer to the plugin's own data structure

(2) a plugin replacement for the readerWriteRecord() which would be given
the flowcap_reader_t, rwRec and the pointer from it's own initialization
function

The plugin SDK header files would then provide everything needed to take
advantage of the flowcap_reader_t and rwRec structure.

It seems to me that having to depend on the file system, even if you use a
SSD or RAM disk, can greatly limit applications that depend on near
realtime performance.

There is also a steep learning curve to libfixbuf which shouldn't be the
case for a flowcap plugin SDK.  I am not looking to have anything changed
about libfixbuf but from the perspective of having used libpcap, it seems
like libfixbuf is at an abstraction layers which results in a magnitude
much more complex API.

Instead, I think a plugin system to flowcap provides a much easier to learn
abstraction layer where the programmer only needs to really learn the
libsilk concept of rwRec to proceed.

Using such a plugin style system where I am feeding flowcap about 75Mbps of
IPFIX data for an initial benchmark, I am processing over 100,000 flow per
second (which represent 2.8 million packets per second or 11Gbps of network
traffic).

It would also be nice for purposes of benchmarking to be able to
approximate number of dropped flows but fbSessionGetSequence() is currently
a private libfixbuf function.  The log warnings aren't helpful because they
include out of sequence events (and I have also suppressed them being
logged to improve performance).  Instead, knowing the raw sequence numbers
should allow for getting an expected total and comparing it with the actual
total.  However, modifying libfixbuf and libflowsource to expose the
sequence numbers at the flowcap layer goes beyond the scope of my project.

Anyways, thanks to all the CERT NetSA developers and I am really looking
forward to seeing what SiLK 4 will provide!

On Mon, Aug 17, 2015 at 1:28 PM, Mark Thomas <mthomas at cert.org> wrote:

> I think the short answer to your question is no, we do not have any
> publicly-available, ready-made scripts that will use a SiLK data
> repository to answer these sorts of questions.
>
> The Analysis Pipeline
> <http://tools.netsa.cert.org/analysis-pipeline/index.html> is a
> separate tool that is designed to process SiLK data as it arrives
> and send alerts for unusual conditions (such as receiving traffic
> from an IP on a watchlist or processing data volume over a certain
> limit).
>
> You may find some analysis techniques mentioned in documents on the
> SEI web site <http://resources.sei.cmu.edu/library/results.cfm>.
> Unfortunately, often times you will that that the technique is
> mentioned but the details about how it works are not available.
>
> I hope that helps.
>
> -Mark
>
>
> -----Original Message-----
> From: Hosam Hittini <hosam.hittini at ies.etisalat.ae>
> Date: Sun, 16 Aug 2015 07:40:02 +0400
> To: 'Mark Thomas' <mthomas at cert.org>
> Cc: <netsa-help at cert.org>, <netsa-tools-discuss at cert.org>, 'Majid Qureshi'
>         <mmajid at ies.etisalat.ae>
> Subject: RE: [netsa-tools-discuss] SiLK Plugins
>
> Dear Mr. Mark,
>
> Thank you, you got what I mean
> But what I am looking for is plugins to detect spoofing for example, or
> DDoS
> attacks
> Are you aware of such developed plugins?
>
> Regards,
> Hosam Hittini
> System Security, Security Operations Center
> Etisalat
> -----Original Message-----
> From: Mark Thomas [mailto:mthomas at cert.org]
> Sent: Tuesday, August 11, 2015 7:58 PM
> To: Hosam Hittini <hosam.hittini at ies.etisalat.ae>
> Cc: netsa-help at cert.org; netsa-tools-discuss at cert.org; 'Majid Qureshi'
> <mmajid at ies.etisalat.ae>
> Subject: Re: [netsa-tools-discuss] SiLK Plugins
>
> A list of C plug-ins that may be used with the SiLK analysis tools is
> documented at http://tools.netsa.cert.org/silk/docs.html#analysis-plugins
>
> The silk/src/plugins directory contains some additional plug-ins for use in
> the analysis tools which may serve as example code for building your own.
> See the silk-plugin[1] manual page for more information on creating
> plug-ins
> from C.
>
> See the silkpython[2] manual page for building plug-ins from python.
>
> There are two plug-ins that work with rwflowpack.  Their source code is
> under the silk/site directory, and their manual pages are
> packlogic-twoway[3] and packlogic-generic[4].
>
> If I have misunderstood what you mean my plugins, I am sorry and I ask that
> you clarify what you mean.
>
> Regards,
>
> -Mark
>
> [1]http://tools.netsa.cert.org/silk/silk-plugin.html
> [2]http://tools.netsa.cert.org/silk/silkpython.html
> [3]http://tools.netsa.cert.org/silk/packlogic-twoway.html
> [4[http://tools.netsa.cert.org/silk/packlogic-generic.html
>
>
> -----Original Message-----
> From: Hosam Hittini <hosam.hittini at ies.etisalat.ae>
> Date: Tue, 11 Aug 2015 12:29:59 +0400
> To: <netsa-help at cert.org>, <netsa-tools-discuss at cert.org>
> Cc: 'Majid Qureshi' <mmajid at ies.etisalat.ae>
> Subject: [netsa-tools-discuss] SiLK Plugins
>
> Hi,
>
>
>
> I wonder if you can provide me with a list of plugins that were developed
> for SiLK along with their documentation
>
> Thank you
>
>
>
>
>
>
>
-------------- next part --------------
HTML attachment scrubbed and removed