[netsa-tools-discuss] Analysis Pipeline

[Daniel J Ruef]

The filter and evals get placed in a configuration file that is specified on the command line by: --configuration-file

Json alerts can go into elasticSearch for viewing with Kibana. LogStash can be used to process the Json alerts into a format for elasticSearch. We have played with it a little, but haven't produced a formal product to handle it.

Dan

-----Original Message-----
From: chris frazier <frazrcc19 at yahoo.com> 
Sent: Wednesday, June 24, 2020 1:11 AM
To: Daniel J Ruef <druef at cert.org>
Cc: Chris Frazier <chris200712 at icloud.com>; netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] Analysis Pipeline

Is it possible to send json alert data to Kibana

Sent from my iPhone

> On 24 Jun 2020, at 06:54, chris frazier <frazrcc19 at yahoo.com> wrote:
> 
> Where do the filters and evals get placed?
> 
> Sent from my iPhone
> 
>> On 24 Jun 2020, at 06:47, chris frazier <frazrcc19 at yahoo.com> wrote:
>> 
>> Let me give that a shot. Thank you
>> 
>> Sent from my iPhone
>> 
>>>> On 23 Jun 2020, at 22:31, Daniel J Ruef <druef at cert.org> wrote:
>>> 
>>> Chris,
>>> Thank you for your interest in Analysis Pipeline.
>>> 
>>> It sounds like you want to have it ingest silk data, so you'll need to be sure and specify --silk and --incoming-directory on the command line, or use a data source configuration file with --data-source-configuration-file with the contents being similar to:
>>> PRIMARY DATA SOURCE silkPolling
>>>  SILK BUILDER
>>>  INCOMING DIRECTORY "/data/pipelineIncoming"
>>>  ERROR DIRECTORY "/data/pipelineError"
>>> END DATA SOURCE
>>> 
>>> In general...it sounds like from your errors, you're not specifying where pipeline gets its data from properly. You can either use command line settings to specify everything (if there is only one data source), or specify a data source configuration file using a command line switch (for any number of data sources).
>>> 
>>> When specifying the data source, you have to tell if what type of data (silk, yaf, or ipfix), and how it will get it (socket, single file, poll a directory).
>>> 
>>> This data source configuration file is different than the one used to specify filters, evaluations, statistic, etc. This part of the documentation isn't that clear, sorry about that. 
>>> 
>>> If you let me know what you're trying to do, and what the explicit error you're getting is, I can help you further.
>>> 
>>> Dan
>>> 
>>> -----Original Message-----
>>> From: netsa-tools-discuss-bounces+druef=cert.org at cert.org <netsa-tools-discuss-bounces+druef=cert.org at cert.org> On Behalf Of Chris Frazier
>>> Sent: Tuesday, June 23, 2020 1:15 AM
>>> To: netsa-tools-discuss at cert.org
>>> Subject: [netsa-tools-discuss] Analysis Pipeline
>>> 
>>> Using rwflowpack only option where rwflowpack is sending to rwflowappend and creates the pipeline incoming directory for pipelines data source
>>> 
>>> When i try to verify-config, I get the error data source file variable not set
>>> 
>>> In the conf file I am providing absolute paths to the variable in question
>>> 
>>> Sent from my iPhone