[netsa-tools-discuss] arch options for separating analysis-pipeline from (collection+storage)

[Mark Thomas]

asad-

The analysis pipeline is designed to work with the incremental files
produced by rwflowpack, and rwflowpack only produces those files
when it is paired with the rwflowappend process.

If the analysis pipeline were running on the same machine as
rwflowpack, you would add pipeline and rwflowappend to the
configuration as mentioned in the pipeline manual page:
http://tools.netsa.cert.org/analysis-pipeline/pipeline-manual.html#rwflowpack_only

Since pipeline is running on a different machine, you need to modify
that configuration.

Assuming your current configuration for rwflowpack is:

 rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
       --log-directory=/var/silk/rwflowpack/log
       --root-directory=/data

You want to modify it as follows:

 rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
       --log-directory=/var/silk/rwflowpack/log
       --output-mode=sending
       --incremental-dir=/var/silk/rwflowpack/incremental
       --sender-dir=/var/silk/rwflowappend/incoming

 rwflowappend --root-directory=/data
       --log-directory=/var/silk/rwflowappend/log
       --incoming-dir=/var/silk/rwflowappend/incoming
       --error-dir=/var/silk/rwflowappend/error
       --archive-dir=/var/silk/rwflowappend/archive
       --flat-archive

Have rwsender read the files from "/var/silk/rwflowappend/archive"
and send them to the rwreceiver process on the Live CD.  Here I use
port "34567".

 rwsender --mode=server --server-port=34567
       --identifier=SENDER --client-ident RECEIVER
       --log-directory=/var/silk/rwsender/log
       --incoming-directory=/var/silk/rwflowappend/archive
       --processing-directory=/var/silk/rwsender/processing
       --error-directory=/var/silk/rwsender/error

On the Live CD:

 rwreceiver --mode=client --server-address=SENDER:10.2.3.4:34567
       --identifier=RECEIVER --client-ident RECEIVER
       --log-directory=/var/silk/rwreceiver/log
       --destination-directory=/var/pipeline/incoming

where "10.2.3.4" is the IP address (or hostname) of the machine were
rwsender is running.  If necessary, open a hole in the firewall to
allow receiver to connect to rwsender.

Finally, configure pipeline to read the files from the
/var/pipeline/incoming directory on the Live CD.

 pipeline --incoming-directory=/var/pipeline/incoming
       --error-directory=/var/pipeline/error
       --log-directory=/var/pipeline/log
       --configuration-file=/var/pipeline/pipeline.conf ...

-Mark


-----Original Message-----
From: asad <a.alii85 at gmail.com>
Date: Sat, 24 Oct 2015 16:09:11 +0500
To: <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] arch options for separating analysis-pipeline
	from (collection+storage)

Hey,


I have successfully running SILK with little issues, now I want to bring in
use of analysis - pipeline, but on a separating machine which infact is
live dvd provided by silk.

Please view attach (diagram) for understand current deployment settings.

Ideally, I just wanted use of rwsender as a server running on silk which is
doing collection and storage to send logs to analysis-pipeline box running
rwreceiver.

What I have read so far, that I have to use flowcap If I want to turn into
distributed model but so far this is not my cases.

So, my question essentially is that by using rwsender running on rwflowpack
machine can it send incremental flow records to analysis-pipeline machine.
If yes, how can it be done? Do I need to have flowcap running ?

I appreciate clarity on this issue. Thanks

regards
asad