[netsa-tools-discuss] arch options for separating analysis-pipeline from (collection+storage)

[asad]

Thanks Mark, I'm running these config on production sys as soon as possible
and notifying you about the results as they come.  You are great help as
always:).

regards
Asad

On Mon, Oct 26, 2015 at 7:52 PM, Mark Thomas <mthomas at cert.org> wrote:

> asad-
>
> The analysis pipeline is designed to work with the incremental files
> produced by rwflowpack, and rwflowpack only produces those files
> when it is paired with the rwflowappend process.
>
> If the analysis pipeline were running on the same machine as
> rwflowpack, you would add pipeline and rwflowappend to the
> configuration as mentioned in the pipeline manual page:
>
> http://tools.netsa.cert.org/analysis-pipeline/pipeline-manual.html#rwflowpack_only
>
> Since pipeline is running on a different machine, you need to modify
> that configuration.
>
> Assuming your current configuration for rwflowpack is:
>
>  rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
>        --log-directory=/var/silk/rwflowpack/log
>        --root-directory=/data
>
> You want to modify it as follows:
>
>  rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
>        --log-directory=/var/silk/rwflowpack/log
>        --output-mode=sending
>        --incremental-dir=/var/silk/rwflowpack/incremental
>        --sender-dir=/var/silk/rwflowappend/incoming
>
>  rwflowappend --root-directory=/data
>        --log-directory=/var/silk/rwflowappend/log
>        --incoming-dir=/var/silk/rwflowappend/incoming
>        --error-dir=/var/silk/rwflowappend/error
>        --archive-dir=/var/silk/rwflowappend/archive
>        --flat-archive
>
> Have rwsender read the files from "/var/silk/rwflowappend/archive"
> and send them to the rwreceiver process on the Live CD.  Here I use
> port "34567".
>
>  rwsender --mode=server --server-port=34567
>        --identifier=SENDER --client-ident RECEIVER
>        --log-directory=/var/silk/rwsender/log
>        --incoming-directory=/var/silk/rwflowappend/archive
>        --processing-directory=/var/silk/rwsender/processing
>        --error-directory=/var/silk/rwsender/error
>
> On the Live CD:
>
>  rwreceiver --mode=client --server-address=SENDER:10.2.3.4:34567
>        --identifier=RECEIVER --client-ident RECEIVER
>        --log-directory=/var/silk/rwreceiver/log
>        --destination-directory=/var/pipeline/incoming
>
> where "10.2.3.4" is the IP address (or hostname) of the machine were
> rwsender is running.  If necessary, open a hole in the firewall to
> allow receiver to connect to rwsender.
>
> Finally, configure pipeline to read the files from the
> /var/pipeline/incoming directory on the Live CD.
>
>  pipeline --incoming-directory=/var/pipeline/incoming
>        --error-directory=/var/pipeline/error
>        --log-directory=/var/pipeline/log
>        --configuration-file=/var/pipeline/pipeline.conf ...
>
> -Mark
>
>
> -----Original Message-----
> From: asad <a.alii85 at gmail.com>
> Date: Sat, 24 Oct 2015 16:09:11 +0500
> To: <netsa-tools-discuss at cert.org>
> Subject: [netsa-tools-discuss] arch options for separating
> analysis-pipeline
>         from (collection+storage)
>
> Hey,
>
>
> I have successfully running SILK with little issues, now I want to bring in
> use of analysis - pipeline, but on a separating machine which infact is
> live dvd provided by silk.
>
> Please view attach (diagram) for understand current deployment settings.
>
> Ideally, I just wanted use of rwsender as a server running on silk which is
> doing collection and storage to send logs to analysis-pipeline box running
> rwreceiver.
>
> What I have read so far, that I have to use flowcap If I want to turn into
> distributed model but so far this is not my cases.
>
> So, my question essentially is that by using rwsender running on rwflowpack
> machine can it send incremental flow records to analysis-pipeline machine.
> If yes, how can it be done? Do I need to have flowcap running ?
>
> I appreciate clarity on this issue. Thanks
>
> regards
> asad
>
-------------- next part --------------
HTML attachment scrubbed and removed