[netsa-tools-discuss] arch options for separating analysis-pipeline from (collection+storage)

Tue Oct 27 09:59:47 EDT 2015

asad-

The logs you include below indicate that rwflowpack is not receiving
any flow data.  Since rwflowpack is not receiving data, I am not
certain how you see data appearing in the /data directory.

Is there another instance of rwflowpack running?

-Mark

-----Original Message-----
From: asad <a.alii85 at gmail.com>
Date: Tue, 27 Oct 2015 18:43:37 +0500
To: Mark Thomas <mthomas at cert.org>
Cc: <netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] arch options for separating
 analysis-pipeline from (collection+storage)

Mark-

I did as told, I gave whole day but the logs at the specified dirs couldn't
be written here are the important results:-

Rwflowpack log
Oct 27 15:24:42 netflow rwflowpack[19538]: Closing incremental files...
Oct 27 15:24:42 netflow rwflowpack[19538]: Moving incremental files...
Oct 27 15:24:42 netflow rwflowpack[19538]: No incremental files to move.
Oct 27 15:24:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
ignored 0, nf9: missing-pkts 0
Oct 27 15:26:42 netflow rwflowpack[19538]: Preparing to move incremental
files...
Oct 27 15:26:42 netflow rwflowpack[19538]: Closing incremental files...
Oct 27 15:26:42 netflow rwflowpack[19538]: Moving incremental files...
Oct 27 15:26:42 netflow rwflowpack[19538]: No incremental files to move.
Oct 27 15:26:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
ignored 0, nf9: missing-pkts 0
Oct 27 15:28:42 netflow rwflowpack[19538]: Preparing to move incremental
files...
Oct 27 15:28:42 netflow rwflowpack[19538]: Closing incremental files...
Oct 27 15:28:42 netflow rwflowpack[19538]: Moving incremental files...
Oct 27 15:28:42 netflow rwflowpack[19538]: No incremental files to move.
Oct 27 15:28:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
ignored 0, nf9: missing-pkts 0
Oct 27 15:30:42 netflow rwflowpack[19538]: Preparing to move incremental
files...
Oct 27 15:30:42 netflow rwflowpack[19538]: Closing incremental files...
Oct 27 15:30:42 netflow rwflowpack[19538]: Moving incremental files...
Oct 27 15:30:42 netflow rwflowpack[19538]: No incremental files to move.
Oct 27 15:30:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
ignored 0, nf9: missing-pkts 0
Oct 27 15:32:42 netflow rwflowpack[19538]: Preparing to move incremental
files...
Oct 27 15:32:42 netflow rwflowpack[19538]: Closing incremental files...
Oct 27 15:32:42 netflow rwflowpack[19538]: Moving incremental files...
Oct 27 15:32:42 netflow rwflowpack[19538]: No incremental files to move.
Oct 27 15:32:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
ignored 0, nf9: missing-pkts 0
Oct 27 15:34:42 netflow rwflowpack[19538]: Preparing to move incremental
files...
Oct 27 15:34:42 netflow rwflowpack[19538]: Closing incremental files...
Oct 27 15:34:42 netflow rwflowpack[19538]: Moving incremental files...
Oct 27 15:34:42 netflow rwflowpack[19538]: No incremental files to move.

Rwflowappend logs
farhan at netflow:/var/silk/rwflowappend/log$ cat rwflowappend-20151027.log
Oct 27 13:49:06 netflow rwflowappend[19583]: '/usr/local/sbin/rwflowappend'
'--incoming-directory=/var/silk/rwflowappend/incoming'
'--error-directory=/var/silk/rwflowappend/error'
'--root-directory=/home/farhan/data'
'--site-config-file=/home/farhan/data/silk.conf'
'--archive-directory=/var/silk/rwflowappend/archive' '--flat-archive'
'--pidfile=/var/silk/rwflowappend/log/rwflowappend.pid' '--log-level=info'
'--log-directory=/var/silk/rwflowappend/log' '--log-basename=rwflowappend'
Oct 27 13:49:06 netflow rwflowappend[19583]: Forked child 19585.  Parent
exiting
Oct 27 13:49:06 netflow rwflowappend[19585]: Starting 1 appender thread...
Oct 27 13:49:06 netflow rwflowappend[19585]: Started all appender threads.
Oct 27 13:49:06 netflow rwflowappend[19585]: Started appender thread #1.

Rwsender logs
Oct 27 13:49:17 netflow rwsender[19678]: '/usr/local/sbin/rwsender'
'--identifier=sender-1' '--server-port=34567' '--client-ident=receiver-1'
'--client-ident=receiver-2' '--filter=receiver-2:^[^_]*_netflow_'
'--client-ident=receiver-3' '--mode=server'
'--incoming-directory=/var/silk/rwflowappend/archive'
'--processing-directory=/var/silk/rwsender/processing'
'--error-directory=/var/silk/rwsender/error' '--priority=100:^S[0-3]_'
'--priority=25:^S[7-9]_' '--local-directory=:/var/rwsender/local-dir1'
'--local-directory=auto-ident1:/var/rwsender/local-dir2'
'--filter=auto-ident1:^S[7-9]_'
'--pidfile=/var/silk/rwsender/log/rwsender.pid' '--log-level=info'
'--log-directory=/var/silk/rwsender/log' '--log-basename=rwsender'
Oct 27 13:49:17 netflow rwsender[19678]: Forked child 19680.  Parent exiting
Oct 27 13:49:17 netflow rwsender[19680]: Incoming file handling thread
started.
Oct 27 13:49:17 netflow rwsender[19680]: Bound to 34567 for listening (TCP)

Even , I'm getting data written in /data dir but I don't think rwflowappend
is doing its job (from the logs at least it seems so).

Thanks.
regards
asad

On Mon, Oct 26, 2015 at 8:17 PM, asad <a.alii85 at gmail.com> wrote:

> Thanks Mark, I'm running these config on production sys as soon as
> possible and notifying you about the results as they come.  You are great
> help as always:).
>
> regards
> Asad
>
> On Mon, Oct 26, 2015 at 7:52 PM, Mark Thomas <mthomas at cert.org> wrote:
>
>> asad-
>>
>> The analysis pipeline is designed to work with the incremental files
>> produced by rwflowpack, and rwflowpack only produces those files
>> when it is paired with the rwflowappend process.
>>
>> If the analysis pipeline were running on the same machine as
>> rwflowpack, you would add pipeline and rwflowappend to the
>> configuration as mentioned in the pipeline manual page:
>>
>> http://tools.netsa.cert.org/analysis-pipeline/pipeline-manual.html#rwflowpack_only
>>
>> Since pipeline is running on a different machine, you need to modify
>> that configuration.
>>
>> Assuming your current configuration for rwflowpack is:
>>
>>  rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
>>        --log-directory=/var/silk/rwflowpack/log
>>        --root-directory=/data
>>
>> You want to modify it as follows:
>>
>>  rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
>>        --log-directory=/var/silk/rwflowpack/log
>>        --output-mode=sending
>>        --incremental-dir=/var/silk/rwflowpack/incremental
>>        --sender-dir=/var/silk/rwflowappend/incoming
>>
>>  rwflowappend --root-directory=/data
>>        --log-directory=/var/silk/rwflowappend/log
>>        --incoming-dir=/var/silk/rwflowappend/incoming
>>        --error-dir=/var/silk/rwflowappend/error
>>        --archive-dir=/var/silk/rwflowappend/archive
>>        --flat-archive
>>
>> Have rwsender read the files from "/var/silk/rwflowappend/archive"
>> and send them to the rwreceiver process on the Live CD.  Here I use
>> port "34567".
>>
>>  rwsender --mode=server --server-port=34567
>>        --identifier=SENDER --client-ident RECEIVER
>>        --log-directory=/var/silk/rwsender/log
>>        --incoming-directory=/var/silk/rwflowappend/archive
>>        --processing-directory=/var/silk/rwsender/processing
>>        --error-directory=/var/silk/rwsender/error
>>
>> On the Live CD:
>>
>>  rwreceiver --mode=client --server-address=SENDER:10.2.3.4:34567
>>        --identifier=RECEIVER --client-ident RECEIVER
>>        --log-directory=/var/silk/rwreceiver/log
>>        --destination-directory=/var/pipeline/incoming
>>
>> where "10.2.3.4" is the IP address (or hostname) of the machine were
>> rwsender is running.  If necessary, open a hole in the firewall to
>> allow receiver to connect to rwsender.
>>
>> Finally, configure pipeline to read the files from the
>> /var/pipeline/incoming directory on the Live CD.
>>
>>  pipeline --incoming-directory=/var/pipeline/incoming
>>        --error-directory=/var/pipeline/error
>>        --log-directory=/var/pipeline/log
>>        --configuration-file=/var/pipeline/pipeline.conf ...
>>
>> -Mark
>>
>>
>> -----Original Message-----
>> From: asad <a.alii85 at gmail.com>
>> Date: Sat, 24 Oct 2015 16:09:11 +0500
>> To: <netsa-tools-discuss at cert.org>
>> Subject: [netsa-tools-discuss] arch options for separating
>> analysis-pipeline
>>         from (collection+storage)
>>
>> Hey,
>>
>>
>> I have successfully running SILK with little issues, now I want to bring
>> in
>> use of analysis - pipeline, but on a separating machine which infact is
>> live dvd provided by silk.
>>
>> Please view attach (diagram) for understand current deployment settings.
>>
>> Ideally, I just wanted use of rwsender as a server running on silk which
>> is
>> doing collection and storage to send logs to analysis-pipeline box running
>> rwreceiver.
>>
>> What I have read so far, that I have to use flowcap If I want to turn into
>> distributed model but so far this is not my cases.
>>
>> So, my question essentially is that by using rwsender running on
>> rwflowpack
>> machine can it send incremental flow records to analysis-pipeline machine.
>> If yes, how can it be done? Do I need to have flowcap running ?
>>
>> I appreciate clarity on this issue. Thanks
>>
>> regards
>> asad
>>
>
>