[netsa-tools-discuss] arch options for separating analysis-pipeline from (collection+storage)

Tue Oct 27 10:20:19 EDT 2015

I am sorry if my use of the command line options to demonstrate how
to run the daemons made things unclear.  Since I typically run the
tools for testing or debugging, I use the command line options
instead of the configuration file.

You will want to change your rwflowpack.conf file to use sending
mode, and then restart the rwflowpack service.

-Mark

On Tue, 27 Oct 2015 19:04:03 +0500, asad wrote:

> Mark-
>
> Darn I remember there is a difference between running rwflowpack from
> etc/init.d versus service rwflowpack I don't have access to system now but
> you are right some other instance is populating /data. I can ps -aux | grep
> -i "rwflowpack" to confirm also.
>
> This also begs the question /etc/init.d/rwflowpack will be using a
> different config file then from service rwflowpack that I didn't check or
> change for this new configuration.
>
>
>
> On Tue, Oct 27, 2015 at 6:59 PM, Mark Thomas <mthomas at cert.org> wrote:
>
>> asad-
>>
>> The logs you include below indicate that rwflowpack is not receiving
>> any flow data.  Since rwflowpack is not receiving data, I am not
>> certain how you see data appearing in the /data directory.
>>
>> Is there another instance of rwflowpack running?
>>
>> -Mark
>>
>>
>> -----Original Message-----
>> From: asad <a.alii85 at gmail.com>
>> Date: Tue, 27 Oct 2015 18:43:37 +0500
>> To: Mark Thomas <mthomas at cert.org>
>> Cc: <netsa-tools-discuss at cert.org>
>> Subject: Re: [netsa-tools-discuss] arch options for separating
>>  analysis-pipeline from (collection+storage)
>>
>> Mark-
>>
>> I did as told, I gave whole day but the logs at the specified dirs couldn't
>> be written here are the important results:-
>>
>> Rwflowpack log
>> Oct 27 15:24:42 netflow rwflowpack[19538]: Closing incremental files...
>> Oct 27 15:24:42 netflow rwflowpack[19538]: Moving incremental files...
>> Oct 27 15:24:42 netflow rwflowpack[19538]: No incremental files to move.
>> Oct 27 15:24:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> ignored 0, nf9: missing-pkts 0
>> Oct 27 15:26:42 netflow rwflowpack[19538]: Preparing to move incremental
>> files...
>> Oct 27 15:26:42 netflow rwflowpack[19538]: Closing incremental files...
>> Oct 27 15:26:42 netflow rwflowpack[19538]: Moving incremental files...
>> Oct 27 15:26:42 netflow rwflowpack[19538]: No incremental files to move.
>> Oct 27 15:26:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> ignored 0, nf9: missing-pkts 0
>> Oct 27 15:28:42 netflow rwflowpack[19538]: Preparing to move incremental
>> files...
>> Oct 27 15:28:42 netflow rwflowpack[19538]: Closing incremental files...
>> Oct 27 15:28:42 netflow rwflowpack[19538]: Moving incremental files...
>> Oct 27 15:28:42 netflow rwflowpack[19538]: No incremental files to move.
>> Oct 27 15:28:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> ignored 0, nf9: missing-pkts 0
>> Oct 27 15:30:42 netflow rwflowpack[19538]: Preparing to move incremental
>> files...
>> Oct 27 15:30:42 netflow rwflowpack[19538]: Closing incremental files...
>> Oct 27 15:30:42 netflow rwflowpack[19538]: Moving incremental files...
>> Oct 27 15:30:42 netflow rwflowpack[19538]: No incremental files to move.
>> Oct 27 15:30:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> ignored 0, nf9: missing-pkts 0
>> Oct 27 15:32:42 netflow rwflowpack[19538]: Preparing to move incremental
>> files...
>> Oct 27 15:32:42 netflow rwflowpack[19538]: Closing incremental files...
>> Oct 27 15:32:42 netflow rwflowpack[19538]: Moving incremental files...
>> Oct 27 15:32:42 netflow rwflowpack[19538]: No incremental files to move.
>> Oct 27 15:32:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> ignored 0, nf9: missing-pkts 0
>> Oct 27 15:34:42 netflow rwflowpack[19538]: Preparing to move incremental
>> files...
>> Oct 27 15:34:42 netflow rwflowpack[19538]: Closing incremental files...
>> Oct 27 15:34:42 netflow rwflowpack[19538]: Moving incremental files...
>> Oct 27 15:34:42 netflow rwflowpack[19538]: No incremental files to move.
>>
>> Rwflowappend logs
>> farhan at netflow:/var/silk/rwflowappend/log$ cat rwflowappend-20151027.log
>> Oct 27 13:49:06 netflow rwflowappend[19583]: '/usr/local/sbin/rwflowappend'
>> '--incoming-directory=/var/silk/rwflowappend/incoming'
>> '--error-directory=/var/silk/rwflowappend/error'
>> '--root-directory=/home/farhan/data'
>> '--site-config-file=/home/farhan/data/silk.conf'
>> '--archive-directory=/var/silk/rwflowappend/archive' '--flat-archive'
>> '--pidfile=/var/silk/rwflowappend/log/rwflowappend.pid' '--log-level=info'
>> '--log-directory=/var/silk/rwflowappend/log' '--log-basename=rwflowappend'
>> Oct 27 13:49:06 netflow rwflowappend[19583]: Forked child 19585.  Parent
>> exiting
>> Oct 27 13:49:06 netflow rwflowappend[19585]: Starting 1 appender thread...
>> Oct 27 13:49:06 netflow rwflowappend[19585]: Started all appender threads.
>> Oct 27 13:49:06 netflow rwflowappend[19585]: Started appender thread #1.
>>
>> Rwsender logs
>> Oct 27 13:49:17 netflow rwsender[19678]: '/usr/local/sbin/rwsender'
>> '--identifier=sender-1' '--server-port=34567' '--client-ident=receiver-1'
>> '--client-ident=receiver-2' '--filter=receiver-2:^[^_]*_netflow_'
>> '--client-ident=receiver-3' '--mode=server'
>> '--incoming-directory=/var/silk/rwflowappend/archive'
>> '--processing-directory=/var/silk/rwsender/processing'
>> '--error-directory=/var/silk/rwsender/error' '--priority=100:^S[0-3]_'
>> '--priority=25:^S[7-9]_' '--local-directory=:/var/rwsender/local-dir1'
>> '--local-directory=auto-ident1:/var/rwsender/local-dir2'
>> '--filter=auto-ident1:^S[7-9]_'
>> '--pidfile=/var/silk/rwsender/log/rwsender.pid' '--log-level=info'
>> '--log-directory=/var/silk/rwsender/log' '--log-basename=rwsender'
>> Oct 27 13:49:17 netflow rwsender[19678]: Forked child 19680.  Parent
>> exiting
>> Oct 27 13:49:17 netflow rwsender[19680]: Incoming file handling thread
>> started.
>> Oct 27 13:49:17 netflow rwsender[19680]: Bound to 34567 for listening (TCP)
>>
>>
>> Even , I'm getting data written in /data dir but I don't think rwflowappend
>> is doing its job (from the logs at least it seems so).
>>
>> Thanks.
>> regards
>> asad
>>
>>
>> On Mon, Oct 26, 2015 at 8:17 PM, asad <a.alii85 at gmail.com> wrote:
>>
>> > Thanks Mark, I'm running these config on production sys as soon as
>> > possible and notifying you about the results as they come.  You are great
>> > help as always:).
>> >
>> > regards
>> > Asad
>> >
>> > On Mon, Oct 26, 2015 at 7:52 PM, Mark Thomas <mthomas at cert.org> wrote:
>> >
>> >> asad-
>> >>
>> >> The analysis pipeline is designed to work with the incremental files
>> >> produced by rwflowpack, and rwflowpack only produces those files
>> >> when it is paired with the rwflowappend process.
>> >>
>> >> If the analysis pipeline were running on the same machine as
>> >> rwflowpack, you would add pipeline and rwflowappend to the
>> >> configuration as mentioned in the pipeline manual page:
>> >>
>> >>
>> http://tools.netsa.cert.org/analysis-pipeline/pipeline-manual.html#rwflowpack_only
>> >>
>> >> Since pipeline is running on a different machine, you need to modify
>> >> that configuration.
>> >>
>> >> Assuming your current configuration for rwflowpack is:
>> >>
>> >>  rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
>> >>        --log-directory=/var/silk/rwflowpack/log
>> >>        --root-directory=/data
>> >>
>> >> You want to modify it as follows:
>> >>
>> >>  rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
>> >>        --log-directory=/var/silk/rwflowpack/log
>> >>        --output-mode=sending
>> >>        --incremental-dir=/var/silk/rwflowpack/incremental
>> >>        --sender-dir=/var/silk/rwflowappend/incoming
>> >>
>> >>  rwflowappend --root-directory=/data
>> >>        --log-directory=/var/silk/rwflowappend/log
>> >>        --incoming-dir=/var/silk/rwflowappend/incoming
>> >>        --error-dir=/var/silk/rwflowappend/error
>> >>        --archive-dir=/var/silk/rwflowappend/archive
>> >>        --flat-archive
>> >>
>> >> Have rwsender read the files from "/var/silk/rwflowappend/archive"
>> >> and send them to the rwreceiver process on the Live CD.  Here I use
>> >> port "34567".
>> >>
>> >>  rwsender --mode=server --server-port=34567
>> >>        --identifier=SENDER --client-ident RECEIVER
>> >>        --log-directory=/var/silk/rwsender/log
>> >>        --incoming-directory=/var/silk/rwflowappend/archive
>> >>        --processing-directory=/var/silk/rwsender/processing
>> >>        --error-directory=/var/silk/rwsender/error
>> >>
>> >> On the Live CD:
>> >>
>> >>  rwreceiver --mode=client --server-address=SENDER:10.2.3.4:34567
>> >>        --identifier=RECEIVER --client-ident RECEIVER
>> >>        --log-directory=/var/silk/rwreceiver/log
>> >>        --destination-directory=/var/pipeline/incoming
>> >>
>> >> where "10.2.3.4" is the IP address (or hostname) of the machine were
>> >> rwsender is running.  If necessary, open a hole in the firewall to
>> >> allow receiver to connect to rwsender.
>> >>
>> >> Finally, configure pipeline to read the files from the
>> >> /var/pipeline/incoming directory on the Live CD.
>> >>
>> >>  pipeline --incoming-directory=/var/pipeline/incoming
>> >>        --error-directory=/var/pipeline/error
>> >>        --log-directory=/var/pipeline/log
>> >>        --configuration-file=/var/pipeline/pipeline.conf ...
>> >>
>> >> -Mark
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: asad <a.alii85 at gmail.com>
>> >> Date: Sat, 24 Oct 2015 16:09:11 +0500
>> >> To: <netsa-tools-discuss at cert.org>
>> >> Subject: [netsa-tools-discuss] arch options for separating
>> >> analysis-pipeline
>> >>         from (collection+storage)
>> >>
>> >> Hey,
>> >>
>> >>
>> >> I have successfully running SILK with little issues, now I want to bring
>> >> in
>> >> use of analysis - pipeline, but on a separating machine which infact is
>> >> live dvd provided by silk.
>> >>
>> >> Please view attach (diagram) for understand current deployment settings.
>> >>
>> >> Ideally, I just wanted use of rwsender as a server running on silk which
>> >> is
>> >> doing collection and storage to send logs to analysis-pipeline box
>> running
>> >> rwreceiver.
>> >>
>> >> What I have read so far, that I have to use flowcap If I want to turn
>> into
>> >> distributed model but so far this is not my cases.
>> >>
>> >> So, my question essentially is that by using rwsender running on
>> >> rwflowpack
>> >> machine can it send incremental flow records to analysis-pipeline
>> machine.
>> >> If yes, how can it be done? Do I need to have flowcap running ?
>> >>
>> >> I appreciate clarity on this issue. Thanks
>> >>
>> >> regards
>> >> asad
>> >>
>> >
>> >
>>