[netsa-tools-discuss] arch options for separating analysis-pipeline from (collection+storage)

Tue Oct 27 10:04:03 EDT 2015

Mark-

Darn I remember there is a difference between running rwflowpack from
etc/init.d versus service rwflowpack I don't have access to system now but
you are right some other instance is populating /data. I can ps -aux | grep
-i "rwflowpack" to confirm also.

This also begs the question /etc/init.d/rwflowpack will be using a
different config file then from service rwflowpack that I didn't check or
change for this new configuration.

On Tue, Oct 27, 2015 at 6:59 PM, Mark Thomas <mthomas at cert.org> wrote:

> asad-
>
> The logs you include below indicate that rwflowpack is not receiving
> any flow data.  Since rwflowpack is not receiving data, I am not
> certain how you see data appearing in the /data directory.
>
> Is there another instance of rwflowpack running?
>
> -Mark
>
>
> -----Original Message-----
> From: asad <a.alii85 at gmail.com>
> Date: Tue, 27 Oct 2015 18:43:37 +0500
> To: Mark Thomas <mthomas at cert.org>
> Cc: <netsa-tools-discuss at cert.org>
> Subject: Re: [netsa-tools-discuss] arch options for separating
>  analysis-pipeline from (collection+storage)
>
> Mark-
>
> I did as told, I gave whole day but the logs at the specified dirs couldn't
> be written here are the important results:-
>
> Rwflowpack log
> Oct 27 15:24:42 netflow rwflowpack[19538]: Closing incremental files...
> Oct 27 15:24:42 netflow rwflowpack[19538]: Moving incremental files...
> Oct 27 15:24:42 netflow rwflowpack[19538]: No incremental files to move.
> Oct 27 15:24:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> ignored 0, nf9: missing-pkts 0
> Oct 27 15:26:42 netflow rwflowpack[19538]: Preparing to move incremental
> files...
> Oct 27 15:26:42 netflow rwflowpack[19538]: Closing incremental files...
> Oct 27 15:26:42 netflow rwflowpack[19538]: Moving incremental files...
> Oct 27 15:26:42 netflow rwflowpack[19538]: No incremental files to move.
> Oct 27 15:26:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> ignored 0, nf9: missing-pkts 0
> Oct 27 15:28:42 netflow rwflowpack[19538]: Preparing to move incremental
> files...
> Oct 27 15:28:42 netflow rwflowpack[19538]: Closing incremental files...
> Oct 27 15:28:42 netflow rwflowpack[19538]: Moving incremental files...
> Oct 27 15:28:42 netflow rwflowpack[19538]: No incremental files to move.
> Oct 27 15:28:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> ignored 0, nf9: missing-pkts 0
> Oct 27 15:30:42 netflow rwflowpack[19538]: Preparing to move incremental
> files...
> Oct 27 15:30:42 netflow rwflowpack[19538]: Closing incremental files...
> Oct 27 15:30:42 netflow rwflowpack[19538]: Moving incremental files...
> Oct 27 15:30:42 netflow rwflowpack[19538]: No incremental files to move.
> Oct 27 15:30:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> ignored 0, nf9: missing-pkts 0
> Oct 27 15:32:42 netflow rwflowpack[19538]: Preparing to move incremental
> files...
> Oct 27 15:32:42 netflow rwflowpack[19538]: Closing incremental files...
> Oct 27 15:32:42 netflow rwflowpack[19538]: Moving incremental files...
> Oct 27 15:32:42 netflow rwflowpack[19538]: No incremental files to move.
> Oct 27 15:32:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> ignored 0, nf9: missing-pkts 0
> Oct 27 15:34:42 netflow rwflowpack[19538]: Preparing to move incremental
> files...
> Oct 27 15:34:42 netflow rwflowpack[19538]: Closing incremental files...
> Oct 27 15:34:42 netflow rwflowpack[19538]: Moving incremental files...
> Oct 27 15:34:42 netflow rwflowpack[19538]: No incremental files to move.
>
> Rwflowappend logs
> farhan at netflow:/var/silk/rwflowappend/log$ cat rwflowappend-20151027.log
> Oct 27 13:49:06 netflow rwflowappend[19583]: '/usr/local/sbin/rwflowappend'
> '--incoming-directory=/var/silk/rwflowappend/incoming'
> '--error-directory=/var/silk/rwflowappend/error'
> '--root-directory=/home/farhan/data'
> '--site-config-file=/home/farhan/data/silk.conf'
> '--archive-directory=/var/silk/rwflowappend/archive' '--flat-archive'
> '--pidfile=/var/silk/rwflowappend/log/rwflowappend.pid' '--log-level=info'
> '--log-directory=/var/silk/rwflowappend/log' '--log-basename=rwflowappend'
> Oct 27 13:49:06 netflow rwflowappend[19583]: Forked child 19585.  Parent
> exiting
> Oct 27 13:49:06 netflow rwflowappend[19585]: Starting 1 appender thread...
> Oct 27 13:49:06 netflow rwflowappend[19585]: Started all appender threads.
> Oct 27 13:49:06 netflow rwflowappend[19585]: Started appender thread #1.
>
> Rwsender logs
> Oct 27 13:49:17 netflow rwsender[19678]: '/usr/local/sbin/rwsender'
> '--identifier=sender-1' '--server-port=34567' '--client-ident=receiver-1'
> '--client-ident=receiver-2' '--filter=receiver-2:^[^_]*_netflow_'
> '--client-ident=receiver-3' '--mode=server'
> '--incoming-directory=/var/silk/rwflowappend/archive'
> '--processing-directory=/var/silk/rwsender/processing'
> '--error-directory=/var/silk/rwsender/error' '--priority=100:^S[0-3]_'
> '--priority=25:^S[7-9]_' '--local-directory=:/var/rwsender/local-dir1'
> '--local-directory=auto-ident1:/var/rwsender/local-dir2'
> '--filter=auto-ident1:^S[7-9]_'
> '--pidfile=/var/silk/rwsender/log/rwsender.pid' '--log-level=info'
> '--log-directory=/var/silk/rwsender/log' '--log-basename=rwsender'
> Oct 27 13:49:17 netflow rwsender[19678]: Forked child 19680.  Parent
> exiting
> Oct 27 13:49:17 netflow rwsender[19680]: Incoming file handling thread
> started.
> Oct 27 13:49:17 netflow rwsender[19680]: Bound to 34567 for listening (TCP)
>
>
> Even , I'm getting data written in /data dir but I don't think rwflowappend
> is doing its job (from the logs at least it seems so).
>
> Thanks.
> regards
> asad
>
>
> On Mon, Oct 26, 2015 at 8:17 PM, asad <a.alii85 at gmail.com> wrote:
>
> > Thanks Mark, I'm running these config on production sys as soon as
> > possible and notifying you about the results as they come.  You are great
> > help as always:).
> >
> > regards
> > Asad
> >
> > On Mon, Oct 26, 2015 at 7:52 PM, Mark Thomas <mthomas at cert.org> wrote:
> >
> >> asad-
> >>
> >> The analysis pipeline is designed to work with the incremental files
> >> produced by rwflowpack, and rwflowpack only produces those files
> >> when it is paired with the rwflowappend process.
> >>
> >> If the analysis pipeline were running on the same machine as
> >> rwflowpack, you would add pipeline and rwflowappend to the
> >> configuration as mentioned in the pipeline manual page:
> >>
> >>
> http://tools.netsa.cert.org/analysis-pipeline/pipeline-manual.html#rwflowpack_only
> >>
> >> Since pipeline is running on a different machine, you need to modify
> >> that configuration.
> >>
> >> Assuming your current configuration for rwflowpack is:
> >>
> >>  rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
> >>        --log-directory=/var/silk/rwflowpack/log
> >>        --root-directory=/data
> >>
> >> You want to modify it as follows:
> >>
> >>  rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
> >>        --log-directory=/var/silk/rwflowpack/log
> >>        --output-mode=sending
> >>        --incremental-dir=/var/silk/rwflowpack/incremental
> >>        --sender-dir=/var/silk/rwflowappend/incoming
> >>
> >>  rwflowappend --root-directory=/data
> >>        --log-directory=/var/silk/rwflowappend/log
> >>        --incoming-dir=/var/silk/rwflowappend/incoming
> >>        --error-dir=/var/silk/rwflowappend/error
> >>        --archive-dir=/var/silk/rwflowappend/archive
> >>        --flat-archive
> >>
> >> Have rwsender read the files from "/var/silk/rwflowappend/archive"
> >> and send them to the rwreceiver process on the Live CD.  Here I use
> >> port "34567".
> >>
> >>  rwsender --mode=server --server-port=34567
> >>        --identifier=SENDER --client-ident RECEIVER
> >>        --log-directory=/var/silk/rwsender/log
> >>        --incoming-directory=/var/silk/rwflowappend/archive
> >>        --processing-directory=/var/silk/rwsender/processing
> >>        --error-directory=/var/silk/rwsender/error
> >>
> >> On the Live CD:
> >>
> >>  rwreceiver --mode=client --server-address=SENDER:10.2.3.4:34567
> >>        --identifier=RECEIVER --client-ident RECEIVER
> >>        --log-directory=/var/silk/rwreceiver/log
> >>        --destination-directory=/var/pipeline/incoming
> >>
> >> where "10.2.3.4" is the IP address (or hostname) of the machine were
> >> rwsender is running.  If necessary, open a hole in the firewall to
> >> allow receiver to connect to rwsender.
> >>
> >> Finally, configure pipeline to read the files from the
> >> /var/pipeline/incoming directory on the Live CD.
> >>
> >>  pipeline --incoming-directory=/var/pipeline/incoming
> >>        --error-directory=/var/pipeline/error
> >>        --log-directory=/var/pipeline/log
> >>        --configuration-file=/var/pipeline/pipeline.conf ...
> >>
> >> -Mark
> >>
> >>
> >> -----Original Message-----
> >> From: asad <a.alii85 at gmail.com>
> >> Date: Sat, 24 Oct 2015 16:09:11 +0500
> >> To: <netsa-tools-discuss at cert.org>
> >> Subject: [netsa-tools-discuss] arch options for separating
> >> analysis-pipeline
> >>         from (collection+storage)
> >>
> >> Hey,
> >>
> >>
> >> I have successfully running SILK with little issues, now I want to bring
> >> in
> >> use of analysis - pipeline, but on a separating machine which infact is
> >> live dvd provided by silk.
> >>
> >> Please view attach (diagram) for understand current deployment settings.
> >>
> >> Ideally, I just wanted use of rwsender as a server running on silk which
> >> is
> >> doing collection and storage to send logs to analysis-pipeline box
> running
> >> rwreceiver.
> >>
> >> What I have read so far, that I have to use flowcap If I want to turn
> into
> >> distributed model but so far this is not my cases.
> >>
> >> So, my question essentially is that by using rwsender running on
> >> rwflowpack
> >> machine can it send incremental flow records to analysis-pipeline
> machine.
> >> If yes, how can it be done? Do I need to have flowcap running ?
> >>
> >> I appreciate clarity on this issue. Thanks
> >>
> >> regards
> >> asad
> >>
> >
> >
>
-------------- next part --------------
HTML attachment scrubbed and removed