[netsa-tools-discuss] arch options for separating analysis-pipeline from (collection+storage)
asad
a.alii85 at gmail.com
Tue Oct 27 10:37:20 EDT 2015
Mark-
No, confusion is on my end, I just need to translate your command line
parms to config file which shouldn't be an issue. Rwflowpack.conf using
--output-mode =sending is changed as told. But what I believe rwflowpack
was started from /etc/init.d/ which uses different rwflowpack.conf.
On Tue, Oct 27, 2015 at 7:20 PM, Mark Thomas <mthomas at cert.org> wrote:
> I am sorry if my use of the command line options to demonstrate how
> to run the daemons made things unclear. Since I typically run the
> tools for testing or debugging, I use the command line options
> instead of the configuration file.
>
> You will want to change your rwflowpack.conf file to use sending
> mode, and then restart the rwflowpack service.
>
> -Mark
>
>
> On Tue, 27 Oct 2015 19:04:03 +0500, asad wrote:
>
> > Mark-
> >
> > Darn I remember there is a difference between running rwflowpack from
> > etc/init.d versus service rwflowpack I don't have access to system now
> but
> > you are right some other instance is populating /data. I can ps -aux |
> grep
> > -i "rwflowpack" to confirm also.
> >
> > This also begs the question /etc/init.d/rwflowpack will be using a
> > different config file then from service rwflowpack that I didn't check or
> > change for this new configuration.
> >
> >
> >
> > On Tue, Oct 27, 2015 at 6:59 PM, Mark Thomas <mthomas at cert.org> wrote:
> >
> >> asad-
> >>
> >> The logs you include below indicate that rwflowpack is not receiving
> >> any flow data. Since rwflowpack is not receiving data, I am not
> >> certain how you see data appearing in the /data directory.
> >>
> >> Is there another instance of rwflowpack running?
> >>
> >> -Mark
> >>
> >>
> >> -----Original Message-----
> >> From: asad <a.alii85 at gmail.com>
> >> Date: Tue, 27 Oct 2015 18:43:37 +0500
> >> To: Mark Thomas <mthomas at cert.org>
> >> Cc: <netsa-tools-discuss at cert.org>
> >> Subject: Re: [netsa-tools-discuss] arch options for separating
> >> analysis-pipeline from (collection+storage)
> >>
> >> Mark-
> >>
> >> I did as told, I gave whole day but the logs at the specified dirs
> couldn't
> >> be written here are the important results:-
> >>
> >> Rwflowpack log
> >> Oct 27 15:24:42 netflow rwflowpack[19538]: Closing incremental files...
> >> Oct 27 15:24:42 netflow rwflowpack[19538]: Moving incremental files...
> >> Oct 27 15:24:42 netflow rwflowpack[19538]: No incremental files to move.
> >> Oct 27 15:24:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> >> ignored 0, nf9: missing-pkts 0
> >> Oct 27 15:26:42 netflow rwflowpack[19538]: Preparing to move incremental
> >> files...
> >> Oct 27 15:26:42 netflow rwflowpack[19538]: Closing incremental files...
> >> Oct 27 15:26:42 netflow rwflowpack[19538]: Moving incremental files...
> >> Oct 27 15:26:42 netflow rwflowpack[19538]: No incremental files to move.
> >> Oct 27 15:26:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> >> ignored 0, nf9: missing-pkts 0
> >> Oct 27 15:28:42 netflow rwflowpack[19538]: Preparing to move incremental
> >> files...
> >> Oct 27 15:28:42 netflow rwflowpack[19538]: Closing incremental files...
> >> Oct 27 15:28:42 netflow rwflowpack[19538]: Moving incremental files...
> >> Oct 27 15:28:42 netflow rwflowpack[19538]: No incremental files to move.
> >> Oct 27 15:28:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> >> ignored 0, nf9: missing-pkts 0
> >> Oct 27 15:30:42 netflow rwflowpack[19538]: Preparing to move incremental
> >> files...
> >> Oct 27 15:30:42 netflow rwflowpack[19538]: Closing incremental files...
> >> Oct 27 15:30:42 netflow rwflowpack[19538]: Moving incremental files...
> >> Oct 27 15:30:42 netflow rwflowpack[19538]: No incremental files to move.
> >> Oct 27 15:30:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> >> ignored 0, nf9: missing-pkts 0
> >> Oct 27 15:32:42 netflow rwflowpack[19538]: Preparing to move incremental
> >> files...
> >> Oct 27 15:32:42 netflow rwflowpack[19538]: Closing incremental files...
> >> Oct 27 15:32:42 netflow rwflowpack[19538]: Moving incremental files...
> >> Oct 27 15:32:42 netflow rwflowpack[19538]: No incremental files to move.
> >> Oct 27 15:32:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
> >> ignored 0, nf9: missing-pkts 0
> >> Oct 27 15:34:42 netflow rwflowpack[19538]: Preparing to move incremental
> >> files...
> >> Oct 27 15:34:42 netflow rwflowpack[19538]: Closing incremental files...
> >> Oct 27 15:34:42 netflow rwflowpack[19538]: Moving incremental files...
> >> Oct 27 15:34:42 netflow rwflowpack[19538]: No incremental files to move.
> >>
> >> Rwflowappend logs
> >> farhan at netflow:/var/silk/rwflowappend/log$ cat
> rwflowappend-20151027.log
> >> Oct 27 13:49:06 netflow rwflowappend[19583]:
> '/usr/local/sbin/rwflowappend'
> >> '--incoming-directory=/var/silk/rwflowappend/incoming'
> >> '--error-directory=/var/silk/rwflowappend/error'
> >> '--root-directory=/home/farhan/data'
> >> '--site-config-file=/home/farhan/data/silk.conf'
> >> '--archive-directory=/var/silk/rwflowappend/archive' '--flat-archive'
> >> '--pidfile=/var/silk/rwflowappend/log/rwflowappend.pid'
> '--log-level=info'
> >> '--log-directory=/var/silk/rwflowappend/log'
> '--log-basename=rwflowappend'
> >> Oct 27 13:49:06 netflow rwflowappend[19583]: Forked child 19585. Parent
> >> exiting
> >> Oct 27 13:49:06 netflow rwflowappend[19585]: Starting 1 appender
> thread...
> >> Oct 27 13:49:06 netflow rwflowappend[19585]: Started all appender
> threads.
> >> Oct 27 13:49:06 netflow rwflowappend[19585]: Started appender thread #1.
> >>
> >> Rwsender logs
> >> Oct 27 13:49:17 netflow rwsender[19678]: '/usr/local/sbin/rwsender'
> >> '--identifier=sender-1' '--server-port=34567'
> '--client-ident=receiver-1'
> >> '--client-ident=receiver-2' '--filter=receiver-2:^[^_]*_netflow_'
> >> '--client-ident=receiver-3' '--mode=server'
> >> '--incoming-directory=/var/silk/rwflowappend/archive'
> >> '--processing-directory=/var/silk/rwsender/processing'
> >> '--error-directory=/var/silk/rwsender/error' '--priority=100:^S[0-3]_'
> >> '--priority=25:^S[7-9]_' '--local-directory=:/var/rwsender/local-dir1'
> >> '--local-directory=auto-ident1:/var/rwsender/local-dir2'
> >> '--filter=auto-ident1:^S[7-9]_'
> >> '--pidfile=/var/silk/rwsender/log/rwsender.pid' '--log-level=info'
> >> '--log-directory=/var/silk/rwsender/log' '--log-basename=rwsender'
> >> Oct 27 13:49:17 netflow rwsender[19678]: Forked child 19680. Parent
> >> exiting
> >> Oct 27 13:49:17 netflow rwsender[19680]: Incoming file handling thread
> >> started.
> >> Oct 27 13:49:17 netflow rwsender[19680]: Bound to 34567 for listening
> (TCP)
> >>
> >>
> >> Even , I'm getting data written in /data dir but I don't think
> rwflowappend
> >> is doing its job (from the logs at least it seems so).
> >>
> >> Thanks.
> >> regards
> >> asad
> >>
> >>
> >> On Mon, Oct 26, 2015 at 8:17 PM, asad <a.alii85 at gmail.com> wrote:
> >>
> >> > Thanks Mark, I'm running these config on production sys as soon as
> >> > possible and notifying you about the results as they come. You are
> great
> >> > help as always:).
> >> >
> >> > regards
> >> > Asad
> >> >
> >> > On Mon, Oct 26, 2015 at 7:52 PM, Mark Thomas <mthomas at cert.org>
> wrote:
> >> >
> >> >> asad-
> >> >>
> >> >> The analysis pipeline is designed to work with the incremental files
> >> >> produced by rwflowpack, and rwflowpack only produces those files
> >> >> when it is paired with the rwflowappend process.
> >> >>
> >> >> If the analysis pipeline were running on the same machine as
> >> >> rwflowpack, you would add pipeline and rwflowappend to the
> >> >> configuration as mentioned in the pipeline manual page:
> >> >>
> >> >>
> >>
> http://tools.netsa.cert.org/analysis-pipeline/pipeline-manual.html#rwflowpack_only
> >> >>
> >> >> Since pipeline is running on a different machine, you need to modify
> >> >> that configuration.
> >> >>
> >> >> Assuming your current configuration for rwflowpack is:
> >> >>
> >> >> rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
> >> >> --log-directory=/var/silk/rwflowpack/log
> >> >> --root-directory=/data
> >> >>
> >> >> You want to modify it as follows:
> >> >>
> >> >> rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
> >> >> --log-directory=/var/silk/rwflowpack/log
> >> >> --output-mode=sending
> >> >> --incremental-dir=/var/silk/rwflowpack/incremental
> >> >> --sender-dir=/var/silk/rwflowappend/incoming
> >> >>
> >> >> rwflowappend --root-directory=/data
> >> >> --log-directory=/var/silk/rwflowappend/log
> >> >> --incoming-dir=/var/silk/rwflowappend/incoming
> >> >> --error-dir=/var/silk/rwflowappend/error
> >> >> --archive-dir=/var/silk/rwflowappend/archive
> >> >> --flat-archive
> >> >>
> >> >> Have rwsender read the files from "/var/silk/rwflowappend/archive"
> >> >> and send them to the rwreceiver process on the Live CD. Here I use
> >> >> port "34567".
> >> >>
> >> >> rwsender --mode=server --server-port=34567
> >> >> --identifier=SENDER --client-ident RECEIVER
> >> >> --log-directory=/var/silk/rwsender/log
> >> >> --incoming-directory=/var/silk/rwflowappend/archive
> >> >> --processing-directory=/var/silk/rwsender/processing
> >> >> --error-directory=/var/silk/rwsender/error
> >> >>
> >> >> On the Live CD:
> >> >>
> >> >> rwreceiver --mode=client --server-address=SENDER:10.2.3.4:34567
> >> >> --identifier=RECEIVER --client-ident RECEIVER
> >> >> --log-directory=/var/silk/rwreceiver/log
> >> >> --destination-directory=/var/pipeline/incoming
> >> >>
> >> >> where "10.2.3.4" is the IP address (or hostname) of the machine were
> >> >> rwsender is running. If necessary, open a hole in the firewall to
> >> >> allow receiver to connect to rwsender.
> >> >>
> >> >> Finally, configure pipeline to read the files from the
> >> >> /var/pipeline/incoming directory on the Live CD.
> >> >>
> >> >> pipeline --incoming-directory=/var/pipeline/incoming
> >> >> --error-directory=/var/pipeline/error
> >> >> --log-directory=/var/pipeline/log
> >> >> --configuration-file=/var/pipeline/pipeline.conf ...
> >> >>
> >> >> -Mark
> >> >>
> >> >>
> >> >> -----Original Message-----
> >> >> From: asad <a.alii85 at gmail.com>
> >> >> Date: Sat, 24 Oct 2015 16:09:11 +0500
> >> >> To: <netsa-tools-discuss at cert.org>
> >> >> Subject: [netsa-tools-discuss] arch options for separating
> >> >> analysis-pipeline
> >> >> from (collection+storage)
> >> >>
> >> >> Hey,
> >> >>
> >> >>
> >> >> I have successfully running SILK with little issues, now I want to
> bring
> >> >> in
> >> >> use of analysis - pipeline, but on a separating machine which infact
> is
> >> >> live dvd provided by silk.
> >> >>
> >> >> Please view attach (diagram) for understand current deployment
> settings.
> >> >>
> >> >> Ideally, I just wanted use of rwsender as a server running on silk
> which
> >> >> is
> >> >> doing collection and storage to send logs to analysis-pipeline box
> >> running
> >> >> rwreceiver.
> >> >>
> >> >> What I have read so far, that I have to use flowcap If I want to turn
> >> into
> >> >> distributed model but so far this is not my cases.
> >> >>
> >> >> So, my question essentially is that by using rwsender running on
> >> >> rwflowpack
> >> >> machine can it send incremental flow records to analysis-pipeline
> >> machine.
> >> >> If yes, how can it be done? Do I need to have flowcap running ?
> >> >>
> >> >> I appreciate clarity on this issue. Thanks
> >> >>
> >> >> regards
> >> >> asad
> >> >>
> >> >
> >> >
> >>
>
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list