[netsa-tools-discuss] arch options for separating analysis-pipeline from (collection+storage)
asad
a.alii85 at gmail.com
Fri Oct 30 09:17:34 EDT 2015
Mark-
Thanks for the correct problem identification, as I found out that there
two instances of "rwflowpack" running on system each one with different set
of rwflowpack.conf.
After correcting the problem to run single instance everything went smooth
to the point that I was able to finally log the alert in the log file
specified in pipeline.conf.
Once again, thanking you for support and guidance.
regards
asad
On Tue, Oct 27, 2015 at 7:37 PM, asad <a.alii85 at gmail.com> wrote:
> Mark-
>
> No, confusion is on my end, I just need to translate your command line
> parms to config file which shouldn't be an issue. Rwflowpack.conf using
> --output-mode =sending is changed as told. But what I believe rwflowpack
> was started from /etc/init.d/ which uses different rwflowpack.conf.
>
>
>
> On Tue, Oct 27, 2015 at 7:20 PM, Mark Thomas <mthomas at cert.org> wrote:
>
>> I am sorry if my use of the command line options to demonstrate how
>> to run the daemons made things unclear. Since I typically run the
>> tools for testing or debugging, I use the command line options
>> instead of the configuration file.
>>
>> You will want to change your rwflowpack.conf file to use sending
>> mode, and then restart the rwflowpack service.
>>
>> -Mark
>>
>>
>> On Tue, 27 Oct 2015 19:04:03 +0500, asad wrote:
>>
>> > Mark-
>> >
>> > Darn I remember there is a difference between running rwflowpack from
>> > etc/init.d versus service rwflowpack I don't have access to system now
>> but
>> > you are right some other instance is populating /data. I can ps -aux |
>> grep
>> > -i "rwflowpack" to confirm also.
>> >
>> > This also begs the question /etc/init.d/rwflowpack will be using a
>> > different config file then from service rwflowpack that I didn't check
>> or
>> > change for this new configuration.
>> >
>> >
>> >
>> > On Tue, Oct 27, 2015 at 6:59 PM, Mark Thomas <mthomas at cert.org> wrote:
>> >
>> >> asad-
>> >>
>> >> The logs you include below indicate that rwflowpack is not receiving
>> >> any flow data. Since rwflowpack is not receiving data, I am not
>> >> certain how you see data appearing in the /data directory.
>> >>
>> >> Is there another instance of rwflowpack running?
>> >>
>> >> -Mark
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: asad <a.alii85 at gmail.com>
>> >> Date: Tue, 27 Oct 2015 18:43:37 +0500
>> >> To: Mark Thomas <mthomas at cert.org>
>> >> Cc: <netsa-tools-discuss at cert.org>
>> >> Subject: Re: [netsa-tools-discuss] arch options for separating
>> >> analysis-pipeline from (collection+storage)
>> >>
>> >> Mark-
>> >>
>> >> I did as told, I gave whole day but the logs at the specified dirs
>> couldn't
>> >> be written here are the important results:-
>> >>
>> >> Rwflowpack log
>> >> Oct 27 15:24:42 netflow rwflowpack[19538]: Closing incremental files...
>> >> Oct 27 15:24:42 netflow rwflowpack[19538]: Moving incremental files...
>> >> Oct 27 15:24:42 netflow rwflowpack[19538]: No incremental files to
>> move.
>> >> Oct 27 15:24:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> >> ignored 0, nf9: missing-pkts 0
>> >> Oct 27 15:26:42 netflow rwflowpack[19538]: Preparing to move
>> incremental
>> >> files...
>> >> Oct 27 15:26:42 netflow rwflowpack[19538]: Closing incremental files...
>> >> Oct 27 15:26:42 netflow rwflowpack[19538]: Moving incremental files...
>> >> Oct 27 15:26:42 netflow rwflowpack[19538]: No incremental files to
>> move.
>> >> Oct 27 15:26:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> >> ignored 0, nf9: missing-pkts 0
>> >> Oct 27 15:28:42 netflow rwflowpack[19538]: Preparing to move
>> incremental
>> >> files...
>> >> Oct 27 15:28:42 netflow rwflowpack[19538]: Closing incremental files...
>> >> Oct 27 15:28:42 netflow rwflowpack[19538]: Moving incremental files...
>> >> Oct 27 15:28:42 netflow rwflowpack[19538]: No incremental files to
>> move.
>> >> Oct 27 15:28:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> >> ignored 0, nf9: missing-pkts 0
>> >> Oct 27 15:30:42 netflow rwflowpack[19538]: Preparing to move
>> incremental
>> >> files...
>> >> Oct 27 15:30:42 netflow rwflowpack[19538]: Closing incremental files...
>> >> Oct 27 15:30:42 netflow rwflowpack[19538]: Moving incremental files...
>> >> Oct 27 15:30:42 netflow rwflowpack[19538]: No incremental files to
>> move.
>> >> Oct 27 15:30:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> >> ignored 0, nf9: missing-pkts 0
>> >> Oct 27 15:32:42 netflow rwflowpack[19538]: Preparing to move
>> incremental
>> >> files...
>> >> Oct 27 15:32:42 netflow rwflowpack[19538]: Closing incremental files...
>> >> Oct 27 15:32:42 netflow rwflowpack[19538]: Moving incremental files...
>> >> Oct 27 15:32:42 netflow rwflowpack[19538]: No incremental files to
>> move.
>> >> Oct 27 15:32:42 netflow rwflowpack[19538]: 'S1': forward 0, reverse 0,
>> >> ignored 0, nf9: missing-pkts 0
>> >> Oct 27 15:34:42 netflow rwflowpack[19538]: Preparing to move
>> incremental
>> >> files...
>> >> Oct 27 15:34:42 netflow rwflowpack[19538]: Closing incremental files...
>> >> Oct 27 15:34:42 netflow rwflowpack[19538]: Moving incremental files...
>> >> Oct 27 15:34:42 netflow rwflowpack[19538]: No incremental files to
>> move.
>> >>
>> >> Rwflowappend logs
>> >> farhan at netflow:/var/silk/rwflowappend/log$ cat
>> rwflowappend-20151027.log
>> >> Oct 27 13:49:06 netflow rwflowappend[19583]:
>> '/usr/local/sbin/rwflowappend'
>> >> '--incoming-directory=/var/silk/rwflowappend/incoming'
>> >> '--error-directory=/var/silk/rwflowappend/error'
>> >> '--root-directory=/home/farhan/data'
>> >> '--site-config-file=/home/farhan/data/silk.conf'
>> >> '--archive-directory=/var/silk/rwflowappend/archive' '--flat-archive'
>> >> '--pidfile=/var/silk/rwflowappend/log/rwflowappend.pid'
>> '--log-level=info'
>> >> '--log-directory=/var/silk/rwflowappend/log'
>> '--log-basename=rwflowappend'
>> >> Oct 27 13:49:06 netflow rwflowappend[19583]: Forked child 19585.
>> Parent
>> >> exiting
>> >> Oct 27 13:49:06 netflow rwflowappend[19585]: Starting 1 appender
>> thread...
>> >> Oct 27 13:49:06 netflow rwflowappend[19585]: Started all appender
>> threads.
>> >> Oct 27 13:49:06 netflow rwflowappend[19585]: Started appender thread
>> #1.
>> >>
>> >> Rwsender logs
>> >> Oct 27 13:49:17 netflow rwsender[19678]: '/usr/local/sbin/rwsender'
>> >> '--identifier=sender-1' '--server-port=34567'
>> '--client-ident=receiver-1'
>> >> '--client-ident=receiver-2' '--filter=receiver-2:^[^_]*_netflow_'
>> >> '--client-ident=receiver-3' '--mode=server'
>> >> '--incoming-directory=/var/silk/rwflowappend/archive'
>> >> '--processing-directory=/var/silk/rwsender/processing'
>> >> '--error-directory=/var/silk/rwsender/error' '--priority=100:^S[0-3]_'
>> >> '--priority=25:^S[7-9]_' '--local-directory=:/var/rwsender/local-dir1'
>> >> '--local-directory=auto-ident1:/var/rwsender/local-dir2'
>> >> '--filter=auto-ident1:^S[7-9]_'
>> >> '--pidfile=/var/silk/rwsender/log/rwsender.pid' '--log-level=info'
>> >> '--log-directory=/var/silk/rwsender/log' '--log-basename=rwsender'
>> >> Oct 27 13:49:17 netflow rwsender[19678]: Forked child 19680. Parent
>> >> exiting
>> >> Oct 27 13:49:17 netflow rwsender[19680]: Incoming file handling thread
>> >> started.
>> >> Oct 27 13:49:17 netflow rwsender[19680]: Bound to 34567 for listening
>> (TCP)
>> >>
>> >>
>> >> Even , I'm getting data written in /data dir but I don't think
>> rwflowappend
>> >> is doing its job (from the logs at least it seems so).
>> >>
>> >> Thanks.
>> >> regards
>> >> asad
>> >>
>> >>
>> >> On Mon, Oct 26, 2015 at 8:17 PM, asad <a.alii85 at gmail.com> wrote:
>> >>
>> >> > Thanks Mark, I'm running these config on production sys as soon as
>> >> > possible and notifying you about the results as they come. You are
>> great
>> >> > help as always:).
>> >> >
>> >> > regards
>> >> > Asad
>> >> >
>> >> > On Mon, Oct 26, 2015 at 7:52 PM, Mark Thomas <mthomas at cert.org>
>> wrote:
>> >> >
>> >> >> asad-
>> >> >>
>> >> >> The analysis pipeline is designed to work with the incremental files
>> >> >> produced by rwflowpack, and rwflowpack only produces those files
>> >> >> when it is paired with the rwflowappend process.
>> >> >>
>> >> >> If the analysis pipeline were running on the same machine as
>> >> >> rwflowpack, you would add pipeline and rwflowappend to the
>> >> >> configuration as mentioned in the pipeline manual page:
>> >> >>
>> >> >>
>> >>
>> http://tools.netsa.cert.org/analysis-pipeline/pipeline-manual.html#rwflowpack_only
>> >> >>
>> >> >> Since pipeline is running on a different machine, you need to modify
>> >> >> that configuration.
>> >> >>
>> >> >> Assuming your current configuration for rwflowpack is:
>> >> >>
>> >> >> rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
>> >> >> --log-directory=/var/silk/rwflowpack/log
>> >> >> --root-directory=/data
>> >> >>
>> >> >> You want to modify it as follows:
>> >> >>
>> >> >> rwflowpack --sensor-conf=/var/silk/rwflowpack/sensor.conf
>> >> >> --log-directory=/var/silk/rwflowpack/log
>> >> >> --output-mode=sending
>> >> >> --incremental-dir=/var/silk/rwflowpack/incremental
>> >> >> --sender-dir=/var/silk/rwflowappend/incoming
>> >> >>
>> >> >> rwflowappend --root-directory=/data
>> >> >> --log-directory=/var/silk/rwflowappend/log
>> >> >> --incoming-dir=/var/silk/rwflowappend/incoming
>> >> >> --error-dir=/var/silk/rwflowappend/error
>> >> >> --archive-dir=/var/silk/rwflowappend/archive
>> >> >> --flat-archive
>> >> >>
>> >> >> Have rwsender read the files from "/var/silk/rwflowappend/archive"
>> >> >> and send them to the rwreceiver process on the Live CD. Here I use
>> >> >> port "34567".
>> >> >>
>> >> >> rwsender --mode=server --server-port=34567
>> >> >> --identifier=SENDER --client-ident RECEIVER
>> >> >> --log-directory=/var/silk/rwsender/log
>> >> >> --incoming-directory=/var/silk/rwflowappend/archive
>> >> >> --processing-directory=/var/silk/rwsender/processing
>> >> >> --error-directory=/var/silk/rwsender/error
>> >> >>
>> >> >> On the Live CD:
>> >> >>
>> >> >> rwreceiver --mode=client --server-address=SENDER:10.2.3.4:34567
>> >> >> --identifier=RECEIVER --client-ident RECEIVER
>> >> >> --log-directory=/var/silk/rwreceiver/log
>> >> >> --destination-directory=/var/pipeline/incoming
>> >> >>
>> >> >> where "10.2.3.4" is the IP address (or hostname) of the machine were
>> >> >> rwsender is running. If necessary, open a hole in the firewall to
>> >> >> allow receiver to connect to rwsender.
>> >> >>
>> >> >> Finally, configure pipeline to read the files from the
>> >> >> /var/pipeline/incoming directory on the Live CD.
>> >> >>
>> >> >> pipeline --incoming-directory=/var/pipeline/incoming
>> >> >> --error-directory=/var/pipeline/error
>> >> >> --log-directory=/var/pipeline/log
>> >> >> --configuration-file=/var/pipeline/pipeline.conf ...
>> >> >>
>> >> >> -Mark
>> >> >>
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: asad <a.alii85 at gmail.com>
>> >> >> Date: Sat, 24 Oct 2015 16:09:11 +0500
>> >> >> To: <netsa-tools-discuss at cert.org>
>> >> >> Subject: [netsa-tools-discuss] arch options for separating
>> >> >> analysis-pipeline
>> >> >> from (collection+storage)
>> >> >>
>> >> >> Hey,
>> >> >>
>> >> >>
>> >> >> I have successfully running SILK with little issues, now I want to
>> bring
>> >> >> in
>> >> >> use of analysis - pipeline, but on a separating machine which
>> infact is
>> >> >> live dvd provided by silk.
>> >> >>
>> >> >> Please view attach (diagram) for understand current deployment
>> settings.
>> >> >>
>> >> >> Ideally, I just wanted use of rwsender as a server running on silk
>> which
>> >> >> is
>> >> >> doing collection and storage to send logs to analysis-pipeline box
>> >> running
>> >> >> rwreceiver.
>> >> >>
>> >> >> What I have read so far, that I have to use flowcap If I want to
>> turn
>> >> into
>> >> >> distributed model but so far this is not my cases.
>> >> >>
>> >> >> So, my question essentially is that by using rwsender running on
>> >> >> rwflowpack
>> >> >> machine can it send incremental flow records to analysis-pipeline
>> >> machine.
>> >> >> If yes, how can it be done? Do I need to have flowcap running ?
>> >> >>
>> >> >> I appreciate clarity on this issue. Thanks
>> >> >>
>> >> >> regards
>> >> >> asad
>> >> >>
>> >> >
>> >> >
>> >>
>>
>
>
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list