[netsa-tools-discuss] Rwflowpack config questions

[Chacko P]

Hi Mark,

I ran a few tests and the attached PDF document captures some of my observations. The observations are that rwflowpack buffering still prevents me from getting an accurate view into the ‘real-time’ flows (please see the observations in the first run).   
Could you please confirm that the assumptions (guesses) that I have made in my questions? Thank you.
Separately, I am facing challenges with the rwfilter | rwsort | rwuniq query embedded within the Python code of the application, but I’ll have to test that a bit more before I ask questions.
Thanks,
Chacko


________________________________________
From: Mark Thomas <mthomas at cert.org>
Sent: Monday, December 7, 2015 9:04 PM
To: Chacko P
Cc: netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] Rwflowpack config questions

YAF is the tool that reads packet data and converts the packets into
flow records.  By default, YAF uses a 30 minute active timeout (for
long-lived flow records such as ssh sessions) and a 5 minute idle
timeout.

To get data closer to "real time", configure YAF to use 30 second
timeouts by either specifying these switches on the command line:

 --active-timeout=30 --idle-timeout=30

or adding those switches to the YAF_EXTRAFLAGS variable in the
yaf.conf file:

 YAF_EXTRAFLAGS="--silk --active-timeout=30 --idle-timeout=30"

Reducing these timeouts will increase the number of flow records
that YAF produces and the number of records that rwflowpack must
process.

The rwflowpack tool reads flow records and writes them to disk.  The
FLUSH_TIMEOUT setting (which sets the --flush-timeout switch)
determines how often the flow records in rwflowpack's memory are
written to disk.  When rwflowpack is processing flow records from a
busy link, rwflowpack is regularly copying records from its memory
to disk.


Setting LOG_POST_ROTATE to 'rm %s' should remove the previous day's
log file.  Since the post-rotate process runs with the same
privileges as rwflowpack, the leading "sudo" is not necessary.


I hope that helps.

-Mark


-----Original Message-----
From: Chacko P <chacko.p at incism.com>
Date: Sun, 6 Dec 2015 12:57:52 +0000
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Rwflowpack config questions

Hello,

I have a couple of questions regarding a single machine install of
rwflowpack and yaf.

My questions are as follows:

  1.  Our application that uses SiLK has some measure of real time
flow visualization.  Toward this end the config file has the entry
FLUSH_TIMEOUT=30 with the intent to get recent flow records. This
setting does not have an apparent effect as traffic that that flowed 5
minutes prior does not register with the rwfilter command with a
relevant time filter. It does, though, if the rwflowpack process is
cycled, which I expect cause a flush to disk. Is there some other
setting that needs to be configured as well to achieve this? Is there
an alternate way to get flow details up to, say, the most recent
minute?

  2.  The log directory is being filled with rwflowpack....gz
files. So I introduced the following line in the conf
file. LOG_POST_ROTATE='sudo rm %'. It's a guess at best since I was
not able to locate a working example. Will this work? Or does the
statement '(Old log files are not removed by rwflowpack; the
administrator should use another tool to remove them.)' supersede even
this and should look for another tool.

Any help would be greatly appreciated. If there is a document that has
the answers to these questions, then please point me to it.


 Thanks,



Chacko P., CISSP

Principal Consultant, Information Security.

Incism

email<mailto:chacko.p at incism.com>| web<http://www.incism.com/> |
profile<https://in.linkedin.com/in/chackopallathucheril> | social

Incisive views. Truly.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SiLK_tests.pdf
Type: application/pdf
Size: 275853 bytes
Desc: SiLK_tests.pdf
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20151210/7abe80bc/attachment.pdf>