[netsa-tools-discuss] Rwflowpack config questions

[Mark Thomas]

Chacko-

The difference in the byte counts reported by scapy and YAF is that
YAF does not include the 14-byte Ethernet header in its byte counts.
You can see this clearly if you look at the values returned in
second run when you use a timeout of 3600 seconds:

  $ perl -lwe 'print((3880464 - 3813726) / 4767)'
  14

The lower byte count in the first run (with the 30 second timeouts)
is a reflection of the lower packet count in that run.  My guess is
that the lower packet count is because rwflowpack failed to process
all the flow records that YAF generated.  To confirm this, read the
rwflowpack log messages (in either the system log or the rwflowpack
log file) and search for "message out of sequence" errors.

To confirm that YAF is processing all the packet data, include the
--verbose switch on the YAF command line.  When you exit YAF or send
it the USR1 signal, it reports the amount of data it is processed
and the number of flows it generated.  (These counts may already be
present in the rwflowpack log file since YAF sends a "stats" packet
to rwflowpack every so often.)

Your assumption for Q4 is correct, the lower record count in the
second run is because YAF was able create flow records with longer
durations and more packets.

Depending on how close to real time you want, SiLK may not be the
best solution.  Since YAF and SiLK are combine packets into flow
records, there is always going to be some amount of delay between
the time when a packet arrives and when a record containing that
summarizes that connection is available.

Cheers,

-Mark


-----Original Message-----
From: Chacko P <chacko.p at incism.com>
Date: Thu, 10 Dec 2015 15:37:38 +0000
To: Mark Thomas <mthomas at cert.org>
Cc: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] Rwflowpack config questions

Hi Mark,

I ran a few tests and the attached PDF document captures some of my
observations. The observations are that rwflowpack buffering still
prevents me from getting an accurate view into the ‘real-time’ flows
(please see the observations in the first run).
Could you please confirm that the assumptions (guesses) that I have made in my questions? Thank you.
Separately, I am facing challenges with the rwfilter | rwsort | rwuniq
query embedded within the Python code of the application, but I’ll
have to test that a bit more before I ask questions.
Thanks,
Chacko


________________________________________
From: Mark Thomas <mthomas at cert.org>
Sent: Monday, December 7, 2015 9:04 PM
To: Chacko P
Cc: netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] Rwflowpack config questions

YAF is the tool that reads packet data and converts the packets into
flow records.  By default, YAF uses a 30 minute active timeout (for
long-lived flow records such as ssh sessions) and a 5 minute idle
timeout.

To get data closer to "real time", configure YAF to use 30 second
timeouts by either specifying these switches on the command line:

 --active-timeout=30 --idle-timeout=30

or adding those switches to the YAF_EXTRAFLAGS variable in the
yaf.conf file:

 YAF_EXTRAFLAGS="--silk --active-timeout=30 --idle-timeout=30"

Reducing these timeouts will increase the number of flow records
that YAF produces and the number of records that rwflowpack must
process.

The rwflowpack tool reads flow records and writes them to disk.  The
FLUSH_TIMEOUT setting (which sets the --flush-timeout switch)
determines how often the flow records in rwflowpack's memory are
written to disk.  When rwflowpack is processing flow records from a
busy link, rwflowpack is regularly copying records from its memory
to disk.


Setting LOG_POST_ROTATE to 'rm %s' should remove the previous day's
log file.  Since the post-rotate process runs with the same
privileges as rwflowpack, the leading "sudo" is not necessary.


I hope that helps.

-Mark


-----Original Message-----
From: Chacko P <chacko.p at incism.com>
Date: Sun, 6 Dec 2015 12:57:52 +0000
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Rwflowpack config questions

Hello,

I have a couple of questions regarding a single machine install of
rwflowpack and yaf.

My questions are as follows:

  1.  Our application that uses SiLK has some measure of real time
flow visualization.  Toward this end the config file has the entry
FLUSH_TIMEOUT=30 with the intent to get recent flow records. This
setting does not have an apparent effect as traffic that that flowed 5
minutes prior does not register with the rwfilter command with a
relevant time filter. It does, though, if the rwflowpack process is
cycled, which I expect cause a flush to disk. Is there some other
setting that needs to be configured as well to achieve this? Is there
an alternate way to get flow details up to, say, the most recent
minute?

  2.  The log directory is being filled with rwflowpack....gz
files. So I introduced the following line in the conf
file. LOG_POST_ROTATE='sudo rm %'. It's a guess at best since I was
not able to locate a working example. Will this work? Or does the
statement '(Old log files are not removed by rwflowpack; the
administrator should use another tool to remove them.)' supersede even
this and should look for another tool.

Any help would be greatly appreciated. If there is a document that has
the answers to these questions, then please point me to it.


 Thanks,



Chacko P., CISSP

Principal Consultant, Information Security.

Incism

email<mailto:chacko.p at incism.com>| web<http://www.incism.com/> |
profile<https://in.linkedin.com/in/chackopallathucheril> | social

Incisive views. Truly.