[netsa-tools-discuss] alernate use of rwstats --percentage
Mark Thomas
mthomas at cert.org
Wed Sep 23 16:26:35 EDT 2015
Thank you for providing the versions of the tools you are running
and the command lines you are using to invoke the tools.
The order in which things are started is important.
rwflowpack only checks the SILK_IPFIX_PRINT_TEMPLATES variable once
when it is invoked, so the variable must be set prior to starting
rwflowpack. The variable must be set to a non-empty value, and the
first character should not be 0.
You are running YAF over a TCP connection. In this case, YAF only
sends the IPFIX templates one time--typically when YAF first
connects to rwflowpack.
For rwflowpack to print the templates, you must:
1. Set the SILK_IPFIX_PRINT_TEMPLATES environment variable to 1.
2. Start rwflowpack.
3. Start YAF.
When YAF connects to rwflowpack, the log file will contain lines
that look similar to:
Sep 23 16:14:42 mac rwflowpack[65061]: 'Irouter': accepted connection from 127.0.0.1:55901
Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID 0xC013, Contains 6 Elements, Enabled by SILK_IPFIX_PRINT_TEMPLATES
Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID 0xC013, Position 0, Length 4, IE 184, Name tcpSequenceNumber
Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID 0xC013, Position 1, Length 1, IE 6871/ 14, Name initialTCPFlags
Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID 0xC013, Position 2, Length 1, IE 6871/ 15, Name unionTCPFlags
...
In my set-up, I saw 11 templates:
TemplateID 0xC013, Contains 6 Elements
TemplateID 0xC015, Contains 23 Elements
TemplateID 0xC016, Contains 6 Elements
TemplateID 0xC018, Contains 2 Elements
TemplateID 0xB800, Contains 39 Elements
TemplateID 0xC003, Contains 3 Elements
TemplateID 0xC004, Contains 2 Elements
TemplateID 0xC005, Contains 11 Elements
TemplateID 0xC006, Contains 3 Elements
TemplateID 0xD000, Contains 14 Elements
TemplateID 0xC008, Contains 1 Elements
I hope that helps.
-Mark
On Thu, 24 Sep 2015 00:29:10 +0500, asad wrote:
> Thanks Thomas, for such comprehensive explanation. From silk-summary.txt
> file I can verify that my
>
> SILK 3.10.2
> libfixbuf-1.7.0 >= 1.6.0
>
> The command I used is
>
> rwflowpack \
> --sensor-configuration=/data/sensors.conf \
> --site-config-file=/data/silk.conf \
> --archive-directory=/usr/local/var/lib/rwflowpack/archive \
> --output-mode=local-storage \
> --root-directory=/data \
> --pidfile=/var/log/rwflowpack.pid --log-level=info \
> --log-directory=/var/log --log-basename=rwflowpack \
>
> But I don't see in logs any "information element". I'm using tcpreplay
> tool, to replay a sample pcap files (since I'm home now), but in logs I see
> as
>
> "Sep 24 00:24:04 Silky-flows rwflowpack[812]: 'S0': forward 0, reverse 0,
> ignored 0; yaf: recs 0, pkts 0, dropped-pkts 0, ignored-pkts 0,
> bad-sequence-pkts 0, expired-frags 0"
>
> I have tested with cmd
>
> "
>
> yaf --silk --ipfix=tcp --live=pcap --out=127.0.0.1 \
> --ipfix-port=18001 --in=eth0 --applabel --max-payload=384 &"
>
>
> and it works fine logs are made as they should.
>
>
> On Wed, Sep 23, 2015 at 10:26 PM, Mark Thomas <mthomas at cert.org> wrote:
>
>> The way that you set the environment variable depends on how your
>> are starting rwflowpack or flowcap.
>>
>> 1. Using the start-up scripts.
>>
>> Follow these instructions if you set variables in the the
>> rwflowpack.conf configuration file and then run the rwflowpack shell
>> script as
>>
>> rwflowpack start
>>
>> The rwflowpack shell script is typically installed in the directory
>> $prefix/share/silk/etc/init.d. In the RedHat RPMs, the start-up
>> script is installed in /etc/init.d/rwflowpack.
>>
>> These instructions also apply for flowcap.
>>
>> Find the rwflowpack start-up script or the flowcap start-up script.
>> Within that script, find the start() subroutine. In that
>> subroutine, find the following:
>>
>> if [ X`whoami` = "X${USER}" ] ; then
>> eval "${PROG_PATH} ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>> else
>> su - ${USER} -c "${PROG_PATH} ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>> fi
>>
>> Insert SILK_IPFIX_PRINT_TEMPLATES=1 after the initial double quote.
>> The result should read:
>>
>> if [ X`whoami` = "X${USER}" ] ; then
>> eval "SILK_IPFIX_PRINT_TEMPLATES=1 ${PROG_PATH} ${PROG_OPTIONS}
>> ${EXTRA_OPTIONS} &"
>> else
>> su - ${USER} -c "SILK_IPFIX_PRINT_TEMPLATES=1 ${PROG_PATH}
>> ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>> fi
>>
>>
>>
>> 2. Starting from the command line.
>>
>> Follow these instructions if you invoke the rwflowpack or flowcap
>> binary from the command line and specify its command line options as
>> part of the command, such as
>>
>> rwflowpack --root-directory=/data ...
>>
>> These binaries are typically installed in $prefix/sbin.
>>
>> In this case, you can set the SILK_IPFIX_PRINT_TEMPLATES environment
>> variable as you set any other environment variable in a modern
>> Bourne-compatible shell:
>>
>> export SILK_IPFIX_PRINT_TEMPLATES=1
>> rwflowpack --root-directory=/data ...
>>
>>
>>
>> This feature requires SiLK 3.8.2 or newer and libfixbuf-1.4.0 or
>> newer.
>>
>> -Mark
>>
>>
>> On Wed, 23 Sep 2015 13:33:06 +0500, asad wrote:
>>
>> > Thomas,
>> >
>> > Can you educate me how to set env variable "SILK_IPFIX_PRINT_TEMPLATES"?
>> >
>> > On 9/21/15, Mark Thomas <mthomas at cert.org> wrote:
>> >> I have a couple of PCAP files that contain data from a Cisco ASA,
>> >> and the NetFlow v9 templates do not include Information Element 6,
>> >> TCP_FLAGS.
>> >>
>> >> If you wish to confirm this for yourself, set the
>> >> SILK_IPFIX_PRINT_TEMPLATES environment variable to 1 prior to
>> >> starting rwflowpack or flowcap. With that variable set, rwflowpack
>> >> or flowcap print to its log file each IPFIX/NetFlow v9 template it
>> >> receives.
>> >>
>> >> When the tool prints the template, it uses the IPFIX names for the
>> >> information elements, which you can find at
>> >> http://www.iana.org/assignments/ipfix/ipfix.xhtml
>> >>
>> >> The IPFIX name for TCP_FLAGS is tcpControlBits.
>> >>
>> >> -Mark
>> >>
>> >>
>> >> On Fri, 18 Sep 2015 10:54:56 +0500, asad wrote:
>> >>
>> >>> This is update.
>> >>>
>> >>> I have edited the cmd as told by "Angela", now I get a very useful
>> >>> output which shows that the %Bytes value is never greater then
>> >>> 0.025112, this explains why the percentage =1 was not working.
>> >>>
>> >>> But what more strange is now, I start to question the suitability of
>> >>> cisco asa asel netflows logs here is reason why ..
>> >>>
>> >>>
>> >>> changing the command and adding params e.g --packets=4- --ack-flag=1
>> >>> delivers me zero output. Even more strange (please see attachment)
>> >>> that flags columns is empty. ( it is even empty when there is no
>> >>> --ack-flag=1 value set).
>> >>>
>> >>>
>> >>>
>> >>> On 9/16/15, asad <a.alii85 at gmail.com> wrote:
>> >>>> Thanks Angela and Evgeniy. I believe I have been unfair to part I have
>> >>>> explained my case effectively.
>> >>>>
>> >>>> There is a critical server on enterprise network who traffic I want to
>> >>>> monitor for following usage :-
>> >>>>
>> >>>> " to monitor connection to and from the server w.r.t to bytes "
>> >>>>
>> >>>> This is done to get some way of knowing "normal" behavior for the
>> >>>> traffic
>> >>>> i.e # between server and client no of bytes send per day.
>> >>>>
>> >>>> I will try out the suggestions as soon i get access to office network
>> >>>> (currently I'm at home) and will update accordingly.
>> >>>>
>> >>>> On Wed, Sep 16, 2015 at 6:22 PM, Angela Horneman <ahorneman at cert.org>
>> >>>> wrote:
>> >>>>
>> >>>>> Asad,
>> >>>>>
>> >>>>> If you replace --percentage=1 with --count=10 in your first example,
>> >>>>> there
>> >>>>> will be a column "%Bytes" in the output. You can use that column to
>> >>>>> check
>> >>>>> if any of the 10 DIPs with the greatest byte volumes have a volume
>> that
>> >>>>> is
>> >>>>> at least 1% of the total.
>> >>>>>
>> >>>>> Angela
>> >>>>>
>> >>>>> -----Original Message-----
>> >>>>> From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org
>> [mailto:
>> >>>>> netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org] On Behalf
>> Of
>> >>>>> asad
>> >>>>> Sent: Wednesday, September 16, 2015 12:49 AM
>> >>>>> To: Evgeniy Sudyr <eject.in.ua at gmail.com>
>> >>>>> Cc: netsa-tools-discuss at cert.org
>> >>>>> Subject: Re: [netsa-tools-discuss] alernate use of rwstats
>> --percentage
>> >>>>>
>> >>>>> Thanks Eugene,
>> >>>>>
>> >>>>> My output is:-
>> >>>>>
>> >>>>> sIP |sPort| dIP| dPort| bytes|
>> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>> >>>>>
>> >>>>>
>> >>>>> With command
>> >>>>>
>> >>>>> rwfilter --sensor=S0 --type=all --pass=stdout --saddress=10.10.13.152
>> >>>>> --start-date=2015/09/13:15 --end-date=2015/09/16:15 | rwsort
>> >>>>> --fields=bytes | rwcut --fields=sip,sport,dip,dport,bytes
>> >>>>>
>> >>>>> I'm getting bytes in last column, but as a percentage of total bytes
>> >>>>> from
>> >>>>> all records I don't know how to get that.
>> >>>>>
>> >>>>> thanks.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On 9/15/15, Evgeniy Sudyr <eject.in.ua at gmail.com> wrote:
>> >>>>> > Ai,
>> >>>>> >
>> >>>>> > are you sure that in your rwfilter results you have more than 1%
>> of
>> >>>>> Bytes?
>> >>>>> >
>> >>>>> > From rwstats man page:
>> >>>>> >
>> >>>>> > *--percentage*=*N* Print the bins where the primary value is
>> >>>>> > greater-than (or less-than) *N* percent of the sum of the primary
>> >>>>> > values across all bins.
>> >>>>> >
>> >>>>> >
>> >>>>> > I think it will be useful to see --count --Packets
>> >>>>> >
>> >>>>> >
>> >>>>> > On Tue, Sep 15, 2015 at 5:31 PM, asad <a.alii85 at gmail.com> wrote:
>> >>>>> >
>> >>>>> >> Hi,
>> >>>>> >>
>> >>>>> >> I want to know what "alternate options" exists for following:-
>> >>>>> >>
>> >>>>> >> rwfilter --sensor=Vrouter1 --type=out --sport=3306 --pass=stdout
>> >>>>> >> --start-date=2012/11/13:00 --end-date=2012/11/13:23
>> >>>>> >> --saddress=172.31.253.102 | rwstats --percentage=1 --bytes
>> >>>>> >> --fields=dip
>> >>>>> >>
>> >>>>> >> I don't know why but using --percentage=1 flag, I get zero
>> results,
>> >>>>> >> even when in records I know this IP is present. Is there any
>> reason
>> >>>>> >> why would such happpen?
>> >>>>> >>
>> >>>>> >> Or I can move to another rwstats switch parameters to perform same
>> >>>>> >> task as trying to achieve with percentage=1
>> >>>>> >>
>> >>>>> >> Thanks.
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >
>> >>>>> >
>> >>>>> > --
>> >>>>> > --
>> >>>>> > With regards,
>> >>>>> > Eugene Sudyr
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>
>> >>
>>
More information about the netsa-tools-discuss
mailing list