[netsa-tools-discuss] alernate use of rwstats --percentage
Mark Thomas
mthomas at cert.org
Fri Sep 25 14:06:56 EDT 2015
With the environment variable set, you will see the template
information in the rwflowpack log file whenever rwflowpack receives
an IPFIX template from yaf or a NetFlow v9 template from your
router.
The template information printed by rwflowpack uses the IPFIX names
for each information element. The list of IPFIX information
elements is maintained by IANA
http://www.iana.org/assignments/ipfix/ipfix.xhtml
In rwflowpack, a source IPv4 address will be printed as:
Domain 0x0000, TemplateID 0x0108, Position 6, Length 4, IE 8, Name sourceIPv4Address
where "IE 8" is the numeric identifier for the information element.
The NetFlow v9 names for information elements 1 to 79 can be found
in the NetFlow v9 RFC. http://tools.ietf.org/html/rfc3954
When using the CFLOW filter in Wireshark to view either IPFIX or
NetFlow v9 template, a source IPv4 address will be printed as:
Field (6/21): IP_SRC_ADDR
Type: IP_SRC_ADDR (8)
Length: 4
The names used by Wireshark are sometimes the same as the NetFlow v9
names.
Information element 6 contains the union of TCP flags across all
packets in a TCP flow record. The information element is called
tcpControlBits by IPFIX, and it is called TCP_FLAGS by Cisco and
Wireshark.
However, if the template does not contain this element, there is no
way for SiLK to get the TCP Flags.
Note that Cisco says they do not populate the TCP Flags fields on
the ASA.
https://supportforums.cisco.com/document/30471/netflow-asa
NetFlow on the ASA vs IOS
The ASA only supports NetFlow version 9 and there are no plans to
support NetFlow version 5. NetFlow on the ASA is event
driven. Unlike routing platforms we do not send incremental
updates; NSEL records are only sent during flow creation, teardown
or ACL deny events. Also unlike the routing platforms we will not
populate the ToS bits or the TCP flags. Lastly, all flows on the
ASA are bidirectional. All counters for a flow will increase for
traffic flowing from A->B or B->A.
I hope that helps.
-Mark
On Thu, 24 Sep 2015 13:55:37 +0500, asad wrote:
> You were right,there was problem with the order I was running the commands,
> Now I got it to work and I see the logs are being populated, now the only
> thing missing is " Information Element 6,TCP_FLAGS." which I'm interested
> in.
>
> I'm using pcap files which is non-cisco asa based, I just couldn't find one
> that matches my production env e.g cisco 5585-x. If you can safely if size
> and privacy allows you can send me some sample pcap files to replay with.
>
> Or perhaps without the required pcap files I'm still supposed to see some
> "template info" in the logs, In this case I go back to point of config
> SILK_IPFIX_PRINT_TEMPLATES.
>
> Also, in the ps -auxx | grep rwflowpack
>
> I also see /etc/init.d/rwflowpack though I never used rwflowpack.conf. I
> hope there is no conflict.
>
> On Thu, Sep 24, 2015 at 1:26 AM, Mark Thomas <mthomas at cert.org> wrote:
>
>> Thank you for providing the versions of the tools you are running
>> and the command lines you are using to invoke the tools.
>>
>> The order in which things are started is important.
>>
>> rwflowpack only checks the SILK_IPFIX_PRINT_TEMPLATES variable once
>> when it is invoked, so the variable must be set prior to starting
>> rwflowpack. The variable must be set to a non-empty value, and the
>> first character should not be 0.
>>
>> You are running YAF over a TCP connection. In this case, YAF only
>> sends the IPFIX templates one time--typically when YAF first
>> connects to rwflowpack.
>>
>> For rwflowpack to print the templates, you must:
>>
>> 1. Set the SILK_IPFIX_PRINT_TEMPLATES environment variable to 1.
>>
>> 2. Start rwflowpack.
>>
>> 3. Start YAF.
>>
>> When YAF connects to rwflowpack, the log file will contain lines
>> that look similar to:
>>
>> Sep 23 16:14:42 mac rwflowpack[65061]: 'Irouter': accepted connection
>> from 127.0.0.1:55901
>> Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID 0xC013,
>> Contains 6 Elements, Enabled by SILK_IPFIX_PRINT_TEMPLATES
>> Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID 0xC013,
>> Position 0, Length 4, IE 184, Name tcpSequenceNumber
>> Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID 0xC013,
>> Position 1, Length 1, IE 6871/ 14, Name initialTCPFlags
>> Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID 0xC013,
>> Position 2, Length 1, IE 6871/ 15, Name unionTCPFlags
>> ...
>>
>> In my set-up, I saw 11 templates:
>>
>> TemplateID 0xC013, Contains 6 Elements
>> TemplateID 0xC015, Contains 23 Elements
>> TemplateID 0xC016, Contains 6 Elements
>> TemplateID 0xC018, Contains 2 Elements
>> TemplateID 0xB800, Contains 39 Elements
>> TemplateID 0xC003, Contains 3 Elements
>> TemplateID 0xC004, Contains 2 Elements
>> TemplateID 0xC005, Contains 11 Elements
>> TemplateID 0xC006, Contains 3 Elements
>> TemplateID 0xD000, Contains 14 Elements
>> TemplateID 0xC008, Contains 1 Elements
>>
>> I hope that helps.
>>
>> -Mark
>>
>>
>> On Thu, 24 Sep 2015 00:29:10 +0500, asad wrote:
>>
>> > Thanks Thomas, for such comprehensive explanation. From silk-summary.txt
>> > file I can verify that my
>> >
>> > SILK 3.10.2
>> > libfixbuf-1.7.0 >= 1.6.0
>> >
>> > The command I used is
>> >
>> > rwflowpack \
>> > --sensor-configuration=/data/sensors.conf \
>> > --site-config-file=/data/silk.conf \
>> > --archive-directory=/usr/local/var/lib/rwflowpack/archive \
>> > --output-mode=local-storage \
>> > --root-directory=/data \
>> > --pidfile=/var/log/rwflowpack.pid --log-level=info \
>> > --log-directory=/var/log --log-basename=rwflowpack \
>> >
>> > But I don't see in logs any "information element". I'm using tcpreplay
>> > tool, to replay a sample pcap files (since I'm home now), but in logs I
>> see
>> > as
>> >
>> > "Sep 24 00:24:04 Silky-flows rwflowpack[812]: 'S0': forward 0, reverse 0,
>> > ignored 0; yaf: recs 0, pkts 0, dropped-pkts 0, ignored-pkts 0,
>> > bad-sequence-pkts 0, expired-frags 0"
>> >
>> > I have tested with cmd
>> >
>> > "
>> >
>> > yaf --silk --ipfix=tcp --live=pcap --out=127.0.0.1 \
>> > --ipfix-port=18001 --in=eth0 --applabel --max-payload=384 &"
>> >
>> >
>> > and it works fine logs are made as they should.
>> >
>> >
>> > On Wed, Sep 23, 2015 at 10:26 PM, Mark Thomas <mthomas at cert.org> wrote:
>> >
>> >> The way that you set the environment variable depends on how your
>> >> are starting rwflowpack or flowcap.
>> >>
>> >> 1. Using the start-up scripts.
>> >>
>> >> Follow these instructions if you set variables in the the
>> >> rwflowpack.conf configuration file and then run the rwflowpack shell
>> >> script as
>> >>
>> >> rwflowpack start
>> >>
>> >> The rwflowpack shell script is typically installed in the directory
>> >> $prefix/share/silk/etc/init.d. In the RedHat RPMs, the start-up
>> >> script is installed in /etc/init.d/rwflowpack.
>> >>
>> >> These instructions also apply for flowcap.
>> >>
>> >> Find the rwflowpack start-up script or the flowcap start-up script.
>> >> Within that script, find the start() subroutine. In that
>> >> subroutine, find the following:
>> >>
>> >> if [ X`whoami` = "X${USER}" ] ; then
>> >> eval "${PROG_PATH} ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>> >> else
>> >> su - ${USER} -c "${PROG_PATH} ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>> >> fi
>> >>
>> >> Insert SILK_IPFIX_PRINT_TEMPLATES=1 after the initial double quote.
>> >> The result should read:
>> >>
>> >> if [ X`whoami` = "X${USER}" ] ; then
>> >> eval "SILK_IPFIX_PRINT_TEMPLATES=1 ${PROG_PATH} ${PROG_OPTIONS}
>> >> ${EXTRA_OPTIONS} &"
>> >> else
>> >> su - ${USER} -c "SILK_IPFIX_PRINT_TEMPLATES=1 ${PROG_PATH}
>> >> ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>> >> fi
>> >>
>> >>
>> >>
>> >> 2. Starting from the command line.
>> >>
>> >> Follow these instructions if you invoke the rwflowpack or flowcap
>> >> binary from the command line and specify its command line options as
>> >> part of the command, such as
>> >>
>> >> rwflowpack --root-directory=/data ...
>> >>
>> >> These binaries are typically installed in $prefix/sbin.
>> >>
>> >> In this case, you can set the SILK_IPFIX_PRINT_TEMPLATES environment
>> >> variable as you set any other environment variable in a modern
>> >> Bourne-compatible shell:
>> >>
>> >> export SILK_IPFIX_PRINT_TEMPLATES=1
>> >> rwflowpack --root-directory=/data ...
>> >>
>> >>
>> >>
>> >> This feature requires SiLK 3.8.2 or newer and libfixbuf-1.4.0 or
>> >> newer.
>> >>
>> >> -Mark
>> >>
>> >>
>> >> On Wed, 23 Sep 2015 13:33:06 +0500, asad wrote:
>> >>
>> >> > Thomas,
>> >> >
>> >> > Can you educate me how to set env variable
>> "SILK_IPFIX_PRINT_TEMPLATES"?
>> >> >
>> >> > On 9/21/15, Mark Thomas <mthomas at cert.org> wrote:
>> >> >> I have a couple of PCAP files that contain data from a Cisco ASA,
>> >> >> and the NetFlow v9 templates do not include Information Element 6,
>> >> >> TCP_FLAGS.
>> >> >>
>> >> >> If you wish to confirm this for yourself, set the
>> >> >> SILK_IPFIX_PRINT_TEMPLATES environment variable to 1 prior to
>> >> >> starting rwflowpack or flowcap. With that variable set, rwflowpack
>> >> >> or flowcap print to its log file each IPFIX/NetFlow v9 template it
>> >> >> receives.
>> >> >>
>> >> >> When the tool prints the template, it uses the IPFIX names for the
>> >> >> information elements, which you can find at
>> >> >> http://www.iana.org/assignments/ipfix/ipfix.xhtml
>> >> >>
>> >> >> The IPFIX name for TCP_FLAGS is tcpControlBits.
>> >> >>
>> >> >> -Mark
>> >> >>
>> >> >>
>> >> >> On Fri, 18 Sep 2015 10:54:56 +0500, asad wrote:
>> >> >>
>> >> >>> This is update.
>> >> >>>
>> >> >>> I have edited the cmd as told by "Angela", now I get a very useful
>> >> >>> output which shows that the %Bytes value is never greater then
>> >> >>> 0.025112, this explains why the percentage =1 was not working.
>> >> >>>
>> >> >>> But what more strange is now, I start to question the suitability of
>> >> >>> cisco asa asel netflows logs here is reason why ..
>> >> >>>
>> >> >>>
>> >> >>> changing the command and adding params e.g --packets=4- --ack-flag=1
>> >> >>> delivers me zero output. Even more strange (please see attachment)
>> >> >>> that flags columns is empty. ( it is even empty when there is no
>> >> >>> --ack-flag=1 value set).
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On 9/16/15, asad <a.alii85 at gmail.com> wrote:
>> >> >>>> Thanks Angela and Evgeniy. I believe I have been unfair to part I
>> have
>> >> >>>> explained my case effectively.
>> >> >>>>
>> >> >>>> There is a critical server on enterprise network who traffic I
>> want to
>> >> >>>> monitor for following usage :-
>> >> >>>>
>> >> >>>> " to monitor connection to and from the server w.r.t to bytes "
>> >> >>>>
>> >> >>>> This is done to get some way of knowing "normal" behavior for the
>> >> >>>> traffic
>> >> >>>> i.e # between server and client no of bytes send per day.
>> >> >>>>
>> >> >>>> I will try out the suggestions as soon i get access to office
>> network
>> >> >>>> (currently I'm at home) and will update accordingly.
>> >> >>>>
>> >> >>>> On Wed, Sep 16, 2015 at 6:22 PM, Angela Horneman <
>> ahorneman at cert.org>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>>> Asad,
>> >> >>>>>
>> >> >>>>> If you replace --percentage=1 with --count=10 in your first
>> example,
>> >> >>>>> there
>> >> >>>>> will be a column "%Bytes" in the output. You can use that column
>> to
>> >> >>>>> check
>> >> >>>>> if any of the 10 DIPs with the greatest byte volumes have a volume
>> >> that
>> >> >>>>> is
>> >> >>>>> at least 1% of the total.
>> >> >>>>>
>> >> >>>>> Angela
>> >> >>>>>
>> >> >>>>> -----Original Message-----
>> >> >>>>> From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org
>> >> [mailto:
>> >> >>>>> netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org] On
>> Behalf
>> >> Of
>> >> >>>>> asad
>> >> >>>>> Sent: Wednesday, September 16, 2015 12:49 AM
>> >> >>>>> To: Evgeniy Sudyr <eject.in.ua at gmail.com>
>> >> >>>>> Cc: netsa-tools-discuss at cert.org
>> >> >>>>> Subject: Re: [netsa-tools-discuss] alernate use of rwstats
>> >> --percentage
>> >> >>>>>
>> >> >>>>> Thanks Eugene,
>> >> >>>>>
>> >> >>>>> My output is:-
>> >> >>>>>
>> >> >>>>> sIP |sPort| dIP| dPort| bytes|
>> >> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>> >> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>> >> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> With command
>> >> >>>>>
>> >> >>>>> rwfilter --sensor=S0 --type=all --pass=stdout
>> --saddress=10.10.13.152
>> >> >>>>> --start-date=2015/09/13:15 --end-date=2015/09/16:15 | rwsort
>> >> >>>>> --fields=bytes | rwcut --fields=sip,sport,dip,dport,bytes
>> >> >>>>>
>> >> >>>>> I'm getting bytes in last column, but as a percentage of total
>> bytes
>> >> >>>>> from
>> >> >>>>> all records I don't know how to get that.
>> >> >>>>>
>> >> >>>>> thanks.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On 9/15/15, Evgeniy Sudyr <eject.in.ua at gmail.com> wrote:
>> >> >>>>> > Ai,
>> >> >>>>> >
>> >> >>>>> > are you sure that in your rwfilter results you have more than
>> 1%
>> >> of
>> >> >>>>> Bytes?
>> >> >>>>> >
>> >> >>>>> > From rwstats man page:
>> >> >>>>> >
>> >> >>>>> > *--percentage*=*N* Print the bins where the primary value is
>> >> >>>>> > greater-than (or less-than) *N* percent of the sum of the
>> primary
>> >> >>>>> > values across all bins.
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > I think it will be useful to see --count --Packets
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > On Tue, Sep 15, 2015 at 5:31 PM, asad <a.alii85 at gmail.com>
>> wrote:
>> >> >>>>> >
>> >> >>>>> >> Hi,
>> >> >>>>> >>
>> >> >>>>> >> I want to know what "alternate options" exists for following:-
>> >> >>>>> >>
>> >> >>>>> >> rwfilter --sensor=Vrouter1 --type=out --sport=3306
>> --pass=stdout
>> >> >>>>> >> --start-date=2012/11/13:00 --end-date=2012/11/13:23
>> >> >>>>> >> --saddress=172.31.253.102 | rwstats --percentage=1 --bytes
>> >> >>>>> >> --fields=dip
>> >> >>>>> >>
>> >> >>>>> >> I don't know why but using --percentage=1 flag, I get zero
>> >> results,
>> >> >>>>> >> even when in records I know this IP is present. Is there any
>> >> reason
>> >> >>>>> >> why would such happpen?
>> >> >>>>> >>
>> >> >>>>> >> Or I can move to another rwstats switch parameters to perform
>> same
>> >> >>>>> >> task as trying to achieve with percentage=1
>> >> >>>>> >>
>> >> >>>>> >> Thanks.
>> >> >>>>> >>
>> >> >>>>> >>
>> >> >>>>> >>
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > --
>> >> >>>>> > --
>> >> >>>>> > With regards,
>> >> >>>>> > Eugene Sudyr
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>
>> >>
>>
More information about the netsa-tools-discuss
mailing list