[netsa-tools-discuss] alernate use of rwstats --percentage
asad
a.alii85 at gmail.com
Mon Sep 28 02:45:42 EDT 2015
Thomas,
I have supplied the same configuration at my production systems at
office with the following minor changes. I'm not using YAF to get
flows to SILK, infact I'm using
probe S0 netflow-v9
listen-on-port 2055
protocol udp
quirks zero-packets
Also, I'm running rwflowpack from "/usr/local/sbin/rwflowpack". I have
supplied the
export SILK_IPFIX_PRINT_TEMPLATES=1 as required.
I'm looking for text in /var/log/rwflowpack-* for
" Domain 0x0000, TemplateID 0x0108, Position 6, Length 4, IE 8, Name
sourceIPv4Address"
But not finding anything in related to it. Cisco ASA is configred to
send the template for every minute. I even delete today log file in
/var/log to lower the size and making searching easy, but still I see
no "template info".
Kindly help.
regards
asad
On 9/25/15, Mark Thomas <mthomas at cert.org> wrote:
> With the environment variable set, you will see the template
> information in the rwflowpack log file whenever rwflowpack receives
> an IPFIX template from yaf or a NetFlow v9 template from your
> router.
>
> The template information printed by rwflowpack uses the IPFIX names
> for each information element. The list of IPFIX information
> elements is maintained by IANA
> http://www.iana.org/assignments/ipfix/ipfix.xhtml
>
> In rwflowpack, a source IPv4 address will be printed as:
>
> Domain 0x0000, TemplateID 0x0108, Position 6, Length 4, IE 8, Name
> sourceIPv4Address
>
> where "IE 8" is the numeric identifier for the information element.
>
> The NetFlow v9 names for information elements 1 to 79 can be found
> in the NetFlow v9 RFC. http://tools.ietf.org/html/rfc3954
>
> When using the CFLOW filter in Wireshark to view either IPFIX or
> NetFlow v9 template, a source IPv4 address will be printed as:
>
> Field (6/21): IP_SRC_ADDR
> Type: IP_SRC_ADDR (8)
> Length: 4
>
> The names used by Wireshark are sometimes the same as the NetFlow v9
> names.
>
> Information element 6 contains the union of TCP flags across all
> packets in a TCP flow record. The information element is called
> tcpControlBits by IPFIX, and it is called TCP_FLAGS by Cisco and
> Wireshark.
>
> However, if the template does not contain this element, there is no
> way for SiLK to get the TCP Flags.
>
> Note that Cisco says they do not populate the TCP Flags fields on
> the ASA.
> https://supportforums.cisco.com/document/30471/netflow-asa
>
> NetFlow on the ASA vs IOS
>
> The ASA only supports NetFlow version 9 and there are no plans to
> support NetFlow version 5. NetFlow on the ASA is event
> driven. Unlike routing platforms we do not send incremental
> updates; NSEL records are only sent during flow creation, teardown
> or ACL deny events. Also unlike the routing platforms we will not
> populate the ToS bits or the TCP flags. Lastly, all flows on the
> ASA are bidirectional. All counters for a flow will increase for
> traffic flowing from A->B or B->A.
>
>
> I hope that helps.
>
> -Mark
>
>
> On Thu, 24 Sep 2015 13:55:37 +0500, asad wrote:
>
>> You were right,there was problem with the order I was running the
>> commands,
>> Now I got it to work and I see the logs are being populated, now the only
>> thing missing is " Information Element 6,TCP_FLAGS." which I'm interested
>> in.
>>
>> I'm using pcap files which is non-cisco asa based, I just couldn't find
>> one
>> that matches my production env e.g cisco 5585-x. If you can safely if
>> size
>> and privacy allows you can send me some sample pcap files to replay with.
>>
>> Or perhaps without the required pcap files I'm still supposed to see some
>> "template info" in the logs, In this case I go back to point of config
>> SILK_IPFIX_PRINT_TEMPLATES.
>>
>> Also, in the ps -auxx | grep rwflowpack
>>
>> I also see /etc/init.d/rwflowpack though I never used rwflowpack.conf. I
>> hope there is no conflict.
>>
>> On Thu, Sep 24, 2015 at 1:26 AM, Mark Thomas <mthomas at cert.org> wrote:
>>
>>> Thank you for providing the versions of the tools you are running
>>> and the command lines you are using to invoke the tools.
>>>
>>> The order in which things are started is important.
>>>
>>> rwflowpack only checks the SILK_IPFIX_PRINT_TEMPLATES variable once
>>> when it is invoked, so the variable must be set prior to starting
>>> rwflowpack. The variable must be set to a non-empty value, and the
>>> first character should not be 0.
>>>
>>> You are running YAF over a TCP connection. In this case, YAF only
>>> sends the IPFIX templates one time--typically when YAF first
>>> connects to rwflowpack.
>>>
>>> For rwflowpack to print the templates, you must:
>>>
>>> 1. Set the SILK_IPFIX_PRINT_TEMPLATES environment variable to 1.
>>>
>>> 2. Start rwflowpack.
>>>
>>> 3. Start YAF.
>>>
>>> When YAF connects to rwflowpack, the log file will contain lines
>>> that look similar to:
>>>
>>> Sep 23 16:14:42 mac rwflowpack[65061]: 'Irouter': accepted connection
>>> from 127.0.0.1:55901
>>> Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID
>>> 0xC013,
>>> Contains 6 Elements, Enabled by SILK_IPFIX_PRINT_TEMPLATES
>>> Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID
>>> 0xC013,
>>> Position 0, Length 4, IE 184, Name tcpSequenceNumber
>>> Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID
>>> 0xC013,
>>> Position 1, Length 1, IE 6871/ 14, Name initialTCPFlags
>>> Sep 23 16:14:42 mac rwflowpack[65061]: Domain 0x0000, TemplateID
>>> 0xC013,
>>> Position 2, Length 1, IE 6871/ 15, Name unionTCPFlags
>>> ...
>>>
>>> In my set-up, I saw 11 templates:
>>>
>>> TemplateID 0xC013, Contains 6 Elements
>>> TemplateID 0xC015, Contains 23 Elements
>>> TemplateID 0xC016, Contains 6 Elements
>>> TemplateID 0xC018, Contains 2 Elements
>>> TemplateID 0xB800, Contains 39 Elements
>>> TemplateID 0xC003, Contains 3 Elements
>>> TemplateID 0xC004, Contains 2 Elements
>>> TemplateID 0xC005, Contains 11 Elements
>>> TemplateID 0xC006, Contains 3 Elements
>>> TemplateID 0xD000, Contains 14 Elements
>>> TemplateID 0xC008, Contains 1 Elements
>>>
>>> I hope that helps.
>>>
>>> -Mark
>>>
>>>
>>> On Thu, 24 Sep 2015 00:29:10 +0500, asad wrote:
>>>
>>> > Thanks Thomas, for such comprehensive explanation. From
>>> > silk-summary.txt
>>> > file I can verify that my
>>> >
>>> > SILK 3.10.2
>>> > libfixbuf-1.7.0 >= 1.6.0
>>> >
>>> > The command I used is
>>> >
>>> > rwflowpack \
>>> > --sensor-configuration=/data/sensors.conf \
>>> > --site-config-file=/data/silk.conf \
>>> > --archive-directory=/usr/local/var/lib/rwflowpack/archive \
>>> > --output-mode=local-storage \
>>> > --root-directory=/data \
>>> > --pidfile=/var/log/rwflowpack.pid --log-level=info \
>>> > --log-directory=/var/log --log-basename=rwflowpack \
>>> >
>>> > But I don't see in logs any "information element". I'm using tcpreplay
>>> > tool, to replay a sample pcap files (since I'm home now), but in logs
>>> > I
>>> see
>>> > as
>>> >
>>> > "Sep 24 00:24:04 Silky-flows rwflowpack[812]: 'S0': forward 0, reverse
>>> > 0,
>>> > ignored 0; yaf: recs 0, pkts 0, dropped-pkts 0, ignored-pkts 0,
>>> > bad-sequence-pkts 0, expired-frags 0"
>>> >
>>> > I have tested with cmd
>>> >
>>> > "
>>> >
>>> > yaf --silk --ipfix=tcp --live=pcap --out=127.0.0.1 \
>>> > --ipfix-port=18001 --in=eth0 --applabel --max-payload=384 &"
>>> >
>>> >
>>> > and it works fine logs are made as they should.
>>> >
>>> >
>>> > On Wed, Sep 23, 2015 at 10:26 PM, Mark Thomas <mthomas at cert.org>
>>> > wrote:
>>> >
>>> >> The way that you set the environment variable depends on how your
>>> >> are starting rwflowpack or flowcap.
>>> >>
>>> >> 1. Using the start-up scripts.
>>> >>
>>> >> Follow these instructions if you set variables in the the
>>> >> rwflowpack.conf configuration file and then run the rwflowpack shell
>>> >> script as
>>> >>
>>> >> rwflowpack start
>>> >>
>>> >> The rwflowpack shell script is typically installed in the directory
>>> >> $prefix/share/silk/etc/init.d. In the RedHat RPMs, the start-up
>>> >> script is installed in /etc/init.d/rwflowpack.
>>> >>
>>> >> These instructions also apply for flowcap.
>>> >>
>>> >> Find the rwflowpack start-up script or the flowcap start-up script.
>>> >> Within that script, find the start() subroutine. In that
>>> >> subroutine, find the following:
>>> >>
>>> >> if [ X`whoami` = "X${USER}" ] ; then
>>> >> eval "${PROG_PATH} ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>>> >> else
>>> >> su - ${USER} -c "${PROG_PATH} ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>>> >> fi
>>> >>
>>> >> Insert SILK_IPFIX_PRINT_TEMPLATES=1 after the initial double quote.
>>> >> The result should read:
>>> >>
>>> >> if [ X`whoami` = "X${USER}" ] ; then
>>> >> eval "SILK_IPFIX_PRINT_TEMPLATES=1 ${PROG_PATH} ${PROG_OPTIONS}
>>> >> ${EXTRA_OPTIONS} &"
>>> >> else
>>> >> su - ${USER} -c "SILK_IPFIX_PRINT_TEMPLATES=1 ${PROG_PATH}
>>> >> ${PROG_OPTIONS} ${EXTRA_OPTIONS} &"
>>> >> fi
>>> >>
>>> >>
>>> >>
>>> >> 2. Starting from the command line.
>>> >>
>>> >> Follow these instructions if you invoke the rwflowpack or flowcap
>>> >> binary from the command line and specify its command line options as
>>> >> part of the command, such as
>>> >>
>>> >> rwflowpack --root-directory=/data ...
>>> >>
>>> >> These binaries are typically installed in $prefix/sbin.
>>> >>
>>> >> In this case, you can set the SILK_IPFIX_PRINT_TEMPLATES environment
>>> >> variable as you set any other environment variable in a modern
>>> >> Bourne-compatible shell:
>>> >>
>>> >> export SILK_IPFIX_PRINT_TEMPLATES=1
>>> >> rwflowpack --root-directory=/data ...
>>> >>
>>> >>
>>> >>
>>> >> This feature requires SiLK 3.8.2 or newer and libfixbuf-1.4.0 or
>>> >> newer.
>>> >>
>>> >> -Mark
>>> >>
>>> >>
>>> >> On Wed, 23 Sep 2015 13:33:06 +0500, asad wrote:
>>> >>
>>> >> > Thomas,
>>> >> >
>>> >> > Can you educate me how to set env variable
>>> "SILK_IPFIX_PRINT_TEMPLATES"?
>>> >> >
>>> >> > On 9/21/15, Mark Thomas <mthomas at cert.org> wrote:
>>> >> >> I have a couple of PCAP files that contain data from a Cisco ASA,
>>> >> >> and the NetFlow v9 templates do not include Information Element 6,
>>> >> >> TCP_FLAGS.
>>> >> >>
>>> >> >> If you wish to confirm this for yourself, set the
>>> >> >> SILK_IPFIX_PRINT_TEMPLATES environment variable to 1 prior to
>>> >> >> starting rwflowpack or flowcap. With that variable set,
>>> >> >> rwflowpack
>>> >> >> or flowcap print to its log file each IPFIX/NetFlow v9 template it
>>> >> >> receives.
>>> >> >>
>>> >> >> When the tool prints the template, it uses the IPFIX names for the
>>> >> >> information elements, which you can find at
>>> >> >> http://www.iana.org/assignments/ipfix/ipfix.xhtml
>>> >> >>
>>> >> >> The IPFIX name for TCP_FLAGS is tcpControlBits.
>>> >> >>
>>> >> >> -Mark
>>> >> >>
>>> >> >>
>>> >> >> On Fri, 18 Sep 2015 10:54:56 +0500, asad wrote:
>>> >> >>
>>> >> >>> This is update.
>>> >> >>>
>>> >> >>> I have edited the cmd as told by "Angela", now I get a very
>>> >> >>> useful
>>> >> >>> output which shows that the %Bytes value is never greater then
>>> >> >>> 0.025112, this explains why the percentage =1 was not working.
>>> >> >>>
>>> >> >>> But what more strange is now, I start to question the suitability
>>> >> >>> of
>>> >> >>> cisco asa asel netflows logs here is reason why ..
>>> >> >>>
>>> >> >>>
>>> >> >>> changing the command and adding params e.g --packets=4-
>>> >> >>> --ack-flag=1
>>> >> >>> delivers me zero output. Even more strange (please see
>>> >> >>> attachment)
>>> >> >>> that flags columns is empty. ( it is even empty when there is no
>>> >> >>> --ack-flag=1 value set).
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> On 9/16/15, asad <a.alii85 at gmail.com> wrote:
>>> >> >>>> Thanks Angela and Evgeniy. I believe I have been unfair to part
>>> >> >>>> I
>>> have
>>> >> >>>> explained my case effectively.
>>> >> >>>>
>>> >> >>>> There is a critical server on enterprise network who traffic I
>>> want to
>>> >> >>>> monitor for following usage :-
>>> >> >>>>
>>> >> >>>> " to monitor connection to and from the server w.r.t to bytes "
>>> >> >>>>
>>> >> >>>> This is done to get some way of knowing "normal" behavior for
>>> >> >>>> the
>>> >> >>>> traffic
>>> >> >>>> i.e # between server and client no of bytes send per day.
>>> >> >>>>
>>> >> >>>> I will try out the suggestions as soon i get access to office
>>> network
>>> >> >>>> (currently I'm at home) and will update accordingly.
>>> >> >>>>
>>> >> >>>> On Wed, Sep 16, 2015 at 6:22 PM, Angela Horneman <
>>> ahorneman at cert.org>
>>> >> >>>> wrote:
>>> >> >>>>
>>> >> >>>>> Asad,
>>> >> >>>>>
>>> >> >>>>> If you replace --percentage=1 with --count=10 in your first
>>> example,
>>> >> >>>>> there
>>> >> >>>>> will be a column "%Bytes" in the output. You can use that
>>> >> >>>>> column
>>> to
>>> >> >>>>> check
>>> >> >>>>> if any of the 10 DIPs with the greatest byte volumes have a
>>> >> >>>>> volume
>>> >> that
>>> >> >>>>> is
>>> >> >>>>> at least 1% of the total.
>>> >> >>>>>
>>> >> >>>>> Angela
>>> >> >>>>>
>>> >> >>>>> -----Original Message-----
>>> >> >>>>> From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org
>>> >> [mailto:
>>> >> >>>>> netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org] On
>>> Behalf
>>> >> Of
>>> >> >>>>> asad
>>> >> >>>>> Sent: Wednesday, September 16, 2015 12:49 AM
>>> >> >>>>> To: Evgeniy Sudyr <eject.in.ua at gmail.com>
>>> >> >>>>> Cc: netsa-tools-discuss at cert.org
>>> >> >>>>> Subject: Re: [netsa-tools-discuss] alernate use of rwstats
>>> >> --percentage
>>> >> >>>>>
>>> >> >>>>> Thanks Eugene,
>>> >> >>>>>
>>> >> >>>>> My output is:-
>>> >> >>>>>
>>> >> >>>>> sIP |sPort| dIP| dPort| bytes|
>>> >> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>>> >> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>>> >> >>>>> 10.10.13.152| 0| 10.10.4.145| 0| 78|
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> With command
>>> >> >>>>>
>>> >> >>>>> rwfilter --sensor=S0 --type=all --pass=stdout
>>> --saddress=10.10.13.152
>>> >> >>>>> --start-date=2015/09/13:15 --end-date=2015/09/16:15 | rwsort
>>> >> >>>>> --fields=bytes | rwcut --fields=sip,sport,dip,dport,bytes
>>> >> >>>>>
>>> >> >>>>> I'm getting bytes in last column, but as a percentage of total
>>> bytes
>>> >> >>>>> from
>>> >> >>>>> all records I don't know how to get that.
>>> >> >>>>>
>>> >> >>>>> thanks.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On 9/15/15, Evgeniy Sudyr <eject.in.ua at gmail.com> wrote:
>>> >> >>>>> > Ai,
>>> >> >>>>> >
>>> >> >>>>> > are you sure that in your rwfilter results you have more
>>> >> >>>>> > than
>>> 1%
>>> >> of
>>> >> >>>>> Bytes?
>>> >> >>>>> >
>>> >> >>>>> > From rwstats man page:
>>> >> >>>>> >
>>> >> >>>>> > *--percentage*=*N* Print the bins where the primary value is
>>> >> >>>>> > greater-than (or less-than) *N* percent of the sum of the
>>> primary
>>> >> >>>>> > values across all bins.
>>> >> >>>>> >
>>> >> >>>>> >
>>> >> >>>>> > I think it will be useful to see --count --Packets
>>> >> >>>>> >
>>> >> >>>>> >
>>> >> >>>>> > On Tue, Sep 15, 2015 at 5:31 PM, asad <a.alii85 at gmail.com>
>>> wrote:
>>> >> >>>>> >
>>> >> >>>>> >> Hi,
>>> >> >>>>> >>
>>> >> >>>>> >> I want to know what "alternate options" exists for
>>> >> >>>>> >> following:-
>>> >> >>>>> >>
>>> >> >>>>> >> rwfilter --sensor=Vrouter1 --type=out --sport=3306
>>> --pass=stdout
>>> >> >>>>> >> --start-date=2012/11/13:00 --end-date=2012/11/13:23
>>> >> >>>>> >> --saddress=172.31.253.102 | rwstats --percentage=1 --bytes
>>> >> >>>>> >> --fields=dip
>>> >> >>>>> >>
>>> >> >>>>> >> I don't know why but using --percentage=1 flag, I get zero
>>> >> results,
>>> >> >>>>> >> even when in records I know this IP is present. Is there any
>>> >> reason
>>> >> >>>>> >> why would such happpen?
>>> >> >>>>> >>
>>> >> >>>>> >> Or I can move to another rwstats switch parameters to
>>> >> >>>>> >> perform
>>> same
>>> >> >>>>> >> task as trying to achieve with percentage=1
>>> >> >>>>> >>
>>> >> >>>>> >> Thanks.
>>> >> >>>>> >>
>>> >> >>>>> >>
>>> >> >>>>> >>
>>> >> >>>>> >
>>> >> >>>>> >
>>> >> >>>>> > --
>>> >> >>>>> > --
>>> >> >>>>> > With regards,
>>> >> >>>>> > Eugene Sudyr
>>> >> >>>>> >
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>
>>> >> >>
>>> >>
>>>
>
More information about the netsa-tools-discuss
mailing list