[netsa-tools-discuss] Flows per second in SiLK

Wed Aug 12 16:03:46 EDT 2015

In answer your three questions...

1. Missing packet count.

The missing packet count is an estimate by libfixbuf of the number
of packets it did not receive.  libfixbuf increments this count
whenever the sequence number of the packet it receives is different
than the number it was expecting.

NetFlow v9 is delivered over UDP which is an unreliable protocol.
If a packet gets dropped at some point during its journey from the
router to SiLK, the sequence numbers of the packets will indicate
that one or more packets were lost.  Occasionally, a packet has not
actually been dropped, but it has arrived out of sequence.

Packets are dropped when rwflowpack is too busy to accept them, the
router is too busy to send them, or an intermediate point is too
busy to process them.

2. Explanation of rwflowpack log line.

Periodically rwflowpack prints a line similar to the following

 's1_probe': forward 722703, reverse 0, ignored 0, nf9: missing-pkts 2990

Although SiLK stores flow records in as uni-directional, it can read
bi-directional flows created by some flow generators such as YAF.
http://tools.netsa.cert.org/yaf/

In the log message, the value after "reverse" is the number of
bi-directional flow records SiLK received.  The value after
"forward" is the sum of the uni-directional and bi-directional flow
records.  The sum of the forward and reverse values should be the
total number of records SiLK wrote to disk during the current
period.

The value after "ignored" denotes records that SiLK is unable to
process.  For example, if SiLK is configured without IPv6 support,
any IPv6 record it receives is added to the "ignore" count.

3. Setting command line options in rwflowpack.

Since you start rwflowpack via

  /usr/local/share/silk/etc/init.d/rwflowpack restart

there should be a file named

  /usr/local/share/silk/etc/rwflowpack.conf

that determines the command line options used by that script.  The
comments in that file should specify which command line switch is
affected by the variables in the file.  This is also described in
this section of the Installation Handbook
http://tools.netsa.cert.org/silk/install-handbook.html#x1-470004.3

-Mark

-----Original Message-----
From: Hosam Hittini <hosam.hittini at ies.etisalat.ae>
Date: Wed, 12 Aug 2015 07:47:17 +0400
To: 'Mark Thomas' <mthomas at cert.org>
Cc: <netsa-tools-discuss at cert.org>, <netsa-help at cert.org>, 'Majid Qureshi' <mmajid at ies.etisalat.ae>
Subject: RE: [netsa-tools-discuss] Flows per second in SiLK

Dear Mark,

I found the log file, Thank you
But I have some missing packets as shown below; any idea why?
Aug 12 07:18:43 euc rwflowpack[25292]: s1_probe': forward 722703, reverse 0,
ignored 0, nf9: missing-pkts 2990
I wonder if you can explain the log to me (What is reverse, ignored and
missing packets)
In addition, I'd like to set some flowpack options
(https://tools.netsa.cert.org/silk/rwflowpack.html), can you tell me how to
do that?
I don't mind restarting flowpack, I usually do it with the following command
"sudo sh /usr/local/share/silk/etc/init.d/rwflowpack restart"

Regards,
Hosam Hittini

-----Original Message-----
From: Mark Thomas [mailto:mthomas at cert.org] 
Sent: Tuesday, August 11, 2015 8:14 PM
To: Hosam Hittini <hosam.hittini at ies.etisalat.ae>
Cc: netsa-tools-discuss at cert.org; netsa-help at cert.org; 'Majid Qureshi'
<mmajid at ies.etisalat.ae>
Subject: Re: [netsa-tools-discuss] Flows per second in SiLK

You need to restart rwflowpack to change it settings.

However, your current rwflowpack invocation should be writing its log
messages somewhere.  If it not writing to a file, the messages should be
going into syslog (e.g., /var/log/messages on RedHat Linux).

-Mark

-----Original Message-----
From: Hosam Hittini <hosam.hittini at ies.etisalat.ae>
Date: Tue, 11 Aug 2015 09:14:28 +0400
To: 'Mark Thomas' <mthomas at cert.org>
Cc: <netsa-tools-discuss at cert.org>, <netsa-help at cert.org>, 'Majid Qureshi'
	<mmajid at ies.etisalat.ae>
Subject: RE: [netsa-tools-discuss] Flows per second in SiLK

Dear Mark,

I believe the first answer better suits what I'm looking for When I tried
this command rwflowpack
--sensor-configuration=/usr/local/etc/silk/sensor.conf --sensor-name=S1
--log-directory=`pwd` --root-directory=/silk_data_repository
I got the following errors
's1_probe': couldn't create socket listening to *:9996: Address already in
use Is it possible to get a log without interrupting the operation?
Thanks in advance

Regards,
Hosam Hittini
System Security, Security Operations Center
E: 84/5733
M: +971 555 655 878

-----Original Message-----
From: Mark Thomas [mailto:mthomas at cert.org]
Sent: Monday, August 10, 2015 9:27 PM
To: Hosam Hittini <hosam.hittini at ies.etisalat.ae>
Cc: netsa-tools-discuss at cert.org; 'Ron Bandes' <rbandes at cert.org>;
netsa-help at cert.org; 'Majid Qureshi' <mmajid at ies.etisalat.ae>
Subject: Re: [netsa-tools-discuss] Flows per second in SiLK

There are two answers to the question regarding the number of flow records
being received by SiLK.

1. If you are talking about how many flow records are being received and
processed by the rwflowpack or flowcap tools, the answer is currently No,
there is nothing that directly reports the number of flow records received
per second.

Every so often (described below), the rwflowpack and flowcap tools report to
the log file the number of flow records processed, for example

  Mar 16 16:21:48 host flowcap[99294]: 'S9': forward 14, reverse 0, ignored
0, nf9: missing-pkts 0

If you divide the number that appears after "forward" with the time period,
you can get an average flow rate for the time period.

The time period is determined by the --timeout switch in flowcap and the
--flush-timeout switch in rwflowpack.  The daemons do a lot of work when the
timeout occurs, so I would suggest you not make timeout smaller than 15
seconds.

2. Since the flow record is sent when the flow ends, you could get an
estimate of the number of flow records received per second by using the
rwcount tool.  Set the --bin-size to 1 and choose the end-spike
--load-scheme.

  $ rwcount --bin-size=1 --load-scheme=end --start-time=2009/02/13
                 Date|  Records|   Bytes|  Packets|
  2009/02/12T00:00:02|     2.00|  259.00|     2.00|
  2009/02/12T00:00:03|     1.00|  504.00|     9.00|
  2009/02/12T00:00:04|     0.00|    0.00|     0.00|
  2009/02/12T00:00:05|     0.00|    0.00|     0.00|

I hope that helps,

-Mark

-----Original Message-----
From: Hosam Hittini <hosam.hittini at ies.etisalat.ae>
Date: Mon, 10 Aug 2015 15:22:10 +0400
To: <netsa-tools-discuss at cert.org>, 'Ron Bandes' <rbandes at cert.org>,
	<netsa-help at cert.org>
Cc: 'Majid Qureshi' <mmajid at ies.etisalat.ae>
Subject: [netsa-tools-discuss] Flows per second in SiLK

Hi,

I was wondering if there's a way to get the number of flows per second being
received at SiLK

We have version 3.8.0

Thanks in advance