[netsa-tools-discuss] Flags type count in rwstats
Hossam Zalabany
elzalabany at hu-berlin.de
Tue Jul 23 04:30:55 EDT 2019
Dear Angela
Many Thanks for your reply I highly appreciate it and very thankful for your time and effort.
Sorry if my question was not clear, but Iam grateful for your clarification. the reason for 1 min files is that Iam expecting large files if I used larger time span and I wanted to keep the calculation process fast per file. the idea is also to delete the files after extracting the statistics so we do not run into full disk space issue.
Actually the current netflow probe is configured for 60 sec time out already “I know not optimum and for sure you are right when it comes to TCP conversations that kept a life for for minutes” but that is only the starting point.
now, when I use rwcut I can see per each flow have some flags, my aim with rwstats is to aggregate the number of flags as well as the bytes for all flow records for the top bins sharing same sIP,sPort,dIP, dPort, for instance to achieve something like below :
> sIP|sPort| dIP|dPort|pro|fin|syn|rst|ack|urg| Bytes| %Bytes| cumul_%|
> 10.21.64.133 |60870| 10.5.54.69| 2051| 6| 24| 12| 1| 5| 13| 7257019952| 0.888884| 0.888884|
> 10.5.63.12| 2049| 10.1.223.88| 958| 6| 24| 1| 4| 9| 7| 7136227712| 0.874089| 1.762973|
> 10.21.64.133 |34073| 10.5.54.69| 2051| 6| 24| 1| 4| 9| 7| 5803794764| 0.710884| 2.473857|
> 10.5.63.12| 2049| 10.5.4.31| 1020| 6| 24| 12| 1| 5| 13| 3883330408| 0.475654| 2.949511|
> 10.21.64.133 |34595| 10.5.54.69| 2051| 24| 1| 4| 9| 7| 3| 3857964856| 0.472547| 3.422058|
> 10.5.204.4| 5247| 10.120.30.236|53789| 17| 24| 1| 4| 9| 7| 3674064752| 0.450022| 3.872080|
> 10.21.64.133 |37529| 10.5.54.69| 2051| 6| 24| 12| 1| 5| 13| 3262020960| 0.399552| 4.271632|
> 10.6.8.250| 0| 10.5.204.4| 0| 97| 24| 1| 4| 9| 7| 2994127260| 0.366739| 4.638371|
> 10.134.26.21| 0| 10.3.250.6| 0| 97| 24| 12| 1| 5| 13| 2893997631| 0.354474| 4.992846|
> 10.5.63.12| 2049| 10.134.144.21| 747| 6| 24| 1| 4| 9| 7| 2853011756| 0.349454| 5.342300|
Do you think that is reasonable? doable?
Many Thanks for the support and cooperation
Hossam
> On 18. Jul 2019, at 19:07, Angela Horneman <ahorneman at cert.org> wrote:
>
> Hello Hossam,
>
> I’m not quite sure of your end goal. In your current rwstats command, the output is not showing the count of TCP flags. The “24” is the integer value of AP (ACK, PSH). The values for each flag are:
> FIN = 1
> SYN = 2
> RST = 4
> ACK = 16
> URG = 32
> If you add up the flags that occur in one flow, you get the integer flag value. Network flow only tells the distinct flags that occurred, not the number times they each occurred.
>
> Before I try to help with your output, let’s clarify your set-up. First, as I’m sure you know, network flow summarizes connections over a period of time. For TCP, the summaries will cover all packets of a TCP session and will not be exported as a flow record until the session is terminated (e.g Rst or Fin flags occur) or an active timeout occurs. For many products the active timeout is 30 minutes. Therefore a single flow record may span traffic for several minutes.
>
> If you want to find your most active talkers by minute intervals, the files do not need to be set to be one minute intervals—the SiLK summary commands (rwstats, rwuniq, rwcount) have time bin options. However, you need to think about if you are measuring traffic per minute or counting the completed flows in a minute. If the first, you would need to write a script to process the longer flows into one-minute time lengths, or set the active timeout to be one minute (which might cause performance issues on the appliance that is generating the IPFIX).
>
> Now, the further question is, given the following example data, what do you want the output to show? (This sample is very fake for illustration only.)
>
> sTime |eTime |sIP |sPort |dIP |dPort |proto |flags |bytes
> :01:00 |:01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |PA |10
> :01:00 |:01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |R |10
> :01:00 |:01:03 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |R |10
> :01:00 |:01:00 |x.x.x.1 |33333 |x.x.x.2 |76543 |6 |PA |10
> :01:02 |:01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |PA |10
>
>
> rwstats --fields=sTime,sIP,sPort,dIP,dPort,proto,flags --values=bytes –bin-time=60
> will give
> sTime |sIP |sPort |dIP |dPort |proto |flags |bytes
> :01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |PA |10
> :01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |R |20
> :01:00 |x.x.x.1 |33333 |x.x.x.2 |76543 |6 |PA |10
> :01:02 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |PA |10
>
> What values would you expect to see in your table:
>
> sIP|sPort|dIP|dPort|pro|ACK|SYN|FIN|Bytes|
>
>
> Angela Horneman
> Analysis Team Lead
> Member of the Technical Staff
> CERT Software Engineering Institute
> Carnegie Mellon University
> ahorneman at cert.org <mailto:ahorneman at cert.org>
>
>
>
> From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org [mailto:netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org] On Behalf Of Hossam Zalabany
> Sent: Thursday, July 18, 2019 8:44 AM
> To: netsa-tools-discuss at cert.org
> Subject: [netsa-tools-discuss] Flags type count in rwstats
>
> Dear Cert.
>
> I am trying to run SilK to parse statistics of my IPFIX top talkers per minute, I configured the sensor to have a new file for each minute, and keeping the in and out pairing out of the scoop at the moment, all fine so far except that I am only able to count total number of TCP flags, I need to count each flag time separately like, ACK, SYN, and so.
>
> the current command I use is :
>
> rwstats --fields=sip,sport,dip,dport,protocol,flags --integer-tcp-flags --values=byte --count=10 ext2ext-sens1_20190718.09
> INPUT: 41300277 Records for 25847944 Bins and 816419146922 Total Bytes
> OUTPUT: Top 10 Bins by Bytes
> sIP|sPort| dIP|dPort|pro|fla| Bytes| %Bytes| cumul_%|
> 10.21.64.133 |60870| 10.5.54.69| 2051| 6| 24| 7257019952| 0.888884| 0.888884|
> 10.5.63.12| 2049| 10.1.223.88| 958| 6| 24| 7136227712| 0.874089| 1.762973|
> 10.21.64.133 |34073| 10.5.54.69| 2051| 6| 24| 5803794764| 0.710884| 2.473857|
> 10.5.63.12| 2049| 10.5.4.31| 1020| 6| 24| 3883330408| 0.475654| 2.949511|
> 10.21.64.133 |34595| 10.5.54.69| 2051| 6| 24| 3857964856| 0.472547| 3.422058|
> 10.5.204.4| 5247| 10.120.30.236|53789| 17| 0| 3674064752| 0.450022| 3.872080|
> 10.21.64.133 |37529| 10.5.54.69| 2051| 6| 24| 3262020960| 0.399552| 4.271632|
> 10.6.8.250| 0| 10.5.204.4| 0| 97| 0| 2994127260| 0.366739| 4.638371|
> 10.134.26.21| 0| 10.3.250.6| 0| 97| 0| 2893997631| 0.354474| 4.992846|
> 10.5.63.12| 2049| 10.134.144.21| 747| 6| 24| 2853011756| 0.349454| 5.342300|
>
>
> what I desire to have is sIP|sPort|dIP|dPort|pro|ACK|SYN|FIN|Bytes|
>
>
> is there is any suggested steps ?
>
>
> regards
>
>
> Hossam
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list