[netsa-tools-discuss] Flags type count in rwstats
Angela Horneman
ahorneman at cert.org
Tue Jul 23 08:39:48 EDT 2019
Hassam,
It isn’t possible to count the total number of each flag per flow and then aggregate the counts with SiLK. In the network flow, you can only see if a flag occurred not how many times it occurred during the flow. You can get the number of records that occur for each flag combination and then have a script that post-processes the output to aggregate flows with the same sIP, dIP, sPort, dPort with the counts of the flows for each flag.
For example, if you create a file with a command like this:
rwuniq --fields=sTime,sIP,sPort,dIP,dPort,proto,flags --values=byte,flows --bin-time=60 --no-col --no-title
Then you can use something like this python script
inDict = {}
inFile = open("flowSummary_in.txt","r")
for line in inFile:
#fields[0]|fields[1]|fields[2]|fields[3]|fields[4]|fields[5]|fields[6]|fields[7]|fields[8]
#sTime|sIP|sPort|dIP|dPort|proto|flags|bytes|flows
fields = line.split("|")
key = '|'.join(fields[:5])
if not key in inDict:
inDict[key] = {}
inDict[key]["bytes"] = 0
for flagType in ["F", "S", "R", "A", "U", "P", "E", "C"]:
inDict[key][flagType] = 0
inDict[key]["bytes"] += int(fields[7])
flagVals = list(fields[6])
for flag in flagVals:
inDict[key][flag] += int(fields[8])
inFile.close()
outFile = open("flowSummary_out.txt","w")
outFile.write("sTime|sIP|sPort|dIP|dPort|proto|F|S|R|A|U|P|E|C|bytes\r\n")
for key,value in inDict.items():
newLine = key
for flagType in ["F", "S", "R", "A", "U", "P", "E", "C"]:
newLine += "|" + str(inDict[key][flagType])
newLine += "|" + str(inDict[key]["bytes"])
outFile.write(newLine + "\r\n")
outFile.close()
To change this output:
2019/07/23T07:00:00|10.0.0.1|80|10.0.0.2|12345|6|RA|120|3|
2019/07/23T07:00:00|10.0.0.1|80|10.0.0.2|12345|6|PA|120|3|
2019/07/23T07:00:00|10.0.0.1|80|10.0.0.2|54321|6|RA|120|3|
Into:
sTime|sIP|sPort|dIP|dPort|proto|F|S|R|A|U|P|E|C|bytes
2019/07/23T07:00:00|10.0.0.1|80|10.0.0.2|54321|0|0|3|3|0|0|0|0|120
2019/07/23T07:00:00|10.0.0.1|80|10.0.0.2|12345|0|0|3|6|0|3|0|0|240
Then you would need to sort as desired to take the top talkers.
Angela
From: Hossam Zalabany [mailto:elzalabany at hu-berlin.de]
Sent: Tuesday, July 23, 2019 4:31 AM
To: Angela Horneman <ahorneman at cert.org>
Cc: netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] Flags type count in rwstats
Dear Angela
Many Thanks for your reply I highly appreciate it and very thankful for your time and effort.
Sorry if my question was not clear, but Iam grateful for your clarification. the reason for 1 min files is that Iam expecting large files if I used larger time span and I wanted to keep the calculation process fast per file. the idea is also to delete the files after extracting the statistics so we do not run into full disk space issue.
Actually the current netflow probe is configured for 60 sec time out already “I know not optimum and for sure you are right when it comes to TCP conversations that kept a life for for minutes” but that is only the starting point.
now, when I use rwcut I can see per each flow have some flags, my aim with rwstats is to aggregate the number of flags as well as the bytes for all flow records for the top bins sharing same sIP,sPort,dIP, dPort, for instance to achieve something like below :
sIP|sPort| dIP|dPort|pro|fin|syn|rst|ack|urg| Bytes| %Bytes| cumul_%|
10.21.64.133 |60870| 10.5.54.69| 2051| 6| 24| 12| 1| 5| 13| 7257019952| 0.888884| 0.888884|
10.5.63.12| 2049| 10.1.223.88| 958| 6| 24| 1| 4| 9| 7| 7136227712| 0.874089| 1.762973|
10.21.64.133 |34073| 10.5.54.69| 2051| 6| 24| 1| 4| 9| 7| 5803794764| 0.710884| 2.473857|
10.5.63.12| 2049| 10.5.4.31| 1020| 6| 24| 12| 1| 5| 13| 3883330408| 0.475654| 2.949511|
10.21.64.133 |34595| 10.5.54.69| 2051| 24| 1| 4| 9| 7| 3| 3857964856| 0.472547| 3.422058|
10.5.204.4| 5247| 10.120.30.236|53789| 17| 24| 1| 4| 9| 7| 3674064752| 0.450022| 3.872080|
10.21.64.133 |37529| 10.5.54.69| 2051| 6| 24| 12| 1| 5| 13| 3262020960| 0.399552| 4.271632|
10.6.8.250| 0| 10.5.204.4| 0| 97| 24| 1| 4| 9| 7| 2994127260| 0.366739| 4.638371|
10.134.26.21| 0| 10.3.250.6| 0| 97| 24| 12| 1| 5| 13| 2893997631| 0.354474| 4.992846|
10.5.63.12| 2049| 10.134.144.21| 747| 6| 24| 1| 4| 9| 7| 2853011756| 0.349454| 5.342300|
Do you think that is reasonable? doable?
Many Thanks for the support and cooperation
Hossam
On 18. Jul 2019, at 19:07, Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
Hello Hossam,
I’m not quite sure of your end goal. In your current rwstats command, the output is not showing the count of TCP flags. The “24” is the integer value of AP (ACK, PSH). The values for each flag are:
FIN = 1
SYN = 2
RST = 4
ACK = 16
URG = 32
If you add up the flags that occur in one flow, you get the integer flag value. Network flow only tells the distinct flags that occurred, not the number times they each occurred.
Before I try to help with your output, let’s clarify your set-up. First, as I’m sure you know, network flow summarizes connections over a period of time. For TCP, the summaries will cover all packets of a TCP session and will not be exported as a flow record until the session is terminated (e.g Rst or Fin flags occur) or an active timeout occurs. For many products the active timeout is 30 minutes. Therefore a single flow record may span traffic for several minutes.
If you want to find your most active talkers by minute intervals, the files do not need to be set to be one minute intervals—the SiLK summary commands (rwstats, rwuniq, rwcount) have time bin options. However, you need to think about if you are measuring traffic per minute or counting the completed flows in a minute. If the first, you would need to write a script to process the longer flows into one-minute time lengths, or set the active timeout to be one minute (which might cause performance issues on the appliance that is generating the IPFIX).
Now, the further question is, given the following example data, what do you want the output to show? (This sample is very fake for illustration only.)
sTime |eTime |sIP |sPort |dIP |dPort |proto |flags |bytes
:01:00 |:01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |PA |10
:01:00 |:01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |R |10
:01:00 |:01:03 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |R |10
:01:00 |:01:00 |x.x.x.1 |33333 |x.x.x.2 |76543 |6 |PA |10
:01:02 |:01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |PA |10
rwstats --fields=sTime,sIP,sPort,dIP,dPort,proto,flags --values=bytes –bin-time=60
will give
sTime |sIP |sPort |dIP |dPort |proto |flags |bytes
:01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |PA |10
:01:00 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |R |20
:01:00 |x.x.x.1 |33333 |x.x.x.2 |76543 |6 |PA |10
:01:02 |x.x.x.1 |12345 |x.x.x.2 |98765 |6 |PA |10
What values would you expect to see in your table:
sIP|sPort|dIP|dPort|pro|ACK|SYN|FIN|Bytes|
Angela Horneman
Analysis Team Lead
Member of the Technical Staff
CERT Software Engineering Institute
Carnegie Mellon University
ahorneman at cert.org<mailto:ahorneman at cert.org>
From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org<mailto:netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org> [mailto:netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org] On Behalf Of Hossam Zalabany
Sent: Thursday, July 18, 2019 8:44 AM
To: netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Flags type count in rwstats
Dear Cert.
I am trying to run SilK to parse statistics of my IPFIX top talkers per minute, I configured the sensor to have a new file for each minute, and keeping the in and out pairing out of the scoop at the moment, all fine so far except that I am only able to count total number of TCP flags, I need to count each flag time separately like, ACK, SYN, and so.
the current command I use is :
rwstats --fields=sip,sport,dip,dport,protocol,flags --integer-tcp-flags --values=byte --count=10 ext2ext-sens1_20190718.09
INPUT: 41300277 Records for 25847944 Bins and 816419146922 Total Bytes
OUTPUT: Top 10 Bins by Bytes
sIP|sPort| dIP|dPort|pro|fla| Bytes| %Bytes| cumul_%|
10.21.64.133 |60870| 10.5.54.69| 2051| 6| 24| 7257019952| 0.888884| 0.888884|
10.5.63.12| 2049| 10.1.223.88| 958| 6| 24| 7136227712| 0.874089| 1.762973|
10.21.64.133 |34073| 10.5.54.69| 2051| 6| 24| 5803794764| 0.710884| 2.473857|
10.5.63.12| 2049| 10.5.4.31| 1020| 6| 24| 3883330408| 0.475654| 2.949511|
10.21.64.133 |34595| 10.5.54.69| 2051| 6| 24| 3857964856| 0.472547| 3.422058|
10.5.204.4| 5247| 10.120.30.236|53789| 17| 0| 3674064752| 0.450022| 3.872080|
10.21.64.133 |37529| 10.5.54.69| 2051| 6| 24| 3262020960| 0.399552| 4.271632|
10.6.8.250| 0| 10.5.204.4| 0| 97| 0| 2994127260| 0.366739| 4.638371|
10.134.26.21| 0| 10.3.250.6| 0| 97| 0| 2893997631| 0.354474| 4.992846|
10.5.63.12| 2049| 10.134.144.21| 747| 6| 24| 2853011756| 0.349454| 5.342300|
what I desire to have is sIP|sPort|dIP|dPort|pro|ACK|SYN|FIN|Bytes|
is there is any suggested steps ?
regards
Hossam
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list