[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Angela Horneman
ahorneman at cert.org
Thu Nov 12 12:41:20 EST 2015
It sounds like what you want to do is track SIPs with 250 different DIPs and SIP DIP pairs with 10 flows and alert when a SIP matches both. If so, try something like (explanation below the blocks):
FILTER ourNetwork
SIP == 193.0.0.0/8
END FILTER
EVALUATION ourNetworkSIPDIPsMultiFlows
FILTER ourNetwork
FOREACH SIP DIP
CHECK THRESHOLD
RECORD COUNT > 10
TIME WINDOW 1 HOUR
END CHECK
CLEAR ALWAYS
OUTPUT LIST SIP DIP ourNetworkSIPDIPsMultiFlows_list
OUTPUT TIMEOUT 1 HOUR
DO NOT ALERT
END EVALUATION
FILTER ourNetworkSipDipManyFlows
SIP DIP IN_LIST ourNetworkSIPDIPsMultiFlows_list
END FILTER
EVALUATION ourNetworkSipsToLotsOfDipsWithManyFlows
FILTER sipDipManyFlows
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP > 250
TIME WINDOW 1 HOUR
END CHECK
CLEAR ALWAYS
OUTPUT TIMEOUT 1 HOUR
ALERT ALWAYS
ALERT EACH ONLY ONCE
END EVALUATION
This will take all flows where the source is “ourNetwork” and track how many flows are between each SIP DIP pair. As soon as a pair has at least 10 flows, they get added to a list. Then we filter new flows to get only flows that are between SIP DIP pairs have 10+ flows between them. Of all those flows, we track the count of the unique DIPs each SIP has had 10+ flow with. Once a SIP has at least 250, it becomes a success and alerts.
Angela
From: asad [mailto:a.alii85 at gmail.com]
Sent: Thursday, November 12, 2015 8:56 AM
To: Angela Horneman <ahorneman at cert.org>
Cc: Timur D. Snoke <tdsnoke at cert.org>; netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Hello Angela,
Yes, your last statement is true, i thought RECORD COUNT would be used for that but I'm failing to understand how I can ADD the two operations I cannot used RECORD COUNT in single CHECK statement.
On Thu, Nov 12, 2015 at 6:51 PM, Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
Good morning Asad,
The way it is currently set up, it will become a success as soon as a SIP has at least 10 flows and contacts 250 or more unique DIPs in an hour.
- The TIME WINDOW of each CHECK block is independent.
- The threshold values for each CHECK block must be met for each key (in your case SIP, since you are using FOREACH SIP), before the key becomes a success (i.e. available for alerting or inclusion in lists).
In this case, the RECORD COUNT > 10 is unnecessary as by definition, if there are at least 250 unique DIPs there are at least that number of flows.
Are you trying to detect where a SIP contacts at least 250 different DIPs and where it has a flow to each DIP at least 10 times?
Angela
From: asad [mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>]
Sent: Thursday, November 12, 2015 8:40 AM
To: Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>>
Cc: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>; netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Angela, I need your advice
FILTER ourNetwork
SIP == 193.0.0.0/8<http://193.0.0.0/8>
END FILTER
FILTER ourNetwork
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP > 250
TIME_WINDOW 1 HOUR
END CHECK
CHECK THRESHOLD
RECORD COUNT > 10
TIME_WINDOW 1 HOUR
END CHECK
END EVALUATION
I'm looking for match which happens more then 10 times in second CHECK block do I need to specify TIME WINDOW separate or will it AND the results for e.g more then 250 IPs in 1 hour and PLUS generates / matches at least 10 times. Thanks.
On Wed, Nov 11, 2015 at 9:12 PM, Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
For statistics, the values are directly related to the time window, not state based on successes. This means that the alert amount (e.g. ALERT SINCE LAST TIME) doesn’t apply. The UPDATE statement is all that is needed to tell Analysis Pipeline how often to send the alerts.
From: asad [mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>]
Sent: Wednesday, November 11, 2015 9:34 AM
To: Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>>
Cc: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>; netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Hi Angela,
Thanks for fast response.
Here I want to add earlier you told me to try this construct
STATISTIC checkCountsDistinctDIPs
FILTER non-local-to-remote
FOREACH SIP
DISTINCT DIP
TIME WINDOW 6 MINUTES
SEVERITY 7
UPDATE 5 MINUTES
END STATISTIC
I somehow missed it from equation today, I m wanting to try this since going by the logic in it I should know if in 6 minutes what be count of the distinct destinations IP.
Trying this , I want to know do I need any ALERT clauses as well, unlike EVAL block I don't see any alerting conditions will it default to ALERT SINCE LAST TIME.
Thanks for support.
On Wed, Nov 11, 2015 at 6:55 PM, Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
Hi Asad,
Did you make sure that the address you are checking with the rwfilter is not in either the DMZ.set, voip.set, or rns.set file?
There is not really a way with SiLK to mimic the TIME WINDOW of Analysis Pipeline. Even if you recursively run rwfilter over short intervals, that does not match the Analysis Pipeline's sliding time window. For testing, you might want to use TIME WINDOW 1 DAY if you are comparing with an rwfilter.
Try this:
FILTER toWormPorts
TYPENAME == int2int
DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147]
END FILTER
EVALUATION sipsToManyDipsOnWormPorts
FILTER toWormPorts
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP >= 5
TIME WINDOW 1 DAY
END CHECK
ALERT ALWAYS
ALERT EVERYTHING
CLEAR ALWAYS
END EVALUATION
If it works, you can try decreasing the time window in the check threshold part of the evaluation, then adding in the "whitelist" set files to the filter.
Angela
-----Original Message-----
From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org<mailto:cert.org at cert.org> [mailto:netsa-tools-discuss-bounces+ahorneman<mailto:netsa-tools-discuss-bounces%2Bahorneman>=cert.org at cert.org<mailto:cert.org at cert.org>] On Behalf Of asad
Sent: Wednesday, November 11, 2015 5:34 AM
To: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
Cc: netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Here is the result of the filter
farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
--saddress=10.10.81.74 --start-date=2015/11/6 --dport=137 --pass=stdout | rwstats --fields=sip,dip --values=records --top
--count=5
INPUT: 142555 Records for 2944 Bins and 142555 Total Records
OUTPUT: Top 5 Bins by Records
sIP| dIP| Records| %Records| cumul_%|
10.10.81.74| 192.168.33.74| 96| 0.067342| 0.067342|
10.10.81.74| 192.168.172.1| 96| 0.067342| 0.134685|
10.10.81.74| 192.168.181.45| 96| 0.067342| 0.202027|
10.10.81.74| 192.168.172.17| 96| 0.067342| 0.269370|
10.10.81.74| 10.10.232.21| 96| 0.067342| 0.336712|
Now, I'm not sure why the ALERT will not still be seen.
The auxLog.log shows
"
2015-11-10 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"
My update pipeline.conf says
FILTER non-local-to-remote
TYPENAME IN_LIST [int2int]
SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
SIP NOT_IN_LIST "/root/silkydata/voip.set"
SIP NOT_IN_LIST "/root/silkydata/rns.set"
DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
EVALUATION Systems_using_many_different_protocols
FILTER outgoing-flows
FOREACH SIP
CHECK THRESHOLD
DISTINCT DPORT > 25
TIME_WINDOW 3600 SECONDS
END CHECK
SEVERITY 7
ALERT JUST_NEW_THIS_TIME
ALERT ALWAYS
CLEAR NEVER
END EVALUATION
EVALUATION Common-worm-ports
FILTER non-local-to-remote
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP > 5
TIME_WINDOW 300 SECONDS
END CHECK
SEVERITY 7
ALERT ALWAYS
CLEAR NEVER
END EVALUATION
On 11/10/15, Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>> wrote:
> Asad,
>
> The more information you provide the better our ability to help you
> work through what your configuration will need to be.
>
> It is good to be using TYPENAME as a limiting factor at the start of
> your FILTER, often we see that at least half of the traffic by volume
> is web traffic so excluding that from your EVALUATION will provide a
> performance improvement.
>
> The INT2INT traffic usually reflects an incomplete site definition, it
> would be good to fix that because you might find that you have to make
> special accommodations in your FILTER composition.
>
> I hope this helps,
>
> --
> Timur Snoke
> Network Defense Analyst
> CERT/CC - Network Situational Awareness Software Engineering Institute
> (SEI)
> O: (412) 268-7806
>
> From: asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com><mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>>
> Date: Tuesday, November 10, 2015 at 9:32 AM
> To: timur snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org><mailto:tdsnoke at cert.org<mailto:tdsnoke at cert.org>>>
> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues
> with EVAL block
>
>
>
>
>
> On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
> <tdsnoke at cert.org<mailto:tdsnoke at cert.org><mailto:tdsnoke at cert.org<mailto:tdsnoke at cert.org>>> wrote:
> Hello Asad,
>
> This is an interesting question but I am not sure I understand from
> your description what you are trying to capture.
>
> Thanks Timur,
>
> Let me re-explain it in a clear way.
>
>
>
> You are using type and defining SIP in your filters but do not really
> explain it in your use case.
>
> In my case I want the source IP which is involved in communicating
> with common worm ports to at least x5 different destinations IP.
> Further I want this to match as much as 5 times. I think I need RECORD COUNT >5?
>
>
> Are you looking for outside hosts that are trying to scan these ports
> on multiple hosts inside your network?
> If this is the case you should just use IN for your TYPENAME, there
> are no web ports or icmp traffic that you are concerned with in your
> port list. If the initiating host is outside then you wouldn’t want
> INT2INT, OUT or OUTWEB. This change will potentially limit the total
> number of flows being evaluated.
>
> In my current system which is SIEM the rules are firing for traffic
> direction which is int2int.
>
>
> Can you show example flows that should match but doesn’t?
>
> I will prepare an rwfilter results for you and get back to you. I have
> evidence of its using traffic logs from cisco asa I can show that If
> you want.
>
> --
> Timur Snoke
> Network Defense Analyst
> CERT/CC - Network Situational Awareness Software Engineering Institute
> (SEI)
> O: (412) 268-7806
>
> From:
> <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:cert.org at cert.org><mailto:netsa-to<mailto:netsa-to>
> ols-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:cert.org at cert.org>>>
> on behalf of asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com><mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>>
> Date: Tuesday, November 10, 2015 at 8:30 AM
> To: "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org><mailto:netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>"
> <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org><mailto:netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>>
> Subject: [netsa-tools-discuss] analysis pipeline alerting issues with
> EVAL block
>
> Hello,
>
> I have a very simple alerting requirement
>
> "
> when a destination ports matches ports which are considered as 'worm ports'
> traffic is send to 5 different unique ips in 5 minutes time.
>
> "
>
> I know on traffic level i'm getting required data since using traffic
> logs from the cisco asa (same device is also sending netflows) and it
> works as expected. I'm suppose to see an ip address but on alert.log I see nothing.
> Below is the logic.
>
> Any help?
>
>
>
> FILTER outgoing-flows
> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> END FILTER
>
> FILTER non-local-to-remote
> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
> 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
>
>
> EVALUATION Systems_using_many_different_protocols
> FILTER non-local-to-remote
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DIP > 25
> TIME_WINDOW 3600 SECONDS
> END CHECK
> SEVERITY 7
> ALERT JUST_NEW_THIS_TIME
> ALERT ALWAYS
> CLEAR NEVER
> END EVALUATION
>
>
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list