[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Angela Horneman
ahorneman at cert.org
Wed Nov 11 11:12:51 EST 2015
For statistics, the values are directly related to the time window, not state based on successes. This means that the alert amount (e.g. ALERT SINCE LAST TIME) doesn’t apply. The UPDATE statement is all that is needed to tell Analysis Pipeline how often to send the alerts.
From: asad [mailto:a.alii85 at gmail.com]
Sent: Wednesday, November 11, 2015 9:34 AM
To: Angela Horneman <ahorneman at cert.org>
Cc: Timur D. Snoke <tdsnoke at cert.org>; netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Hi Angela,
Thanks for fast response.
Here I want to add earlier you told me to try this construct
STATISTIC checkCountsDistinctDIPs
FILTER non-local-to-remote
FOREACH SIP
DISTINCT DIP
TIME WINDOW 6 MINUTES
SEVERITY 7
UPDATE 5 MINUTES
END STATISTIC
I somehow missed it from equation today, I m wanting to try this since going by the logic in it I should know if in 6 minutes what be count of the distinct destinations IP.
Trying this , I want to know do I need any ALERT clauses as well, unlike EVAL block I don't see any alerting conditions will it default to ALERT SINCE LAST TIME.
Thanks for support.
On Wed, Nov 11, 2015 at 6:55 PM, Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
Hi Asad,
Did you make sure that the address you are checking with the rwfilter is not in either the DMZ.set, voip.set, or rns.set file?
There is not really a way with SiLK to mimic the TIME WINDOW of Analysis Pipeline. Even if you recursively run rwfilter over short intervals, that does not match the Analysis Pipeline's sliding time window. For testing, you might want to use TIME WINDOW 1 DAY if you are comparing with an rwfilter.
Try this:
FILTER toWormPorts
TYPENAME == int2int
DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147]
END FILTER
EVALUATION sipsToManyDipsOnWormPorts
FILTER toWormPorts
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP >= 5
TIME WINDOW 1 DAY
END CHECK
ALERT ALWAYS
ALERT EVERYTHING
CLEAR ALWAYS
END EVALUATION
If it works, you can try decreasing the time window in the check threshold part of the evaluation, then adding in the "whitelist" set files to the filter.
Angela
-----Original Message-----
From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org<mailto:cert.org at cert.org> [mailto:netsa-tools-discuss-bounces+ahorneman<mailto:netsa-tools-discuss-bounces%2Bahorneman>=cert.org at cert.org<mailto:cert.org at cert.org>] On Behalf Of asad
Sent: Wednesday, November 11, 2015 5:34 AM
To: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
Cc: netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Here is the result of the filter
farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
--saddress=10.10.81.74 --start-date=2015/11/6 --dport=137 --pass=stdout | rwstats --fields=sip,dip --values=records --top
--count=5
INPUT: 142555 Records for 2944 Bins and 142555 Total Records
OUTPUT: Top 5 Bins by Records
sIP| dIP| Records| %Records| cumul_%|
10.10.81.74| 192.168.33.74| 96| 0.067342| 0.067342|
10.10.81.74| 192.168.172.1| 96| 0.067342| 0.134685|
10.10.81.74| 192.168.181.45| 96| 0.067342| 0.202027|
10.10.81.74| 192.168.172.17| 96| 0.067342| 0.269370|
10.10.81.74| 10.10.232.21| 96| 0.067342| 0.336712|
Now, I'm not sure why the ALERT will not still be seen.
The auxLog.log shows
"
2015-11-10 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"
My update pipeline.conf says
FILTER non-local-to-remote
TYPENAME IN_LIST [int2int]
SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
SIP NOT_IN_LIST "/root/silkydata/voip.set"
SIP NOT_IN_LIST "/root/silkydata/rns.set"
DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
EVALUATION Systems_using_many_different_protocols
FILTER outgoing-flows
FOREACH SIP
CHECK THRESHOLD
DISTINCT DPORT > 25
TIME_WINDOW 3600 SECONDS
END CHECK
SEVERITY 7
ALERT JUST_NEW_THIS_TIME
ALERT ALWAYS
CLEAR NEVER
END EVALUATION
EVALUATION Common-worm-ports
FILTER non-local-to-remote
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP > 5
TIME_WINDOW 300 SECONDS
END CHECK
SEVERITY 7
ALERT ALWAYS
CLEAR NEVER
END EVALUATION
On 11/10/15, Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>> wrote:
> Asad,
>
> The more information you provide the better our ability to help you
> work through what your configuration will need to be.
>
> It is good to be using TYPENAME as a limiting factor at the start of
> your FILTER, often we see that at least half of the traffic by volume
> is web traffic so excluding that from your EVALUATION will provide a
> performance improvement.
>
> The INT2INT traffic usually reflects an incomplete site definition, it
> would be good to fix that because you might find that you have to make
> special accommodations in your FILTER composition.
>
> I hope this helps,
>
> --
> Timur Snoke
> Network Defense Analyst
> CERT/CC - Network Situational Awareness Software Engineering Institute
> (SEI)
> O: (412) 268-7806
>
> From: asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com><mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>>
> Date: Tuesday, November 10, 2015 at 9:32 AM
> To: timur snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org><mailto:tdsnoke at cert.org<mailto:tdsnoke at cert.org>>>
> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues
> with EVAL block
>
>
>
>
>
> On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
> <tdsnoke at cert.org<mailto:tdsnoke at cert.org><mailto:tdsnoke at cert.org<mailto:tdsnoke at cert.org>>> wrote:
> Hello Asad,
>
> This is an interesting question but I am not sure I understand from
> your description what you are trying to capture.
>
> Thanks Timur,
>
> Let me re-explain it in a clear way.
>
>
>
> You are using type and defining SIP in your filters but do not really
> explain it in your use case.
>
> In my case I want the source IP which is involved in communicating
> with common worm ports to at least x5 different destinations IP.
> Further I want this to match as much as 5 times. I think I need RECORD COUNT >5?
>
>
> Are you looking for outside hosts that are trying to scan these ports
> on multiple hosts inside your network?
> If this is the case you should just use IN for your TYPENAME, there
> are no web ports or icmp traffic that you are concerned with in your
> port list. If the initiating host is outside then you wouldn’t want
> INT2INT, OUT or OUTWEB. This change will potentially limit the total
> number of flows being evaluated.
>
> In my current system which is SIEM the rules are firing for traffic
> direction which is int2int.
>
>
> Can you show example flows that should match but doesn’t?
>
> I will prepare an rwfilter results for you and get back to you. I have
> evidence of its using traffic logs from cisco asa I can show that If
> you want.
>
> --
> Timur Snoke
> Network Defense Analyst
> CERT/CC - Network Situational Awareness Software Engineering Institute
> (SEI)
> O: (412) 268-7806
>
> From:
> <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:cert.org at cert.org><mailto:netsa-to<mailto:netsa-to>
> ols-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:cert.org at cert.org>>>
> on behalf of asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com><mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>>
> Date: Tuesday, November 10, 2015 at 8:30 AM
> To: "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org><mailto:netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>"
> <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org><mailto:netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>>
> Subject: [netsa-tools-discuss] analysis pipeline alerting issues with
> EVAL block
>
> Hello,
>
> I have a very simple alerting requirement
>
> "
> when a destination ports matches ports which are considered as 'worm ports'
> traffic is send to 5 different unique ips in 5 minutes time.
>
> "
>
> I know on traffic level i'm getting required data since using traffic
> logs from the cisco asa (same device is also sending netflows) and it
> works as expected. I'm suppose to see an ip address but on alert.log I see nothing.
> Below is the logic.
>
> Any help?
>
>
>
> FILTER outgoing-flows
> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> END FILTER
>
> FILTER non-local-to-remote
> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
> 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
>
>
> EVALUATION Systems_using_many_different_protocols
> FILTER non-local-to-remote
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DIP > 25
> TIME_WINDOW 3600 SECONDS
> END CHECK
> SEVERITY 7
> ALERT JUST_NEW_THIS_TIME
> ALERT ALWAYS
> CLEAR NEVER
> END EVALUATION
>
>
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list