[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Angela Horneman
ahorneman at cert.org
Thu Nov 12 08:56:13 EST 2015
Hi Asad,
Can you send the non-working configuration file as an attachment? I don't see anything obvious below.
Angela
-----Original Message-----
From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org [mailto:netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org] On Behalf Of asad
Sent: Thursday, November 12, 2015 2:02 AM
To: Timur D. Snoke <tdsnoke at cert.org>
Cc: netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
Also, another important thing that it worked only when I removed second EVAL block, I don't know when adding two together it stopped working all ALERTING stops.
FILTER outgoing-flows
TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
SIP NOT_IN_LIST "/root/silkydata/voip.set"
SIP NOT_IN_LIST "/root/silkydata/rns.set"
END FILTER
FILTER non-local-to-remote
TYPENAME == int2int
SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
SIP NOT_IN_LIST "/root/silkydata/voip.set"
DPORT IN_LIST [137,6]
END FILTER
EVALUATION Systems_using_many_different_protocols
FILTER outgoing-flows
FOREACH SIP
CHECK THRESHOLD
DISTINCT DPORT > 25
TIME_WINDOW 3600 SECONDS
END CHECK
SEVERITY 7
ALERT ALWAYS
CLEAR NEVER
END EVALUATION
EVALUATION Common-worm-ports
FILTER non-local-to-remote
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP > 5
TIME_WINDOW 360 SECONDS
END CHECK
SEVERITY 7
ALERT ALWAYS
ALERT EVERYTHING
CLEAR ALWAYS
END EVALUATION
What wrong in order?
On 11/12/15, asad <a.alii85 at gmail.com> wrote:
> Good news to all, its working i tested Angela i saw STATISTICS block
> working, compared it with EVAL block I see that I was looking for
> events in 5 minutes window not 6 as in STATISTICS i corrected that and
> its working now:) Thanks Timur and Angela you people are good.
>
> On 11/12/15, asad <a.alii85 at gmail.com> wrote:
>> Timur,
>>
>> This is so strange, consider this
>>
>> farhan at netflow:~/silkydata$ rwfilter --sensor=S0 --type=int2int
>> --saddress=10.10.81.74 --start-date=2015/11/06T09:15
>> --end-date=2015/11/06T09:20 --dport=137 --pass=stdout | rwuniq
>> --field=sip --dip-distinct
>> rwfilter: Warning: start-date precision greater than hours ignored
>> rwfilter: Warning: end-date precision greater than hours ignored
>> sIP|dIP-Distin|
>> 10.10.81.74| 2928|
>>
>>
>> I'm getting 2928 unique destination ip matches and analysis pipeline
>> won't even detect 5?
>> Why?
>>
>> On 11/11/15, Timur D. Snoke <tdsnoke at cert.org> wrote:
>>> Asad, over what interval are those connections?
>>>
>>> You can try this:
>>>
>>> rwfilter \
>>> --sensor=S0 \
>>> --type=int2int \
>>> --saddress=10.10.81.74 \
>>> --start-date=2015/11/6 \
>>> --dport=137
>>> --pass=stdout \
>>> | rwcount \
>>> --bin-size=300 \
>>> --skip-zeroes
>>>
>>>
>>> Your evaluation is that an inside host talks to more than 5 outside
>>> hosts during a five minute window. The previous rwfilter query will
>>> help us find the most active hour and then determine you can try the
>>> following commands to determine if the conditions are met. Assuming
>>> the busiest time slice is 12:30-12:40.
>>>
>>> rwfilter \
>>> --sensor=S0 \
>>> --type=int2int \
>>> --saddress=10.10.81.74 \
>>> --start-date=2015/11/6T12:30 \
>>> --end-date=2015/11/16T12:40 \
>>> --dport=137
>>> --pass=stdout \
>>> | rwuniq --field=sip —dip-distinct
>>>
>>> And to see what the flows look like for that time period ordered by
>>> time try
>>> this:
>>>
>>> rwfilter \
>>> --sensor=S0 \
>>> --type=int2int \
>>> --saddress=10.10.81.74 \
>>> --start-date=2015/11/6T12:30 \
>>> --end-date=2015/11/16T12:40 \
>>> --dport=137
>>> --pass=stdout \
>>> | rwsort --field=stime \
>>> | rwcut
>>>
>>> I think you are seeing the traffic but not the 5 dips in 5 minutes.
>>> If you get results that are different please include them in your
>>> response.
>>>
>>>
>>>
>>> I hope this helps,
>>>
>>> --
>>> Timur Snoke
>>> Network Defense Analyst
>>> CERT/CC - Network Situational Awareness Software Engineering
>>> Institute (SEI)
>>> O: (412) 268-7806
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 11/11/15, 5:34 AM, "asad" <a.alii85 at gmail.com> wrote:
>>>
>>>>Here is the result of the filter
>>>>
>>>>farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
>>>>--saddress=10.10.81.74 --start-date=2015/11/6 --dport=137
>>>>--pass=stdout | rwstats --fields=sip,dip --values=records --top
>>>>--count=5
>>>>INPUT: 142555 Records for 2944 Bins and 142555 Total Records
>>>>OUTPUT: Top 5 Bins by Records
>>>> sIP| dIP| Records| %Records| cumul_%|
>>>> 10.10.81.74| 192.168.33.74| 96| 0.067342| 0.067342|
>>>> 10.10.81.74| 192.168.172.1| 96| 0.067342| 0.134685|
>>>> 10.10.81.74| 192.168.181.45| 96| 0.067342| 0.202027|
>>>> 10.10.81.74| 192.168.172.17| 96| 0.067342| 0.269370|
>>>> 10.10.81.74| 10.10.232.21| 96| 0.067342| 0.336712|
>>>>
>>>>
>>>>Now, I'm not sure why the ALERT will not still be seen.
>>>>
>>>>The auxLog.log shows
>>>>
>>>>"
>>>>2015-11-10
>>>> 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"
>>>>
>>>>
>>>>My update pipeline.conf says
>>>>
>>>>FILTER non-local-to-remote
>>>>TYPENAME IN_LIST [int2int]
>>>>SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
>>>>SIP NOT_IN_LIST "/root/silkydata/voip.set"
>>>>SIP NOT_IN_LIST "/root/silkydata/rns.set"
>>>>DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997,
>>>>998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134,
>>>>3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$
>>>>END FILTER
>>>>
>>>>
>>>>EVALUATION Systems_using_many_different_protocols
>>>>FILTER outgoing-flows
>>>>FOREACH SIP
>>>>CHECK THRESHOLD
>>>>DISTINCT DPORT > 25
>>>>TIME_WINDOW 3600 SECONDS
>>>>END CHECK
>>>>SEVERITY 7
>>>>ALERT JUST_NEW_THIS_TIME
>>>>ALERT ALWAYS
>>>>CLEAR NEVER
>>>>END EVALUATION
>>>>
>>>>
>>>>EVALUATION Common-worm-ports
>>>>FILTER non-local-to-remote
>>>>FOREACH SIP
>>>>CHECK THRESHOLD
>>>>DISTINCT DIP > 5
>>>>TIME_WINDOW 300 SECONDS
>>>>END CHECK
>>>>SEVERITY 7
>>>>ALERT ALWAYS
>>>>CLEAR NEVER
>>>>END EVALUATION
>>>>
>>>>On 11/10/15, Timur D. Snoke <tdsnoke at cert.org> wrote:
>>>>> Asad,
>>>>>
>>>>> The more information you provide the better our ability to help
>>>>> you work through what your configuration will need to be.
>>>>>
>>>>> It is good to be using TYPENAME as a limiting factor at the start
>>>>> of your FILTER, often we see that at least half of the traffic by
>>>>> volume is web traffic so excluding that from your EVALUATION will
>>>>> provide a performance improvement.
>>>>>
>>>>> The INT2INT traffic usually reflects an incomplete site
>>>>> definition, it would be good to fix that because you might find
>>>>> that you have to make special accommodations in your FILTER
>>>>> composition.
>>>>>
>>>>> I hope this helps,
>>>>>
>>>>> --
>>>>> Timur Snoke
>>>>> Network Defense Analyst
>>>>> CERT/CC - Network Situational Awareness Software Engineering
>>>>> Institute (SEI)
>>>>> O: (412) 268-7806
>>>>>
>>>>> From: asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
>>>>> Date: Tuesday, November 10, 2015 at 9:32 AM
>>>>> To: timur snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
>>>>> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting
>>>>> issues with EVAL block
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
>>>>> <tdsnoke at cert.org<mailto:tdsnoke at cert.org>> wrote:
>>>>> Hello Asad,
>>>>>
>>>>> This is an interesting question but I am not sure I understand
>>>>> from your description what you are trying to capture.
>>>>>
>>>>> Thanks Timur,
>>>>>
>>>>> Let me re-explain it in a clear way.
>>>>>
>>>>>
>>>>>
>>>>> You are using type and defining SIP in your filters but do not
>>>>> really explain it in your use case.
>>>>>
>>>>> In my case I want the source IP which is involved in communicating
>>>>> with common worm ports to at least x5 different destinations IP.
>>>>> Further I want this to match as much as 5 times. I think I need
>>>>> RECORD COUNT >5?
>>>>>
>>>>>
>>>>> Are you looking for outside hosts that are trying to scan these
>>>>> ports on multiple hosts inside your network?
>>>>> If this is the case you should just use IN for your TYPENAME,
>>>>> there are no web ports or icmp traffic that you are concerned with
>>>>> in your port list.
>>>>> If
>>>>> the initiating host is outside then you wouldn’t want INT2INT, OUT
>>>>> or OUTWEB. This change will potentially limit the total number of
>>>>> flows being evaluated.
>>>>>
>>>>> In my current system which is SIEM the rules are firing for
>>>>> traffic direction which is int2int.
>>>>>
>>>>>
>>>>> Can you show example flows that should match but doesn’t?
>>>>>
>>>>> I will prepare an rwfilter results for you and get back to you. I
>>>>> have evidence of its using traffic logs from cisco asa I can show
>>>>> that If you want.
>>>>>
>>>>> --
>>>>> Timur Snoke
>>>>> Network Defense Analyst
>>>>> CERT/CC - Network Situational Awareness Software Engineering
>>>>> Institute (SEI)
>>>>> O: (412) 268-7806
>>>>>
>>>>> From:
>>>>> <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:nets
>>>>> a-tools-discuss-bounces+tdsnoke=cert.org at cert.org>>
>>>>> on behalf of asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
>>>>> Date: Tuesday, November 10, 2015 at 8:30 AM
>>>>> To:
>>>>> "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>"
>>>>> <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
>>>>> >
>>>>> Subject: [netsa-tools-discuss] analysis pipeline alerting issues
>>>>> with EVAL block
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have a very simple alerting requirement
>>>>>
>>>>> "
>>>>> when a destination ports matches ports which are considered as
>>>>> 'worm ports'
>>>>> traffic is send to 5 different unique ips in 5 minutes time.
>>>>>
>>>>> "
>>>>>
>>>>> I know on traffic level i'm getting required data since using
>>>>> traffic logs from the cisco asa (same device is also sending
>>>>> netflows) and it works as expected. I'm suppose to see an ip
>>>>> address but on alert.log I see nothing.
>>>>> Below is the logic.
>>>>>
>>>>> Any help?
>>>>>
>>>>>
>>>>>
>>>>> FILTER outgoing-flows
>>>>> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
>>>>> "/root/silkydata/DMZ.set"
>>>>> SIP NOT_IN_LIST "/root/silkydata/voip.set"
>>>>> SIP NOT_IN_LIST "/root/silkydata/rns.set"
>>>>> END FILTER
>>>>>
>>>>> FILTER non-local-to-remote
>>>>> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
>>>>> "/root/silkydata/DMZ.set"
>>>>> SIP NOT_IN_LIST "/root/silkydata/voip.set"
>>>>> SIP NOT_IN_LIST "/root/silkydata/rns.set"
>>>>> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997,
>>>>> 998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134,
>>>>> 3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147,
>>>>> 31$ END FILTER
>>>>>
>>>>>
>>>>> EVALUATION Systems_using_many_different_protocols
>>>>> FILTER non-local-to-remote
>>>>> FOREACH SIP
>>>>> CHECK THRESHOLD
>>>>> DISTINCT DIP > 25
>>>>> TIME_WINDOW 3600 SECONDS
>>>>> END CHECK
>>>>> SEVERITY 7
>>>>> ALERT JUST_NEW_THIS_TIME
>>>>> ALERT ALWAYS
>>>>> CLEAR NEVER
>>>>> END EVALUATION
>>>>>
>>>>>
>>>>
>>>
>>
>
More information about the netsa-tools-discuss
mailing list