[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

asad a.alii85 at gmail.com
Thu Nov 12 02:01:52 EST 2015


Also, another important thing that it worked only when I removed
second EVAL block, I don't know when adding two together it stopped
working all ALERTING stops.

FILTER outgoing-flows
TYPENAME IN_LIST [in,int2int,out,outweb,outicmp]
SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
SIP NOT_IN_LIST "/root/silkydata/voip.set"
SIP NOT_IN_LIST "/root/silkydata/rns.set"
END FILTER


FILTER non-local-to-remote
TYPENAME == int2int
SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
SIP NOT_IN_LIST "/root/silkydata/voip.set"
DPORT IN_LIST [137,6]
END FILTER

EVALUATION  Systems_using_many_different_protocols
FILTER outgoing-flows
FOREACH SIP
CHECK THRESHOLD
DISTINCT DPORT > 25
TIME_WINDOW 3600 SECONDS
END CHECK
SEVERITY 7
ALERT ALWAYS
CLEAR NEVER
END EVALUATION

EVALUATION  Common-worm-ports
FILTER non-local-to-remote
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP > 5
TIME_WINDOW 360 SECONDS
END CHECK
SEVERITY 7
ALERT ALWAYS
ALERT EVERYTHING
CLEAR ALWAYS
END EVALUATION

What wrong in order?

On 11/12/15, asad <a.alii85 at gmail.com> wrote:
> Good news to all, its working i tested Angela i saw STATISTICS block
> working, compared it with EVAL block I see that I was looking for
> events in 5 minutes window not 6 as in STATISTICS i corrected that and
> its working now:) Thanks Timur and Angela you people are good.
>
> On 11/12/15, asad <a.alii85 at gmail.com> wrote:
>> Timur,
>>
>> This is so strange, consider this
>>
>> farhan at netflow:~/silkydata$ rwfilter   --sensor=S0   --type=int2int
>> --saddress=10.10.81.74   --start-date=2015/11/06T09:15
>> --end-date=2015/11/06T09:20   --dport=137  --pass=stdout  | rwuniq
>> --field=sip --dip-distinct
>> rwfilter: Warning: start-date precision greater than hours ignored
>> rwfilter: Warning: end-date precision greater than hours ignored
>>             sIP|dIP-Distin|
>>     10.10.81.74|      2928|
>>
>>
>> I'm getting 2928 unique destination ip matches and analysis pipeline
>> won't even detect 5?
>> Why?
>>
>> On 11/11/15, Timur D. Snoke <tdsnoke at cert.org> wrote:
>>> Asad, over what interval are those connections?
>>>
>>> You can try this:
>>>
>>> rwfilter \
>>>   --sensor=S0 \
>>>   --type=int2int \
>>>   --saddress=10.10.81.74 \
>>>   --start-date=2015/11/6 \
>>>   --dport=137
>>>   --pass=stdout \
>>>  | rwcount \
>>>   --bin-size=300 \
>>>   --skip-zeroes
>>>
>>>
>>> Your evaluation is that an inside host talks to more than 5 outside
>>> hosts
>>> during a five minute window. The previous rwfilter query will help us
>>> find
>>> the most active hour and then determine you can try the following
>>> commands
>>> to determine if the conditions are met. Assuming the busiest time slice
>>> is
>>> 12:30-12:40.
>>>
>>> rwfilter \
>>>   --sensor=S0 \
>>>   --type=int2int \
>>>   --saddress=10.10.81.74 \
>>>   --start-date=2015/11/6T12:30 \
>>> --end-date=2015/11/16T12:40 \
>>>   --dport=137
>>>   --pass=stdout \
>>>  | rwuniq --field=sip —dip-distinct
>>>
>>> And to see what the flows look like for that time period ordered by time
>>> try
>>> this:
>>>
>>> rwfilter \
>>>   --sensor=S0 \
>>>   --type=int2int \
>>>   --saddress=10.10.81.74 \
>>>   --start-date=2015/11/6T12:30 \
>>>   --end-date=2015/11/16T12:40 \
>>>   --dport=137
>>>   --pass=stdout \
>>>  | rwsort --field=stime \
>>>  | rwcut
>>>
>>> I think you are seeing the traffic but not the 5 dips in 5 minutes. If
>>> you
>>> get results that are different please include them in your response.
>>>
>>>
>>>
>>> I hope this helps,
>>>
>>> --
>>> Timur Snoke
>>> Network Defense Analyst
>>> CERT/CC - Network Situational Awareness
>>> Software Engineering Institute (SEI)
>>> O: (412) 268-7806
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 11/11/15, 5:34 AM, "asad" <a.alii85 at gmail.com> wrote:
>>>
>>>>Here is the result of the filter
>>>>
>>>>farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
>>>>--saddress=10.10.81.74 --start-date=2015/11/6  --dport=137
>>>>--pass=stdout  | rwstats --fields=sip,dip --values=records --top
>>>>--count=5
>>>>INPUT: 142555 Records for 2944 Bins and 142555 Total Records
>>>>OUTPUT: Top 5 Bins by Records
>>>>            sIP|            dIP|   Records|  %Records|   cumul_%|
>>>>    10.10.81.74|  192.168.33.74|        96|  0.067342|  0.067342|
>>>>    10.10.81.74|  192.168.172.1|        96|  0.067342|  0.134685|
>>>>    10.10.81.74| 192.168.181.45|        96|  0.067342|  0.202027|
>>>>    10.10.81.74| 192.168.172.17|        96|  0.067342|  0.269370|
>>>>    10.10.81.74|   10.10.232.21|        96|  0.067342|  0.336712|
>>>>
>>>>
>>>>Now, I'm not sure why the ALERT will not still be seen.
>>>>
>>>>The auxLog.log shows
>>>>
>>>>"
>>>>2015-11-10
>>>> 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"
>>>>
>>>>
>>>>My update pipeline.conf says
>>>>
>>>>FILTER non-local-to-remote
>>>>TYPENAME IN_LIST [int2int]
>>>>SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
>>>>SIP NOT_IN_LIST "/root/silkydata/voip.set"
>>>>SIP NOT_IN_LIST "/root/silkydata/rns.set"
>>>>DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
>>>>999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
>>>>3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$
>>>>END FILTER
>>>>
>>>>
>>>>EVALUATION  Systems_using_many_different_protocols
>>>>FILTER outgoing-flows
>>>>FOREACH SIP
>>>>CHECK THRESHOLD
>>>>DISTINCT DPORT > 25
>>>>TIME_WINDOW 3600 SECONDS
>>>>END CHECK
>>>>SEVERITY 7
>>>>ALERT JUST_NEW_THIS_TIME
>>>>ALERT ALWAYS
>>>>CLEAR NEVER
>>>>END EVALUATION
>>>>
>>>>
>>>>EVALUATION  Common-worm-ports
>>>>FILTER non-local-to-remote
>>>>FOREACH SIP
>>>>CHECK THRESHOLD
>>>>DISTINCT DIP > 5
>>>>TIME_WINDOW 300 SECONDS
>>>>END CHECK
>>>>SEVERITY 7
>>>>ALERT ALWAYS
>>>>CLEAR NEVER
>>>>END EVALUATION
>>>>
>>>>On 11/10/15, Timur D. Snoke <tdsnoke at cert.org> wrote:
>>>>> Asad,
>>>>>
>>>>> The more information you provide the better our ability to help you
>>>>> work
>>>>> through what your configuration will need to be.
>>>>>
>>>>> It is good to be using TYPENAME as a limiting factor at the start of
>>>>> your
>>>>> FILTER, often we see that at least half of the traffic by volume is
>>>>> web
>>>>> traffic so excluding that from your EVALUATION will provide a
>>>>> performance
>>>>> improvement.
>>>>>
>>>>> The INT2INT traffic usually reflects an incomplete site definition, it
>>>>> would
>>>>> be good to fix that because you might find that you have to make
>>>>> special
>>>>> accommodations in your FILTER composition.
>>>>>
>>>>> I  hope this helps,
>>>>>
>>>>> --
>>>>> Timur Snoke
>>>>> Network Defense Analyst
>>>>> CERT/CC - Network Situational Awareness
>>>>> Software Engineering Institute (SEI)
>>>>> O: (412) 268-7806
>>>>>
>>>>> From: asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
>>>>> Date: Tuesday, November 10, 2015 at 9:32 AM
>>>>> To: timur snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
>>>>> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues
>>>>> with
>>>>> EVAL block
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
>>>>> <tdsnoke at cert.org<mailto:tdsnoke at cert.org>> wrote:
>>>>> Hello Asad,
>>>>>
>>>>> This is an interesting question but I am not sure I understand from
>>>>> your
>>>>> description what you are trying to capture.
>>>>>
>>>>> Thanks Timur,
>>>>>
>>>>> Let me re-explain it in a clear way.
>>>>>
>>>>>
>>>>>
>>>>> You are using type and defining SIP in your filters but do not really
>>>>> explain it in your use case.
>>>>>
>>>>> In my case I want the source IP which is involved in communicating
>>>>> with
>>>>> common worm ports to at least x5 different destinations IP. Further I
>>>>> want
>>>>> this to match as much as 5 times. I think I need RECORD COUNT >5?
>>>>>
>>>>>
>>>>> Are you looking for outside hosts that are trying to scan these ports
>>>>> on
>>>>> multiple hosts inside your network?
>>>>> If this is the case you should just use IN for your TYPENAME, there
>>>>> are
>>>>> no
>>>>> web ports or icmp traffic that you are concerned with in your port
>>>>> list.
>>>>> If
>>>>> the initiating host is outside then you wouldn’t want INT2INT, OUT or
>>>>> OUTWEB. This change will potentially limit the total number of flows
>>>>> being
>>>>> evaluated.
>>>>>
>>>>> In my current system which is SIEM the rules are firing for traffic
>>>>> direction which is int2int.
>>>>>
>>>>>
>>>>> Can you show example flows that should match but doesn’t?
>>>>>
>>>>> I will prepare an rwfilter results for you and get back to you. I have
>>>>> evidence of its using traffic logs from cisco asa I can show that If
>>>>> you
>>>>> want.
>>>>>
>>>>> --
>>>>> Timur Snoke
>>>>> Network Defense Analyst
>>>>> CERT/CC - Network Situational Awareness
>>>>> Software Engineering Institute (SEI)
>>>>> O: (412) 268-7806
>>>>>
>>>>> From:
>>>>> <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org>>
>>>>> on behalf of asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
>>>>> Date: Tuesday, November 10, 2015 at 8:30 AM
>>>>> To:
>>>>> "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>"
>>>>> <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>
>>>>> Subject: [netsa-tools-discuss] analysis pipeline alerting issues with
>>>>> EVAL
>>>>> block
>>>>>
>>>>> Hello,
>>>>>
>>>>> I have a very simple alerting requirement
>>>>>
>>>>> "
>>>>> when a destination ports matches ports which are considered as 'worm
>>>>> ports'
>>>>> traffic is send to 5 different unique ips in 5 minutes time.
>>>>>
>>>>> "
>>>>>
>>>>> I know on traffic level i'm getting required data since using traffic
>>>>> logs
>>>>> from the cisco asa (same device is also sending netflows) and it works
>>>>> as
>>>>> expected. I'm suppose to see an ip address but on alert.log I see
>>>>> nothing.
>>>>> Below is the logic.
>>>>>
>>>>> Any help?
>>>>>
>>>>>
>>>>>
>>>>> FILTER outgoing-flows
>>>>> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp]
>>>>> SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
>>>>> SIP NOT_IN_LIST "/root/silkydata/voip.set"
>>>>> SIP NOT_IN_LIST "/root/silkydata/rns.set"
>>>>> END FILTER
>>>>>
>>>>> FILTER non-local-to-remote
>>>>> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp]
>>>>> SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
>>>>> SIP NOT_IN_LIST "/root/silkydata/voip.set"
>>>>> SIP NOT_IN_LIST "/root/silkydata/rns.set"
>>>>> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
>>>>> 999,
>>>>> 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
>>>>> 3137,
>>>>> 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$
>>>>> END FILTER
>>>>>
>>>>>
>>>>> EVALUATION  Systems_using_many_different_protocols
>>>>> FILTER  non-local-to-remote
>>>>> FOREACH SIP
>>>>> CHECK THRESHOLD
>>>>> DISTINCT DIP > 25
>>>>> TIME_WINDOW 3600 SECONDS
>>>>> END CHECK
>>>>> SEVERITY 7
>>>>> ALERT JUST_NEW_THIS_TIME
>>>>> ALERT ALWAYS
>>>>> CLEAR NEVER
>>>>> END EVALUATION
>>>>>
>>>>>
>>>>
>>>
>>
>


More information about the netsa-tools-discuss mailing list