[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block
asad
a.alii85 at gmail.com
Thu Nov 12 12:17:11 EST 2015
Hi Angela,
This is my intended configuration in regard to use to another check
statement to satisfy the requirement "where a SIP contacts at least 250
different DIPs and where it has a flow to each DIP at least 10 times".
FILTER non-local-to-remote
TYPENAME == int2int
SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
SIP NOT_IN_LIST "/root/silkydata/voip.set"
DPORT IN_LIST [137,6]
END FILTER
EVALUATION Common-worm-ports
FILTER non-local-to-remote
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP > 5
TIME_WINDOW 360 SECONDS
END CHECK
CHECK THRESHOLD
RECORD COUNT > 50
END CHECK
SEVERITY 7
ALERT ALWAYS
ALERT EVERYTHING
CLEAR ALWAYS
END EVALUATION
I have changed the logical order for pipeline.conf made it as a master file
which INCLUDES different configuration e.g common-worm-ports. I will
provide you the file when I reach office tomorrow.
On Thu, Nov 12, 2015 at 6:56 PM, Angela Horneman <ahorneman at cert.org> wrote:
> Hi Asad,
>
> Can you send the non-working configuration file as an attachment? I don't
> see anything obvious below.
>
> Angela
>
> -----Original Message-----
> From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org [mailto:
> netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org] On Behalf Of asad
> Sent: Thursday, November 12, 2015 2:02 AM
> To: Timur D. Snoke <tdsnoke at cert.org>
> Cc: netsa-tools-discuss at cert.org
> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with
> EVAL block
>
> Also, another important thing that it worked only when I removed second
> EVAL block, I don't know when adding two together it stopped working all
> ALERTING stops.
>
> FILTER outgoing-flows
> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> END FILTER
>
>
> FILTER non-local-to-remote
> TYPENAME == int2int
> SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> DPORT IN_LIST [137,6]
> END FILTER
>
> EVALUATION Systems_using_many_different_protocols
> FILTER outgoing-flows
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DPORT > 25
> TIME_WINDOW 3600 SECONDS
> END CHECK
> SEVERITY 7
> ALERT ALWAYS
> CLEAR NEVER
> END EVALUATION
>
> EVALUATION Common-worm-ports
> FILTER non-local-to-remote
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DIP > 5
> TIME_WINDOW 360 SECONDS
> END CHECK
> SEVERITY 7
> ALERT ALWAYS
> ALERT EVERYTHING
> CLEAR ALWAYS
> END EVALUATION
>
> What wrong in order?
>
> On 11/12/15, asad <a.alii85 at gmail.com> wrote:
> > Good news to all, its working i tested Angela i saw STATISTICS block
> > working, compared it with EVAL block I see that I was looking for
> > events in 5 minutes window not 6 as in STATISTICS i corrected that and
> > its working now:) Thanks Timur and Angela you people are good.
> >
> > On 11/12/15, asad <a.alii85 at gmail.com> wrote:
> >> Timur,
> >>
> >> This is so strange, consider this
> >>
> >> farhan at netflow:~/silkydata$ rwfilter --sensor=S0 --type=int2int
> >> --saddress=10.10.81.74 --start-date=2015/11/06T09:15
> >> --end-date=2015/11/06T09:20 --dport=137 --pass=stdout | rwuniq
> >> --field=sip --dip-distinct
> >> rwfilter: Warning: start-date precision greater than hours ignored
> >> rwfilter: Warning: end-date precision greater than hours ignored
> >> sIP|dIP-Distin|
> >> 10.10.81.74| 2928|
> >>
> >>
> >> I'm getting 2928 unique destination ip matches and analysis pipeline
> >> won't even detect 5?
> >> Why?
> >>
> >> On 11/11/15, Timur D. Snoke <tdsnoke at cert.org> wrote:
> >>> Asad, over what interval are those connections?
> >>>
> >>> You can try this:
> >>>
> >>> rwfilter \
> >>> --sensor=S0 \
> >>> --type=int2int \
> >>> --saddress=10.10.81.74 \
> >>> --start-date=2015/11/6 \
> >>> --dport=137
> >>> --pass=stdout \
> >>> | rwcount \
> >>> --bin-size=300 \
> >>> --skip-zeroes
> >>>
> >>>
> >>> Your evaluation is that an inside host talks to more than 5 outside
> >>> hosts during a five minute window. The previous rwfilter query will
> >>> help us find the most active hour and then determine you can try the
> >>> following commands to determine if the conditions are met. Assuming
> >>> the busiest time slice is 12:30-12:40.
> >>>
> >>> rwfilter \
> >>> --sensor=S0 \
> >>> --type=int2int \
> >>> --saddress=10.10.81.74 \
> >>> --start-date=2015/11/6T12:30 \
> >>> --end-date=2015/11/16T12:40 \
> >>> --dport=137
> >>> --pass=stdout \
> >>> | rwuniq --field=sip —dip-distinct
> >>>
> >>> And to see what the flows look like for that time period ordered by
> >>> time try
> >>> this:
> >>>
> >>> rwfilter \
> >>> --sensor=S0 \
> >>> --type=int2int \
> >>> --saddress=10.10.81.74 \
> >>> --start-date=2015/11/6T12:30 \
> >>> --end-date=2015/11/16T12:40 \
> >>> --dport=137
> >>> --pass=stdout \
> >>> | rwsort --field=stime \
> >>> | rwcut
> >>>
> >>> I think you are seeing the traffic but not the 5 dips in 5 minutes.
> >>> If you get results that are different please include them in your
> >>> response.
> >>>
> >>>
> >>>
> >>> I hope this helps,
> >>>
> >>> --
> >>> Timur Snoke
> >>> Network Defense Analyst
> >>> CERT/CC - Network Situational Awareness Software Engineering
> >>> Institute (SEI)
> >>> O: (412) 268-7806
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 11/11/15, 5:34 AM, "asad" <a.alii85 at gmail.com> wrote:
> >>>
> >>>>Here is the result of the filter
> >>>>
> >>>>farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
> >>>>--saddress=10.10.81.74 --start-date=2015/11/6 --dport=137
> >>>>--pass=stdout | rwstats --fields=sip,dip --values=records --top
> >>>>--count=5
> >>>>INPUT: 142555 Records for 2944 Bins and 142555 Total Records
> >>>>OUTPUT: Top 5 Bins by Records
> >>>> sIP| dIP| Records| %Records| cumul_%|
> >>>> 10.10.81.74| 192.168.33.74| 96| 0.067342| 0.067342|
> >>>> 10.10.81.74| 192.168.172.1| 96| 0.067342| 0.134685|
> >>>> 10.10.81.74| 192.168.181.45| 96| 0.067342| 0.202027|
> >>>> 10.10.81.74| 192.168.172.17| 96| 0.067342| 0.269370|
> >>>> 10.10.81.74| 10.10.232.21| 96| 0.067342| 0.336712|
> >>>>
> >>>>
> >>>>Now, I'm not sure why the ALERT will not still be seen.
> >>>>
> >>>>The auxLog.log shows
> >>>>
> >>>>"
> >>>>2015-11-10
> >>>>
> 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"
> >>>>
> >>>>
> >>>>My update pipeline.conf says
> >>>>
> >>>>FILTER non-local-to-remote
> >>>>TYPENAME IN_LIST [int2int]
> >>>>SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
> >>>>SIP NOT_IN_LIST "/root/silkydata/voip.set"
> >>>>SIP NOT_IN_LIST "/root/silkydata/rns.set"
> >>>>DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997,
> >>>>998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134,
> >>>>3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$
> >>>>END FILTER
> >>>>
> >>>>
> >>>>EVALUATION Systems_using_many_different_protocols
> >>>>FILTER outgoing-flows
> >>>>FOREACH SIP
> >>>>CHECK THRESHOLD
> >>>>DISTINCT DPORT > 25
> >>>>TIME_WINDOW 3600 SECONDS
> >>>>END CHECK
> >>>>SEVERITY 7
> >>>>ALERT JUST_NEW_THIS_TIME
> >>>>ALERT ALWAYS
> >>>>CLEAR NEVER
> >>>>END EVALUATION
> >>>>
> >>>>
> >>>>EVALUATION Common-worm-ports
> >>>>FILTER non-local-to-remote
> >>>>FOREACH SIP
> >>>>CHECK THRESHOLD
> >>>>DISTINCT DIP > 5
> >>>>TIME_WINDOW 300 SECONDS
> >>>>END CHECK
> >>>>SEVERITY 7
> >>>>ALERT ALWAYS
> >>>>CLEAR NEVER
> >>>>END EVALUATION
> >>>>
> >>>>On 11/10/15, Timur D. Snoke <tdsnoke at cert.org> wrote:
> >>>>> Asad,
> >>>>>
> >>>>> The more information you provide the better our ability to help
> >>>>> you work through what your configuration will need to be.
> >>>>>
> >>>>> It is good to be using TYPENAME as a limiting factor at the start
> >>>>> of your FILTER, often we see that at least half of the traffic by
> >>>>> volume is web traffic so excluding that from your EVALUATION will
> >>>>> provide a performance improvement.
> >>>>>
> >>>>> The INT2INT traffic usually reflects an incomplete site
> >>>>> definition, it would be good to fix that because you might find
> >>>>> that you have to make special accommodations in your FILTER
> >>>>> composition.
> >>>>>
> >>>>> I hope this helps,
> >>>>>
> >>>>> --
> >>>>> Timur Snoke
> >>>>> Network Defense Analyst
> >>>>> CERT/CC - Network Situational Awareness Software Engineering
> >>>>> Institute (SEI)
> >>>>> O: (412) 268-7806
> >>>>>
> >>>>> From: asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
> >>>>> Date: Tuesday, November 10, 2015 at 9:32 AM
> >>>>> To: timur snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
> >>>>> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting
> >>>>> issues with EVAL block
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
> >>>>> <tdsnoke at cert.org<mailto:tdsnoke at cert.org>> wrote:
> >>>>> Hello Asad,
> >>>>>
> >>>>> This is an interesting question but I am not sure I understand
> >>>>> from your description what you are trying to capture.
> >>>>>
> >>>>> Thanks Timur,
> >>>>>
> >>>>> Let me re-explain it in a clear way.
> >>>>>
> >>>>>
> >>>>>
> >>>>> You are using type and defining SIP in your filters but do not
> >>>>> really explain it in your use case.
> >>>>>
> >>>>> In my case I want the source IP which is involved in communicating
> >>>>> with common worm ports to at least x5 different destinations IP.
> >>>>> Further I want this to match as much as 5 times. I think I need
> >>>>> RECORD COUNT >5?
> >>>>>
> >>>>>
> >>>>> Are you looking for outside hosts that are trying to scan these
> >>>>> ports on multiple hosts inside your network?
> >>>>> If this is the case you should just use IN for your TYPENAME,
> >>>>> there are no web ports or icmp traffic that you are concerned with
> >>>>> in your port list.
> >>>>> If
> >>>>> the initiating host is outside then you wouldn’t want INT2INT, OUT
> >>>>> or OUTWEB. This change will potentially limit the total number of
> >>>>> flows being evaluated.
> >>>>>
> >>>>> In my current system which is SIEM the rules are firing for
> >>>>> traffic direction which is int2int.
> >>>>>
> >>>>>
> >>>>> Can you show example flows that should match but doesn’t?
> >>>>>
> >>>>> I will prepare an rwfilter results for you and get back to you. I
> >>>>> have evidence of its using traffic logs from cisco asa I can show
> >>>>> that If you want.
> >>>>>
> >>>>> --
> >>>>> Timur Snoke
> >>>>> Network Defense Analyst
> >>>>> CERT/CC - Network Situational Awareness Software Engineering
> >>>>> Institute (SEI)
> >>>>> O: (412) 268-7806
> >>>>>
> >>>>> From:
> >>>>> <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:nets
> >>>>> a-tools-discuss-bounces+tdsnoke=cert.org at cert.org>>
> >>>>> on behalf of asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
> >>>>> Date: Tuesday, November 10, 2015 at 8:30 AM
> >>>>> To:
> >>>>> "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>"
> >>>>> <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
> >>>>> >
> >>>>> Subject: [netsa-tools-discuss] analysis pipeline alerting issues
> >>>>> with EVAL block
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> I have a very simple alerting requirement
> >>>>>
> >>>>> "
> >>>>> when a destination ports matches ports which are considered as
> >>>>> 'worm ports'
> >>>>> traffic is send to 5 different unique ips in 5 minutes time.
> >>>>>
> >>>>> "
> >>>>>
> >>>>> I know on traffic level i'm getting required data since using
> >>>>> traffic logs from the cisco asa (same device is also sending
> >>>>> netflows) and it works as expected. I'm suppose to see an ip
> >>>>> address but on alert.log I see nothing.
> >>>>> Below is the logic.
> >>>>>
> >>>>> Any help?
> >>>>>
> >>>>>
> >>>>>
> >>>>> FILTER outgoing-flows
> >>>>> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> >>>>> "/root/silkydata/DMZ.set"
> >>>>> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> >>>>> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> >>>>> END FILTER
> >>>>>
> >>>>> FILTER non-local-to-remote
> >>>>> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> >>>>> "/root/silkydata/DMZ.set"
> >>>>> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> >>>>> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> >>>>> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997,
> >>>>> 998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134,
> >>>>> 3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147,
> >>>>> 31$ END FILTER
> >>>>>
> >>>>>
> >>>>> EVALUATION Systems_using_many_different_protocols
> >>>>> FILTER non-local-to-remote
> >>>>> FOREACH SIP
> >>>>> CHECK THRESHOLD
> >>>>> DISTINCT DIP > 25
> >>>>> TIME_WINDOW 3600 SECONDS
> >>>>> END CHECK
> >>>>> SEVERITY 7
> >>>>> ALERT JUST_NEW_THIS_TIME
> >>>>> ALERT ALWAYS
> >>>>> CLEAR NEVER
> >>>>> END EVALUATION
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list