[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

Wed Nov 11 09:34:19 EST 2015

Hi Angela,
Thanks for fast response.

Here I want to add earlier you told me to try this construct

STATISTIC checkCountsDistinctDIPs

  FILTER non-local-to-remote

  FOREACH SIP

  DISTINCT DIP

  TIME WINDOW 6 MINUTES

  SEVERITY 7

  UPDATE 5 MINUTES
END STATISTIC

I somehow missed it from equation today, I m wanting to try this since
going by the logic in it I should know if in 6 minutes what be count of the
distinct destinations IP.

Trying this , I want to know do I need any ALERT clauses as well, unlike
EVAL block I don't see any alerting conditions will it default to ALERT
SINCE LAST TIME.

Thanks for support.

On Wed, Nov 11, 2015 at 6:55 PM, Angela Horneman <ahorneman at cert.org> wrote:

> Hi Asad,
>
> Did you make sure that the address you are checking with the rwfilter is
> not in either the DMZ.set, voip.set, or rns.set file?
>
> There is not really a way with SiLK to mimic the TIME WINDOW of Analysis
> Pipeline. Even if you recursively run rwfilter over short intervals, that
> does not match the Analysis Pipeline's sliding time window. For testing,
> you might want to use TIME WINDOW 1 DAY if you are comparing with an
> rwfilter.
>
> Try this:
>
> FILTER toWormPorts
>   TYPENAME == int2int
>   DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
> 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147]
> END FILTER
>
> EVALUATION sipsToManyDipsOnWormPorts
>   FILTER toWormPorts
>   FOREACH SIP
>   CHECK THRESHOLD
>     DISTINCT DIP >= 5
>     TIME WINDOW 1 DAY
>   END CHECK
>   ALERT ALWAYS
>   ALERT EVERYTHING
>   CLEAR ALWAYS
> END EVALUATION
>
> If it works, you can try decreasing the time window in the check threshold
> part of the evaluation, then adding in the "whitelist" set files to the
> filter.
>
> Angela
>
> -----Original Message-----
> From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org [mailto:
> netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org] On Behalf Of asad
> Sent: Wednesday, November 11, 2015 5:34 AM
> To: Timur D. Snoke <tdsnoke at cert.org>
> Cc: netsa-tools-discuss at cert.org
> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with
> EVAL block
>
> Here is the result of the filter
>
> farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
> --saddress=10.10.81.74 --start-date=2015/11/6  --dport=137 --pass=stdout
> | rwstats --fields=sip,dip --values=records --top
> --count=5
> INPUT: 142555 Records for 2944 Bins and 142555 Total Records
> OUTPUT: Top 5 Bins by Records
>             sIP|            dIP|   Records|  %Records|   cumul_%|
>     10.10.81.74|  192.168.33.74|        96|  0.067342|  0.067342|
>     10.10.81.74|  192.168.172.1|        96|  0.067342|  0.134685|
>     10.10.81.74| 192.168.181.45|        96|  0.067342|  0.202027|
>     10.10.81.74| 192.168.172.17|        96|  0.067342|  0.269370|
>     10.10.81.74|   10.10.232.21|        96|  0.067342|  0.336712|
>
>
> Now, I'm not sure why the ALERT will not still be seen.
>
> The auxLog.log shows
>
> "
> 2015-11-10
> 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"
>
>
> My update pipeline.conf says
>
> FILTER non-local-to-remote
> TYPENAME IN_LIST [int2int]
> SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998, 999,
> 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137,
> 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
>
>
> EVALUATION  Systems_using_many_different_protocols
> FILTER outgoing-flows
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DPORT > 25
> TIME_WINDOW 3600 SECONDS
> END CHECK
> SEVERITY 7
> ALERT JUST_NEW_THIS_TIME
> ALERT ALWAYS
> CLEAR NEVER
> END EVALUATION
>
>
> EVALUATION  Common-worm-ports
> FILTER non-local-to-remote
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DIP > 5
> TIME_WINDOW 300 SECONDS
> END CHECK
> SEVERITY 7
> ALERT ALWAYS
> CLEAR NEVER
> END EVALUATION
>
> On 11/10/15, Timur D. Snoke <tdsnoke at cert.org> wrote:
> > Asad,
> >
> > The more information you provide the better our ability to help you
> > work through what your configuration will need to be.
> >
> > It is good to be using TYPENAME as a limiting factor at the start of
> > your FILTER, often we see that at least half of the traffic by volume
> > is web traffic so excluding that from your EVALUATION will provide a
> > performance improvement.
> >
> > The INT2INT traffic usually reflects an incomplete site definition, it
> > would be good to fix that because you might find that you have to make
> > special accommodations in your FILTER composition.
> >
> > I  hope this helps,
> >
> > --
> > Timur Snoke
> > Network Defense Analyst
> > CERT/CC - Network Situational Awareness Software Engineering Institute
> > (SEI)
> > O: (412) 268-7806
> >
> > From: asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
> > Date: Tuesday, November 10, 2015 at 9:32 AM
> > To: timur snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
> > Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues
> > with EVAL block
> >
> >
> >
> >
> >
> > On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
> > <tdsnoke at cert.org<mailto:tdsnoke at cert.org>> wrote:
> > Hello Asad,
> >
> > This is an interesting question but I am not sure I understand from
> > your description what you are trying to capture.
> >
> > Thanks Timur,
> >
> > Let me re-explain it in a clear way.
> >
> >
> >
> > You are using type and defining SIP in your filters but do not really
> > explain it in your use case.
> >
> > In my case I want the source IP which is involved in communicating
> > with common worm ports to at least x5 different destinations IP.
> > Further I want this to match as much as 5 times. I think I need RECORD
> COUNT >5?
> >
> >
> > Are you looking for outside hosts that are trying to scan these ports
> > on multiple hosts inside your network?
> > If this is the case you should just use IN for your TYPENAME, there
> > are no web ports or icmp traffic that you are concerned with in your
> > port list. If the initiating host is outside then you wouldn’t want
> > INT2INT, OUT or OUTWEB. This change will potentially limit the total
> > number of flows being evaluated.
> >
> > In my current system which is SIEM the rules are firing for traffic
> > direction which is int2int.
> >
> >
> > Can you show example flows that should match but doesn’t?
> >
> > I will prepare an rwfilter results for you and get back to you. I have
> > evidence of its using traffic logs from cisco asa I can show that If
> > you want.
> >
> > --
> > Timur Snoke
> > Network Defense Analyst
> > CERT/CC - Network Situational Awareness Software Engineering Institute
> > (SEI)
> > O: (412) 268-7806
> >
> > From:
> > <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:netsa-to
> > ols-discuss-bounces+tdsnoke=cert.org at cert.org>>
> > on behalf of asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
> > Date: Tuesday, November 10, 2015 at 8:30 AM
> > To: "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>"
> > <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>
> > Subject: [netsa-tools-discuss] analysis pipeline alerting issues with
> > EVAL block
> >
> > Hello,
> >
> > I have a very simple alerting requirement
> >
> > "
> > when a destination ports matches ports which are considered as 'worm
> ports'
> > traffic is send to 5 different unique ips in 5 minutes time.
> >
> > "
> >
> > I know on traffic level i'm getting required data since using traffic
> > logs from the cisco asa (same device is also sending netflows) and it
> > works as expected. I'm suppose to see an ip address but on alert.log I
> see nothing.
> > Below is the logic.
> >
> > Any help?
> >
> >
> >
> > FILTER outgoing-flows
> > TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> > "/root/silkydata/DMZ.set"
> > SIP NOT_IN_LIST "/root/silkydata/voip.set"
> > SIP NOT_IN_LIST "/root/silkydata/rns.set"
> > END FILTER
> >
> > FILTER non-local-to-remote
> > TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> > "/root/silkydata/DMZ.set"
> > SIP NOT_IN_LIST "/root/silkydata/voip.set"
> > SIP NOT_IN_LIST "/root/silkydata/rns.set"
> > DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> > 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
> > 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
> >
> >
> > EVALUATION  Systems_using_many_different_protocols
> > FILTER  non-local-to-remote
> > FOREACH SIP
> > CHECK THRESHOLD
> > DISTINCT DIP > 25
> > TIME_WINDOW 3600 SECONDS
> > END CHECK
> > SEVERITY 7
> > ALERT JUST_NEW_THIS_TIME
> > ALERT ALWAYS
> > CLEAR NEVER
> > END EVALUATION
> >
> >
>
>
-------------- next part --------------
HTML attachment scrubbed and removed