[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

Wed Nov 11 06:06:06 EST 2015

I have also verified through second filter results that there are at
least 2944 unique destinations communication since 6th of this month.

I still don't know what the more clever of telling the time-windows
for example 300 seconds, I try doing it with rwcount --bin-size but it
won't work in combination with rwfilter.

farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
--saddress=10.10.81.74 --start-date=2015/11/6 --dport=137
--pass=stdout  | rwuniq --fields=dport --dip-distinct=6
dPort|dIP-Distin|
  137|      2944|

On 11/11/15, asad <a.alii85 at gmail.com> wrote:
> Here is the result of the filter
>
> farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
> --saddress=10.10.81.74 --start-date=2015/11/6  --dport=137
> --pass=stdout  | rwstats --fields=sip,dip --values=records --top
> --count=5
> INPUT: 142555 Records for 2944 Bins and 142555 Total Records
> OUTPUT: Top 5 Bins by Records
>             sIP|            dIP|   Records|  %Records|   cumul_%|
>     10.10.81.74|  192.168.33.74|        96|  0.067342|  0.067342|
>     10.10.81.74|  192.168.172.1|        96|  0.067342|  0.134685|
>     10.10.81.74| 192.168.181.45|        96|  0.067342|  0.202027|
>     10.10.81.74| 192.168.172.17|        96|  0.067342|  0.269370|
>     10.10.81.74|   10.10.232.21|        96|  0.067342|  0.336712|
>
>
> Now, I'm not sure why the ALERT will not still be seen.
>
> The auxLog.log shows
>
> "
> 2015-11-10
> 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"
>
>
> My update pipeline.conf says
>
> FILTER non-local-to-remote
> TYPENAME IN_LIST [int2int]
> SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
> 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$
> END FILTER
>
>
> EVALUATION  Systems_using_many_different_protocols
> FILTER outgoing-flows
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DPORT > 25
> TIME_WINDOW 3600 SECONDS
> END CHECK
> SEVERITY 7
> ALERT JUST_NEW_THIS_TIME
> ALERT ALWAYS
> CLEAR NEVER
> END EVALUATION
>
>
> EVALUATION  Common-worm-ports
> FILTER non-local-to-remote
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DIP > 5
> TIME_WINDOW 300 SECONDS
> END CHECK
> SEVERITY 7
> ALERT ALWAYS
> CLEAR NEVER
> END EVALUATION
>
> On 11/10/15, Timur D. Snoke <tdsnoke at cert.org> wrote:
>> Asad,
>>
>> The more information you provide the better our ability to help you work
>> through what your configuration will need to be.
>>
>> It is good to be using TYPENAME as a limiting factor at the start of your
>> FILTER, often we see that at least half of the traffic by volume is web
>> traffic so excluding that from your EVALUATION will provide a performance
>> improvement.
>>
>> The INT2INT traffic usually reflects an incomplete site definition, it
>> would
>> be good to fix that because you might find that you have to make special
>> accommodations in your FILTER composition.
>>
>> I  hope this helps,
>>
>> --
>> Timur Snoke
>> Network Defense Analyst
>> CERT/CC - Network Situational Awareness
>> Software Engineering Institute (SEI)
>> O: (412) 268-7806
>>
>> From: asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
>> Date: Tuesday, November 10, 2015 at 9:32 AM
>> To: timur snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
>> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with
>> EVAL block
>>
>>
>>
>>
>>
>> On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
>> <tdsnoke at cert.org<mailto:tdsnoke at cert.org>> wrote:
>> Hello Asad,
>>
>> This is an interesting question but I am not sure I understand from your
>> description what you are trying to capture.
>>
>> Thanks Timur,
>>
>> Let me re-explain it in a clear way.
>>
>>
>>
>> You are using type and defining SIP in your filters but do not really
>> explain it in your use case.
>>
>> In my case I want the source IP which is involved in communicating with
>> common worm ports to at least x5 different destinations IP. Further I
>> want
>> this to match as much as 5 times. I think I need RECORD COUNT >5?
>>
>>
>> Are you looking for outside hosts that are trying to scan these ports on
>> multiple hosts inside your network?
>> If this is the case you should just use IN for your TYPENAME, there are
>> no
>> web ports or icmp traffic that you are concerned with in your port list.
>> If
>> the initiating host is outside then you wouldn’t want INT2INT, OUT or
>> OUTWEB. This change will potentially limit the total number of flows
>> being
>> evaluated.
>>
>> In my current system which is SIEM the rules are firing for traffic
>> direction which is int2int.
>>
>>
>> Can you show example flows that should match but doesn’t?
>>
>> I will prepare an rwfilter results for you and get back to you. I have
>> evidence of its using traffic logs from cisco asa I can show that If you
>> want.
>>
>> --
>> Timur Snoke
>> Network Defense Analyst
>> CERT/CC - Network Situational Awareness
>> Software Engineering Institute (SEI)
>> O: (412) 268-7806
>>
>> From:
>> <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org>>
>> on behalf of asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>
>> Date: Tuesday, November 10, 2015 at 8:30 AM
>> To: "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>"
>> <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>
>> Subject: [netsa-tools-discuss] analysis pipeline alerting issues with
>> EVAL
>> block
>>
>> Hello,
>>
>> I have a very simple alerting requirement
>>
>> "
>> when a destination ports matches ports which are considered as 'worm
>> ports'
>> traffic is send to 5 different unique ips in 5 minutes time.
>>
>> "
>>
>> I know on traffic level i'm getting required data since using traffic
>> logs
>> from the cisco asa (same device is also sending netflows) and it works as
>> expected. I'm suppose to see an ip address but on alert.log I see
>> nothing.
>> Below is the logic.
>>
>> Any help?
>>
>>
>>
>> FILTER outgoing-flows
>> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp]
>> SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
>> SIP NOT_IN_LIST "/root/silkydata/voip.set"
>> SIP NOT_IN_LIST "/root/silkydata/rns.set"
>> END FILTER
>>
>> FILTER non-local-to-remote
>> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp]
>> SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
>> SIP NOT_IN_LIST "/root/silkydata/voip.set"
>> SIP NOT_IN_LIST "/root/silkydata/rns.set"
>> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
>> 999,
>> 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137,
>> 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$
>> END FILTER
>>
>>
>> EVALUATION  Systems_using_many_different_protocols
>> FILTER  non-local-to-remote
>> FOREACH SIP
>> CHECK THRESHOLD
>> DISTINCT DIP > 25
>> TIME_WINDOW 3600 SECONDS
>> END CHECK
>> SEVERITY 7
>> ALERT JUST_NEW_THIS_TIME
>> ALERT ALWAYS
>> CLEAR NEVER
>> END EVALUATION
>>
>>
>