[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

asad a.alii85 at gmail.com
Tue Nov 17 10:28:07 EST 2015


Hi Angela,

Thanks for your ongoing support.

I tried the config you shared with me, but it was not firing ,I noticed
that with SIP DIP combo doesn't constitute record count greater then >2.

In current state, I'm looking for config to identify a situation where the
same source IP has matched top filter condition at least *N *times. For e.g
same source IP seen communicating with common worms ports provided in the
filter block 5 times over.

The rest of logic is as it ... "across more than 50 destination IP
within 5 minutes."

I want to know using the earlier config you share would just removing DIP
from 'SIP DIP' do the trick

As for complete example,

A -> X1 : 135
A -> X2.: 136

To show here for source IP (A) if it communicates with 135,136 (common
worms ports) at least x5 times over in 5 minutes with at least x2 distinct
IP addresses (x1,x2) fire an alert.

Hope this help. Please let me know I can provide better example.

On Tue, Nov 17, 2015 at 2:15 AM, Angela Horneman <ahorneman at cert.org> wrote:

> Statistics do not let you create output lists or do thresholding, so you
> do need the first EVALUATION block. I'm not sure I understand the second
> part of your question. Can you provide an example and what you want to see
> in the alert?
>
> -----Original Message-----
> From: asad [mailto:a.alii85 at gmail.com]
> Sent: Friday, November 13, 2015 7:07 AM
> To: Angela Horneman <ahorneman at cert.org>
> Cc: Timur D. Snoke <tdsnoke at cert.org>; netsa-tools-discuss at cert.org
> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with
> EVAL block
>
> Angela,
>
> Here is the update, after like 6 hours of SILK i realized the reason this
> won't work is because on checking the rwfilter results the records count
> are never greater then 5. See this,
>
> rwfilter   --sensor=S0   --type=int2int   --saddress=10.10.81.74
> --daddress=1.1.1.1  --start-date=2015/11/13T12:15   --dport=137
> --pass=stdout  | rwuniq --field=sip --values=dip-distinct,records
> rwfilter: Warning: start-date precision greater than hours ignored
>             sIP|dIP-Distin|   Records|
>     10.10.81.74|         1|         2|
>
> So, I checked the original rule defined in Qradar this time for your
> understanding I'm coping it as it is Below
>
> "
> and when any of these BB:PortDefinition: Common Worm Ports with the same
> source IP more than 5 times, across more than 50 destination IP within 5
> minutes.
>
> "
> What I want is a same source IP communicating using the common worm ports
> at least x5 times. Its reputation. So , I changed the config as follows
>
> FILTER non-local-to-remote
> TYPENAME IN_LIST [int2int,outnull,innull] SIP NOT_IN_LIST
> "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> DPORT IN_LIST
> [135,136,137,138,139,444,445,995,996,997,998,999,8998,3127,3128,3129,3130,3131,3132,3133,3134,3135,3136,3137,3138,3139,3140,3141,3142,3143,3144,3145,3146,3147,3148,3149,3389,2,20,25,21,81,99]
> END FILTER
>
> EVALUATION ourNetworkSIPDIPsMultiFlows
> FILTER non-local-to-remote
> FOREACH SIP
> CHECK THRESHOLD
> RECORD COUNT > 5
> TIME WINDOW 360 SECONDS
> END CHECK
> CLEAR ALWAYS
> OUTPUT LIST SIP CommonwormPORTSMultiflow_list OUTPUT_TIMEOUT 360 SECONDS
> DO NOT ALERT END EVALUATION
>
> FILTER CommonwormportsMultiflow_list
> SIP IN_LIST CommonwormPORTSMultiflow_list END FILTER
>
> EVALUATION Common-worm-ports
> FILTER non-local-to-remote
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DIP > 50
> TIME WINDOW 360 SECONDS
> END CHECK
> CLEAR ALWAYS
> OUTPUT_TIMEOUT 360 SECONDS
> ALERT ALWAYS
> ALERT EACH ONLY ONCE
> END EVALUATION
>
> It is working now, do you think a STATISTICS block can be used instead of
> first EVAL block.? Also, when the condition matches with unique destination
> ip greater then 25 then in backlog memory pipelines outputs value of dip
> that was present at time of match. What about I want to see all the list of
> distinct IPs I need ALERT EVERYTHING.
>
>
> On 11/12/15, Angela Horneman <ahorneman at cert.org> wrote:
> > It sounds like what you want to do is track SIPs with 250 different
> > DIPs and SIP DIP pairs with 10 flows and alert when a SIP matches
> > both. If so, try something like (explanation below the blocks):
> >
> > FILTER ourNetwork
> >   SIP == 193.0.0.0/8
> > END FILTER
> >
> > EVALUATION ourNetworkSIPDIPsMultiFlows
> >   FILTER ourNetwork
> >   FOREACH SIP DIP
> >   CHECK THRESHOLD
> >     RECORD COUNT > 10
> >     TIME WINDOW 1 HOUR
> >   END CHECK
> >   CLEAR ALWAYS
> >   OUTPUT LIST SIP DIP ourNetworkSIPDIPsMultiFlows_list  OUTPUT TIMEOUT
> > 1 HOUR
> >   DO NOT ALERT
> > END EVALUATION
> >
> > FILTER ourNetworkSipDipManyFlows
> >   SIP DIP IN_LIST ourNetworkSIPDIPsMultiFlows_list END FILTER
> >
> > EVALUATION ourNetworkSipsToLotsOfDipsWithManyFlows
> >   FILTER sipDipManyFlows
> >   FOREACH SIP
> >   CHECK THRESHOLD
> >     DISTINCT DIP > 250
> >     TIME WINDOW 1 HOUR
> >   END CHECK
> >   CLEAR ALWAYS
> >  OUTPUT TIMEOUT 1 HOUR
> >   ALERT ALWAYS
> >   ALERT EACH ONLY ONCE
> > END EVALUATION
> >
> > This will take all flows where the source is “ourNetwork” and track
> > how many flows are between each SIP DIP pair. As soon as a pair has at
> > least 10 flows, they get added to a list. Then we filter new flows to
> > get only flows that are between SIP DIP pairs have 10+ flows between
> > them. Of all those flows, we track the count of the unique DIPs each SIP
> has had 10+ flow with.
> >  Once a SIP has at least 250, it becomes a success and alerts.
> >
> > Angela
> >
> >
> > From: asad [mailto:a.alii85 at gmail.com]
> > Sent: Thursday, November 12, 2015 8:56 AM
> > To: Angela Horneman <ahorneman at cert.org>
> > Cc: Timur D. Snoke <tdsnoke at cert.org>; netsa-tools-discuss at cert.org
> > Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues
> > with EVAL block
> >
> > Hello Angela,
> >
> > Yes, your last statement is true, i thought RECORD COUNT would be used
> > for that but I'm failing to understand how I can ADD the two
> > operations I cannot used RECORD COUNT in single CHECK statement.
> >
> > On Thu, Nov 12, 2015 at 6:51 PM, Angela Horneman
> > <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
> > Good morning Asad,
> >
> > The way it is currently set up, it will become a success as soon as a
> > SIP has at least 10 flows and contacts 250 or more unique DIPs in an
> hour.
> >
> > -          The TIME WINDOW of each CHECK block is independent.
> >
> > -          The threshold values for each CHECK block must be met for each
> > key (in your case SIP, since you are using FOREACH SIP), before the
> > key becomes a success (i.e. available for alerting or inclusion in
> lists).
> > In this case, the RECORD COUNT > 10 is unnecessary as by definition,
> > if there are at least 250 unique DIPs there are at least that number of
> flows.
> >
> > Are you trying to detect where a SIP contacts at least 250 different
> > DIPs and where it has a flow to each DIP at least 10 times?
> >
> > Angela
> >
> > From: asad [mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>]
> > Sent: Thursday, November 12, 2015 8:40 AM
> >
> > To: Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>>
> > Cc: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>;
> > netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
> > Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues
> > with EVAL block
> >
> > Angela, I need your advice
> >
> > FILTER ourNetwork
> >     SIP == 193.0.0.0/8<http://193.0.0.0/8> END FILTER
> >
> >
> > FILTER ourNetwork
> >     FOREACH SIP
> >     CHECK THRESHOLD
> >         DISTINCT DIP > 250
> >         TIME_WINDOW 1 HOUR
> >     END CHECK
> >     CHECK THRESHOLD
> >         RECORD COUNT > 10
> >         TIME_WINDOW 1 HOUR
> >     END CHECK
> > END EVALUATION
> >
> > I'm looking for match which happens more then 10 times  in second CHECK
> > block do I need to specify TIME WINDOW separate or will it AND the
> results
> > for e.g more then 250 IPs in 1 hour and PLUS generates / matches at
> least 10
> > times. Thanks.
> >
> >
> > On Wed, Nov 11, 2015 at 9:12 PM, Angela Horneman
> > <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
> > For statistics, the values are directly related to the time window, not
> > state based on successes. This means that the alert amount (e.g. ALERT
> SINCE
> > LAST TIME) doesn’t apply. The UPDATE statement is all that is needed to
> tell
> > Analysis Pipeline how often to send the alerts.
> >
> > From: asad [mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>]
> > Sent: Wednesday, November 11, 2015 9:34 AM
> > To: Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>>
> > Cc: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>;
> > netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
> >
> > Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with
> > EVAL block
> >
> > Hi Angela,
> > Thanks for fast response.
> > Here I want to add earlier you told me to try this construct
> > STATISTIC checkCountsDistinctDIPs
> >   FILTER non-local-to-remote
> >   FOREACH SIP
> >   DISTINCT DIP
> >   TIME WINDOW 6 MINUTES
> >   SEVERITY 7
> >   UPDATE 5 MINUTES
> > END STATISTIC
> >
> > I somehow missed it from equation today, I m wanting to try this since
> going
> > by the logic in it I should know if in 6 minutes what be count of the
> > distinct destinations IP.
> > Trying this , I want to know do I need any ALERT clauses as well, unlike
> > EVAL block I don't see any alerting conditions will it default to ALERT
> > SINCE LAST TIME.
> > Thanks for support.
> >
> > On Wed, Nov 11, 2015 at 6:55 PM, Angela Horneman
> > <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
> > Hi Asad,
> >
> > Did you make sure that the address you are checking with the rwfilter is
> not
> > in either the DMZ.set, voip.set, or rns.set file?
> >
> > There is not really a way with SiLK to mimic the TIME WINDOW of Analysis
> > Pipeline. Even if you recursively run rwfilter over short intervals, that
> > does not match the Analysis Pipeline's sliding time window. For testing,
> you
> > might want to use TIME WINDOW 1 DAY if you are comparing with an
> rwfilter.
> >
> > Try this:
> >
> > FILTER toWormPorts
> >   TYPENAME == int2int
> >   DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> 999,
> > 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137,
> > 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147]
> > END FILTER
> >
> > EVALUATION sipsToManyDipsOnWormPorts
> >   FILTER toWormPorts
> >   FOREACH SIP
> >   CHECK THRESHOLD
> >     DISTINCT DIP >= 5
> >     TIME WINDOW 1 DAY
> >   END CHECK
> >   ALERT ALWAYS
> >   ALERT EVERYTHING
> >   CLEAR ALWAYS
> > END EVALUATION
> >
> > If it works, you can try decreasing the time window in the check
> threshold
> > part of the evaluation, then adding in the "whitelist" set files to the
> > filter.
> >
> > Angela
> >
> > -----Original Message-----
> > From:
> > netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org<mailto:
> cert.org at cert.org>
> > [mailto:netsa-tools-discuss-bounces+ahorneman<mailto:
> netsa-tools-discuss-bounces%2Bahorneman>=cert.org at cert.org<mailto:
> cert.org at cert.org>]
> > On Behalf Of asad
> > Sent: Wednesday, November 11, 2015 5:34 AM
> > To: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
> > Cc: netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
> > Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with
> > EVAL block
> >
> > Here is the result of the filter
> >
> > farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
> > --saddress=10.10.81.74 --start-date=2015/11/6  --dport=137
> --pass=stdout  |
> > rwstats --fields=sip,dip --values=records --top
> > --count=5
> > INPUT: 142555 Records for 2944 Bins and 142555 Total Records
> > OUTPUT: Top 5 Bins by Records
> >             sIP|            dIP|   Records|  %Records|   cumul_%|
> >     10.10.81.74|  192.168.33.74|        96|  0.067342|  0.067342|
> >     10.10.81.74|  192.168.172.1|        96|  0.067342|  0.134685|
> >     10.10.81.74| 192.168.181.45|        96|  0.067342|  0.202027|
> >     10.10.81.74| 192.168.172.17|        96|  0.067342|  0.269370|
> >     10.10.81.74|   10.10.232.21|        96|  0.067342|  0.336712|
> >
> >
> > Now, I'm not sure why the ALERT will not still be seen.
> >
> > The auxLog.log shows
> >
> > "
> > 2015-11-10
> >
> 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"
> >
> >
> > My update pipeline.conf says
> >
> > FILTER non-local-to-remote
> > TYPENAME IN_LIST [int2int]
> > SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
> > SIP NOT_IN_LIST "/root/silkydata/voip.set"
> > SIP NOT_IN_LIST "/root/silkydata/rns.set"
> > DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> 999,
> > 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137,
> > 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
> >
> >
> > EVALUATION  Systems_using_many_different_protocols
> > FILTER outgoing-flows
> > FOREACH SIP
> > CHECK THRESHOLD
> > DISTINCT DPORT > 25
> > TIME_WINDOW 3600 SECONDS
> > END CHECK
> > SEVERITY 7
> > ALERT JUST_NEW_THIS_TIME
> > ALERT ALWAYS
> > CLEAR NEVER
> > END EVALUATION
> >
> >
> > EVALUATION  Common-worm-ports
> > FILTER non-local-to-remote
> > FOREACH SIP
> > CHECK THRESHOLD
> > DISTINCT DIP > 5
> > TIME_WINDOW 300 SECONDS
> > END CHECK
> > SEVERITY 7
> > ALERT ALWAYS
> > CLEAR NEVER
> > END EVALUATION
> >
> > On 11/10/15, Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
> > wrote:
> >> Asad,
> >>
> >> The more information you provide the better our ability to help you
> >> work through what your configuration will need to be.
> >>
> >> It is good to be using TYPENAME as a limiting factor at the start of
> >> your FILTER, often we see that at least half of the traffic by volume
> >> is web traffic so excluding that from your EVALUATION will provide a
> >> performance improvement.
> >>
> >> The INT2INT traffic usually reflects an incomplete site definition, it
> >> would be good to fix that because you might find that you have to make
> >> special accommodations in your FILTER composition.
> >>
> >> I  hope this helps,
> >>
> >> --
> >> Timur Snoke
> >> Network Defense Analyst
> >> CERT/CC - Network Situational Awareness Software Engineering Institute
> >> (SEI)
> >> O: (412) 268-7806
> >>
> >> From: asad
> >> <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com><mailto:
> a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>>
> >> Date: Tuesday, November 10, 2015 at 9:32 AM
> >> To: timur snoke
> >> <tdsnoke at cert.org<mailto:tdsnoke at cert.org><mailto:tdsnoke at cert.org
> <mailto:tdsnoke at cert.org>>>
> >> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues
> >> with EVAL block
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
> >> <tdsnoke at cert.org<mailto:tdsnoke at cert.org><mailto:tdsnoke at cert.org
> <mailto:tdsnoke at cert.org>>>
> >> wrote:
> >> Hello Asad,
> >>
> >> This is an interesting question but I am not sure I understand from
> >> your description what you are trying to capture.
> >>
> >> Thanks Timur,
> >>
> >> Let me re-explain it in a clear way.
> >>
> >>
> >>
> >> You are using type and defining SIP in your filters but do not really
> >> explain it in your use case.
> >>
> >> In my case I want the source IP which is involved in communicating
> >> with common worm ports to at least x5 different destinations IP.
> >> Further I want this to match as much as 5 times. I think I need RECORD
> >> COUNT >5?
> >>
> >>
> >> Are you looking for outside hosts that are trying to scan these ports
> >> on multiple hosts inside your network?
> >> If this is the case you should just use IN for your TYPENAME, there
> >> are no web ports or icmp traffic that you are concerned with in your
> >> port list. If the initiating host is outside then you wouldn’t want
> >> INT2INT, OUT or OUTWEB. This change will potentially limit the total
> >> number of flows being evaluated.
> >>
> >> In my current system which is SIEM the rules are firing for traffic
> >> direction which is int2int.
> >>
> >>
> >> Can you show example flows that should match but doesn’t?
> >>
> >> I will prepare an rwfilter results for you and get back to you. I have
> >> evidence of its using traffic logs from cisco asa I can show that If
> >> you want.
> >>
> >> --
> >> Timur Snoke
> >> Network Defense Analyst
> >> CERT/CC - Network Situational Awareness Software Engineering Institute
> >> (SEI)
> >> O: (412) 268-7806
> >>
> >> From:
> >> <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:
> cert.org at cert.org><mailto:netsa-to<mailto:netsa-to>
> >> ols-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:cert.org at cert.org
> >>>
> >> on behalf of asad
> >> <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com><mailto:
> a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>>
> >> Date: Tuesday, November 10, 2015 at 8:30 AM
> >> To:
> >> "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org
> ><mailto:netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org
> >>"
> >> <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org
> ><mailto:netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org
> >>>
> >> Subject: [netsa-tools-discuss] analysis pipeline alerting issues with
> >> EVAL block
> >>
> >> Hello,
> >>
> >> I have a very simple alerting requirement
> >>
> >> "
> >> when a destination ports matches ports which are considered as 'worm
> >> ports'
> >> traffic is send to 5 different unique ips in 5 minutes time.
> >>
> >> "
> >>
> >> I know on traffic level i'm getting required data since using traffic
> >> logs from the cisco asa (same device is also sending netflows) and it
> >> works as expected. I'm suppose to see an ip address but on alert.log I
> see
> >> nothing.
> >> Below is the logic.
> >>
> >> Any help?
> >>
> >>
> >>
> >> FILTER outgoing-flows
> >> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> >> "/root/silkydata/DMZ.set"
> >> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> >> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> >> END FILTER
> >>
> >> FILTER non-local-to-remote
> >> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> >> "/root/silkydata/DMZ.set"
> >> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> >> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> >> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> >> 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
> >> 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
> >>
> >>
> >> EVALUATION  Systems_using_many_different_protocols
> >> FILTER  non-local-to-remote
> >> FOREACH SIP
> >> CHECK THRESHOLD
> >> DISTINCT DIP > 25
> >> TIME_WINDOW 3600 SECONDS
> >> END CHECK
> >> SEVERITY 7
> >> ALERT JUST_NEW_THIS_TIME
> >> ALERT ALWAYS
> >> CLEAR NEVER
> >> END EVALUATION
> >>
> >>
> >
> >
> >
> >
>
>
-------------- next part --------------
HTML attachment scrubbed and removed


More information about the netsa-tools-discuss mailing list