[netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

Angela Horneman ahorneman at cert.org
Thu Nov 12 12:41:20 EST 2015


It sounds like what you want to do is track SIPs with 250 different DIPs and SIP DIP pairs with 10 flows and alert when a SIP matches both. If so, try something like (explanation below the blocks):

FILTER ourNetwork
  SIP == 193.0.0.0/8
END FILTER

EVALUATION ourNetworkSIPDIPsMultiFlows
  FILTER ourNetwork
  FOREACH SIP DIP
  CHECK THRESHOLD
    RECORD COUNT > 10
    TIME WINDOW 1 HOUR
  END CHECK
  CLEAR ALWAYS
  OUTPUT LIST SIP DIP ourNetworkSIPDIPsMultiFlows_list
 OUTPUT TIMEOUT 1 HOUR
  DO NOT ALERT
END EVALUATION

FILTER ourNetworkSipDipManyFlows
  SIP DIP IN_LIST ourNetworkSIPDIPsMultiFlows_list
END FILTER

EVALUATION ourNetworkSipsToLotsOfDipsWithManyFlows
  FILTER sipDipManyFlows
  FOREACH SIP
  CHECK THRESHOLD
    DISTINCT DIP > 250
    TIME WINDOW 1 HOUR
  END CHECK
  CLEAR ALWAYS
 OUTPUT TIMEOUT 1 HOUR
  ALERT ALWAYS
  ALERT EACH ONLY ONCE
END EVALUATION

This will take all flows where the source is “ourNetwork” and track how many flows are between each SIP DIP pair. As soon as a pair has at least 10 flows, they get added to a list. Then we filter new flows to get only flows that are between SIP DIP pairs have 10+ flows between them. Of all those flows, we track the count of the unique DIPs each SIP has had 10+ flow with.  Once a SIP has at least 250, it becomes a success and alerts.

Angela


From: asad [mailto:a.alii85 at gmail.com]
Sent: Thursday, November 12, 2015 8:56 AM
To: Angela Horneman <ahorneman at cert.org>
Cc: Timur D. Snoke <tdsnoke at cert.org>; netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

Hello Angela,

Yes, your last statement is true, i thought RECORD COUNT would be used for that but I'm failing to understand how I can ADD the two operations I cannot used RECORD COUNT in single CHECK statement.

On Thu, Nov 12, 2015 at 6:51 PM, Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
Good morning Asad,

The way it is currently set up, it will become a success as soon as a SIP has at least 10 flows and contacts 250 or more unique DIPs in an hour.

-          The TIME WINDOW of each CHECK block is independent.

-          The threshold values for each CHECK block must be met for each key (in your case SIP, since you are using FOREACH SIP), before the key becomes a success (i.e. available for alerting or inclusion in lists).
In this case, the RECORD COUNT > 10 is unnecessary as by definition, if there are at least 250 unique DIPs there are at least that number of flows.

Are you trying to detect where a SIP contacts at least 250 different DIPs and where it has a flow to each DIP at least 10 times?

Angela

From: asad [mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>]
Sent: Thursday, November 12, 2015 8:40 AM

To: Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>>
Cc: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>; netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

Angela, I need your advice

FILTER ourNetwork
    SIP == 193.0.0.0/8<http://193.0.0.0/8>
END FILTER


FILTER ourNetwork
    FOREACH SIP
    CHECK THRESHOLD
        DISTINCT DIP > 250
        TIME_WINDOW 1 HOUR
    END CHECK
    CHECK THRESHOLD
        RECORD COUNT > 10
        TIME_WINDOW 1 HOUR
    END CHECK
END EVALUATION

I'm looking for match which happens more then 10 times  in second CHECK block do I need to specify TIME WINDOW separate or will it AND the results for e.g more then 250 IPs in 1 hour and PLUS generates / matches at least 10 times. Thanks.


On Wed, Nov 11, 2015 at 9:12 PM, Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
For statistics, the values are directly related to the time window, not state based on successes. This means that the alert amount (e.g. ALERT SINCE LAST TIME) doesn’t apply. The UPDATE statement is all that is needed to tell Analysis Pipeline how often to send the alerts.

From: asad [mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>]
Sent: Wednesday, November 11, 2015 9:34 AM
To: Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>>
Cc: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>; netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>

Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

Hi Angela,
Thanks for fast response.
Here I want to add earlier you told me to try this construct
STATISTIC checkCountsDistinctDIPs
  FILTER non-local-to-remote
  FOREACH SIP
  DISTINCT DIP
  TIME WINDOW 6 MINUTES
  SEVERITY 7
  UPDATE 5 MINUTES
END STATISTIC

I somehow missed it from equation today, I m wanting to try this since going by the logic in it I should know if in 6 minutes what be count of the distinct destinations IP.
Trying this , I want to know do I need any ALERT clauses as well, unlike EVAL block I don't see any alerting conditions will it default to ALERT SINCE LAST TIME.
Thanks for support.

On Wed, Nov 11, 2015 at 6:55 PM, Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>> wrote:
Hi Asad,

Did you make sure that the address you are checking with the rwfilter is not in either the DMZ.set, voip.set, or rns.set file?

There is not really a way with SiLK to mimic the TIME WINDOW of Analysis Pipeline. Even if you recursively run rwfilter over short intervals, that does not match the Analysis Pipeline's sliding time window. For testing, you might want to use TIME WINDOW 1 DAY if you are comparing with an rwfilter.

Try this:

FILTER toWormPorts
  TYPENAME == int2int
  DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147]
END FILTER

EVALUATION sipsToManyDipsOnWormPorts
  FILTER toWormPorts
  FOREACH SIP
  CHECK THRESHOLD
    DISTINCT DIP >= 5
    TIME WINDOW 1 DAY
  END CHECK
  ALERT ALWAYS
  ALERT EVERYTHING
  CLEAR ALWAYS
END EVALUATION

If it works, you can try decreasing the time window in the check threshold part of the evaluation, then adding in the "whitelist" set files to the filter.

Angela

-----Original Message-----
From: netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org<mailto:cert.org at cert.org> [mailto:netsa-tools-discuss-bounces+ahorneman<mailto:netsa-tools-discuss-bounces%2Bahorneman>=cert.org at cert.org<mailto:cert.org at cert.org>] On Behalf Of asad
Sent: Wednesday, November 11, 2015 5:34 AM
To: Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>>
Cc: netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues with EVAL block

Here is the result of the filter

farhan at netflow:~$ rwfilter --sensor=S0 --type=int2int
--saddress=10.10.81.74 --start-date=2015/11/6  --dport=137 --pass=stdout  | rwstats --fields=sip,dip --values=records --top
--count=5
INPUT: 142555 Records for 2944 Bins and 142555 Total Records
OUTPUT: Top 5 Bins by Records
            sIP|            dIP|   Records|  %Records|   cumul_%|
    10.10.81.74|  192.168.33.74|        96|  0.067342|  0.067342|
    10.10.81.74|  192.168.172.1|        96|  0.067342|  0.134685|
    10.10.81.74| 192.168.181.45|        96|  0.067342|  0.202027|
    10.10.81.74| 192.168.172.17|        96|  0.067342|  0.269370|
    10.10.81.74|   10.10.232.21|        96|  0.067342|  0.336712|


Now, I'm not sure why the ALERT will not still be seen.

The auxLog.log shows

"
2015-11-10 05:22:08|Memory_Reset|5|Systems_using_many_different_protocols|130396|Common-worm-portS|0|Excessive-firwall-accepts-From-Multiple-Sources-to-a-Single-Destination|0|"


My update pipeline.conf says

FILTER non-local-to-remote
TYPENAME IN_LIST [int2int]
SIP NOT_IN_LIST "/root/silkydata/DMZ.set"
SIP NOT_IN_LIST "/root/silkydata/voip.set"
SIP NOT_IN_LIST "/root/silkydata/rns.set"
DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998, 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136, 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER


EVALUATION  Systems_using_many_different_protocols
FILTER outgoing-flows
FOREACH SIP
CHECK THRESHOLD
DISTINCT DPORT > 25
TIME_WINDOW 3600 SECONDS
END CHECK
SEVERITY 7
ALERT JUST_NEW_THIS_TIME
ALERT ALWAYS
CLEAR NEVER
END EVALUATION


EVALUATION  Common-worm-ports
FILTER non-local-to-remote
FOREACH SIP
CHECK THRESHOLD
DISTINCT DIP > 5
TIME_WINDOW 300 SECONDS
END CHECK
SEVERITY 7
ALERT ALWAYS
CLEAR NEVER
END EVALUATION

On 11/10/15, Timur D. Snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org>> wrote:
> Asad,
>
> The more information you provide the better our ability to help you
> work through what your configuration will need to be.
>
> It is good to be using TYPENAME as a limiting factor at the start of
> your FILTER, often we see that at least half of the traffic by volume
> is web traffic so excluding that from your EVALUATION will provide a
> performance improvement.
>
> The INT2INT traffic usually reflects an incomplete site definition, it
> would be good to fix that because you might find that you have to make
> special accommodations in your FILTER composition.
>
> I  hope this helps,
>
> --
> Timur Snoke
> Network Defense Analyst
> CERT/CC - Network Situational Awareness Software Engineering Institute
> (SEI)
> O: (412) 268-7806
>
> From: asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com><mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>>
> Date: Tuesday, November 10, 2015 at 9:32 AM
> To: timur snoke <tdsnoke at cert.org<mailto:tdsnoke at cert.org><mailto:tdsnoke at cert.org<mailto:tdsnoke at cert.org>>>
> Subject: Re: [netsa-tools-discuss] analysis pipeline alerting issues
> with EVAL block
>
>
>
>
>
> On Tue, Nov 10, 2015 at 6:56 PM, Timur D. Snoke
> <tdsnoke at cert.org<mailto:tdsnoke at cert.org><mailto:tdsnoke at cert.org<mailto:tdsnoke at cert.org>>> wrote:
> Hello Asad,
>
> This is an interesting question but I am not sure I understand from
> your description what you are trying to capture.
>
> Thanks Timur,
>
> Let me re-explain it in a clear way.
>
>
>
> You are using type and defining SIP in your filters but do not really
> explain it in your use case.
>
> In my case I want the source IP which is involved in communicating
> with common worm ports to at least x5 different destinations IP.
> Further I want this to match as much as 5 times. I think I need RECORD COUNT >5?
>
>
> Are you looking for outside hosts that are trying to scan these ports
> on multiple hosts inside your network?
> If this is the case you should just use IN for your TYPENAME, there
> are no web ports or icmp traffic that you are concerned with in your
> port list. If the initiating host is outside then you wouldn’t want
> INT2INT, OUT or OUTWEB. This change will potentially limit the total
> number of flows being evaluated.
>
> In my current system which is SIEM the rules are firing for traffic
> direction which is int2int.
>
>
> Can you show example flows that should match but doesn’t?
>
> I will prepare an rwfilter results for you and get back to you. I have
> evidence of its using traffic logs from cisco asa I can show that If
> you want.
>
> --
> Timur Snoke
> Network Defense Analyst
> CERT/CC - Network Situational Awareness Software Engineering Institute
> (SEI)
> O: (412) 268-7806
>
> From:
> <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:cert.org at cert.org><mailto:netsa-to<mailto:netsa-to>
> ols-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:cert.org at cert.org>>>
> on behalf of asad <a.alii85 at gmail.com<mailto:a.alii85 at gmail.com><mailto:a.alii85 at gmail.com<mailto:a.alii85 at gmail.com>>>
> Date: Tuesday, November 10, 2015 at 8:30 AM
> To: "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org><mailto:netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>"
> <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org><mailto:netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>>
> Subject: [netsa-tools-discuss] analysis pipeline alerting issues with
> EVAL block
>
> Hello,
>
> I have a very simple alerting requirement
>
> "
> when a destination ports matches ports which are considered as 'worm ports'
> traffic is send to 5 different unique ips in 5 minutes time.
>
> "
>
> I know on traffic level i'm getting required data since using traffic
> logs from the cisco asa (same device is also sending netflows) and it
> works as expected. I'm suppose to see an ip address but on alert.log I see nothing.
> Below is the logic.
>
> Any help?
>
>
>
> FILTER outgoing-flows
> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> END FILTER
>
> FILTER non-local-to-remote
> TYPENAME IN_LIST [in,int2int,out,outweb,outicmp] SIP NOT_IN_LIST
> "/root/silkydata/DMZ.set"
> SIP NOT_IN_LIST "/root/silkydata/voip.set"
> SIP NOT_IN_LIST "/root/silkydata/rns.set"
> DPORT IN_LIST [135, 136, 137, 138, 139, 444, 445, 995, 996, 997, 998,
> 999, 8998, 3127, 3128, 3129, 3130, 3131, 3132, 3133, 3134, 3135, 3136,
> 3137, 3138, 3139, 3140, 3141, 3142, 3143,3146, 3147, 31$ END FILTER
>
>
> EVALUATION  Systems_using_many_different_protocols
> FILTER  non-local-to-remote
> FOREACH SIP
> CHECK THRESHOLD
> DISTINCT DIP > 25
> TIME_WINDOW 3600 SECONDS
> END CHECK
> SEVERITY 7
> ALERT JUST_NEW_THIS_TIME
> ALERT ALWAYS
> CLEAR NEVER
> END EVALUATION
>
>



-------------- next part --------------
HTML attachment scrubbed and removed


More information about the netsa-tools-discuss mailing list