[netsa-tools-discuss] Point in Time Data in Silk

Wed Dec 2 06:58:43 EST 2015

Drew,

You said, "and keep going until I reach --bin-size=1."  Be careful with this.  The typical is an active timeout for a flow is 30 minutes, so long running flows don’t even get written to the database until they are quite old.  With second-granularity, you have to make lots of assumptions about how the packets behaved over those 30 minutes.  This is true of 1 minute flows too.

From the rwcount man page:

   "There is no
       way to know how the bytes and packets were distributed during the duration of the record:
       their distribution could be front-loaded, back-loaded, uniform, et cetera."

I would recommend you stick with 30 or 60 minute granularity, "gigabytes per 30 minutes".  From there you can say "an average of X megabytes per minute".  At least realize it is an approximation.

I hope this helps.

George

-----Original Message-----
From: netsa-tools-discuss-bounces+george.warnagiris=ii-vi.com at cert.org [mailto:netsa-tools-discuss-bounces+george.warnagiris=ii-vi.com at cert.org] On Behalf Of Chris Inacio
Sent: Tuesday, December 01, 2015 9:50 PM
To: Drew Morrigan
Cc: netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] Point in Time Data in Silk

> On Dec 1, 2015, at 8:55 PM, Drew Morrigan <drewm at landesa.org> wrote:
> 
> 
> “Instantaneous peak bandwidth” is a brilliantly succinct way to say what I took a few paragraphs to illustrate.  So thank you for that phrase, if nothing else!
>  
> While Prism sounds pretty close to what I am looking for, my servers are CLI only.  Is it possible to run it in such an environment, and perhaps export the results so that they can be viewed graphically on another computer?
>  
> I’ll also poke about with the other tools you suggested.  Thank you for those, as well.
>  
> 
> 
> Drew Morrigan | drewm at landesa.org
> Systems Admin / Support Specialist
> 
> Landesa
> 1424 Fourth Ave., Suite 300, Seattle, WA  98101
> T: 206-257-6158 | F: 206-528-5881
> Skype: drewm_landesa
> 
> www.landesa.org
> 
> Recipient of the 2015 Hilton Humanitarian Award
> 

I honestly don’t know that much about the backend implementations of FlowBat or FlowViewer - so I won’t particularly theorize about if they can run CLI only.  I will note that both of them are web UI based front ends to SiLK.

Prism can run without a display.  In this case, the key thing would be the summary data that would get stored in the database.  I wouldn’t get too gung ho yet though.  I’ll make sure some folks take a look at this thread and can give you the real relevant details on this.

Hopefully the FlowBat and FlowViewer folks will jump in as well.

regards,
--
Chris Inacio
inacio at cert.org

> From: Chris Inacio [mailto:inacio at cert.org] 
> Sent: Tuesday, December 1, 2015 5:46 PM
> To: Drew Morrigan
> Cc: netsa-tools-discuss at cert.org
> Subject: Re: [netsa-tools-discuss] Point in Time Data in Silk
>  
> Drew,
>  
> First, let me make the disclaimer that this isn't exactly my area of expertise.
>  
> Let me point you at a script we have on our site: prism. Prism will create a database with traffic time series data and allow for plots of that information to be created from the database as time series trend lines. You can define the bin types (in web, in, http, etc.) as you like. Using the stored prism data would greatly speed your search. (At the cost of disk space, of course.  :) but you can get nice plots too.)
>  
> Http://tools.netsa.cert.org/script-prism/
>  
> Although I got the impression you don't want something like prism. (I'm willing to bet tools like flowviewer and flowbat also have very similar features.) 
> 
> http://sourceforge.net/p/flowviewer/wiki/Home/
> Http://flowbat.com
>  
> When I first read this, I thought you were searching for your TopTalker, but after rereading I'm second guessing and wondering if what you really want is instantaneous peak bandwidth. Those would obviously be very different things. 
>  
> Maybe you can answer that and an analyst will respond with better search foo than I have. 
>  
> Regards,
> --
> Chris Inacio
> Inacio at cert.org
>  
>  
> Sent from my iPad
> 
> On Dec 1, 2015, at 6:33 PM, Drew Morrigan <drewm at landesa.org> wrote:
> 
> Greetings,
>  
>   I am using Silk 3.11 on Ubuntu Server 14.04.  I have it happily collecting data from our FWs and am currently fumbling around with the analysis tools.  I’ve been able to get some cool/useful information from those fumblings, but there is something I need I haven’t been able to put together easily on my own.
>  
>   Due to some changes to our environment, we will soon be making more use out of our ISP’s upload bandwidth than we have been.  We want to know how much outgoing traffic we are currently transmitting, but from a ‘snapshot’ perspective, not the total amount used.  To hopefully clarify things, here’s what I’m doing currently:
>  
> rwfilter --start-date=2015/10/28T00 --end-date=2015/11/30T18 --type=out,outweb --sensors=S2 --saddress=10.0.0.0/24 --pass=stdout | rwcount --bin-size=86400 --skip-zeroes
>  
> As I’ve come to understand things, this would give me the total data transmitted for each 24-hour period over the specified range.  I’d then drill down into those periods and determine which hours had the highest transmissions, and keep going until I reach –bin-size=1.  That would be my ‘snapshot’ upload bandwidth usage (less compression, WAN optimization, etc., of course).
>  
> This is, suffice to say, not an efficient process, especially since collectors are running at more than one office location.  How else could I be going about this?
> 
> 
> Drew Morrigan | drewm at landesa.org
> Systems Admin / Support Specialist
> 
> Landesa
> 1424 Fourth Ave., Suite 300, Seattle, WA  98101
> T: 206-257-6158 | F: 206-528-5881
> Skype: drewm_landesa
> 
> www.landesa.org
> 
> Recipient of the 2015 Hilton Humanitarian Award
> 

The information contained in this transmission is intended only for the person or entity
to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive
material. If you are not the intended recipient, please contact the sender immediately
and destroy the material in its entirety, whether electronic or hard copy. You are
notified that any review, retransmission, copying, disclosure, dissemination or other
use of, or taking of any action in reliance upon this information by persons or entities 
other than the intended recipient is prohibited.