[netsa-tools-discuss] More beginner questions

[Mark Thomas]

Keith-

Thank you for your email.

My answers to your questions are in-line.

On Fri, 19 Sep 2014 12:20:37 -0700, Keith Miller wrote:

> Greetings,
>
> We've definitely decided to start utilizing your ecosystem, but
> have a few questions.

It is always good to hear from new users.

> Is there a native way to ingest sFlow records, or will I need to
> use a tool like sflowtool to convert them to PCAP before feeding
> to yaf as a file?

The next release of SiLK (3.9.0) when linked against the next
release of libfixbuf (1.6.0) will have support for sFlow v5.  Both
SiLK and libfixbuf are currently in the final stages of testing.

> Is there any documentation about the practical limits of Analysis
> Pipeline and watchlist scalability?  My initial target is to
> monitor for 200K ip's with an initial rate of 50MB/s peak.  But is
> there anything that would stop me from scaling that up to millions
> of IP's at 1GB/s?

The watchlists in the Analysis Pipeline are implemented using SiLK
IPsets, which are very efficient when handling IPv4 addresses and
good when handling IPv6.  If all the IPs are single watchlist (and
therefore within a single IPset), performance will be very good.

At the other extreme, if every IP is on a separate watchlist,
performance will suffer as the Pipeline loops over all those IPset
files.

The Analysis Pipeline processes files produced by rwflowpack.  These
files act as a capacitor to dampen the peaks and valleys of traffic
crossing the monitoring point.

> Last, are there any reference architectures available that speak
> to HA deployments at the aforementioned scales?

When we test for performance under network load, we are primarily
testing the capabilities of yaf, our flow exporter.  yaf has been
tested on 10Gbit networks, and higher network speeds when using
high-speed capture cards (Endace/Emulex, Napatech, and Netronome).

I do not believe there is a reference that mentions using SiLK
without using yaf as the main capture tool.

> Thanks in advance!

I hope I have answered your questions.

> Keith Miller

Regards,

-Mark