[netsa-tools-discuss] Benchmarking of flowcap

[Mark Thomas]

Matt-

Thank you for your comments; they are appreciated.

The short answer is that I do not believe we have any numbers that
benchmark flowcap.  The YAF developer does most of our benchmarking,
and her goal is to measure how quickly YAF turns packets into flow
records.

I am impressed with the numbers you are seeing.  When I ran some
informal tests back in March, I saw NetFlow V9 record processing
peaking near 300,000 records per second.

In my experiments, I used an older RedHat EL5 machine running SiLK
3.10.1 linked against libfixbuf-1.6.2.  The data I used came from
packet capture (libpcap) files containing NetFlow V9 packets.  My
tests used smallish pcap files, which prevented me from running
prolonged tests.  I sent the packets over the IPv4 loopback address.

I used flowcap in my testing, since it does less work than
rwflowpack.  I also ensured that all the data flowcap received was
written to a single file.  In production, there would be times when
flowcap would need to close and reopen the data files.

First I checked processing of IPFIX (v10) data as generated by the
yaf[2] tool.  flowcap was normally able to keep up with yaf, often
processing 475,000 or more records per second.

When replaying NetFlow v9 data, I could comfortably process 200,000
records per second, and in one test I successful processed 300,000
records per second.  However, when I replayed data as fast as
possible, performance dropped to about 110,000 records/second.

There seems to be a sudden drop in performance between fast-enough
and too-fast: I assume this is because when the first packet is
lost, SiLK writes a message to the log.  Writing that log message
slows processing so that more data is lost.

I definitely like your numbers better than mine.

Thanks again.

-Mark


-----Original Message-----
From: Matthew Markland <marklandm at acm.org>
Date: Sun, 19 Jul 2015 11:05:12 -0500
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Benchmarking of flowcap

All:

Thank you very much for supplying and supporting what looks to be an
excellent set of tools. I have been investigating flow collection and
how well various tools stand up to high rates of export packets. Has
there already been any work done, or anecdotal evidence of flowcap's
scaling? The numbers I have been looking at have ranged up to a peek
of one million flows per second (which I divide by 30 to approximate
the number of packets arriving).

I realize that some of the performance may depend on the underlying
hardware, so if you pass on rates the hardware you are running on
would be useful to know also.

Thank you very much for your time.

Matt Markland