[netsa-tools-discuss] Benchmarking of flowcap

[Mark Thomas]

Matt-

I am sorry to hear that one million flows is your goal and not what
you are seeing.

In my testing, my goal in "replaying" a pcap file is to send the
payload of the packets through SiLK.  I wrote a simple Perl script
that uses the Net::Pcap module to read pcap files and resend their
content to host:port pair.  It has a command line option to
specify the sleep time between sending each packet.  This script is
sufficient for what I need.  It is attached.

Have you tried to increase the size of the receiving socket buffers?
Assuming Linux:

 sysctl -w net.core.rmem_max=<value>
 sysctl -w net.core.rmem_default=<value>

In my testing, the number of packets received by flowcap matches the
values given by netstat -s.  I have taken this to indicate that
dropped packets are outside of flowcap's control.

The SILK_LIBFIXBUF_SUPPRESS_WARNINGS environment variable suppresses
the printing of most messages from libfixbuf (such as those about
out-of-sequence packets).  In my limited testing I have not found it
to make much of difference to overall performance.

I do not like the sounds of garbled v9 output.  I certainly hope the
errors are in the source data.  I assume you can discover this by
examining the source data with Wireshark (or its command-line only
version, tshark).

Good luck.

-Mark


On Fri, 24 Jul 2015 10:38:21 -0500, Matthew Markland wrote:

> Mark:
> Thank you so much for the response and feedback. I really appreciate it.
> I probably should have been a little clearer; my current testing with
> flowcap does not handle 1 million flows per second. That is the
> predicted peak that we would have to handle. My group is having a lot
> of trouble evaluating flowcap right now because it does appear to fall
> off a cliff when SiLK starts having troubles, either with dropped
> packets causing sequence number failures, or other error messages
> being generated.
> We also seem to be seeing garbled v9 output when data is dropped, but
> we haven't yet been able to determine whether that is a result of the
> dropped data or if some of the v9 data we are receiving is corrupt.
> I'm curious how you are replaying your pcap files. I have been trying
> to get tcpreplay working but I can't seem to get it to do what I
> want. If you have any advice I would welcome it.
> Thanks again.
> Matt----Matthew Markland
> mwmarkland at outlook.com
>
>
>
>> From: mthomas at cert.org
>> To: marklandm at acm.org
>> Date: Fri, 24 Jul 2015 11:28:06 -0400
>> CC: netsa-tools-discuss at cert.org
>> Subject: Re: [netsa-tools-discuss] Benchmarking of flowcap
>> 
>> Matt-
>> 
>> Thank you for your comments; they are appreciated.
>> 
>> The short answer is that I do not believe we have any numbers that
>> benchmark flowcap.  The YAF developer does most of our benchmarking,
>> and her goal is to measure how quickly YAF turns packets into flow
>> records.
>> 
>> I am impressed with the numbers you are seeing.  When I ran some
>> informal tests back in March, I saw NetFlow V9 record processing
>> peaking near 300,000 records per second.
>> 
>> In my experiments, I used an older RedHat EL5 machine running SiLK
>> 3.10.1 linked against libfixbuf-1.6.2.  The data I used came from
>> packet capture (libpcap) files containing NetFlow V9 packets.  My
>> tests used smallish pcap files, which prevented me from running
>> prolonged tests.  I sent the packets over the IPv4 loopback address.
>> 
>> I used flowcap in my testing, since it does less work than
>> rwflowpack.  I also ensured that all the data flowcap received was
>> written to a single file.  In production, there would be times when
>> flowcap would need to close and reopen the data files.
>> 
>> First I checked processing of IPFIX (v10) data as generated by the
>> yaf[2] tool.  flowcap was normally able to keep up with yaf, often
>> processing 475,000 or more records per second.
>> 
>> When replaying NetFlow v9 data, I could comfortably process 200,000
>> records per second, and in one test I successful processed 300,000
>> records per second.  However, when I replayed data as fast as
>> possible, performance dropped to about 110,000 records/second.
>> 
>> There seems to be a sudden drop in performance between fast-enough
>> and too-fast: I assume this is because when the first packet is
>> lost, SiLK writes a message to the log.  Writing that log message
>> slows processing so that more data is lost.
>> 
>> I definitely like your numbers better than mine.
>> 
>> Thanks again.
>> 
>> -Mark
>> 
>> 
>> -----Original Message-----
>> From: Matthew Markland <marklandm at acm.org>
>> Date: Sun, 19 Jul 2015 11:05:12 -0500
>> To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
>> Subject: [netsa-tools-discuss] Benchmarking of flowcap
>> 
>> All:
>> 
>> Thank you very much for supplying and supporting what looks to be an
>> excellent set of tools. I have been investigating flow collection and
>> how well various tools stand up to high rates of export packets. Has
>> there already been any work done, or anecdotal evidence of flowcap's
>> scaling? The numbers I have been looking at have ranged up to a peek
>> of one million flows per second (which I divide by 30 to approximate
>> the number of packets arriving).
>> 
>> I realize that some of the performance may depend on the underlying
>> hardware, so if you pass on rates the hardware you are running on
>> would be useful to know also.
>> 
>> Thank you very much for your time.
>> 
>> Matt Markland
>> 
>>  		 	   		  
>  		 	   		  

-------------- next part --------------
A non-text attachment was scrubbed...
Name: resend-nf-from-pcap.pl
Type: application/octet-stream
Size: 14555 bytes
Desc: not available
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20150724/32a22df4/attachment.obj>