[netsa-tools-discuss] Testing flowcap

Mark Thomas mthomas at cert.org
Tue Sep 29 14:15:14 EDT 2015


Matt-

For load testing, all of our team's testing has used YAF as the
source of the flow data.

For sustained testing of YAF and flowcap, we have used a traffic
generator.  YAF assembles the packets into IPFIX flow records that
then get sent to flowcap.  YAF and flowcap would connect over TCP,
and since YAF sends the templates once the connection is
established, flowcap was certain to receive the templates.

We also have a couple of large-ish PCAP files that we can process
with YAF to produce IPFIX records for testing.  YAF's --caplist
switch allows it to process multiple PCAP files.

The problem with using YAF to test UDP connectivity when YAF is
reading from PCAP files is that YAF floods flowcap with IPFIX
records and there is no way to tell YAF to back-off.  Sometimes I
use a Perl script to send chucks of the PCAP file to YAF's standard
input, and I have Perl sleep between each chunk to reduce the rates
at which packets are generated.

I regularly use the Perl script that I shared with you back in July.
That script uses the Net::Pcap Perl module to process PCAP file(s)
of NetFlow v9 records.  The script strips the UDP header from each
packet and resends the NetFlow v9 payload to flowcap.  Sometimes I
send the same set of PCAP files through the Perl script multiple
times.

If I am really desperate, I write a Perl script to create NetFlow
records with a single template and the same set of flow records over
and over again.  (That was much easier in the days of NetFlow v5.)

The NetFlow templates are certainly an issue when using UDP.
Perhaps you could copy the packets containing the templates out of
the PCAP file, send those packets to flowcap first to ensure it has
them, and the send the deluge of NetFlow v9 records.  You will get
warnings regarding out-of-sequence packets, but at least flowcap
will have the templates.

I am somewhat surprised you are having difficulty getting consistent
results.  Even though the code uses threads and you are dealing with
the network, I would have expected less variation in your test
results.

Good luck!

-Mark


-----Original Message-----
From: Matthew Markland <mwmarkland at outlook.com>
Date: Fri, 25 Sep 2015 15:50:54 -0500
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Testing flowcap

All:

I've been struggling with some testing of flowcap that I've been
attempting. What I have is a pcap file containing actual netflow
packets (i.e. the UDP packets with NetFlow records) which I then
replay onto the network targeting a flowcap instance. I do all the
dirty work to spoof addresses so that the destinations look right. I
then vary the delay between sending UDP packets to try to simulate
different rates of traffic arriving at flowcap.

I know of one significant problem with this plan; that is
templates. If my recording doesn't happen to catch needed templates,
flowcap will drop (a possibly large number of) flows. However, I'm
also seeing inconsistent behavior between runs with the same delay and
input file. It appears that sometimes flowcap doesn't recognize or
expires the templates. I have not been able to get any consistency
with the above testing process.

So, I'm going to ask; how does your group test flowcap? My guess is
that you have a pcap file of traffic that you run through YAF and then
feed that output into flowcap. That doesn't seem to give you a way of
testing the maximum flow count you can handle unless you have a way of
generating enough traffic in the pcap file to overload it.

A secondary question is whether the method we are currently using
could be made to work better. Do we need to rewrite the pcap file time
stamps to make things work out better?

Thanks for your time!

Matt
----
Matthew Markland
mwmarkland at outlook.com

 		 	   		  


More information about the netsa-tools-discuss mailing list