[netsa-tools-discuss] rwflowpack

[Mark Thomas]

John-

My reply is in-line.

On Mon, 13 Oct 2014 15:45:48 +0000, John Green wrote:

> On Wed, 2014-10-08 at 13:18 -0400, Mark Thomas wrote:
>> John-
>> 
>> I am not surprised at the behavior you are seeing.
>> 
>> The -ipblock code was intended to handle a few CIDR blocks, not
>> thousands of blocks, and the search though the blocks is not
>> efficient.
>
> Hi Mark,
> Thanks for getting back to me.   I'll take a look at writing a more
> efficient packing logic module to handle my atypical usage.

If there is some way you can use either SNMP interfaces or VLAN ids
to categorize the flow records, you will see noticeable improvement
in rwflowpack performance.

If you know that all incoming traffic is seen by one NIC and all
outgoing traffic is seen by a different NIC, I would suggest using
the configuration described here.
http://tools.netsa.cert.org/silk/sensor.conf.html#Multiple-Sources-Becoming-One-Sensor-Specific-Directions-

> Are there any figures available anywhere for the quantity of
> netflow typically pushed through a single Silk processing chain?

Off the top of my head I am not aware that we have numbers like
these.  I will ask some of my colleagues.

> I've been
> hitting various issues with rwsender/rwreceiver and
> rwflowpack/append as well.

Some things I would suggest:

* Make certain compression is enabled.  gzip tends to produce
  tighter compression but it is slower that LZO, which is why we
  prefer LZO.

* For rwflowappend, use the --threads switch (added in SiLK 3.8.2).
  Without that switch, the default is a single thread.

* If your configuration has a collection process (that is, flowcap
  or rwflowpack) near the sensor that is sending files to a central
  repository...

  ** Consider increasing the time-out or maximum file size limit
     used on this process.  This reduces the ratio of file-header to
     file-data.

  ** The bytes/record value is smaller for rwflowpack than for
     flowcap.  This would suggest running rwflowpack near the
     sensor, but given the CIDR-block performance issues you are
     seeing, this may not be the best decision.  On the other hand,
     if each collection process sees only a small part of the
     network, this may solve your CIDR-block performance issue.

* For rwsender, try adjusting the --block-size.  There is a small
  amount of overhead for each block.

* Communication between an rwsender and an rwreceiver handles one
  file at a time.  You could try adding a second rwreceiver process
  and using the --filter switch on rwsender to split the files
  between the two rwreceivers.

> I am trying to process around 500GB/day.

Is that 500GB of the raw traffic, or 500GB/day of NetFlow?

> I am trying to get an idea on how many systems this would need to
> be distributed across.
>
> Thanks
> John
>
> Janet(UK) is a trading name of Jisc Collections and Janet Limited, a 
> not-for-profit company which is registered in England under No. 2881024 
> and whose Registered Office is at Lumen House, Library Avenue,
> Harwell Oxford, Didcot, Oxfordshire. OX11 0SG. VAT No. 614944238

Cheers,

-Mark