[netsa-tools-discuss] rwflowpack

[John Green]

Hi Mark,

Responses inline.

On Wed, 2014-10-15 at 13:36 -0400, Mark Thomas wrote:
> If there is some way you can use either SNMP interfaces or VLAN ids
> to categorize the flow records, you will see noticeable improvement
> in rwflowpack performance.

I can for some of my probes which are on the network edge.  This has
helped quite a bit as this includes some of the busiest.

> * Make certain compression is enabled.  gzip tends to produce
>   tighter compression but it is slower that LZO, which is why we
>   prefer LZO.

I am using LZO.

> * For rwflowappend, use the --threads switch (added in SiLK 3.8.2).
>   Without that switch, the default is a single thread.

I've added that.

> * If your configuration has a collection process (that is, flowcap
>   or rwflowpack) near the sensor that is sending files to a central
>   repository...

My initial approach was

netflow -> box1 <- box2
box1 running rwflowpack --incremental and rwsender --server
box2 running rwreceiver --client, rwflowappend & pipeline & rwpollexec

with defaults for polling intervals and flush timeout

rwflowpack produced a large number of files which slowly backed up in
the rwsender processing directory.

I am now doing
box1 running flowcap and rwsender --server
box2 running rwreceiver --client, rwflowpack --incremental, rwflowappend
& pipeline & rwpollexec

and now there is very little queuing on box1, which makes me think it
was number of files rather than volume which was the issue.  I will try
tweaking the polling and timeout values further.

> * For rwsender, try adjusting the --block-size.  There is a small
>   amount of overhead for each block.

I increased this to 65535 with little noticeable difference

> > I am trying to process around 500GB/day.
> 
> Is that 500GB of the raw traffic, or 500GB/day of NetFlow?

500GB of uncompressed netflow.

Is there a reason why you don't use something like inotify rather than
polling?  Portability?

Thanks
John



Janet(UK) is a trading name of Jisc Collections and Janet Limited, a 
not-for-profit company which is registered in England under No. 2881024 
and whose Registered Office is at Lumen House, Library Avenue,
Harwell Oxford, Didcot, Oxfordshire. OX11 0SG. VAT No. 614944238