[netsa-tools-discuss] rwflowpack

[Mark Thomas]

John-

I am not surprised at the behavior you are seeing.

The -ipblock code was intended to handle a few CIDR blocks, not
thousands of blocks, and the search though the blocks is not
efficient.

Adding the ability to split records using IPsets has been on our
radar.  However, issues of higher importance have delayed this
feature, especially when the existing code works well-enough most of
the time for most users.

We intentionally chose to use the less CPU-efficient CIDR block
implementation since the in-core representation of an IPset can be
large when dealing with large CIDR blocks.  However, that trade-off
made more sense when the -ipblock support was initially added
(nearly nine years ago) than it does today.

Thank you for your input.

-Mark


On Mon, 6 Oct 2014 15:38:24 +0000, John Green wrote:

> Hi,
> Is anyone using large numbers of internal/external-ipblocks with Silk?
>
> I've found CPU usage increases considerably when using lots of CIDR
> blocks with both twoway and generic.   With over 2000 CIDR blocks
> rwflowpack maxes out all my 4 CPUs (400%).    With 180 CIDR blocks CPU
> is down to around 150%.   Reducing to just one CIDR block results in CPU
> of 30%, which is similar to an existing flow-capture process
> (flow-tools).
>
> Has anyone developed an alternate packlogic which does ip lookups more
> efficiently, perhaps using a Trie?   Not many of my exporters are on the
> network edge, so using interfaces isn't really feasible.
>
> Thanks
> John
>
>
> Janet(UK) is a trading name of Jisc Collections and Janet Limited, a 
> not-for-profit company which is registered in England under No. 2881024 
> and whose Registered Office is at Lumen House, Library Avenue,
> Harwell Oxford, Didcot, Oxfordshire. OX11 0SG. VAT No. 614944238