[netsa-tools-discuss] Buggy netflow sensors

[David Hoelzer]

I've been spending a fair amount of time with Silk lately, more 
specifically in creating some visualization tools for our data in real 
time, and have run into some interesting issues.  I cannot imagine that 
I am the first to find these, so I'm interested in thoughts on my 
approach to dealing with these things and I am interested to know if the 
community at large would have interest in my proposed fixes/patches.

I'm finding that there are a number of buggy NetFlow and IPFIX 
implementations.  For example, I have Avaya NetFlowV9 ERS devices that 
are intermittently generating flows with start times far in the future 
of the end times, resulting in crazy results for elapsed flow time 
(since these are cast as unsigned values). (Yes, I've restarted the 
switch, yes I've confirmed the values in the packets through raw hex 
decodes with the template).

My current approach is a patch in the rwflowpack.c code that discards 
flows where the elapsed time is insane (significantly larger than the 
automatic logging timeout) or the starting time is greater than the 
current UTC timestamp.  In the short term, this has dramatically cleaned 
up my visualization output because view of flows that have started in 
the last 30 days no longer includes flows that start two months from 
now. :)  It has also cleaned up the lower end and explains why I would 
periodically find log files from the 1970s floating around.

Has anyone else found this and devised some other/better strategy for 
dealing with it?  I really dislike discarding flows, but since the 
devices are already doing 1 packet for every thousand per port, it 
bothers me less.  I am rapidly coming to a point where I will simply 
span everything to my gigamon and pull a YAF flow meter from there; 
again, the loss is less significant at this point since the devices are 
already sampling (cannot be changed) and are clearly (randomly) 
generating garbage data anyway.

Thoughts?

-- 

----
David Hoelzer
Chief of Operations
Enclave Forensics, Inc.