[netsa-tools-discuss] SiLK/libfixbuf ignoring sFlow records

[Kent Kuriyama]

Mark,

Thank you for your reply.  The problem of not collecting data is related to
my rwflowpack startup file.  When I start up use the command line:

/usr/local/sbin/rwflowpack --sensor-configuration=/usr/
local/share/silk/etc/sensor.conf --root-directory=/data/silk
--log-destination=syslog

data is indeed being collected.  I suspect that I was distracted by the
"sequence number" messages and did not bother to check the data directories.

Again, thank you very much for your assistance.

Kent

On Tue, Jun 16, 2015 at 3:41 AM, Mark Thomas <mthomas at cert.org> wrote:

>
> Thank you for your query, and we especially thank you for including
> the sample pcap file and your sensor.conf.
>
> The "sequence number mismatch" messages are not an indication of an
> error.  Typically the message means that a UDP packet has arrived
> out of sequence or gotten lost, or that data is arriving too quickly
> for rwflowpack to process.  They are meant to be informative
> warnings and do not prevent the records from being written to disk.
>
> Given the sequence numbers that I am seeing in both the log messages
> and the pcap file, it appears that there at least are three
> "streams" of data being intermingled.  Looking at the log messages,
> I see a
>
> * 0x11f8b4 stream (decimal 1177780)
>
> * 0x219a9 stream (decimal 137641)
>
> * 0xd9c88f3, 0xd9c8900 stream (decimal 228362483)
>
> The streams are more pronounced in when you examine the sequence
> numbers in pcap file:
>
>   $ tshark -V -n -r file.tcpdump  | grep 'sample, seq' | sort
>       Flow sample, seq 1179103
>       Flow sample, seq 1179104
>       Flow sample, seq 1179105
>       Flow sample, seq 1179106
>       Flow sample, seq 1432328
>       Flow sample, seq 228375879
>       ...
>       Flow sample, seq 228375905
>       Flow sample, seq 614783
>       Flow sample, seq 614784
>
> The "sequence number mismatch" message is going to appear whenever
> rwflowpack/libfixbuf is expecting a packet from one stream but
> receives a packet from a different stream.
>
> A solution to the sequence number mismatch issue is to configure
> each of those streams to use a different agent number.  Alternately,
> you may disable those warnings by setting the
> SILK_LIBFIXBUF_SUPPRESS_WARNINGS environment variable to 1 before
> starting rwflowpack.
>
> However, it is unusual that you are not seeing any data files.
>
> My colleague and I independently examined your pcap file and we were
> both able to store the sFlow data in that file using rwflowpack.
>
> Is it possible that other syslog messages from rwflowpack are being
> hidden by the "sequence number mismatch" messages?
>
> Every two minutes (the --flush-timeout default), you should see a
> syslog message that looks like:
>
>   rwflowpack[45266]: 'IPA-Core': forward 31, reverse 0, ignored 0, sflow:
> missing-pkts 0
>
> showing that the 'IPA-Core' probe generated 31 flow records from the
> sFlow packets it received.  rwflowpack also flushes any in-memory
> records to disk during at the flush-timeout, and this generates
> messages such as:
>
>   rwflowpack[45266]: /data/silk/int2int/2015/06/15/int2int-S0_20150615.19:
> 4 recs
>
> showing the full path to the file that was written to and the number
> of records written.
>
> Are you seeing either of these sorts of message in syslog?
>
> Thanks again.
>
> -Mark
>
>
> On Sun, 14 Jun 2015 10:39:34 -1000, Kent Kuriyama wrote:
>
> > I too am trying to collect sFlow records using Silk 3.10 and
> > libfixbuf version 1.6.2.  I am running rwflowpack using the
> > following command line:
> >
> >
> > /usr/local/sbin/rwflowpack
> > --sensor-configuration=/usr/local/share/silk/etc/sensor.conf
> > --root-directory=/data/silk --log-destination=syslog
> >
> > Looking at the syslog I get a bunch of entries
> >
> > Jun 14 10:12:43 IPAmonitor rwflowpack[40932]: sFlow Sample sequence
> number mismatch for agent 0x0001, expecting 0x11f8b4 received 0xd9c88f3
> > Jun 14 10:12:44 IPAmonitor rwflowpack[40932]: sFlow Sample sequence
> number mismatch for agent 0x0001, expecting 0xd9c8900 received 0x219a9
> > Jun 14 10:12:44 IPAmonitor rwflowpack[40932]: sFlow Sample sequence
> number mismatch for agent 0x0001, expecting 0x219aa received 0xd9c8900
> > Jun 14 10:12:44 IPAmonitor rwflowpack[40932]: sFlow Sample sequence
> number mismatch for agent 0x0001, expecting 0xd9c890a received 0x11f8b4
> >
> > I have not encountered any of the "Ignoring sFlow record: sFlow Record
> > Length Mismatch" messages mentioned in this thread, applying the patch to
> > fbsflow.c did not make any difference.
> >
> > No data is being collected in /data/silk. I have attached a 10 packet
> > capture of the sFlow traffic. Has anyone seen this problem? Thanks.
> >
> > Kent Kuriyama
> >
> > -------------- sensor.conf ---------------------
> > probe IPA-Core sflow listen-on-port 6343 protocol udp end probe sensor S0
> > sflow-probes IPA-Core external-interface 216 internal-interface remainder
> > end sensor
>
-------------- next part --------------
HTML attachment scrubbed and removed