[netsa-tools-discuss] NetFlow V9 sequence number mismatch

Mark Thomas mthomas at cert.org
Thu Feb 27 18:15:21 EST 2020 

Kirk-

I think I may know the problem.

You did not mention the version of SiLK you are using, but I bet (and hope) that you are using the silk-rwflowpack-3.19.0 RPMs from https://forensics.cert.org/

SiLK 3.19.0 was initially released on 2019-10-24, but that version contained a bug when reading IPFIX/NetFlowV9 over UDP.  The bug was fixed and a new release was created on 2019-10-28, but I reused the SiLK 3.19.0 version number.

The RPMs on the forensics site were not updated when SiLK 3.19.0 was re-issued.

I will contact the forensics team to update the SiLK RPMs.  To solve the problem yourself, you can build your own RPMs using the instructions on this page: https://tools.netsa.cert.org/silk/silk-on-box-rpm.html#BuildingRPMs

If the version mix-up I describe above does not describe your situation, please let me know so that I may find the source of your bug.

Regardless of whether this the source of your problem, THANK YOU very much for bringing this matter to my attention, and I am sorry for the trouble it has caused you.

Cheers,

-Mark

P.S.  One of the source files that was changed between the two versions was ipfixsource.c.  To determine if you have the correct version or the buggy version, run

  strings -n 20 /usr/lib64/libflowsource.so* | grep ipfixsource.c

If the output is

  $SiLK: ipfixsource.c 17d730af39a6 2019-10-28 15:44:53Z mthomas $

you have the correct file.  If the output is

  $SiLK: ipfixsource.c c317db03ed37 2019-10-11 14:47:18Z mthomas $

you have the buggy file.  If you have the source code, run this command instead to get one of the two lines above:

  grep RCSIDENT silk-3.19.0/src/libflowsource/ipfixsource.c


-----Original Message-----
From: Kirk Olson <Kirk_Olson at secura.net>
Date: Thu, 27 Feb 2020 20:59:59 +0000
To: Mark Thomas <mthomas at cert.org>
Cc: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: RE: [netsa-tools-discuss] NetFlow V9 sequence number mismatch

Mark thank you for your thoughtful reply. I made the recommended
changes and rebooted the system. I first went to /var/log/messages and
found the following:

Feb 26 09:34:20 ho-nflo-p01 kernel: rwflowpack[22322]: segfault at 8
ip 00007fce79852ad0 sp 00007fce77871b58 error 4 in
libfixbuf.so.9.0.2[7fce7983f000+23000]
Feb 26 09:38:32 ho-nflo-p01 kernel: rwflowpack[1783]: segfault at 8 ip
00007f43f9f92ad0 sp 00007f43f7fb1b58 error 4 in
libfixbuf.so.9.0.2[7f43f9f7f000+23000]
Feb 26 09:42:12 ho-nflo-p01 kernel: rwflowpack[2217]: segfault at 8 ip
00007fc05c2f7ad0 sp 00007fc05a316b58 error 4 in
libfixbuf.so.9.0.2[7fc05c2e4000+23000]
Feb 26 09:42:51 ho-nflo-p01 kernel: rwflowpack[2344]: segfault at 8 ip
00007f1b263fbad0 sp 00007f1b2441ab58 error 4 in
libfixbuf.so.9.0.2[7f1b263e8000+23000]
Feb 26 09:54:40 ho-nflo-p01 kernel: rwflowpack[2855]: segfault at 8 ip
00007faa48999ad0 sp 00007faa469b8b58 error 4 in
libfixbuf.so.9.0.2[7faa48986000+23000]
Feb 26 09:57:28 ho-nflo-p01 kernel: rwflowpack[3527]: segfault at 8 ip
00007f228ead5ad0 sp 00007f228caf4b58 error 4 in
libfixbuf.so.9.0.2[7f228eac2000+23000]
Feb 26 09:59:38 ho-nflo-p01 kernel: rwflowpack[3664]: segfault at 8 ip
00007f9d0842bad0 sp 00007f9d0644ab58 error 4 in
libfixbuf.so.9.0.2[7f9d08418000+23000]
Feb 26 10:03:39 ho-nflo-p01 kernel: rwflowpack[1791]: segfault at 8 ip
00007f476c473ad0 sp 00007f476a492b58 error 4 in
libfixbuf.so.9.0.2[7f476c460000+23000]
Feb 26 10:36:37 ho-nflo-p01 kernel: rwflowpack[2205]: segfault at 8 ip
00007ff000382ad0 sp 00007feffe3a1b58 error 4 in
libfixbuf.so.9.0.2[7ff00036f000+23000]
Feb 26 10:37:29 ho-nflo-p01 kernel: rwflowpack[2299]: segfault at 8 ip
00007f316b2d8ad0 sp 00007f31692f7b58 error 4 in
libfixbuf.so.9.0.2[7f316b2c5000+23000]
Feb 26 10:38:08 ho-nflo-p01 kernel: rwflowpack[1770]: segfault at 8 ip
00007f09f0430ad0 sp 00007f09ee44fb58 error 4 in
libfixbuf.so.9.0.2[7f09f041d000+23000]
Feb 26 10:42:44 ho-nflo-p01 kernel: rwflowpack[2224]: segfault at 8 ip
00007f9307b0aad0 sp 00007f9305b29b58 error 4 in
libfixbuf.so.9.0.2[7f9307af7000+23000]
Feb 26 14:29:13 ho-nflo-p01 kernel: rwflowpack[1787]: segfault at 8 ip
00007f078fcf2ad0 sp 00007f078dd11b58 error 4 in
libfixbuf.so.9.0.2[7f078fcdf000+23000]
Feb 27 14:47:45 ho-nflo-p01 kernel: rwflowpack[1793]: segfault at 8 ip
00007f69ca7daad0 sp 00007f69c87f9b58 error 4 in
libfixbuf.so.9.0.2[7f69ca7c7000+23000]

Apparently this has been failing for some time. Can you provide further guidance based on this error?

Cheers
-Kirk


-----Original Message-----
From: Mark Thomas <mthomas at cert.org> 
Sent: Thursday, February 27, 2020 1:32 PM
To: Kirk Olson <Kirk_Olson at secura.net>
Cc: netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] NetFlow V9 sequence number mismatch

Kirk-

Thank you for providing your configuration information.

rwflowpack produces log messages very two minutes that specify the
number of records received for each sensor and the number of records
written.  Do those messages show that data is being received?

Some things to try that will either resolve the issue or help to debug it:

Set the LOG_LEVEL to debug to provide more detailed logging.

Add a "quirks" statement to your "probe" block:

  probe S0 netflow-v9
  listen-on-port 18001
  protocol udp
  quirks firewall-event zero-packets
  end probe

For additional background on the quirks statement, see the sensor.conf manual page and this SiLK FAQ:
https://tools.netsa.cert.org/silk/faq.html#process-asa

Set the SILK_IPFIX_PRINT_TEMPLATES environment variable to 1 to have
rwflowpack record the template records it sees.  It may be that
something about the templates is causing rwflowpack to ignore them
(such as having no IP addresses; there is a quirk setting for this as
well).

I hope that the above suggestions help you debug the issue.  Please let me know if I can provide additional assistance.

Cheers,

-Mark


-----Original Message-----
From: Kirk Olson <Kirk_Olson at secura.net>
Date: Thu, 27 Feb 2020 17:25:18 +0000
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] NetFlow V9 sequence number mismatch

Follow-up…

We are using the following rwflowpack.conf:
ENABLED=yes
statedirectory=/var/silk
CREATE_DIRECTORIES=yes
BIN_DIR=/usr/sbin
SENSOR_CONFIG=/var/silk/sensors.conf
DATA_ROOTDIR=/var/silk/data
SITE_CONFIG=
PACKING_LOGIC=
INPUT_MODE=stream
INCOMING_DIR=${statedirectory}/incoming
ARCHIVE_DIR= #empty
FLAT_ARCHIVE=0
ERROR_DIR=  #${statedirectory}/error
OUTPUT_MODE=local-storage
SENDER_DIR=${statedirectory}/sender-incoming
INCREMENTAL_DIR=${statedirectory}/sender-incoming
COMPRESSION_TYPE=
POLLING_INTERVAL=
FLUSH_TIMEOUT=
FILE_CACHE_SIZE=
FILE_LOCKING=1
PACK_INTERFACES=0
SILK_IPFIX_PRINT_TEMPLATES=
LOG_TYPE=legacy
LOG_LEVEL=info
LOG_DIR=/var/log
PID_DIR=${statedirectory}
USER=root
EXTRA_OPTIONS=
EXTRA_ENVVAR=

We are using the following sensors.conf:

probe S0 netflow-v9
listen-on-port 18001
protocol udp
end probe

group secura-private-space
ipblocks 10.19.199.0/24  # ACI management network
ipblocks 10.102.0.0/21   # Wireless Guest
ipblocks 10.103.0.0/21   # Wireless Guest
…a bunch more private networks
end group

group secura-internet-space
ipblocks 98.100.228.0/24
end group

sensor S0
netflow-v9-probes S0
internal-ipblock @secura-private-space
external-ipblock remainder #changed this during troubleshooting from secura-internet-space end sensor

We also made the following buffering changes on the system:
  echo 'net.core.wmem_max=12582912' >> /etc/sysctl.conf
  echo 'net.core.rmem_max=12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_rmem= 10240 87380 12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_wmem= 10240 87380 12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_window_scaling = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_timestamps = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_sack = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_no_metrics_save = 1' >> /etc/sysctl.conf
  echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf

These are the libfixbuf versions on the system:
libfixbuf-2.4.0-1.el7.x86_64
libfixbuf-devel-2.4.0-1.el7.x86_64
libfixbuf-ipfixDump-2.4.0-1.el7.x86_64






From: Kirk Olson
Sent: Wednesday, February 26, 2020 2:43 PM
To: 'netsa-tools-discuss at cert.org' <netsa-tools-discuss at cert.org>
Subject: NetFlow V9 sequence number mismatch

Hello and thank you for reviewing my post.

We intend to run a single machine configuration using rwflowpack and SiLK on RHEL ver 7.

The installation on RHEL seems to go fine, as a check of the service status shows:
# systemctl status rwflowpack
● rwflowpack.service - LSB: start and stop SiLK rwflowpack daemon
   Loaded: loaded (/etc/rc.d/init.d/rwflowpack; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2020-02-26 14:29:12 CST; 2min 26s ago
     Docs: man:systemd-sysv-generator(8)
 Process: 1643 ExecStart=/etc/rc.d/init.d/rwflowpack start (code=exited, status=0/SUCCESS)

Feb 26 14:29:11 ho-nflo-p01.intranet.secura.net systemd[1]: Starting LSB: start and stop SiLK rwflowpack daemon...
Feb 26 14:29:12 ho-nflo-p01.intranet.secura.net rwflowpack[1643]: Starting rwflowpack:        [OK]
Feb 26 14:29:12 ho-nflo-p01.intranet.secura.net systemd[1]: Started LSB: start and stop SiLK rwflowpack daemon.

Looking into the log I see the following:
Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: Started logging at 2020-02-26 20:29:11Z Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: '/usr/sbin/rwflowpack' '--sensor-configuration=/var/silk/sensors.conf' '--output-mode=local-storage' '--root-directory=/var/silk/data' '--pidfile=/var/silk/rwflowpack.pid' '--log-level=info' '--log-directory=/var/log' '--log-basename=rwflowpack'
Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: Forked child 1781.  Parent exiting Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Using packing logic from /usr/lib64/silk/packlogic-twoway.so
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Creating stream cache Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Creating NetFlowV9 Reader for probe 'S0' on 18001 Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Starting flush timer Feb 26 14:29:13 ho-nflo-p01 rwflowpack[1781]: 'S0': accepted connection from 172.19.255.31:52600, domain 0x1000001 Feb 26 14:29:13 ho-nflo-p01 rwflowpack[1781]: NetFlow V9 sequence number mismatch for domain 0x1000001, expecting 0x0000 received 0x2f70

We are configured such that rwflowpack would create directories for flows in /var/silk/data/ext2ext/2020/02 but we see no flow directories being created. I am at a loss to understand why we are failing to collect flows from the switch which is a Cisco Catalyst 9K running XE 16.03. Please help!

Thank you for your consideration.
-Kirk


Kirk Olson
Information Security Engineer
Direct: 920-224-7426
Toll Free: 800-558-3405 ext. 7426
[cid:16b8a6c7a1e6d227b41]<https://www.secura.net/>
website | blog | Facebook | Twitter | LinkedIn<https://www.secura.net/>  <https://www.secura.net/> Recognized among Ward’s Top 50 and rated A Excellent by A.M. Best. <https://www.secura.net/> Confidentiality Note: This email may contain confidential and/or private information. If you received this email in error, please delete and notify sender.<https://www.secura.net/>

More information about the netsa-tools-discuss mailing list