[netsa-tools-discuss] NetFlow V9 sequence number mismatch

Thu Feb 27 14:31:45 EST 2020

Kirk-

Thank you for providing your configuration information.

rwflowpack produces log messages very two minutes that specify the number of records received for each sensor and the number of records written.  Do those messages show that data is being received?

Some things to try that will either resolve the issue or help to debug it:

Set the LOG_LEVEL to debug to provide more detailed logging.

Add a "quirks" statement to your "probe" block:

  probe S0 netflow-v9
  listen-on-port 18001
  protocol udp
  quirks firewall-event zero-packets
  end probe

For additional background on the quirks statement, see the sensor.conf manual page and this SiLK FAQ:
https://tools.netsa.cert.org/silk/faq.html#process-asa

Set the SILK_IPFIX_PRINT_TEMPLATES environment variable to 1 to have rwflowpack record the template records it sees.  It may be that something about the templates is causing rwflowpack to ignore them (such as having no IP addresses; there is a quirk setting for this as well).

I hope that the above suggestions help you debug the issue.  Please let me know if I can provide additional assistance.

Cheers,

-Mark

-----Original Message-----
From: Kirk Olson <Kirk_Olson at secura.net>
Date: Thu, 27 Feb 2020 17:25:18 +0000
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] NetFlow V9 sequence number mismatch

Follow-up…

We are using the following rwflowpack.conf:
ENABLED=yes
statedirectory=/var/silk
CREATE_DIRECTORIES=yes
BIN_DIR=/usr/sbin
SENSOR_CONFIG=/var/silk/sensors.conf
DATA_ROOTDIR=/var/silk/data
SITE_CONFIG=
PACKING_LOGIC=
INPUT_MODE=stream
INCOMING_DIR=${statedirectory}/incoming
ARCHIVE_DIR= #empty
FLAT_ARCHIVE=0
ERROR_DIR=  #${statedirectory}/error
OUTPUT_MODE=local-storage
SENDER_DIR=${statedirectory}/sender-incoming
INCREMENTAL_DIR=${statedirectory}/sender-incoming
COMPRESSION_TYPE=
POLLING_INTERVAL=
FLUSH_TIMEOUT=
FILE_CACHE_SIZE=
FILE_LOCKING=1
PACK_INTERFACES=0
SILK_IPFIX_PRINT_TEMPLATES=
LOG_TYPE=legacy
LOG_LEVEL=info
LOG_DIR=/var/log
PID_DIR=${statedirectory}
USER=root
EXTRA_OPTIONS=
EXTRA_ENVVAR=

We are using the following sensors.conf:

probe S0 netflow-v9
listen-on-port 18001
protocol udp
end probe

group secura-private-space
ipblocks 10.19.199.0/24  # ACI management network
ipblocks 10.102.0.0/21   # Wireless Guest
ipblocks 10.103.0.0/21   # Wireless Guest
…a bunch more private networks
end group

group secura-internet-space
ipblocks 98.100.228.0/24
end group

sensor S0
netflow-v9-probes S0
internal-ipblock @secura-private-space
external-ipblock remainder #changed this during troubleshooting from secura-internet-space
end sensor

We also made the following buffering changes on the system:
  echo 'net.core.wmem_max=12582912' >> /etc/sysctl.conf
  echo 'net.core.rmem_max=12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_rmem= 10240 87380 12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_wmem= 10240 87380 12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_window_scaling = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_timestamps = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_sack = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_no_metrics_save = 1' >> /etc/sysctl.conf
  echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf

These are the libfixbuf versions on the system:
libfixbuf-2.4.0-1.el7.x86_64
libfixbuf-devel-2.4.0-1.el7.x86_64
libfixbuf-ipfixDump-2.4.0-1.el7.x86_64

From: Kirk Olson
Sent: Wednesday, February 26, 2020 2:43 PM
To: 'netsa-tools-discuss at cert.org' <netsa-tools-discuss at cert.org>
Subject: NetFlow V9 sequence number mismatch

Hello and thank you for reviewing my post.

We intend to run a single machine configuration using rwflowpack and SiLK on RHEL ver 7.

The installation on RHEL seems to go fine, as a check of the service status shows:
# systemctl status rwflowpack
● rwflowpack.service - LSB: start and stop SiLK rwflowpack daemon
   Loaded: loaded (/etc/rc.d/init.d/rwflowpack; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2020-02-26 14:29:12 CST; 2min 26s ago
     Docs: man:systemd-sysv-generator(8)
 Process: 1643 ExecStart=/etc/rc.d/init.d/rwflowpack start (code=exited, status=0/SUCCESS)

Feb 26 14:29:11 ho-nflo-p01.intranet.secura.net systemd[1]: Starting LSB: start and stop SiLK rwflowpack daemon...
Feb 26 14:29:12 ho-nflo-p01.intranet.secura.net rwflowpack[1643]: Starting rwflowpack:        [OK]
Feb 26 14:29:12 ho-nflo-p01.intranet.secura.net systemd[1]: Started LSB: start and stop SiLK rwflowpack daemon.

Looking into the log I see the following:
Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: Started logging at 2020-02-26 20:29:11Z
Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: '/usr/sbin/rwflowpack' '--sensor-configuration=/var/silk/sensors.conf' '--output-mode=local-storage' '--root-directory=/var/silk/data' '--pidfile=/var/silk/rwflowpack.pid' '--log-level=info' '--log-directory=/var/log' '--log-basename=rwflowpack'
Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: Forked child 1781.  Parent exiting
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Using packing logic from /usr/lib64/silk/packlogic-twoway.so
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Creating stream cache
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Creating NetFlowV9 Reader for probe 'S0' on 18001
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Starting flush timer
Feb 26 14:29:13 ho-nflo-p01 rwflowpack[1781]: 'S0': accepted connection from 172.19.255.31:52600, domain 0x1000001
Feb 26 14:29:13 ho-nflo-p01 rwflowpack[1781]: NetFlow V9 sequence number mismatch for domain 0x1000001, expecting 0x0000 received 0x2f70

We are configured such that rwflowpack would create directories for flows in /var/silk/data/ext2ext/2020/02 but we see no flow directories being created. I am at a loss to understand why we are failing to collect flows from the switch which is a Cisco Catalyst 9K running XE 16.03. Please help!

Thank you for your consideration.
-Kirk

Kirk Olson
Information Security Engineer
Direct: 920-224-7426
Toll Free: 800-558-3405 ext. 7426
[cid:16b8a6c7a1e6d227b41]<https://www.secura.net/>
website | blog | Facebook | Twitter | LinkedIn<https://www.secura.net/>
 <https://www.secura.net/>
Recognized among Ward’s Top 50 and rated A Excellent by A.M. Best. <https://www.secura.net/>
Confidentiality Note: This email may contain confidential and/or private information. If you received this email in error, please delete and notify sender.<https://www.secura.net/>