[netsa-tools-discuss] NetFlow V9 sequence number mismatch

[Kirk Olson]

Follow-up…

We are using the following rwflowpack.conf:
ENABLED=yes
statedirectory=/var/silk
CREATE_DIRECTORIES=yes
BIN_DIR=/usr/sbin
SENSOR_CONFIG=/var/silk/sensors.conf
DATA_ROOTDIR=/var/silk/data
SITE_CONFIG=
PACKING_LOGIC=
INPUT_MODE=stream
INCOMING_DIR=${statedirectory}/incoming
ARCHIVE_DIR= #empty
FLAT_ARCHIVE=0
ERROR_DIR=  #${statedirectory}/error
OUTPUT_MODE=local-storage
SENDER_DIR=${statedirectory}/sender-incoming
INCREMENTAL_DIR=${statedirectory}/sender-incoming
COMPRESSION_TYPE=
POLLING_INTERVAL=
FLUSH_TIMEOUT=
FILE_CACHE_SIZE=
FILE_LOCKING=1
PACK_INTERFACES=0
SILK_IPFIX_PRINT_TEMPLATES=
LOG_TYPE=legacy
LOG_LEVEL=info
LOG_DIR=/var/log
PID_DIR=${statedirectory}
USER=root
EXTRA_OPTIONS=
EXTRA_ENVVAR=

We are using the following sensors.conf:

probe S0 netflow-v9
listen-on-port 18001
protocol udp
end probe

group secura-private-space
ipblocks 10.19.199.0/24  # ACI management network
ipblocks 10.102.0.0/21   # Wireless Guest
ipblocks 10.103.0.0/21   # Wireless Guest
…a bunch more private networks
end group

group secura-internet-space
ipblocks 98.100.228.0/24
end group

sensor S0
netflow-v9-probes S0
internal-ipblock @secura-private-space
external-ipblock remainder #changed this during troubleshooting from secura-internet-space
end sensor

We also made the following buffering changes on the system:
  echo 'net.core.wmem_max=12582912' >> /etc/sysctl.conf
  echo 'net.core.rmem_max=12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_rmem= 10240 87380 12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_wmem= 10240 87380 12582912' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_window_scaling = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_timestamps = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_sack = 1' >> /etc/sysctl.conf
  echo 'net.ipv4.tcp_no_metrics_save = 1' >> /etc/sysctl.conf
  echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf

These are the libfixbuf versions on the system:
libfixbuf-2.4.0-1.el7.x86_64
libfixbuf-devel-2.4.0-1.el7.x86_64
libfixbuf-ipfixDump-2.4.0-1.el7.x86_64






From: Kirk Olson
Sent: Wednesday, February 26, 2020 2:43 PM
To: 'netsa-tools-discuss at cert.org' <netsa-tools-discuss at cert.org>
Subject: NetFlow V9 sequence number mismatch

Hello and thank you for reviewing my post.

We intend to run a single machine configuration using rwflowpack and SiLK on RHEL ver 7.

The installation on RHEL seems to go fine, as a check of the service status shows:
# systemctl status rwflowpack
● rwflowpack.service - LSB: start and stop SiLK rwflowpack daemon
   Loaded: loaded (/etc/rc.d/init.d/rwflowpack; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2020-02-26 14:29:12 CST; 2min 26s ago
     Docs: man:systemd-sysv-generator(8)
 Process: 1643 ExecStart=/etc/rc.d/init.d/rwflowpack start (code=exited, status=0/SUCCESS)

Feb 26 14:29:11 ho-nflo-p01.intranet.secura.net systemd[1]: Starting LSB: start and stop SiLK rwflowpack daemon...
Feb 26 14:29:12 ho-nflo-p01.intranet.secura.net rwflowpack[1643]: Starting rwflowpack:        [OK]
Feb 26 14:29:12 ho-nflo-p01.intranet.secura.net systemd[1]: Started LSB: start and stop SiLK rwflowpack daemon.

Looking into the log I see the following:
Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: Started logging at 2020-02-26 20:29:11Z
Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: '/usr/sbin/rwflowpack' '--sensor-configuration=/var/silk/sensors.conf' '--output-mode=local-storage' '--root-directory=/var/silk/data' '--pidfile=/var/silk/rwflowpack.pid' '--log-level=info' '--log-directory=/var/log' '--log-basename=rwflowpack'
Feb 26 14:29:11 ho-nflo-p01 rwflowpack[1762]: Forked child 1781.  Parent exiting
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Using packing logic from /usr/lib64/silk/packlogic-twoway.so
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Creating stream cache
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Creating NetFlowV9 Reader for probe 'S0' on 18001
Feb 26 14:29:12 ho-nflo-p01 rwflowpack[1781]: Starting flush timer
Feb 26 14:29:13 ho-nflo-p01 rwflowpack[1781]: 'S0': accepted connection from 172.19.255.31:52600, domain 0x1000001
Feb 26 14:29:13 ho-nflo-p01 rwflowpack[1781]: NetFlow V9 sequence number mismatch for domain 0x1000001, expecting 0x0000 received 0x2f70

We are configured such that rwflowpack would create directories for flows in /var/silk/data/ext2ext/2020/02 but we see no flow directories being created. I am at a loss to understand why we are failing to collect flows from the switch which is a Cisco Catalyst 9K running XE 16.03. Please help!

Thank you for your consideration.
-Kirk


Kirk Olson
Information Security Engineer
Direct: 920-224-7426
Toll Free: 800-558-3405 ext. 7426
[cid:16b8a6c7a1e6d227b41]<https://www.secura.net/>
website | blog | Facebook | Twitter | LinkedIn<https://www.secura.net/>
 <https://www.secura.net/>
Recognized among Ward’s Top 50 and rated A Excellent by A.M. Best. <https://www.secura.net/>
Confidentiality Note: This email may contain confidential and/or private information. If you received this email in error, please delete and notify sender.<https://www.secura.net/>
-------------- next part --------------
HTML attachment scrubbed and removed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 3703 bytes
Desc: image001.jpg
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20200227/9fa57cf6/attachment.jpg>