[netsa-tools-discuss] rwflowpack - crash after receiving template

Sat Mar 28 19:31:14 EDT 2015

rwflowpack appears to crash after receiving Netflow 9 template.

SiLK 3.10.1

Libfix 1.6.1

Rflowpack log:

Mar 28 19:50:39 scnrpt01 rwflowpack[35767]: 'S0': Ignoring NetFlowV9 record:
No Templates Present for this session. 8 Flows Lost.

Mar 28 19:50:41 scnrpt01 rwflowpack[35767]: NetFlow V9 Record Count
Discrepancy. Reported: 2. Found: 13.

After this message all logs stop. Service status reports dead.

service rwflowpack status

rwflowpack is dead but pid file exists

I've turned up logging to debug, but there seems to be no further
information in the logs and I'm not sure where else to look.

rwflowpack is configured to receive netflow records from my Fortigate 100D
at version 9.

root at scnrpt01:/etc/nsm/scnrpt01-eth0# cat silk-sensors.conf

probe S0 netflow-v9

   listen-on-port 2055

   protocol udp

end probe

group local-networks

   ipblocks 10.0.0.0/8

   ipblocks 172.16.0.0/12

   ipblocks 192.168.0.0/16

end group

sensor S0

   netflow-v9-probes S0

   internal-ipblocks @local-networks

   external-ipblocks remainder

end sensor

root at scnrpt01:/etc/nsm/scnrpt01-eth0# cat rwflowpack.conf

### Packer configuration file  -*- sh -*-

##

## The canonical pathname for this file is /usr/local/etc/rwflowpack.conf

##

## RCSIDENT("$SiLK: rwflowpack.conf.in 84f29bc8b9af 2013-03-02 12:01:19Z
mthomas $")

##

## This is a /bin/sh file that gets loaded by the init.d/rwflowpack

## wrapper script, and this file must follow /bin/sh syntax rules.

# Set to non-empty value to enable rwflowpack

ENABLED=yes

# These are convenience variables for setting other values in this

# configuration file; their use is not required.

statedirectory=/nsm/sensor_data/scnrpt01-eth0/silk

# If CREATE_DIRECTORIES is set to "yes", the directories named in this

# file will be created automatically if they do not already exist

CREATE_DIRECTORIES=yes

# Full path of the directory containing the "rwflowpack" program

BIN_DIR=/usr/local/sbin

# The full path to the sensor configuration file.  Used by

# --sensor-configuration.  YOU MUST PROVIDE THIS (the value is ignored

# when INPUT_MODE is "respool").

SENSOR_CONFIG=/etc/nsm/scnrpt01-eth0/silk-sensors.conf

# The full path to the root of the tree under which the packed SiLK

# Flow files will be written.  Used by --root-directory.

DATA_ROOTDIR=/nsm/sensor_data/scnrpt01-eth0/silk

# The full path to the site configuration file.  Used by

# --site-config-file.  If not set, defaults to silk.conf in the

# ${DATA_ROOTDIR}.

SITE_CONFIG=/etc/nsm/scnrpt01-eth0/silk.conf

# Specify the path to the packing-logic plug-in that rwflowpack should

# load and use.  The plug-in provides functions that determine into

# which class and type each flow record will be categorized and the

# format of the files that rwflowpack will write.  When SiLK has been

# configured with hard-coded packing logic (i.e., when

# --enable-packing-logic was specified to the configure script), this

# value should be empty.  A default value for this switch may be

# specified in the ${SITE_CONFIG} site configuration file.  This value

# is ignored when INPUT_MODE is "respool".

PACKING_LOGIC=

# Data input mode.  Valid values are:

#  * "stream" mode to read from the network or from probes that have

#    poll-directories

#  * "fcfiles" to process flowcap files on the local disk

#  * "respool" to process SiLK flow files maintaining the sensor and

#    class/type values that already exist on those records.

INPUT_MODE=stream

# Directory in which to look for incoming flowcap files in "fcfiles"

# mode or for incoming SiLK files in "respool" mode

INCOMING_DIR=${statedirectory}/incoming

# Directory to move input files to after successful processing.  When

# in "stream" mode, these are the files passed to any probe with a

# poll-directory directive.  When in "fcfiles" mode, these are the

# flowcap files.  When in "respool" mode, these are the SiLK Flow

# files.  If not set, the input files are not archived but are deleted

# instead.

ARCHIVE_DIR=${statedirectory}/archive

# When using the ARCHIVE_DIR, normally files are stored in

# subdirectories of the ARCHIVE_DIR.  If this variable's value is 1,

# files are stored in ARCHIVE_DIR itself, not in subdirectories of it.

FLAT_ARCHIVE=0

# Directory to move an input file into if there is a problem opening

# the file.  If this value is not set, rwflowpack will exit when it

# encounters a problem file.  When in "fcfiles" mode, these are the

# flowcap files.  When in "stream" mode, these are the files passed to

# any probe with a poll-directory directive.

ERROR_DIR=${statedirectory}/error

# Data output mode.  As of SiLK-3.6.0, valid values are

# "local-storage", "incremental-files", and "sending".

#

# For compatiblity with previous releases prior to SiLK-3.6.0, "local"

# is an alias for "local-storage" and "remote" and is an alias for

# "sending".

#

# In "local-storage" (aka "local") mode, rwflowpack writes the records

# to hourly files in the repository on the local disk.  The root of

# the repository must be specified by the DATA_ROOTDIR variable.

#

# In "incremental-files" mode, rwflowpack creates small files (called

# incremental files) that must be processed by rwflowappend to create

# the hourly files.  The incremental-files are created and stored in a

# single directory named by the INCREMENTAL_DIR variable.

#

# In "sending" (aka "remote") mode, rwflowpack also creates

# incremental files.  The files are created in directory specified by

# the INCREMENTAL_DIR variable and then moved to directory specified

# by the SENDER_DIR variable.

OUTPUT_MODE=local-storage

# When the OUTPUT_MODE is "sending", this is the destination directory

# in which the incremental files are finally stored to await

# processing by rwflowappend, rwsender, or another process.

SENDER_DIR=${statedirectory}/sender-incoming

# When OUTPUT_MODE is "incremental-files" or "sending", this is the

# directory where the incremental files are initially built.  In

# "incremental-files" mode, the files remain in this directory.  In

# "sending" mode, the incremental files are moved to the SENDER_DIR

# directory.

INCREMENTAL_DIR=${statedirectory}/sender-incoming

# The type of compression to use for packed files.  Left empty, the

# value chosen at compilation time will be used.  Valid values are

# "best" and "none".  Other values are system-specific (the available

# values are listed in the description of the --compression-method

# switch in the output of rwflowpack --help).

COMPRESSION_TYPE=

# Interval between attempts to check the INCOMING_DIR or

# poll-directory probe entries for new files, in seconds.  This may be

# left blank, and will default to 15.

POLLING_INTERVAL=

# Interval between periodic flushes of open SiLK Flow files to disk,

# in seconds.  This may be left blank, and will default to 120.

FLUSH_TIMEOUT=

# Maximum number of SiLK Flow files to have open for writing

# simultaneously.  This may be left blank, and will default to 64

FILE_CACHE_SIZE=

# Whether rwflowpack should use advisory write locks.  1=yes, 0=no.

# Set to zero if messages like "Cannot get a write lock on file"

# appear in rwflowpack's log file.

FILE_LOCKING=1

# Whether rwflowpack should include the input and output SNMP

# interfaces and the next-hop-ip in the output files.  1=yes, 0=no.

# The default is no, and these values are not stored to save disk

# space.  (The input and output fields contain VLAN tags when the

# sensor.conf file contains the attribute "interface-values vlan".)

PACK_INTERFACES=0

###

# The type of logging to use.  Valid values are "legacy" and "syslog".

LOG_TYPE=legacy

# The lowest level of logging to actually log.  Valid values are:

# emerg, alert, crit, err, warning, notice, info, debug

LOG_LEVEL=debug

# The full path of the directory where the log files will be written

# when LOG_TYPE is "legacy".

LOG_DIR=/var/log/

# The full path of the directory where the PID file will be written

PID_DIR=${LOG_DIR}

# The user this program runs as; root permission is required only when

# rwflowpack listens on a privileged port.

USER=root

#USER=`whoami`  # run as user invoking the script

# Extra options to pass to rwflowpack

EXTRA_OPTIONS=

root at scnrpt01:/etc/nsm/scnrpt01-eth0# cat silk.conf

# silk.conf for the "twoway" site

# RCSIDENT("$SiLK: silk.conf 52d8f4f62ffd 2012-05-25 21:16:30Z mthomas $")

# For a description of the syntax of this file, see silk.conf(5).

# The syntactic format of this file

#    version 2 supports sensor descriptions, but otherwise identical to 1

version 2

# NOTE: Once data has been collected for a sensor or a flowtype, the

# sensor or flowtype should never be removed or renumbered.  SiLK Flow

# files store the sensor ID and flowtype ID as integers; removing or

# renumbering a sensor or flowtype breaks this mapping.

sensor 0 S0    "Description for sensor S0"

sensor 1 S1

sensor 2 S2    "Optional description for sensor S2"

sensor 3 S3

sensor 4 S4

sensor 5 S5

sensor 6 S6

sensor 7 S7

sensor 8 S8

sensor 9 S9

sensor 10 S10

sensor 11 S11

sensor 12 S12

sensor 13 S13

sensor 14 S14

class all

    sensors S0 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14

end class

# Editing above this line is sufficient for sensor definition.

# Be sure you understand the workings of the packing system before

# editing the class and type definitions below.  In particular, if you

# change or add-to the following, the C code in packlogic-twoway.c

# will need to change as well.

class all

    type  0 in      in

    type  1 out     out

    type  2 inweb   iw

    type  3 outweb  ow

    type  4 innull  innull

    type  5 outnull outnull

    type  6 int2int int2int

    type  7 ext2ext ext2ext

    type  8 inicmp  inicmp

    type  9 outicmp outicmp

    type 10 other   other

    default-types in inweb inicmp

end class

default-class all

# The layout of the tree below SILK_DATA_ROOTDIR.

# Use the default, which assumes a single class.

# path-format "%T/%Y/%m/%d/%x"

# The plug-in to load to get the packing logic to use in rwflowpack.

# The --packing-logic switch to rwflowpack will override this value.

# If SiLK was configured with hard-coded packing logic, this value is

# ignored.

packing-logic "packlogic-twoway.so"

-------------- next part --------------
HTML attachment scrubbed and removed