[netsa-tools-discuss] results with rwfilter

Tue Apr 14 14:48:44 EDT 2020

# cat /etc/sysconfig/rwflowpack.conf
### Packer configuration file  -*- sh -*-
##
## The canonical pathname for this file is
## /etc/rwflowpack.conf
##
## RCSIDENT("$SiLK: rwflowpack.conf.in 60e5dccfed7c 2015-09-23 20:32:54Z mthomas $")
##
## This is a /bin/sh file that gets loaded by the init.d/rwflowpack
## wrapper script, and this file must follow /bin/sh syntax rules.

# Set to non-empty value to enable rwflowpack
ENABLED=yes

# These are convenience variables for setting other values in this
# configuration file; their use is not required.
# statedirectory=/var/lib/rwflowpack
statedirectory=/var/silk

# If CREATE_DIRECTORIES is set to "yes", the directories named in this
# file will be created automatically if they do not already exist
CREATE_DIRECTORIES=yes

# Full path of the directory containing the "rwflowpack" program
BIN_DIR=/usr/sbin

# The full path to the sensor configuration file.  Used by
# --sensor-configuration.  YOU MUST PROVIDE THIS (the value is ignored
# when INPUT_MODE is "respool").
SENSOR_CONFIG=/var/silk/sensors.conf

# The full path to the root of the tree under which the packed SiLK
# Flow files will be written.  Used by --root-directory.
DATA_ROOTDIR=/var/silk/data

# The full path to the site configuration file.  Used by
# --site-config-file.  If not set, defaults to silk.conf in the
# ${DATA_ROOTDIR}.
SITE_CONFIG=

# Specify the path to the packing-logic plug-in that rwflowpack should
# load and use.  The plug-in provides functions that determine into
# which class and type each flow record will be categorized and the
# format of the files that rwflowpack will write.  When SiLK has been
# configured with hard-coded packing logic (i.e., when
# --enable-packing-logic was specified to the configure script), this
# value should be empty.  A default value for this switch may be
# specified in the ${SITE_CONFIG} site configuration file.  This value
# is ignored when INPUT_MODE is "respool".
PACKING_LOGIC=

# Data input mode.  Valid values are:
#  * "stream" mode to read from the network or from probes that have
#    poll-directories
#  * "fcfiles" to process flowcap files on the local disk
#  * "respool" to process SiLK flow files maintaining the sensor and
#    class/type values that already exist on those records.
INPUT_MODE=stream

# Directory in which to look for incoming flowcap files in "fcfiles"
# mode or for incoming SiLK files in "respool" mode
INCOMING_DIR=${statedirectory}/incoming

# Directory to move input files to after successful processing.  When
# in "stream" mode, these are the files passed to any probe with a
# poll-directory directive.  When in "fcfiles" mode, these are the
# flowcap files.  When in "respool" mode, these are the SiLK Flow
# files.  If not set, the input files are not archived but are deleted
# instead.
#ARCHIVE_DIR=${statedirectory}/archive
ARCHIVE_DIR= #empty

# When using the ARCHIVE_DIR, normally files are stored in
# subdirectories of the ARCHIVE_DIR.  If this variable's value is 1,
# files are stored in ARCHIVE_DIR itself, not in subdirectories of it.
FLAT_ARCHIVE=0

# Directory to move an input file into if there is a problem opening
# the file.  If this value is not set, rwflowpack will exit when it
# encounters a problem file.  When in "fcfiles" mode, these are the
# flowcap files.  When in "stream" mode, these are the files passed to
# any probe with a poll-directory directive.
ERROR_DIR=  #${statedirectory}/error

# Data output mode.  As of SiLK-3.6.0, valid values are
# "local-storage", "incremental-files", and "sending".
#
# For compatiblity with previous releases prior to SiLK-3.6.0, "local"
# is an alias for "local-storage" and "remote" and is an alias for
# "sending".
#
# In "local-storage" (aka "local") mode, rwflowpack writes the records
# to hourly files in the repository on the local disk.  The root of
# the repository must be specified by the DATA_ROOTDIR variable.
#
# In "incremental-files" mode, rwflowpack creates small files (called
# incremental files) that must be processed by rwflowappend to create
# the hourly files.  The incremental-files are created and stored in a
# single directory named by the INCREMENTAL_DIR variable.
#
# In "sending" (aka "remote") mode, rwflowpack also creates
# incremental files.  The files are created in directory specified by
# the INCREMENTAL_DIR variable and then moved to directory specified
# by the SENDER_DIR variable.
OUTPUT_MODE=local-storage

# When the OUTPUT_MODE is "sending", this is the destination directory
# in which the incremental files are finally stored to await
# processing by rwflowappend, rwsender, or another process.
SENDER_DIR=${statedirectory}/sender-incoming

# When OUTPUT_MODE is "incremental-files" or "sending", this is the
# directory where the incremental files are initially built.  In
# "incremental-files" mode, the files remain in this directory.  In
# "sending" mode, the incremental files are moved to the SENDER_DIR
# directory.
INCREMENTAL_DIR=${statedirectory}/sender-incoming

# The type of compression to use for packed files.  Left empty, the
# value chosen at compilation time will be used.  Valid values are
# "best" and "none".  Other values are system-specific (the available
# values are listed in the description of the --compression-method
# switch in the output of rwflowpack --help).
COMPRESSION_TYPE=

# Interval between attempts to check the INCOMING_DIR or
# poll-directory probe entries for new files, in seconds.  This may be
# left blank, and will default to 15.
POLLING_INTERVAL=

# Interval between periodic flushes of open SiLK Flow files to disk,
# in seconds.  This may be left blank, and will default to 120.
FLUSH_TIMEOUT=

# Maximum number of SiLK Flow files to have open for writing
# simultaneously.  This may be left blank, and will default to 64
FILE_CACHE_SIZE=

# Whether rwflowpack should use advisory write locks.  1=yes, 0=no.
# Set to zero if messages like "Cannot get a write lock on file"
# appear in rwflowpack's log file.
FILE_LOCKING=1

# Whether rwflowpack should include the input and output SNMP
# interfaces and the next-hop-ip in the output files.  1=yes, 0=no.
# The default is no, and these values are not stored to save disk
# space.  (The input and output fields contain VLAN tags when the
# sensor.conf file contains the attribute "interface-values vlan".)
PACK_INTERFACES=0

# Setting this environment variable to 1 causes rwflowpack to log the
# NetFlowV9/IPFIX templates that it receives.
SILK_IPFIX_PRINT_TEMPLATES=1

###

# The type of logging to use.  Valid values are "legacy" and "syslog".
#LOG_TYPE=syslog
LOG_TYPE=legacy

# The lowest level of logging to actually log.  Valid values are:
# emerg, alert, crit, err, warning, notice, info, debug
LOG_LEVEL=debug

# The full path of the directory where the log files will be written
# when LOG_TYPE is "legacy".
#LOG_DIR=${statedirectory}/log
LOG_DIR=/var/log

# The full path of the directory where the PID file will be written
PID_DIR=${statedirectory}

# The user this program runs as; root permission is required only when
# rwflowpack listens on a privileged port.
USER=root
#USER=`whoami`  # run as user invoking the script

# Extra options to pass to rwflowpack
EXTRA_OPTIONS=

# Extra environment variables to set when running rwflowpack.  These
# should be specified as VAR=value pairs as shown here:
#EXTRA_ENVVAR='FOO=1 BAR=baz'
EXTRA_ENVVAR=

From: Timur David Snoke <tdsnoke at cert.org>
Sent: Tuesday, April 14, 2020 12:24 PM
To: Kirk Olson <Kirk_Olson at secura.net>; Angela Horneman <ahorneman at cert.org>; netsa-tools-discuss at cert.org
Subject: Re: [netsa-tools-discuss] results with rwfilter

Hi Kirk,

Your selection criteria might be too limiting, in the example that you presented you are looking for any UDP traffic to or from 172.18.18.151 between April 10 at 09:00 and April 12 at 11:00.

Try something that you know will get a result for a shorter time period to get a quicker validation that you have content.

rwfilter --start=2020/04/10T09 --sensor=ACIDVS --type=all  --protocol=0- --pass=/tmp/netstat/ACIDVS-2020041009.rw --site-config-file=/var/silk/data/silk.conf

This will query for all traffic seen by the sensor during the hour of April 10 at 09:00 and write it to a file.

You can also look at your data repository for the date/time in question and confirm that you have content.

-Timur Snoke

From: <netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org<mailto:netsa-tools-discuss-bounces+tdsnoke=cert.org at cert.org>> on behalf of Kirk Olson <Kirk_Olson at secura.net<mailto:Kirk_Olson at secura.net>>
Date: Tuesday, April 14, 2020 at 1:07 PM
To: Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>>, "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>" <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>
Subject: Re: [netsa-tools-discuss] results with rwfilter

My apologies Angela, this is the command which returns no data:

rwfilter --start=2020/04/10T09 --end=2020/04/12T11 --sensor=ACIDVS --type=all --any-address=172.18.18.151 --protocol=17 --pass=/tmp/netstat/HOCA01udp.rw --site-config-file=/var/silk/data/silk.conf

From: Angela Horneman <ahorneman at cert.org<mailto:ahorneman at cert.org>>
Sent: Tuesday, April 14, 2020 11:52 AM
To: Kirk Olson <Kirk_Olson at secura.net<mailto:Kirk_Olson at secura.net>>; netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: Re: [netsa-tools-discuss] results with rwfilter

Hi Kirk,

In the command below you have a start year of 2020 and an end of 2015.

Angela Horneman
Situational Awareness Analysis Team Lead
CMU/SEI/CERT

From: <netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org<mailto:netsa-tools-discuss-bounces+ahorneman=cert.org at cert.org>> on behalf of Kirk Olson <Kirk_Olson at secura.net<mailto:Kirk_Olson at secura.net>>
Date: Tuesday, April 14, 2020 at 12:46 PM
To: "netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>" <netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>>
Subject: [netsa-tools-discuss] results with rwfilter

I have been using the following rwfilter command to pull data from a sensor named ACIDVS:

rwfilter --start=2020/04/10T09 --end=2015/06/17T11 --sensor=ACIDVS --type=all --any-address=172.18.18.151 --protocol=17 --pass=HOCA01udp.rw

rwfilter does build a resultant file with headers in the top row but there is no data from the sensor in the file. Is there something simple I am missing here? I have read the docs and it is not obvious to me where I might be going wrong.

Thank you for your time.
-Kirk

Kirk Olson
Information Security Engineer
Direct: 920-224-7426
Toll Free: 800-558-3405 ext. 7426
[cid:16b8a6c7a1e6d227b41]<https://www.secura.net/>
website | blog | Facebook | Twitter | LinkedIn<https://www.secura.net/>
 <https://www.secura.net/>
Recognized among Ward’s Top 50 and rated A Excellent by A.M. Best. <https://www.secura.net/>
Confidentiality Note: This email may contain confidential and/or private information. If you received this email in error, please delete and notify sender.<https://www.secura.net/>
-------------- next part --------------
HTML attachment scrubbed and removed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 3705 bytes
Desc: image001.jpg
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20200414/dddb915d/attachment.jpg>