[netsa-tools-discuss] where do we go from here?

[Mark Thomas]

Kirk-

The directory structure that the SiLK tools use for storing and retrieving records is defined in the silk.conf file with the path-format setting.

It looks like your setting for path-format is the default:

  path-format "%T/%Y/%m/%d/%x"

To have a separate directory per sensor, you need to add a "%N/" to the path:

  path-format "%N/%T/%Y/%m/%d/%x"

which will produce the path:

  /var/silk/data/MPSW1/int2int/2020/03/19

After you edit the silk.conf file, restart rwflowpack and it will begin to write records into the new location.  However, you may want to move the old files to the new location before you restart rwflowpack.

Assuming you are not running rwflowpack while moving files, you can use the following depending on the number of sensors you have collecting data:

If you are only using one sensor at your site, you can simply move the entire directory structure like so:

  cd /var/silk
  mv data MPSW1
  mkdir data
  mv MPSW1/silk.conf data/.
  mv MPSW1 data/. 

If you have multiple sensors, it becomes more cumbersome since the files for each sensor need to be moved into a separate top-level directory.  The following shell code should work, but it is not the most efficient way to move the files:

  cd /var/silk/data
  SENSORS=`rwsiteinfo --site-conf silk.conf --fields=sensor:list --no-title --delimited --list-delim=' '`
  TYPES=`rwsiteinfo --site-conf silk.conf --fields=type:list --no-title --delimited --list-delim=' '`

  mkdir -p $SENSORS
  for sensor in $SENSORS ; do
    find $TYPES -name '*-'$sensor'_*' -print0 | \
      xargs -0 -I '%' mv % $sensor/%
  done
  rmdir -p $SENSORS

The find command will complain about types for which there is no data; this is expected.

The final rmdir removes any sensor directories for which you have no data.  It will complain that it cannot remove directories that contain data; this is also expected.

I hope that helps.

-Mark


-----Original Message-----
From: Kirk Olson <Kirk_Olson at secura.net>
Date: Thu, 19 Mar 2020 19:37:07 +0000
To: "netsa-tools-discuss at cert.org" <netsa-tools-discuss at cert.org>
Cc: Jacob Farrell <Jacob_Farrell at secura.net>
Subject: [netsa-tools-discuss] where do we go from here?

Hello and thank you for reviewing my post.

The rwflowpack service is up and running. We are packing files. Here is an example:

root at ho-nflo-p01:/var/silk/data/int2int/2020/03/19
# ls -l int2int-MPSW1*
-rw-r--r--. 1 root root 639631 Mar 18 20:01 int2int-MPSW1_20200319.00
-rw-r--r--. 1 root root 642971 Mar 18 21:01 int2int-MPSW1_20200319.01
-rw-r--r--. 1 root root 707284 Mar 18 22:01 int2int-MPSW1_20200319.02
-rw-r--r--. 1 root root 631120 Mar 18 23:01 int2int-MPSW1_20200319.03
-rw-r--r--. 1 root root 635771 Mar 19 00:01 int2int-MPSW1_20200319.04
-rw-r--r--. 1 root root 632680 Mar 19 01:01 int2int-MPSW1_20200319.05
-rw-r--r--. 1 root root 627672 Mar 19 02:01 int2int-MPSW1_20200319.06
-rw-r--r--. 1 root root 636847 Mar 19 03:01 int2int-MPSW1_20200319.07
-rw-r--r--. 1 root root 634166 Mar 19 04:01 int2int-MPSW1_20200319.08
-rw-r--r--. 1 root root 630384 Mar 19 05:01 int2int-MPSW1_20200319.09
-rw-r--r--. 1 root root 637808 Mar 19 06:01 int2int-MPSW1_20200319.10
...etc

We intend to use Joe Loiacono's FlowViewer/FlowGrapher/FlowMonitor
toolset to report on the data. However, it is clear the flow
information is not written in a directory structure that this toolset
will understand. It appears that the files are not being packed into a
directory named after the probe device. I am struggling to understand
if that is rwflowpack's job or if we are supposed to be running
rwflowappend to write the final directory and hourly flow files from
the incremental flow files.

I have looked high and low for an architecture to guide me with no luck. Can you folks provide some guidance? Thank you.
-Kirk