[netsa-tools-discuss] Setting up yaf, super_mediator and silk (on FreeBSD)

[andreas scherrer]

Hi

I am trying to set up yaf to collect flows directly from an interface 
(which only sees VLAN tagged traffic) and forward it to super_mediator 
to process the DPI information and forward the flow to rwflowpack/SiLK.

Unfortunately rwflowpack does not seem to be happy with what it gets 
from super_mediator:

-----
rwflowpack[69512]: File does not appear to be a SiLK data file '<filename>'
-----

I saw on [1] that the header of a SiLK file should have "0xDEADBEEF" at 
the beginning. That does not seem to be the case for my files...

-----
$ hexdump <filename>.med| head -1
0000000 0a00 0001 2158 8905 0000 0000 0000 0000
-----

That seems to be consistent.

yaf is running with the following configuration (yaf.init file):

-----
input = {inf = "re1", type="pcap"}
UDP_LOCAL_EXPORT = {host = "localhost", port = "9901", protocol="udp"}
output = UDP_LOCAL_EXPORT
decode = {gre = false, ip4_only=false, ip6_only=false, nofrag=false}
export = {silk = true, mac = true}
applabel = true
applabel_rules = "/usr/local/etc/yafApplabelRules.conf"
maxpayload = 2048
export_payload = false
udp_uniflow = 53
DPI_PLUGIN = {name = "/usr/local/lib/yaf/dpacketplugin.so",
               conf="/usr/local/etc/yafDPIRules.conf"}
DHCP_PLUGIN = {name = "/usr/local/lib/yaf/dhcp_fp_plugin.so"}
plugin = {DPI_PLUGIN, DHCP_PLUGIN}
PCAP_EXPORT = {path = "/tmp/pcap", maxpcap=25, pcap_timer=300, 
meta="/tmp/meta"}
log = {spec = "/tmp/yaflog.log", level="debug"}
-----

And started using the following command line:

-----
yaf -c /usr/local/etc/yaf.init --become-user foo --become-group foo
-----

BTW: when I change "export_payload to yes, I get the following error 
from yaf: "yaf terminating on error: End of message. Overrun on 
variable-length encode (need 2051 bytes, 402 available)"

super_mediator is running with the following config (file):

-----
COLLECTOR UDP
    PORT 9901
COLLECTOR END

EXPORTER FILEHANDLER "S0"
    PATH "/var/spool/silk/destination/"
    ROTATE 10
    FLOW_ONLY
EXPORTER END

LOGLEVEL DEBUG

LOG "/var/log/super_mediator.log"

PIDFILE "/var/run/super_mediator.pid"
-----

And is started with the following command line:

-----
super_mediator -c /usr/local/etc/super_mediator.conf
-----

I am doing this on FreeBSD (10.x).

$ yaf --version
yaf version 2.8.4
$ super_mediator --version
super_mediator version 1.3.0


Any hint would be greatly appreciated!
andreas

Ps.: My set up is working *without* super_mediator (meaning sending data 
directly from yaf to rwflowpack)

[1] https://tools.netsa.cert.org/silk/faq.html#file-header