[netsa-tools-discuss] Setting up yaf, super_mediator and silk (on FreeBSD)

[Emily Sarneso]

Hello Andreas,

Thank you for your interest in our tools.  In order to have rwflowpack poll a directory for IPFIX files, make sure the input mode (--input-mode) is set to stream.  I suspect that it is currently set to fcfiles or respool as the log message indicates that it is expecting SiLK data, not IPFIX data.

Alternatively, you can export IPFIX via TCP/UDP from super_mediator (instead of writing IPFIX to files and having rwflowpack poll a directory) by using the following EXPORTER block:

EXPORTER UDP “S0”
   HOST localhost
   PORT 18001
   FLOW_ONLY
EXPORTER END

There is a good tutorial for configuring the tools here: http://tools.netsa.cert.org/yaf/libyaf/yaf_sm_silk.html

> BTW: when I change "export_payload to yes, I get the following error from yaf: "yaf terminating on error: End of message. Overrun on variable-length encode (need 2051 bytes, 402 available)”

When exporting via UDP, the IPFIX message must be less than the MTU (typically 1420). When you enable export payload and max-payload is set to 2048, you are overrunning the available space in the IPFIX message.  I would suggest using TCP between YAF and super_mediator if you would like to export payload and DPI data.  Alternatively, you can reduce the max-payload OR max-export options in YAF to reduce the amount of data you are exporting. However, even if you reduce the max-payload or max-export options you may run into this same error message when exporting DPI data.  If you are required to use UDP, you should also change the “limit total” value in the yafDPIRules.conf to set a limit on the DPI data that YAF will export.

Hope that helps.  Please let us know if you run into any other problems.

Emily




--------------------
Emily Sarneso
CMU/SEI/CERT
ecoff at cert.org
(412) 268-6313





> On Nov 7, 2016, at 6:23 PM, andreas scherrer <ascherrer at gmail.com> wrote:
> 
> Hi
> 
> I am trying to set up yaf to collect flows directly from an interface (which only sees VLAN tagged traffic) and forward it to super_mediator to process the DPI information and forward the flow to rwflowpack/SiLK.
> 
> Unfortunately rwflowpack does not seem to be happy with what it gets from super_mediator:
> 
> -----
> rwflowpack[69512]: File does not appear to be a SiLK data file '<filename>'
> -----
> 
> I saw on [1] that the header of a SiLK file should have "0xDEADBEEF" at the beginning. That does not seem to be the case for my files...
> 
> -----
> $ hexdump <filename>.med| head -1
> 0000000 0a00 0001 2158 8905 0000 0000 0000 0000
> -----
> 
> That seems to be consistent.
> 
> yaf is running with the following configuration (yaf.init file):
> 
> -----
> input = {inf = "re1", type="pcap"}
> UDP_LOCAL_EXPORT = {host = "localhost", port = "9901", protocol="udp"}
> output = UDP_LOCAL_EXPORT
> decode = {gre = false, ip4_only=false, ip6_only=false, nofrag=false}
> export = {silk = true, mac = true}
> applabel = true
> applabel_rules = "/usr/local/etc/yafApplabelRules.conf"
> maxpayload = 2048
> export_payload = false
> udp_uniflow = 53
> DPI_PLUGIN = {name = "/usr/local/lib/yaf/dpacketplugin.so",
>              conf="/usr/local/etc/yafDPIRules.conf"}
> DHCP_PLUGIN = {name = "/usr/local/lib/yaf/dhcp_fp_plugin.so"}
> plugin = {DPI_PLUGIN, DHCP_PLUGIN}
> PCAP_EXPORT = {path = "/tmp/pcap", maxpcap=25, pcap_timer=300, meta="/tmp/meta"}
> log = {spec = "/tmp/yaflog.log", level="debug"}
> -----
> 
> And started using the following command line:
> 
> -----
> yaf -c /usr/local/etc/yaf.init --become-user foo --become-group foo
> -----
> 
> BTW: when I change "export_payload to yes, I get the following error from yaf: "yaf terminating on error: End of message. Overrun on variable-length encode (need 2051 bytes, 402 available)"
> 
> super_mediator is running with the following config (file):
> 
> -----
> COLLECTOR UDP
>   PORT 9901
> COLLECTOR END
> 
> EXPORTER FILEHANDLER "S0"
>   PATH "/var/spool/silk/destination/"
>   ROTATE 10
>   FLOW_ONLY
> EXPORTER END
> 
> LOGLEVEL DEBUG
> 
> LOG "/var/log/super_mediator.log"
> 
> PIDFILE "/var/run/super_mediator.pid"
> -----
> 
> And is started with the following command line:
> 
> -----
> super_mediator -c /usr/local/etc/super_mediator.conf
> -----
> 
> I am doing this on FreeBSD (10.x).
> 
> $ yaf --version
> yaf version 2.8.4
> $ super_mediator --version
> super_mediator version 1.3.0
> 
> 
> Any hint would be greatly appreciated!
> andreas
> 
> Ps.: My set up is working *without* super_mediator (meaning sending data directly from yaf to rwflowpack)
> 
> [1] https://tools.netsa.cert.org/silk/faq.html#file-header
>