[netsa-tools-discuss] Setting up yaf, super_mediator and silk (on FreeBSD)

andreas scherrer ascherrer at gmail.com
Tue Nov 8 18:27:29 EST 2016


Hi Emily

Thank you very much for the quick and precise answer. Unfortunately I am 
still struggling with my setup; that is probably because I 
oversimplified in my question.

Actually I have two machines, a collector and an analysis machine, and I 
use (try to) rwsender/rwreceiver. Basically I am trying to add 
super_mediator to the mix which is described in figure 1.4 (page 12) in [2].

I believe this should work as follows:
a) yaf running on collector, producing --silk data
b) flowcap running on collector getting yaf data (following your advice 
I have switched to TCP localhost) and writing to a file/directory (i.e. 
producing a silk file)
c) rwsender waiting for rwreceiver to fetch the file
d) rwreceiver running on analysis machine fetches file (still in silk 
format)
e) super_mediator on analysis machine reads the (silk) file and produces 
multiple outputs: an IPFIX file for rwflowpack as well as DPI output (be 
it a file or load it into a database)
f) rwflowpack reads the IPFIX file that super_mediator produced

It seems that I get something conceptually wrong here, but I do not seem 
to grok it.

What I (now) see is super_mediator complaining:
"C1: Ignoring Packet: Illegal IPFIX Message version 0xdead; input is 
probably not an IPFIX Message stream."

=> yes, it is not IPFIX, it is silk... Kind of the inverse issue I had 
before :(

For example, [3] uses the "--silk" option when yaf is started and 
sending data to super_mediator (although not via file, but via TCP). 
This indicates that super_mediator can handle/read data that was 
produced by yaf using --silk, no? On the other hand, [4] and other 
sources seem to indicate that super_mediator only handles IPFIX. At 
least [4] states (section "Collector Block", title "PATH file path") 
"...should be the name of the IPFIX file to read and process.".

I want to do DNS (and maybe HTTP) logging. Does that mean that I need 
--silk or could I also do this with IPFIX "only"? This should be 
possible without --silk, right?

Does it mean I need to export the payload (in/with yaf)? I guess so, 
otherwise super_mediator (running on a different machine!) will not have 
the required information?

Hopefully it is possible to make sense from my text here! Any help/nudge 
in the right direction is greatly appreciated.

On a side note: is there an option to run super_mediator as a different 
user (like --become-user for yaf)? I am not sure if/how I would 
automatically start super_mediator as non-root at boot time without such 
a switch.

[2] http://tools.netsa.cert.org/silk/silk-install-handbook.pdf
[3] http://tools.netsa.cert.org/yaf/libyaf/yaf_sm_silk.html
[4] http://tools.netsa.cert.org/super_mediator/super_mediator.conf.html

On 8.11.16 14:52 , Emily Sarneso wrote:
> Hello Andreas,
>
> Thank you for your interest in our tools.  In order to have rwflowpack poll a directory for IPFIX files, make sure the input mode (--input-mode) is set to stream.  I suspect that it is currently set to fcfiles or respool as the log message indicates that it is expecting SiLK data, not IPFIX data.
>
> Alternatively, you can export IPFIX via TCP/UDP from super_mediator (instead of writing IPFIX to files and having rwflowpack poll a directory) by using the following EXPORTER block:
>
> EXPORTER UDP “S0”
>    HOST localhost
>    PORT 18001
>    FLOW_ONLY
> EXPORTER END
>
> There is a good tutorial for configuring the tools here: http://tools.netsa.cert.org/yaf/libyaf/yaf_sm_silk.html
>
>> BTW: when I change "export_payload to yes, I get the following error from yaf: "yaf terminating on error: End of message. Overrun on variable-length encode (need 2051 bytes, 402 available)”
>
> When exporting via UDP, the IPFIX message must be less than the MTU (typically 1420). When you enable export payload and max-payload is set to 2048, you are overrunning the available space in the IPFIX message.  I would suggest using TCP between YAF and super_mediator if you would like to export payload and DPI data.  Alternatively, you can reduce the max-payload OR max-export options in YAF to reduce the amount of data you are exporting. However, even if you reduce the max-payload or max-export options you may run into this same error message when exporting DPI data.  If you are required to use UDP, you should also change the “limit total” value in the yafDPIRules.conf to set a limit on the DPI data that YAF will export.
>
> Hope that helps.  Please let us know if you run into any other problems.
>
> Emily
>
>
>
>
> --------------------
> Emily Sarneso
> CMU/SEI/CERT
> ecoff at cert.org
> (412) 268-6313
>
>
>
>
>
>> On Nov 7, 2016, at 6:23 PM, andreas scherrer <ascherrer at gmail.com> wrote:
>>
>> Hi
>>
>> I am trying to set up yaf to collect flows directly from an interface (which only sees VLAN tagged traffic) and forward it to super_mediator to process the DPI information and forward the flow to rwflowpack/SiLK.
>>
>> Unfortunately rwflowpack does not seem to be happy with what it gets from super_mediator:
>>
>> -----
>> rwflowpack[69512]: File does not appear to be a SiLK data file '<filename>'
>> -----
>>
>> I saw on [1] that the header of a SiLK file should have "0xDEADBEEF" at the beginning. That does not seem to be the case for my files...
>>
>> -----
>> $ hexdump <filename>.med| head -1
>> 0000000 0a00 0001 2158 8905 0000 0000 0000 0000
>> -----
>>
>> That seems to be consistent.
>>
>> yaf is running with the following configuration (yaf.init file):
>>
>> -----
>> input = {inf = "re1", type="pcap"}
>> UDP_LOCAL_EXPORT = {host = "localhost", port = "9901", protocol="udp"}
>> output = UDP_LOCAL_EXPORT
>> decode = {gre = false, ip4_only=false, ip6_only=false, nofrag=false}
>> export = {silk = true, mac = true}
>> applabel = true
>> applabel_rules = "/usr/local/etc/yafApplabelRules.conf"
>> maxpayload = 2048
>> export_payload = false
>> udp_uniflow = 53
>> DPI_PLUGIN = {name = "/usr/local/lib/yaf/dpacketplugin.so",
>>              conf="/usr/local/etc/yafDPIRules.conf"}
>> DHCP_PLUGIN = {name = "/usr/local/lib/yaf/dhcp_fp_plugin.so"}
>> plugin = {DPI_PLUGIN, DHCP_PLUGIN}
>> PCAP_EXPORT = {path = "/tmp/pcap", maxpcap=25, pcap_timer=300, meta="/tmp/meta"}
>> log = {spec = "/tmp/yaflog.log", level="debug"}
>> -----
>>
>> And started using the following command line:
>>
>> -----
>> yaf -c /usr/local/etc/yaf.init --become-user foo --become-group foo
>> -----
>>
>> BTW: when I change "export_payload to yes, I get the following error from yaf: "yaf terminating on error: End of message. Overrun on variable-length encode (need 2051 bytes, 402 available)"
>>
>> super_mediator is running with the following config (file):
>>
>> -----
>> COLLECTOR UDP
>>   PORT 9901
>> COLLECTOR END
>>
>> EXPORTER FILEHANDLER "S0"
>>   PATH "/var/spool/silk/destination/"
>>   ROTATE 10
>>   FLOW_ONLY
>> EXPORTER END
>>
>> LOGLEVEL DEBUG
>>
>> LOG "/var/log/super_mediator.log"
>>
>> PIDFILE "/var/run/super_mediator.pid"
>> -----
>>
>> And is started with the following command line:
>>
>> -----
>> super_mediator -c /usr/local/etc/super_mediator.conf
>> -----
>>
>> I am doing this on FreeBSD (10.x).
>>
>> $ yaf --version
>> yaf version 2.8.4
>> $ super_mediator --version
>> super_mediator version 1.3.0
>>
>>
>> Any hint would be greatly appreciated!
>> andreas
>>
>> Ps.: My set up is working *without* super_mediator (meaning sending data directly from yaf to rwflowpack)
>>
>> [1] https://tools.netsa.cert.org/silk/faq.html#file-header


More information about the netsa-tools-discuss mailing list