[netsa-tools-discuss] super_mediator TCP connection timeout

[Emily Sarneso]

You are correct about rwsender/rwreceiver.  rwsender simply transfers files to a rwreceiver running on another host.

The attached patch (for version 0.4.0) should solve your problem.  Once applied and super_mediator is recompiled and reinstalled, super_mediator will set any exporters that it cannot connect to inactive and continue running.  It will periodically retry to connect to the exporter.  It will log a message when it attempts to connect and whether the attempt was successful or not.

Hope that helps,

Emily




On Mar 26, 2015, at 1:08 PM, inetjunkmail <inetjunkmail at gmail.com> wrote:

> Thanks for the idea Adam.  Unfortunately, RSYNC would still require the receiver to use SiLK to interpret the data and we're trying to avoid that requirement.
>
> In our environment we function much like an ISP so we have multiple customers traversing our routers.  We are transitioning to SiLK as our IPFIX collector/analyzer but we also offer customers their specific flow data (based on IP address).  Today we use nfcapd to receive flow data from the routers then we run a cron job ever 5 minutes to read in the last 5 mins of flow data off disk and export it via nfreplay which is currently exporting the data in Neflow v9 format to the customers.  From the customer perspective, they just need to stand up any Netflow/IPFIX collector and we shoot them the data.  We can certainly change our export format to IPFIX but we'd like to avoid requiring a specific collector.
>
> Ultimately it looks like we may be able to user super_mediator if we choose to export via UDP; since it's connectionless, it doesn't hang on startup.  Ideally we'd come up with a way to enable the use of TCP for the reliability and the benefit of not having to wait around for a template refresh but I think that option is better than maintaining the two parallel solutions.
>
>
> On Thu, Mar 26, 2015 at 11:55 AM, Robertson, Adam (10020) <adam.robertson at protiviti.com> wrote:
> If I understand your need correctly, could you use ‘rsync’ on a ‘cron’ job to periodically send the data to the destination. It would mean that you would need the admin of the receiving server to make sure ‘rsync’ is installed (which I believe it is by default on a lot of Linux flavors). You can set it up for rsync over SSH and use public/private key pair for authentication – again you would need the admin of the receiving server to load your public key.
>
>
>
> Adam Robertson, Manager
>
> 400 South Hope Street, Suite 900 | Los Angeles, CA 90071
>
> office: 213.327.1460 | fax: 213.327.1367
>
> <image001.jpg>
>
>
>
> From: netsa-tools-discuss-bounces+adam.robertson=protiviti.com at cert.org [mailto:netsa-tools-discuss-bounces+adam.robertson=protiviti.com at cert.org] On Behalf Of inetjunkmail
> Sent: Thursday, March 26, 2015 8:17 AM
> To: netsa-tools-discuss at cert.org
> Subject: Re: [netsa-tools-discuss] super_mediator TCP connection timeout
>
>
>
> Thanks for your reply.  Unfortunately we don't own the receiver and they are not using SiLK.  I'm under the impression that the output of rwsender was not equivalent to IPFIX and it required rwreceiver at the other end to receive the data (rather than a generic IPFIX collector).  Do I understand that correctly?
>
>
>
> Thanks again for your help
>
>
>
> On Thu, Mar 26, 2015 at 10:48 AM, Emily Sarneso <ecoff at sei.cmu.edu> wrote:
>
> Hello,
>
> super_mediator requires the process it is exporting to to be available at startup time.  Once it is running and the exporter process goes down, it will stay up and periodically retry to connect to the exporter - but it must be available when it starts up.
>
> If this is a problem, you could have super_mediator write IPFIX files to a directory and have rwsender (remote host) or rwflowpack (local host) poll the directory:
>
> EXPORTER FILEHANDLER
>    PATH “/path/to/poll”
>    ROTATE 120
>    LOCK
>    FLOW_ONLY
> EXPORTER END
>
> In your case, it looks like you are sending a particular set of flows to a remote host, so you may want to look into using rwsender/rwreceiver to transfer and collect the files.  rwsender would be running on the same host as super_mediator polling the directory you provide in the EXPORTER block of the super_mediator.conf.  rwreceiver would run on your remote host “1.1.1.1” and listen for connections from rwsender.  rwflowpack/flowcap will also run on the remote host and poll the directory where rwreceiver writes the files.
>
> http://tools.netsa.cert.org/silk/rwsender.html
> http://tools.netsa.cert.org/silk/rwreceiver.html
>
> Hope this helps.  Please let us know if you have any other questions.
>
> Emily
>
>
>
>
> On Mar 25, 2015, at 8:19 AM, inetjunkmail <inetjunkmail at gmail.com> wrote:
>
> > We have a need to relay a subset of our IPFIX data to a customer.  When we used the NFDump suite, we used nfreplay to ship them the data.  Now that we use SiLK, we were trying to use super_mediator.  To give a better idea of what we're trying to to, super_mediator.conf is below.  The problem is that when the remote collector at 1.1.1.1 is unreachable, super_mediator never starts because it hangs trying to connect.  Is there a way to have it time out and periodically retry to connect to exporters rather than requiring them to be available at the time the service starts?
> >
> > Alternatively, is there a better way to accomplish this?
> >
> > COLLECTOR TCP
> >    PORT 18000
> > COLLECTOR END
> > EXPORTER TCP
> >    PORT 18001
> >    HOST localhost
> >    FLOW_ONLY
> > EXPORTER END
> > EXPORTER TCP
> >    PORT 2055
> >    HOST "1.1.1.1"
> >    FLOW_ONLY
> >    ANY_IP IN_LIST "/data/silk/sets/customerA.set"
> > EXPORTER END
> > LOGLEVEL DEBUG
> > LOG "/data/silk/log/super_mediator.log"
> > PIDFILE "/data/silk/log/super_mediator.pid"
> >
> > Thanks
>
>
>
> NOTICE: Protiviti is a global consulting and internal audit firm composed of experts specializing in risk and advisory services. Protiviti is not licensed or registered as a public accounting firm and does not issue opinions on financial statements or offer attestation services. This electronic mail message is intended exclusively for the individual or entity to which it is addressed. This message, together with any attachment, may contain confidential and privileged information. Any views, opinions or conclusions expressed in this message are those of the individual sender and do not necessarily reflect the views of Protiviti Inc. or its affiliates. Any unauthorized review, use, printing, copying, retention, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email message to the sender and delete all copies of this message. Thank you.
>

-------------- next part --------------
HTML attachment scrubbed and removed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: super_exp_active.patch
Type: application/octet-stream
Size: 559 bytes
Desc: super_exp_active.patch
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20150326/04e6313e/attachment.obj>