[netsa-tools-discuss] super_mediator TCP connection timeout

[inetjunkmail]

Thanks for the idea Adam.  Unfortunately, RSYNC would still require the
receiver to use SiLK to interpret the data and we're trying to avoid that
requirement.

In our environment we function much like an ISP so we have multiple
customers traversing our routers.  We are transitioning to SiLK as our
IPFIX collector/analyzer but we also offer customers their specific flow
data (based on IP address).  Today we use nfcapd to receive flow data from
the routers then we run a cron job ever 5 minutes to read in the last 5
mins of flow data off disk and export it via nfreplay which is currently
exporting the data in Neflow v9 format to the customers.  From the customer
perspective, they just need to stand up any Netflow/IPFIX collector and we
shoot them the data.  We can certainly change our export format to IPFIX
but we'd like to avoid requiring a specific collector.

Ultimately it looks like we may be able to user super_mediator if we choose
to export via UDP; since it's connectionless, it doesn't hang on startup.
Ideally we'd come up with a way to enable the use of TCP for the
reliability and the benefit of not having to wait around for a template
refresh but I think that option is better than maintaining the two parallel
solutions.


On Thu, Mar 26, 2015 at 11:55 AM, Robertson, Adam (10020) <
adam.robertson at protiviti.com> wrote:

>  If I understand your need correctly, could you use ‘rsync’ on a ‘cron’
> job to periodically send the data to the destination. It would mean that
> you would need the admin of the receiving server to make sure ‘rsync’ is
> installed (which I believe it is by default on a lot of Linux flavors). You
> can set it up for rsync over SSH and use public/private key pair for
> authentication – again you would need the admin of the receiving server to
> load your public key.
>
>
>
> *Adam Robertson, *Manager
>
> 400 South Hope Street, Suite 900 | Los Angeles, CA 90071
>
> office: 213.327.1460 | fax: 213.327.1367
>
> [image: image004_email] <http://www.protiviti.com/>
>
>
>
> *From:* netsa-tools-discuss-bounces+adam.robertson=protiviti.com at cert.org
> [mailto:netsa-tools-discuss-bounces+adam.robertson=protiviti.com at cert.org]
> *On Behalf Of *inetjunkmail
> *Sent:* Thursday, March 26, 2015 8:17 AM
> *To:* netsa-tools-discuss at cert.org
> *Subject:* Re: [netsa-tools-discuss] super_mediator TCP connection timeout
>
>
>
> Thanks for your reply.  Unfortunately we don't own the receiver and they
> are not using SiLK.  I'm under the impression that the output of rwsender
> was not equivalent to IPFIX and it required rwreceiver at the other end to
> receive the data (rather than a generic IPFIX collector).  Do I understand
> that correctly?
>
>
>
> Thanks again for your help
>
>
>
> On Thu, Mar 26, 2015 at 10:48 AM, Emily Sarneso <ecoff at sei.cmu.edu> wrote:
>
> Hello,
>
> super_mediator requires the process it is exporting to to be available at
> startup time.  Once it is running and the exporter process goes down, it
> will stay up and periodically retry to connect to the exporter - but it
> must be available when it starts up.
>
> If this is a problem, you could have super_mediator write IPFIX files to a
> directory and have rwsender (remote host) or rwflowpack (local host) poll
> the directory:
>
> EXPORTER FILEHANDLER
>    PATH “/path/to/poll”
>    ROTATE 120
>    LOCK
>    FLOW_ONLY
> EXPORTER END
>
> In your case, it looks like you are sending a particular set of flows to a
> remote host, so you may want to look into using rwsender/rwreceiver to
> transfer and collect the files.  rwsender would be running on the same host
> as super_mediator polling the directory you provide in the EXPORTER block
> of the super_mediator.conf.  rwreceiver would run on your remote host
> “1.1.1.1” and listen for connections from rwsender.  rwflowpack/flowcap
> will also run on the remote host and poll the directory where rwreceiver
> writes the files.
>
> http://tools.netsa.cert.org/silk/rwsender.html
> http://tools.netsa.cert.org/silk/rwreceiver.html
>
> Hope this helps.  Please let us know if you have any other questions.
>
> Emily
>
>
>
>
> On Mar 25, 2015, at 8:19 AM, inetjunkmail <inetjunkmail at gmail.com> wrote:
>
> > We have a need to relay a subset of our IPFIX data to a customer.  When
> we used the NFDump suite, we used nfreplay to ship them the data.  Now that
> we use SiLK, we were trying to use super_mediator.  To give a better idea
> of what we're trying to to, super_mediator.conf is below.  The problem is
> that when the remote collector at 1.1.1.1 is unreachable, super_mediator
> never starts because it hangs trying to connect.  Is there a way to have it
> time out and periodically retry to connect to exporters rather than
> requiring them to be available at the time the service starts?
> >
> > Alternatively, is there a better way to accomplish this?
> >
> > COLLECTOR TCP
> >    PORT 18000
> > COLLECTOR END
> > EXPORTER TCP
> >    PORT 18001
> >    HOST localhost
> >    FLOW_ONLY
> > EXPORTER END
> > EXPORTER TCP
> >    PORT 2055
> >    HOST "1.1.1.1"
> >    FLOW_ONLY
> >    ANY_IP IN_LIST "/data/silk/sets/customerA.set"
> > EXPORTER END
> > LOGLEVEL DEBUG
> > LOG "/data/silk/log/super_mediator.log"
> > PIDFILE "/data/silk/log/super_mediator.pid"
> >
> > Thanks
>
>
>  NOTICE: Protiviti is a global consulting and internal audit firm
> composed of experts specializing in risk and advisory services. Protiviti
> is not licensed or registered as a public accounting firm and does not
> issue opinions on financial statements or offer attestation services. This
> electronic mail message is intended exclusively for the individual or
> entity to which it is addressed. This message, together with any
> attachment, may contain confidential and privileged information. Any views,
> opinions or conclusions expressed in this message are those of the
> individual sender and do not necessarily reflect the views of Protiviti
> Inc. or its affiliates. Any unauthorized review, use, printing, copying,
> retention, disclosure or distribution is strictly prohibited. If you have
> received this message in error, please immediately advise the sender by
> reply email message to the sender and delete all copies of this message.
> Thank you.
>
-------------- next part --------------
HTML attachment scrubbed and removed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4428 bytes
Desc: not available
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20150326/f0a95a4b/attachment.jpg>