[netsa-tools-discuss] super_mediator TCP connection timeout

[inetjunkmail]

Wow.  Thanks Emily.  We'll give it a try next week.

On Thu, Mar 26, 2015 at 1:46 PM, Emily Sarneso <ecoff at sei.cmu.edu> wrote:

>  You are correct about rwsender/rwreceiver.  rwsender simply transfers
> files to a rwreceiver running on another host.
>
> The attached patch (for version 0.4.0) should solve your problem.  Once
> applied and super_mediator is recompiled and reinstalled, super_mediator
> will set any exporters that it cannot connect to inactive and continue
> running.  It will periodically retry to connect to the exporter.  It will
> log a message when it attempts to connect and whether the attempt was
> successful or not.
>
> Hope that helps,
>
> Emily
>
>
>
>
> On Mar 26, 2015, at 1:08 PM, inetjunkmail <inetjunkmail at gmail.com> wrote:
>
> > Thanks for the idea Adam.  Unfortunately, RSYNC would still require the
> receiver to use SiLK to interpret the data and we're trying to avoid that
> requirement.
> >
> > In our environment we function much like an ISP so we have multiple
> customers traversing our routers.  We are transitioning to SiLK as our
> IPFIX collector/analyzer but we also offer customers their specific flow
> data (based on IP address).  Today we use nfcapd to receive flow data from
> the routers then we run a cron job ever 5 minutes to read in the last 5
> mins of flow data off disk and export it via nfreplay which is currently
> exporting the data in Neflow v9 format to the customers.  From the customer
> perspective, they just need to stand up any Netflow/IPFIX collector and we
> shoot them the data.  We can certainly change our export format to IPFIX
> but we'd like to avoid requiring a specific collector.
> >
> > Ultimately it looks like we may be able to user super_mediator if we
> choose to export via UDP; since it's connectionless, it doesn't hang on
> startup.  Ideally we'd come up with a way to enable the use of TCP for the
> reliability and the benefit of not having to wait around for a template
> refresh but I think that option is better than maintaining the two parallel
> solutions.
> >
> >
> > On Thu, Mar 26, 2015 at 11:55 AM, Robertson, Adam (10020) <
> adam.robertson at protiviti.com> wrote:
> > If I understand your need correctly, could you use ‘rsync’ on a ‘cron’
> job to periodically send the data to the destination. It would mean that
> you would need the admin of the receiving server to make sure ‘rsync’ is
> installed (which I believe it is by default on a lot of Linux flavors). You
> can set it up for rsync over SSH and use public/private key pair for
> authentication – again you would need the admin of the receiving server to
> load your public key.
> >
> >
> >
> > Adam Robertson, Manager
> >
> > 400 South Hope Street, Suite 900 | Los Angeles, CA 90071
> >
> > office: 213.327.1460 | fax: 213.327.1367
> >
> > <image001.jpg>
> >
> >
> >
> > From: netsa-tools-discuss-bounces+adam.robertson=protiviti.com at cert.org
> [mailto:netsa-tools-discuss-bounces+adam.robertson=protiviti.com at cert.org
> <netsa-tools-discuss-bounces+adam.robertson=protiviti.com at cert.org>] On
> Behalf Of inetjunkmail
> > Sent: Thursday, March 26, 2015 8:17 AM
> > To: netsa-tools-discuss at cert.org
> > Subject: Re: [netsa-tools-discuss] super_mediator TCP connection timeout
> >
> >
> >
> > Thanks for your reply.  Unfortunately we don't own the receiver and they
> are not using SiLK.  I'm under the impression that the output of rwsender
> was not equivalent to IPFIX and it required rwreceiver at the other end to
> receive the data (rather than a generic IPFIX collector).  Do I understand
> that correctly?
> >
> >
> >
> > Thanks again for your help
> >
> >
> >
> > On Thu, Mar 26, 2015 at 10:48 AM, Emily Sarneso <ecoff at sei.cmu.edu>
> wrote:
> >
> > Hello,
> >
> > super_mediator requires the process it is exporting to to be available
> at startup time.  Once it is running and the exporter process goes down, it
> will stay up and periodically retry to connect to the exporter - but it
> must be available when it starts up.
> >
> > If this is a problem, you could have super_mediator write IPFIX files to
> a directory and have rwsender (remote host) or rwflowpack (local host) poll
> the directory:
> >
> > EXPORTER FILEHANDLER
> >    PATH “/path/to/poll”
> >    ROTATE 120
> >    LOCK
> >    FLOW_ONLY
> > EXPORTER END
> >
> > In your case, it looks like you are sending a particular set of flows to
> a remote host, so you may want to look into using rwsender/rwreceiver to
> transfer and collect the files.  rwsender would be running on the same host
> as super_mediator polling the directory you provide in the EXPORTER block
> of the super_mediator.conf.  rwreceiver would run on your remote host
> “1.1.1.1” and listen for connections from rwsender.  rwflowpack/flowcap
> will also run on the remote host and poll the directory where rwreceiver
> writes the files.
> >
> > http://tools.netsa.cert.org/silk/rwsender.html
> > http://tools.netsa.cert.org/silk/rwreceiver.html
> >
> > Hope this helps.  Please let us know if you have any other questions.
> >
> > Emily
> >
> >
> >
> >
> > On Mar 25, 2015, at 8:19 AM, inetjunkmail <inetjunkmail at gmail.com>
> wrote:
> >
> > > We have a need to relay a subset of our IPFIX data to a customer.
> When we used the NFDump suite, we used nfreplay to ship them the data.  Now
> that we use SiLK, we were trying to use super_mediator.  To give a better
> idea of what we're trying to to, super_mediator.conf is below.  The problem
> is that when the remote collector at 1.1.1.1 is unreachable, super_mediator
> never starts because it hangs trying to connect.  Is there a way to have it
> time out and periodically retry to connect to exporters rather than
> requiring them to be available at the time the service starts?
> > >
> > > Alternatively, is there a better way to accomplish this?
> > >
> > > COLLECTOR TCP
> > >    PORT 18000
> > > COLLECTOR END
> > > EXPORTER TCP
> > >    PORT 18001
> > >    HOST localhost
> > >    FLOW_ONLY
> > > EXPORTER END
> > > EXPORTER TCP
> > >    PORT 2055
> > >    HOST "1.1.1.1"
> > >    FLOW_ONLY
> > >    ANY_IP IN_LIST "/data/silk/sets/customerA.set"
> > > EXPORTER END
> > > LOGLEVEL DEBUG
> > > LOG "/data/silk/log/super_mediator.log"
> > > PIDFILE "/data/silk/log/super_mediator.pid"
> > >
> > > Thanks
> >
> >
> >
> > NOTICE: Protiviti is a global consulting and internal audit firm
> composed of experts specializing in risk and advisory services. Protiviti
> is not licensed or registered as a public accounting firm and does not
> issue opinions on financial statements or offer attestation services. This
> electronic mail message is intended exclusively for the individual or
> entity to which it is addressed. This message, together with any
> attachment, may contain confidential and privileged information. Any views,
> opinions or conclusions expressed in this message are those of the
> individual sender and do not necessarily reflect the views of Protiviti
> Inc. or its affiliates. Any unauthorized review, use, printing, copying,
> retention, disclosure or distribution is strictly prohibited. If you have
> received this message in error, please immediately advise the sender by
> reply email message to the sender and delete all copies of this message.
> Thank you.
> >
>
>
-------------- next part --------------
HTML attachment scrubbed and removed