[netsa-tools-discuss] super_mediator outputs broken JSON

[Matt Coates]

Leigh,

Attached you will find a patch file that should resolve your issue. Let me know how it works on your data.

Thanks,
Matt Coates

From: Porter, Leigh <leigh.porter at roke.co.uk>
Sent: Wednesday, November 20, 2019 7:22 AM
To: Matt Coates <mfcoates at cert.org>; netsa-tools-discuss at cert.org
Subject: RE: super_mediator outputs broken JSON

Hiya,

Actually yeah if you could send over a fix I can run it across our PCAP library, thanks!

--
Leigh



Follow Us: LinkedIn<http://www.linkedin.com/company/roke-manor-research> | Twitter<https://twitter.com/rokemanor> | Facebook<https://www.facebook.com/rokemanor>

Roke Manor Research Limited, Romsey, Hampshire, SO51 0ZN, United Kingdom. Part of the Chemring Group. Registered in England & Wales. Registered No: 00267550. The information contained in this e-mail and any attachments is proprietary to Roke Manor Research Limited and must not be passed to any third party without permission. This communication is for information only and shall not create or change any contractual relationship.
www.roke.co.uk<http://www.roke.co.uk/?utm_source=Roke&utm_medium=Email&utm_content=Company%20Signature&utm_campaign=Roke>

________________________________
From: Matt Coates <mfcoates at cert.org<mailto:mfcoates at cert.org>>
Sent: 08 November 2019 18:10
To: Porter, Leigh <leigh.porter at roke.co.uk<mailto:leigh.porter at roke.co.uk>>; netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: RE: super_mediator outputs broken JSON

Hi Leigh,

Thanks for reporting this issue.  A fix should be available in the next version of Super Mediator.  If you’re interested in testing a pre-release patch, let me know and that can be provided.  Additionally, if you have additional pcap or ipfix you can send us to validate the fix, it would be greatly appreciated.

Matt Coates
CERT
Software Engineering Institute
Carnegie Mellon University



From: netsa-tools-discuss-bounces+mfcoates=cert.org at cert.org<mailto:netsa-tools-discuss-bounces+mfcoates=cert.org at cert.org> <netsa-tools-discuss-bounces+mfcoates=cert.org at cert.org<mailto:netsa-tools-discuss-bounces+mfcoates=cert.org at cert.org>> On Behalf Of Porter, Leigh
Sent: Monday, November 4, 2019 11:28 AM
To: netsa-tools-discuss at cert.org<mailto:netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] super_mediator outputs broken JSON

Hey All,

It seems that super_mediator is breaking some JSON output, here is an example:

This is a single flow record from yaf into super_mediator:

{"flows":{"flowStartMilliseconds":"2019-10-14 09:49:16.230","flowEndMilliseconds":"2019-10-14 09:50:43.138","flowDurationMilliseconds":86.908,"reverseFlowDeltaMilliseconds":0.031,"protocolIdentifier":6,"sourceIPv4Address":"10.137.137.132","sourceTransportPort":55912,"packetTotalCount":10,"octetTotalCount":756,"flowAttributes":"00","sourceMacAddress":"00:00:00:00:00:00","destinationIPv4Address":"13.225.84.145<http://13.225.84.145>","destinationTransportPort":443,"reversePacketTotalCount":11,"reverseOctetTotalCount":5682,"reverseFlowAttributes":"00","destinationMacAddress":"00:00:00:00:00:00","initialTCPFlags":"S","unionTCPFlags":"APRF","reverseInitialTCPFlags":"AS","reverseUnionTCPFlags":"APF","tcpSequenceNumber":"0x459881a3","reverseTcpSequenceNumber":"0xe6764318","ingressInterface":0,"egressInterface":0,"vlanId":"0x000","silkAppLabel":443,"ipClassOfService":"0x00","flowEndReason":"","collectorName":"C1","sslCertList":[{"sslCertIssuer":{"sslCertIssuerCountryName":"US","sslCertIssuerOrgName":"DigiCert Inc","sslCertIssuerCommonName":"DigiCert Global CA G2","sslCertIssuerOrgUnitName":[]},"sslCertSubject":{"sslCertSubCountryName":"US","sslCertSubState":"Washington","sslCertSubLocalityName":"Seattle","sslCertSubOrgName":"Amazon.com, Inc.","sslCertSubCommonName":"*.cloudfront.net","sslCertSubjectOrgUnitName":[]},"sslCertVersion":2,"sslCertSerialNumber":"09 f4 da 45 55 23 f0 34 18 b0 55 7e 6d ce c0 11","sslCertValidityNotBefore":"190717000000Z","sslCertValidityNotAfter":"200705120000Z","sslPublicKeyLength":271,"sslExtensions":{}},{"sslCertIssuer":{"sslCertIssuerCountryName":"US","sslCertIssuerOrgName":"DigiCert Inc","sslCertIssuerCommonName":"DigiCert Global Root G2","sslCertIssuerOrgUnitName":["www.digicert.com<http://www.digicert.com>"]},"sslCertSubject":{"sslCertSubCountryName":"US","sslCertSubOrgName":"DigiCert Inc","sslCertSubCommonName":"DigiCert Global CA G2","sslCertSubjectOrgUnitName":[]},"sslCertVersion":2,"sslCertSerialNumber":"0c 8e e0 c9 0d 6a 89 15 88 04 06 1e e2 41 f9 af","sslCertValidityNotBefore":"130801120000Z","sslCertValidityNotAfter":"280801120000Z","sslPublicKeyLength":271,"sslExtensions":{}},"sslCertVersion":2,"sslCertSerialNumber":"63 18 0d 38 fb 80 97 78 a9 d0 35 a3 16 18 f8 40"}],"sslServerCipher":49199,"sslClientVersion":3,"sslRecordVersion":771,"sslServerName":"d3c3cq33003psk.cloudfront.net"}}


However, pipe that into something like 'jq' and you will see this:

parse error: ':' not as part of an object at line 1, column 2115

If you remove the last occurrence of ","sslExtensions":{}}" then all is fine.

This feature seems to pop up fairly regularly on lots of flows..




Leigh Porter
Managing Consultant
Roke Manor Research Limited
leigh.porter at roke.co.uk<mailto:leigh.porter at roke.co.uk>

Follow Us: LinkedIn<http://www.linkedin.com/company/roke-manor-research> | Twitter<https://twitter.com/rokemanor> | Facebook<https://www.facebook.com/rokemanor>

Roke Manor Research Limited, Romsey, Hampshire, SO51 0ZN, United Kingdom. Part of the Chemring Group. Registered in England & Wales. Registered No: 00267550. The information contained in this e-mail and any attachments is proprietary to Roke Manor Research Limited and must not be passed to any third party without permission. This communication is for information only and shall not create or change any contractual relationship.
www.roke.co.uk<http://www.roke.co.uk/?utm_source=Roke&utm_medium=Email&utm_content=Company%20Signature&utm_campaign=Roke>

________________________________
-------------- next part --------------
HTML attachment scrubbed and removed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sm-json-ssl.patch
Type: application/octet-stream
Size: 1756 bytes
Desc: sm-json-ssl.patch
URL: <http://lists.sei.cmu.edu/pipermail/netsa-tools-discuss/attachments/20191120/1ab929c8/attachment.obj>