[netsa-tools-discuss] Custom C code reads invalid sTime from newer silk files.

[Mark Thomas]

Otto-

Short answer:

You need to add a call to skAppRegister(argv[0]) near the beginning
of main().  For clean-up, add a call to skAppUnregister() near the
end of main().

Long answer:

Firstly, let me apologize that the libsilk interface does not have
the best documentation for use by outside parties.

When I ran your test program over two files, one in the FT_RWROUTED
format and the other in the FT_RWIPV6 format, I found the results to
be different just like you described.

However, I also noticed that multiple invocations using the same
input file produced different output, which immediately made me
realize there was a memory corruption issue.

With valgrind, I discovered skStream was reading uninitialized
memory when processing the "packed-file-info" header of each file.
I also noticed the header was being handled by a "generic" reader
instead of the packed-file-info reader.

That made me realize the header-reading code had not been
initialized.  In SiLK code, the header-reading code is expected to
be initialized by calling skAppRegister() (which in turn calls
skHeaderInitialize()).

SiLK could certainly be smarter and either (1) initialize the header
routines as needed or (2) complain that the header routines need to
be initialized.  (Since we use C and not C++, there is no way to
have that code initialized automatically at start-up.)  Regardless,
I need to fix the bug that causes SiLK to read uninitialized memory
when the header code has not been initialized.  Fixing that bug
would at least allow the code to be consistent across runs, even if
the values are incorrect.

Perhaps the reason you have not seen this before is because each
record in an FT_RWIPV6 format file stores the complete start-time
while each record in an FT_RWROUTED format file stores the
start-time as a millisecond offset from the time stored in the
packed-file-info header.

I hope that helps.

-Mark


-----Original Message-----
From: mworld <mworld at twbc.net>
Date: Wed, 31 Dec 2014 16:38:44 +1000
To: <netsa-tools-discuss at cert.org>
Subject: [netsa-tools-discuss] Custom C code reads invalid sTime from newer
	silk files.

Hi. Below is a very basic program which just opens and reads all the 
records in a silk file (made for testing this issue). This method has 
been working great, until recently. When I read newer silk files, 
rwrec->sTime does not appear to be milliseconds (it does not appear to 
be an endian issue). I am kinda of stumped since the binaries that come 
with silk work fine and read the date correctly and I'm using the same 
library as them.

Problem: rwrec->sTime is returning as this: 34388381982 instead of 
milliseconds. Other fields seem fine.

The same issue occurs using c++ and printing it out using an output stream.

New SiLK file where sTime reads incorrectly: (created with no ipv6 support)

   format(id)          FT_RWROUTED(0x10)
   version             16
   byte-order          littleEndian
   compression(id)     zlib(1)
   header-length       64
   record-length       32
   record-version      5
   silk-version        3.9.0
   count-records       2165
   file-size           30028
   packed-file-info    2014/12/31T01:00:00 0 0

Older SiLK file where sTime reads correctly: (created with ipv6 support)

   format(id)          FT_RWIPV6(0x0b)
   version             16
   byte-order          littleEndian
   compression(id)     zlib(1)
   header-length       68
   record-length       68
   record-version      1
   silk-version        2.5.0
   count-records       3442787
   file-size           46860014
   packed-file-info    2013/08/01T10:00:00 0 0

SiLK is compiled with the local time flag using the FreeBSD port: 
security/silktools (FreeBSD 9.3 64bit).

Any help would be appreciated.

Regards,
Otto.

make.sh
---------------------------------------
#!/bin/sh

sk_cc=`silk_config --compiler`
sk_cflags=`silk_config --cflags`
sk_libs=`silk_config --libsilk-libs`

$sk_cc $sk_cflags -o silk-decode-test silk-decode-test.c $sk_libs
---------------------------------------

silk-decode-test.c
---------------------------------------
#include <stdio.h>
#include <silk/silk.h>
#include <silk/skstream.h>
#include <silk/rwrec.h>
#include <silk/utils.h>

int main(int argc, char** argv)
{
     if (argc != 2) {
         printf("Usage %s silk-file\n", argv[0]);
         return 1;
     }

     rwRec rwrec;
     skstream_t *rwios = NULL;
     int rv = SKSTREAM_OK;

     rv = skStreamOpenSilkFlow(&rwios, argv[1], SK_IO_READ);

     if (rv != SKSTREAM_OK) {
         printf("Unable to open input file '%s'\n", argv[1]);
         skStreamPrintLastErr(rwios, rv, &skAppPrintErr);
     } else {
         printf("%14s%8s%8s\n", "sTime","pks","bytes");

         while ((rv = skStreamReadRecord(rwios, &rwrec)) == SKSTREAM_OK) {
             printf("%14llu", rwRecGetStartTime(&rwrec));
             printf("%8lu", rwRecGetPkts(&rwrec));
             printf("%8lu\n", rwRecGetBytes(&rwrec));
         }
     }

     skStreamDestroy(&rwios);

     return 0;
}

---------------------------------------