[netsa-tools-discuss] pipeline 5.6 / 5.5 / 5.4 segfault (and change in alert-log-file?)
Daniel Ruef
druef at cert.org
Mon Aug 21 15:57:53 EDT 2017
Euan,
Thanks for using Pipeline, and submitting such a clear bug report.
I’ve found the seg fault bug and it will be fixed in version 5.7 which will be out in a few weeks.
For the short term, if you’re comfortable editing code and rebuilding, if you make the following changes to pipeline.c, you’ll get past the segfault.
Add these lines…
setSchemaGlobals(statistic->validSchemas[0]);
statisticUseThisSchema(statistic, lastSchemaEntry);
…to be lines 3932 and 3948…
Making part of the function “alertStatistic” look like this:
} else if (primitive->possUniqueFLs->fieldList->totalVar) {
statOutputCount = 0;
setSchemaGlobals(statistic->validSchemas[0]);
statisticUseThisSchema(statistic, lastSchemaEntry);
piHashForeach(primitive->table.piHash,
gFE_statAddToOutput,
statistic);
MY_PRINT((("%u %s: There are now %u valid outputs"),
networkTime, statistic->id, statOutputCount));
if (statistic->statTimeType == STAT_TIME_EQUAL) {
statistic->recordsSeen = 0;
primitive->freeState(primitive);
primitive->aggStateSetup(primitive);
}
} else {
statOutputCount = 0;
setSchemaGlobals(statistic->validSchemas[0]);
statisticUseThisSchema(statistic, lastSchemaEntry);
piHashForeach(primitive->table.piHash,
piHashFE_fixedStatToOutput,
statistic);
Version 5.6 not updating the –alert-log-file is a change in behavior that it turns out I forgot to document. Part of the changes in 5.6 is that statistics no longer store the last record that affected their state to be more memory efficient. Knowing the last record that affects that state doesn’t help track down problems like they do for evaluations. As a result of not having a record to output, nothing gets added to the regular alert log, but the state values are still sent to the aux-alert-log.
Sorry for the confusion. The documentation will be updated for version 5.7
Dan Ruef
From: netsa-tools-discuss-bounces+druef=cert.org at cert.org [mailto:netsa-tools-discuss-bounces+druef=cert.org at cert.org] On Behalf Of Euan Galloway
Sent: Monday, August 21, 2017 8:12 AM
To: netsa-tools-discuss at cert.org
Subject: [netsa-tools-discuss] pipeline 5.6 / 5.5 / 5.4 segfault (and change in alert-log-file?)
I've noticed (on upgrade to 5.6, but went down to 5.5 and 5.4 and it is the same) that pipeline segfaults when a FOREACH is used where the FILTER expression matches 0 flows.
Also, 5.6 is not updating the "--alert-log-file", only the "--aux-alert-file" (5.5 and 5.4 still do update both, is this just a change in behavior?)
Replicated with a basic install and a "forced" case (looking for an invalid interface) on a new centos7 VM using small dataset in a single file.
This is just an exaggerated / easy to repeat case, it does actually kill pipeline constantly in live use (a real interface may do no traffic matching its filter for a period for instance).
/usr/sbin/pipeline --configuration-file=/root/pipeline/EG_test.conf --log-level=debug --silk
--alert-log-file=/root/pipeline/EG_pipeline-stats_alert.log
--aux-alert-file=/root/pipeline/EG_alerts_stats_aux.log
--site-config-file=/root/pipeline/silk.conf
--country-code-file=/root/pipeline/country_codes.pmap
--name-files EG.silk
# cat EG_test.conf
FILTER in-match
SENSOR == jul
INPUT == 568 #There are flows matching this interface.
END FILTER
FILTER in-nomatch
SENSOR == jul
INPUT == 111 #There are no flows matching this interface (problem is not unique to matching interfaces though)
END FILTER
STATISTIC in-match
UPDATE 5 MINUTES
FOREACH PROTOCOL
SEVERITY 2
FILTER in-match
SUM BYTES
END STATISTIC
STATISTIC in-nomatch
UPDATE 5 MINUTES
FOREACH PROTOCOL #This statistic causes the segfault, I just #'d out FOREACH for it to not segfault
SEVERITY 2
FILTER in-nomatch
SUM BYTES
END STATISTIC
Broken (segfault on FOREACH against non matching FILTER + alert-log-file does not update (even when FOREACH removed));
analysis-pipeline x86_64 5.6-3.el7 @forensics 1.3 M
libfixbuf x86_64 1.7.1-1.el7 @forensics 760 k
libschemaTools x86_64 1.2.1-1.el7 @forensics 335 k
silk-common x86_64 3.16.0-1.el7 @forensics 3.7 M
Processing input: SiLK File: EG.silk
Opening aux-alert-file '/root/pipeline/EG_alerts_stats_aux.log' for append
1502975334 in-match: There are now 6 valid outputs
1502975334 in-nomatch: There are now 0 valid outputs
Segmentation fault
<EG_alerts_stats_aux.log>
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|50|SUM BYTES|12556736|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|47|SUM BYTES|1527433|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|17|SUM BYTES|64809785|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|1|SUM BYTES|46513|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|51|SUM BYTES|225882|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|6|SUM BYTES|1486236238|
vs (remove second FOREACH)
Processing input: SiLK File: EG.silk
Opening aux-alert-file '/root/pipeline/EG_alerts_stats_aux.log' for append
1502975334 in-match: There are now 6 valid outputs
1502975334 in-nomatch: There is now 1 valid output
File took 3.210000 seconds, matched 460071/6419231 records
Total records read: 6419231
Pipeline took 3.210000 seconds.
<EG_alerts_stats_aux.log>
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|50|SUM BYTES|12556736|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|47|SUM BYTES|1527433|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|17|SUM BYTES|64809785|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|1|SUM BYTES|46513|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|51|SUM BYTES|225882|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|6|SUM BYTES|1486236238|
2017-08-21 11:44:31|Statistic|in-nomatch|2|SUM BYTES|0|
Broken (segfault on FOREACH against non matching FILTER but alert-log-file updates (as does aux-alert-file));
analysis-pipeline x86_64 5.5-2.el7 @forensics 1.3 M
libfixbuf x86_64 1.7.1-1.el7 @forensics 760 k
libschemaTools x86_64 1.2.1-1.el7 @forensics 335 k
silk-common x86_64 3.14.0-1.el7 @forensics 3.5 M
Broken; (segfault on FOREACH against non matching FILTER but alert-log-file updates (as does aux-alert-file));
analysis-pipeline x86_64 5.4.1-1.el7 forensics 591 k
silk-common x86_64 3.12.0-1.el7 forensics 1.1 M
libfixbuf x86_64 1.7.1-1.el7 forensics 198 k
libschemaTools x86_64 1.2.1-1.el7 forensics 93 k
Working (no segfault, both log files update));
analysis-pipeline x86_64 5.3.2-2.el7 @forensics 1.3 M
libfixbuf x86_64 1.7.1-1.el7 @forensics 760 k
libschemaTools x86_64 1.2.1-1.el7 @forensics 335 k
silk-common x86_64 3.12.0-1.el7 @forensics 3.6 M
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list