[netsa-tools-discuss] pipeline 5.6 / 5.5 / 5.4 segfault (and change in alert-log-file?)
Euan Galloway
euan at galloway.cc
Mon Aug 21 08:11:57 EDT 2017
I've noticed (on upgrade to 5.6, but went down to 5.5 and 5.4 and it is the
same) that pipeline segfaults when a FOREACH is used where the FILTER
expression matches 0 flows.
Also, 5.6 is not updating the "--alert-log-file", only the
"--aux-alert-file" (5.5 and 5.4 still do update both, is this just a change
in behavior?)
Replicated with a basic install and a "forced" case (looking for an invalid
interface) on a new centos7 VM using small dataset in a single file.
This is just an exaggerated / easy to repeat case, it does actually kill
pipeline constantly in live use (a real interface may do no traffic
matching its filter for a period for instance).
/usr/sbin/pipeline --configuration-file=/root/pipeline/EG_test.conf
--log-level=debug --silk
--alert-log-file=/root/pipeline/EG_pipeline-stats_alert.log
--aux-alert-file=/root/pipeline/EG_alerts_stats_aux.log
--site-config-file=/root/pipeline/silk.conf
--country-code-file=/root/pipeline/country_codes.pmap
--name-files EG.silk
# cat EG_test.conf
FILTER in-match
SENSOR == jul
INPUT == 568 #There are flows matching this interface.
END FILTER
FILTER in-nomatch
SENSOR == jul
INPUT == 111 #There are no flows matching this interface (problem is not
unique to matching interfaces though)
END FILTER
STATISTIC in-match
UPDATE 5 MINUTES
FOREACH PROTOCOL
SEVERITY 2
FILTER in-match
SUM BYTES
END STATISTIC
STATISTIC in-nomatch
UPDATE 5 MINUTES
FOREACH PROTOCOL #This statistic causes the segfault, I just #'d out
FOREACH for it to not segfault
SEVERITY 2
FILTER in-nomatch
SUM BYTES
END STATISTIC
Broken (segfault on FOREACH against non matching FILTER + alert-log-file
does not update (even when FOREACH removed));
analysis-pipeline x86_64
5.6-3.el7 @forensics
1.3 M
libfixbuf x86_64
1.7.1-1.el7 @forensics
760 k
libschemaTools x86_64
1.2.1-1.el7 @forensics
335 k
silk-common x86_64
3.16.0-1.el7 @forensics
3.7 M
Processing input: SiLK File: EG.silk
Opening aux-alert-file '/root/pipeline/EG_alerts_stats_aux.log' for append
1502975334 in-match: There are now 6 valid outputs
1502975334 in-nomatch: There are now 0 valid outputs
Segmentation fault
<EG_alerts_stats_aux.log>
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|50|SUM BYTES|12556736|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|47|SUM BYTES|1527433|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|17|SUM BYTES|64809785|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|1|SUM BYTES|46513|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|51|SUM BYTES|225882|
2017-08-21 11:40:53|Statistic|in-match|2|PROTOCOL|6|SUM BYTES|1486236238|
vs (remove second FOREACH)
Processing input: SiLK File: EG.silk
Opening aux-alert-file '/root/pipeline/EG_alerts_stats_aux.log' for append
1502975334 in-match: There are now 6 valid outputs
1502975334 in-nomatch: There is now 1 valid output
File took 3.210000 seconds, matched 460071/6419231 records
Total records read: 6419231
Pipeline took 3.210000 seconds.
<EG_alerts_stats_aux.log>
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|50|SUM BYTES|12556736|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|47|SUM BYTES|1527433|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|17|SUM BYTES|64809785|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|1|SUM BYTES|46513|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|51|SUM BYTES|225882|
2017-08-21 11:44:31|Statistic|in-match|2|PROTOCOL|6|SUM BYTES|1486236238|
2017-08-21 11:44:31|Statistic|in-nomatch|2|SUM BYTES|0|
Broken (segfault on FOREACH against non matching FILTER but alert-log-file
updates (as does aux-alert-file));
analysis-pipeline x86_64
5.5-2.el7 @forensics
1.3 M
libfixbuf x86_64
1.7.1-1.el7 @forensics
760 k
libschemaTools x86_64
1.2.1-1.el7 @forensics
335 k
silk-common x86_64
3.14.0-1.el7 @forensics
3.5 M
Broken; (segfault on FOREACH against non matching FILTER but alert-log-file
updates (as does aux-alert-file));
analysis-pipeline x86_64
5.4.1-1.el7 forensics
591 k
silk-common x86_64
3.12.0-1.el7 forensics
1.1 M
libfixbuf x86_64
1.7.1-1.el7 forensics
198 k
libschemaTools x86_64
1.2.1-1.el7 forensics
93 k
Working (no segfault, both log files update));
analysis-pipeline x86_64
5.3.2-2.el7 @forensics
1.3 M
libfixbuf x86_64
1.7.1-1.el7 @forensics
760 k
libschemaTools x86_64
1.2.1-1.el7 @forensics
335 k
silk-common x86_64
3.12.0-1.el7 @forensics
3.6 M
-------------- next part --------------
HTML attachment scrubbed and removed
More information about the netsa-tools-discuss
mailing list