Development Build

Commits

• f48f504: remove rstr dependency (#755) (Jörg Zimmermann) #755

15.1.1

What's Changed

• fix multiarch container build by @ekneg54 in #740

Full Changelog: v15.1.0...v15.1.1

15.1.0

What's Changed

• Fix bug lucene compliance by @fabian-moessner in #734
• add multiarch container build by @ekneg54 in #738

Full Changelog: v15.0.0...v15.1.0

15.0.0

Breaking

• drop support for python 3.10 and add support for python 3.13
• CriticalInputError is raised when the input preprocessor values can't be set, this was so far only true
  for the hmac preprocessor, but is now also applied for all other preprocessors.
• fix delimiter typo in StringSplitterRule configuration
• removed the configuration tld_lists in domain_resolver, domain_label_extractor and pseudonymizer as
  the list is now fixed inside the packaged logprep
• remove SQL feature from generic_adder, fields can only be added from rule config or from file
• use a single rule tree instead of a generic and a specific rule tree
• replace the extend_target_list parameter with merge_with_target for improved naming clarity
  and functionality across FieldManager based processors (e.g., FieldManager, Clusterer,
  GenericAdder).

Features

• configuration of initContainers in logprep helm chart is now possible

Improvements

• fix requester documentation
• replace BaseException with Exception for custom errors
• refactor generic_resolver to validate rules on startup instead of application of each rule
• regex pattern lists for the generic_resolver are pre-compiled
• regex matching from lists in the generic_resolver is cached
• matching in the generic_resolver can be case-insensitive
• rewrite the helper method add_field_to such that it always raises an FieldExistsWarning instead of return a bool.
• add new helper method add_fields_to to directly add multiple fields to one event
• refactored some processors to make use of the new helper methods
• add pre-commit hooks to the repository, install new dev dependency and run pre-commit install in the root dir
• the default securityContextfor the pod is now configurable
• allow TimeParser to get the current time with a specified timezone instead of always using local time and setting the timezone to UTC
• remove tldextract dependency
• remove urlextract dependency
• fix wrong documentation for timestamp_differ
• add container signatures to images build in ci pipeline
• add sbom to images build in ci pipeline
• FieldManager supports merging dictionaries

Bugfix

• fix confluent_kafka.store_offsets if last_valid_record is None, can happen if a rebalancing happens
  before the first message was pulled.
• fix pseudonymizer cache metrics not updated
• fix incorrect timezones for log arrival time and delta time in input preprocessing
• fix _get_value in FilterExpression so that keys don't match on values
• fix auto_rule_tester to work with LOGPREP_BYPASS_RULE_TREE enabled
• fix opensearch_output not draining message_backlog on shutdown
• silence FieldExists warning in metrics when LOGPREP_APPEND_MEASUREMENT_TO_EVENT is active

Details

• add dynamic scaling of time representations by @Malutthias in #690
• Delete real kafka tests by @djkhl in #692
• fix store_offsets call when last_valid_record is None by @dtrai2 in #693
• standardize exception inheritance to Exception by @dtrai2 in #695
• fix pseudonymizer cache metrics by @ekneg54 in #703
• remove requirements.* files by @ekneg54 in #699
• refactor validation of generic_resolver rules to startup by @dtrai2 in #694
• speed up kafka output tests by @ekneg54 in #704
• speed up s3 output tests by @ekneg54 in #706
• update add_field_to function for improved error handling by @dtrai2 in #696
• speed up exporter tests by @ekneg54 in #705
• Dev unit tests speed up by @fabian-moessner in #698
• fix requester documentation by @ekneg54 in #702
• add pre-commit hooks and update documentation by @dtrai2 in #707
• Delete logprep/util/pre_detector_rule_matching_tester.py by @Malutthias in #708
• make pod security contex configurable by @djkhl in #711
• add ability to configure init containers in helm chart by @djkhl in #712
• Add missing SSL config to kafka admin clients by @ppcad in #710
• Pre-compile patterns, add cache and support ignoring case in generic resolver by @ppcad in #716
• Fix filter matching with bypass tree by @ppcad in #719
• Improve test_http_input execution speed by @dtrai2 in #720
• remove-tldextract-and-urlextract-dependencies by @ekneg54 in #718
• Fix log arrival timezone by @ppcad in #715
• Fix timestamp differ rule doc by @dtrai2 in #723
• add shut_down method to opensearch output connector by @ekneg54 in #724
• remove sql from generic_adder by @dtrai2 in #722
• Fix FieldsExist warning using LOGPREP_APPEND_MEASUREMENT_TO_EVENT by @dtrai2 in #725
• add signing to action by @dtrai2 in #729
• add SBOM to images by @dtrai2 in #730
• replace container build steps with reusable workflow by @dtrai2 in #732
• fix build-version in dev release by @dtrai2 in #733
• reduce to one rule tree by @ppcad in #731
• support dict merging in fieldmanager and add_fields_to method by @dtrai2 in #737
• Add rule tree configuration validation by @dtrai2 in #736
• add support for python 3.13 and drop support for python 3.10 by @ekneg54 in #666
• prepare release 15 by @ekneg54 in #739

New Contributors

• @fabian-moessner made their first contribution in #698

Full Changelog: v14.0.0...v15.0.0

logprep-15.0.0

Logprep helm chart

logprep-14.1.0

Logprep helm chart

logprep-14.0.1

Logprep helm chart

v14.0.0

Breaking

• remove AutoRuleCorpusTester
• removes the option to use synchronous bulk or parallel_bulk operation in favor of parallel_bulk in opensearch_output
• reimplement error handling by introducing the option to configure an error output
  • if no error output is configured, failed event will be dropped

Features

• adds health check endpoint to metrics on path /health
• changes helm chart to use new readiness check
• adds healthcheck_timeout option to all components to tweak the timeout of healthchecks
• adds desired_cluster_status option to opensearch output to signal healthy cluster status
• initially run health checks on setup for every configured component
• make imagePullPolicy configurable for helm chart deployments
• it is now possible to use Lucene compliant Filter Expressions
• make terminationGracePeriodSeconds configurable in helm chart values
• adds ability to configure error output
• adds option default_op_type to opensearch_output connector to set the default operation for indexing documents (default: index)
• adds option max_chunk_bytes to opensearch_output connector to set the maximum size of the request in bytes (default: 100MB)
• adds option error_backlog_size to logprep configuration to configure the queue size of the error queue
• the opensearch default index is now only used for processed events, errors will be written to the error output, if configured

Improvements

• remove AutoRuleCorpusTester
• adds support for rust extension development
• adds prebuilt wheels for architectures x86_64 on manylinux and musllinux based linux platforms to releases
• add manual how to use local images with minikube example setup to documentation
• move Configuration to top level of documentation
• add CONTRIBUTING file
• sets the default for flush_timeout and send_timeout in kafka_output connector to 0 seconds
• changed python base image for logprep to bitnami/python in cause of better CVE governance

Bugfix

• ensure logprep.abc.Component.Config is immutable and can be applied multiple times
• remove lost callback reassign behavior from kafka_input connector
• remove manual commit option from kafka_input connector
• pin mysql-connector-python to >=9.1.0 to accommodate for CVE-2024-21272 and update MySQLConnector to work with the new version

Details

• Handle UnicodeDecodeError in kafka input by @ppcad in #663
• make configuration immutable by @ekneg54 in #661
• Remove autorulecorpustester by @ekneg54 in #665
• add rust toolchain by @ekneg54 in #662
• add readiness checks by @ekneg54 in #652
• Fix release pipeline broken for arm images by @ekneg54 in #670
• Fix automatic docs generation by @ekneg54 in #671
• Add CONTRIBUTING file by @ekneg54 in #664
• add minor improvements by @ekneg54 in #676
• Fix exporter restart by @ekneg54 in #677
• Ensure kafka output flushes queue on shutdown by @ekneg54 in #679
• change base image to bitnami/python by @ekneg54 in #680
• Remove unneeded dependencies by @P4sca1 in #682
• Remove broken kafka implementations by @ekneg54 in #678
• Fix docker image and avoid implicit setuptools runtime dependency by @P4sca1 in #683
• test container image before push by @ekneg54 in #684
• make terminationGracePeriodSeconds configurable. by @ekneg54 in #686
• Scan images using Trivy by @P4sca1 in #685
• Dev autoruletester refactoring by @Malutthias in #594
• 629 no support for the mysql connector python library in version 900 by @ppcad in #688
• Add Lucene compliant regex filter expression by @djkhl in #675
• add abiltity to configure error output by @ekneg54 in #668
• prepare release 14 by @ekneg54 in #689

New Contributors

• @P4sca1 made their first contribution in #682
• @Malutthias made their first contribution in #594

Full Changelog: v13.1.2...v14.0.0

logprep-14.0.0

Logprep helm chart

logprep-13.4.0

Logprep helm chart