logprep-13.1.0

Logprep helm chart

v13.0.0

Breaking

• This release limits the maximum python version to 3.12.3 because of the issue
  #612.
• Remove normalizer processor, as it's functionality was replaced by the grokker, timestamper and field_manager processors
• Remove elasticsearch_output connector to reduce maintenance effort

Features

• add a helm chart to install logprep in kubernetes based environments

Improvements

• add documentation about behavior of the timestamper on ISO8601 and UNIX time parsing
• add unit tests for helm chart templates
• add helm to github actions runner
• add helm chart release to release pipeline

Bugfix

• fixes a bug where it could happen that a config value could be overwritten by a default in a later configuration in a multi source config scenario
• fixes a bug in the field_manager where extending a non list target leads to a processing failure
• fixes a bug in pseudonymizer where a missing regex_mapping from an existing config_file causes logprep to crash continuously

Details

• set upper python version limit to <3.12.4 and revise github workflows by @dtrai2 in #614
• configuration overwrites with default values by @ekneg54 in #613
• field_manager does not writes to a non list target field if extend_target_list by @ekneg54 in #616
• remove normalizer processor by @dtrai2 in #400
• improve timestamper processor documentation by @ekneg54 in #618
• Pseudonymizer late error on non existing regexmapping by @ekneg54 in #617
• add helm chart by @ekneg54 in #606
• remove elasticsearch_output connector by @ekneg54 in #620
• add helm chart release to release pipeline by @ekneg54 in #621
• prepare release 13 by @ekneg54 in #623

Full Changelog: v12.0.0...v13.0.0

logprep-13.0.0

Logprep helm chart

v12.0.0

Breaking

• pseudonymizer change rule config field pseudonyms to mapping
• clusterer change rule config field target to source_fields
• generic_resolver change rule config field append_to_list to extend_target_list
• hyperscan_resolver change rule config field append_to_list to extend_target_list
• calculator now adds the error tag _calculator_missing_field_warning to the events tag field instead of _calculator_failure in case of missing field in events
• domain_label_extractor now writes _domain_label_extractor_missing_field_warning tag to event tags in case of missing fields
• geoip_enricher now writes _geoip_enricher_missing_field_warning tag to event tags in case of missing fields
• grokker now writes _grokker_missing_field_warning tag to event tags instead of _grokker_failure in case of missing fields
• requester now writes _requester_missing_field_warning tag to event tags instead of _requester_failure in case of missing fields
• timestamp_differ now writes _timestamp_differ_missing_field_warning tag to event tags instead of _timestamp_differ_failure in case of missing fields
• timestamper now writes _timestamper_missing_field_warning tag to event tags instead of _timestamper_failure in case of missing fields
• rename --thread_count parameter to --thread-count in http generator
• removed --report parameter and feature from http generator
• when using extend_target_list in the field managerthe ordering of the given source fields is now preserved
• logprep now exits with a negative exit code if pipeline restart fails 5 times
  • this was implemented because further restart behavior should be configured on level of a system init service or container orchestrating service like k8s
  • the restart_count parameter is configurable. If you want the old behavior back, you can set this parameter to a negative number
• logprep now exits with a exit code of 2 on configuration errors

Features

• add UCL into the quickstart setup
• add logprep http output connector
• add pseudonymization tools to logprep -> see: logprep pseudo --help
• add restart_count parameter to configuration
• add option mode to pseudonymizer processor and to pseudonymization tools to chose the AES Mode for encryption and decryption
• add retry mechanism to opensearch parallel bulk, if opensearch returns 429 rejected_execution_exception

Improvements

• remove logger from Components and Factory signatures
• align processor architecture to use methods like write_to_target, add_field_to and get_dotted_field_value when reading and writing from and to events
  • required substantial refactoring of the hyperscan_resolver, generic_resolver and template_replacer
• change pseudonymizer, pre_detector, selective_extractor processors and pipeline to handle extra_data the same way
• refactor clusterer, pre_detector and pseudonymizer processors and change rule_tree so that the processor do not require process override
  • required substantial refactoring of the clusterer
• handle missing fields in processors via _handle_missing_fields from the field_manager
• add LogprepMPQueueListener to outsource logging to a separate process
• add a single Queuehandler to root logger to ensure all logs were handled by LogprepMPQueueListener
• refactor http_generator to use a logprep http output connector
• ensure all cached_properties are populated during setup time

Details

• remove logger from signatures by @ekneg54 in #589
• add queuelistener for logging by @ekneg54 in #590
• remove pex build by @ekneg54 in #587
• Align processor architecture by @ppcad in #583
• integrate UCL into quickstart setup by @dtrai2 in #582
• remove arrow dependency by @ekneg54 in #595
• refactor http generator output to http_output connector by @ekneg54 in #591
• make extend_target_list preserve ordering by @ekneg54 in #593
• add depseudo tools by @ekneg54 in #599
• add calling setup during configuration verify by @ekneg54 in #601
• fix quickstart compose setup for WSL by @ekneg54 in #604
• revise restart behavior by @ekneg54 in #602
• add option to pseudonymizer to use different encryption modes by @ekneg54 in #600
• add retry mechanism to write_backlog by @dtrai2 in #605
• Prepare Release v12 by @dtrai2 in #607

Full Changelog: v11.3.0...v12.0.0

v11.3.0

Features

• add gzip handling to http_input connector
• adds advanced logging configuration
  • add configurable log format
  • add configurable datetime formate in logs
  • makes hostname available in custom log formats
  • add fine grained log level configuration for every logger instance

Improvements

• rename logprep.event_generator module to logprep.generator
• shorten logger instance names

Bugfix

• fixes exposing OpenSearch/ElasticSearch stacktraces in log when errors happen by making loglevel configurable for loggers opensearch and elasticsearch
• fixes the logprep quickstart profile

Details

• Handle Opensearch Stacktraces by @dtrai2 in #577
• add gzip handling to http_input by @ekneg54 in #581
• Add Flush Timeout to s3 Connector by @saegel in #553
• rename event generator module by @ekneg54 in #585
• advanced logging configuration by @ekneg54 in #584
• fix logprep quickstart profile by @ekneg54 in #586
• prepare release 11.3.0 by @ekneg54 in #588

Full Changelog: v11.2.1...v11.3.0

v11.2.1

Bugfix

• fixes bug, that leads to spawning exporter http server always on localhost

Details

• fix bug prometheus exporter server uvicorn only on localhost by @ekneg54 in #580

Full Changelog: v11.2.0...v11.2.1

v11.2.0

Features

• expose metrics via uvicorn webserver
  • makes all uvicorn configuration options possible
  • add security best practices to server configuration
• add following metrics to http_input connector
  • nummer_of_http_requests
  • message_backlog_size

Bugfix

• fixes a bug in grokker rules, where common field prefixes wasn't possible
• fixes bug where missing key in credentials file leads to AttributeError

Details

• Fix AttributeError due to missing key in credentials file by @djkhl in #574
• Fix grokker rules that have fields with common prefixes by @dtrai2 in #571
• export metrics via uvicorn asgi app by @ekneg54 in #576
• add http input connector metrics by @ekneg54 in #578

Full Changelog: v11.1.0...v11.2.0

v11.1.0

11.1.0

Features

• new documentation part with security best practices which compiles to user_manual/security/best_practices.html
  • also comes with excel export functionality of given best practices
• add basic auth to http_input

Bugfix

• fixes a bug in http connector leading to only first process working
• fixes the broken gracefull shutdown behaviour

Details

• fix http connector only first process working by @ekneg54 in #566
• Add overview with best practices to docs by @dtrai2 in #559
• fix broken gracefull shutdown by @ekneg54 in #568
• Add Basic Auth to HTTP Connector by @herrfeder in #569

Full Changelog: v11.0.1...v11.1.0

v11.0.1

Bugfix

• fixes a bug where the pipeline index increases on every restart of a failed pipeline
• fixes closed log queue issue by run logging in an extra process

Details

• Fix mtls credentials warning by @djkhl in #558
• remove fastapi from depenencies by @ekneg54 in #557
• make pipeline index stable by @ekneg54 in #562
• fix multiprocessing logging by @ekneg54 in #563

Full Changelog: v11.0.0...v11.0.1

v11.0.0

Breaking

• configuration of Authentication for getters is now done by new introduced credentials file

Features

• introducing an additional file to define the credentials for every configuration source
• retrieve oauth token automatically from different oauth endpoints
• retrieve configruation with mTLS authentication
• reimplementation of HTTP Input Connector with following Features:
  • Wildcard based HTTP Request routing
  • Regex based HTTP Request routing
  • Improvements in thread-based runtime
  • Configuration and possibility to add metadata

Improvements

• remove versioneer dependency in favor of setuptools-scm

Bugfix

• fix version string of release versions
• fix version string of container builds for feature branches
• fix merge of config versions for multiple configs

Details

• fix documentation examples by @ekneg54 in #548
• Update FDA to version 0.21.0 in quickstart example by @clumsy9 in #544
• Automatic oauth2 token retrieval by @dtrai2 in #531
• exchange version string generation from versioneer to setuptools-scm by @ekneg54 in #549
• fix merge of version string for multiple configs by @ekneg54 in #556
• Expand authentication with mTLS by @djkhl in #554
• Reimplementation HTTP Connector by @herrfeder in #538

Full Changelog: v10.0.4...v11.0.0