Skip to content

Latest commit

 

History

History
219 lines (207 loc) · 19.7 KB

CHANGELOG-developer.next.asciidoc

File metadata and controls

219 lines (207 loc) · 19.7 KB
Raw

This changelog is intended for community Beat developers. It covers the major breaking changes to the internal APIs in the official Beats and changes related to developing a Beat like code generators or fields.yml. Only the major changes will be covered in this changelog that are expected to affect community developers. Each breaking change added here should have an explanation on how other Beats should be migrated.

Note: This changelog documents the current changes which are not yet present in an actual release.

Beats version HEAD

The list below covers the major changes between 7.0.0-rc2 and main only.

Breaking changes

  • Move Fields from package libbeat/common to libbeat/mapping. 11198

  • For "metricbeat style" generated custom beats, the mage target GoTestIntegration has changed to GoIntegTest and GoTestUnit has changed to GoUnitTest. 13341

  • Build docker and kubernetes features only on supported platforms. 13509

  • Need to register new processors to be used in the JS processor in their init functions. 13509

  • The custom beat generator now uses mage instead of python, mage GenerateCustomBeat can be used to create a new beat, and mage vendorUpdate to update the vendored libbeat in a custom beat. 13610

  • Altered all remaining uses of mapval to use the renamed and enhanced version: go-lookslike instead, which is a separate project. The mapval tree is now gone. 14165

  • Move light modules to OSS. 14369

  • Deprecate test flags, generate and update_expected, in favor of data. 15292

  • Python 3 is required now to run python tests and tools. 14798

  • The type memqueue.Broker is no longer exported; instead of memqueue.NewBroker, call memqueue.NewQueue (which provides the same public interface). 16667

  • The disk spool types spool.Spool and spool.Settings have been renamed to the internal types spool.diskSpool and spool.settings. 16693

  • queue.Eventer has been renamed to queue.ACKListener 16691

  • Require logger as first parameter for outputs.transport.transport#ProxyDialer and outputs.elasticsearch.client#BulkReadItemStatus. 16761

  • The libbeat/outputs/transport package has been moved to libbeat/common/transport. 16734

  • The libbeat/outputs/tls.go file has been removed. All exported symbols in that file (libbeat/outputs.) are now available as libbeat/common/tlscommon.. 16734

  • The newly generated Beats are using go modules to manage dependencies. 16288

  • Extract Elasticsearch client logic from outputs/elasticsearch package into new esclientleg package. 16150

  • Rename queue.BufferConfig.Events to queue.BufferConfig.MaxEvents. 17622

  • Remove queue.Feature and replace queue.RegisterType with queue.RegisterQueueType. 17666

  • Introduce APM libbeat instrumentation. Publish method on Client interface now takes a Context as first argument. 17938

  • The way configuration files are generated has changed to make it easier to customize parts of the config without requiring changes to libbeat config templates. Generation is now fully based on Go text/template and no longer uses file concatenation to generate the config. Your magefile.go will require a change to adapt the devtool API. See the pull request for more details. 18148

  • The Elasticsearch client settings expect the API key to be raw (not base64-encoded). 18939 18945

  • management.ConfigManager has been renamed to management.Manager. 19114

  • UpdateStatus has been added to the management.Manager interface. 19114

  • Remove common.MapStrPointer parameter from cfgfile.Runnerfactory interface. 19135

  • Replace ACKCount, ACKEvents, and ACKLastEvent callbacks with ACKHandler and interface in beat.ClientConfig. 19632

  • Remove global ACK handler support via SetACKHandler from publisher pipeline. 19632

  • Make implementing Close required for reader.Reader interfaces. 20455

  • Remove NumCPU as clients should update the CPU count on the fly in case of config changes in a VM. 23154

  • Remove Metricbeat EventFetcher and EventsFetcher interface. Use the reporter interface instead. 25093

  • Update Darwin build image to a debian 10 base that increases the MacOS SDK and minimum supported version used in build to 10.14. 24193

  • Removed the common.Float type. 28279 28280 28376

  • Removed Beat generators. 28816

  • libbeat.logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. 15544 28573

  • Removed deprecated disk spool from Beats. Use disk queue instead. 28869

  • Wildcard fields no longer have a default ignore_above setting of 1024. 30096 30668

  • Remove common.MapStr and use mapstr.M from github.com/elastic/elastic-agent-libs instead. 31420

  • Remove queue.Consumer. Queues can now be read via a Get call directly on the queue object. 31502

  • The queue.Batch API now provides access to individual events instead of an array. 31699

  • Rename queue.Batch.ACK() to queue.Batch.Done(). 31903

  • queue.ACKListener has been removed. Queue configurations now accept an explicit callback function for ACK handling. 35078

  • Split split httpmon out of x-pack/filebeat/input/internal/httplog. 36385

  • Beats publishing pipeline does not propagate the close signal to its clients any more. It’s responsibility of the user to close the pipeline client. 38197 38556

  • Debug log entries from the acker (stateful ack …​ or stateless ack …​) removed. 39672

  • Rename x-pack/filebeat websocket input to streaming. 40264 40421

  • Journald input now calls journalctl instead of using github.com/coreos/go-systemd/[email protected]/sdjournal, the CGO dependency has been removed from Filebeat 40061

  • System module events now contain input.type: systemlogs instead of input.type: log when harvesting log files, however the ingest pipeline sets it back to the original input (log or journald). 41246

Bugfixes

  • Handle the starting of namespace and node watchers for metadata enrichment according to add_resource_metadata configuration.https://github.com/elastic/beats/pull/38762[38762]

  • Fix multiple metricbeat instances reporting same metrics when using autodiscover with provider kubernetes, and ensure leader elector is always running in autodiscover mode.https://github.com/elastic/beats/pull/38471[38471]

  • Fix how Prometheus histograms are calculated when percentiles are provide.https://github.com/elastic/beats/pull/36537[36537]

  • Stop using mage:import in community beats. This was ignoring the vendorized beats directory for some mage targets, using the code available in GOPATH, this causes inconsistencies and compilation problems if the version of the code in the GOPATH is different to the vendored one. Use of mage:import will continue to be unsupported in custom beats till beats is migrated to go modules, or mage supports vendored dependencies. 13998 14162

  • Metricbeat module builders call host parser only once when instantiating light modules. 20149

  • Fix export dashboard command when running against Elastic Cloud hosted Kibana. 22746

  • Remove event.dataset (ECS) annotion from libbeat.logp. 27404

  • Errors should be thrown as errors. Metricsets inside Metricbeat will now throw errors as the error log level. 27804

  • Avoid panicking in add_fields processor when input event.Fields is a nil map. 28219

  • Drop event batch when get HTTP status 413 from Elasticsearch to avoid infinite loop 14350 29368

  • Allow to use metricbeat for named mssql instances. 24076 30859

  • Setting DEV=true when running mage build now correctly generates binaries without optimisations and with debug symbols 31955

  • The beat.cgroup.memory.mem.usage.bytes metric is now a gauge 31582 32652

  • Fix the integration testcase docker port mapping for sql and oracle modules 34221

  • Fix the ingest pipeline for mysql slowlog to parse schema name with dash 34372

  • Fix the multiple host support for mongodb module 34624

  • Skip HTTPJSON flakey test. 34929 35138

  • Fix ingest pipeline for panw module to parse url scheme correctly 35757

  • Renamed an httpjson input metric to follow naming conventions. httpjson_interval_pages_total was renamed to httpjson_interval_pages because the _total suffix is reserved for counters. 35933 36169

  • Fixed some race conditions in tests 36185

  • Fix Stringer implementation of fingerprint processor 35174

  • Re-enable HTTPJSON fixed flakey test. 34929 36525

  • Make winlogbeat/sys/wineventlog follow the unsafe.Pointer rules. 36650

  • Cleaned up documentation errors & fixed a minor bug in Filebeat Azure blob storage input. 36714

  • Fix copy arguments for strict aligned architectures. 36976

  • Fix panic when more than 32767 pipeline clients are active. 38197 38556

  • Skip flakey metrics test on windows in filebeat httpjson input. 39676 39678

  • Fix flakey test on Windows 2022 in packetbeat/route. 39698 39822

  • Fix bug in minimum length for request trace logging. 39834

  • Close connections properly in Filbeat’s HTTPJSON input. 39790

  • Add the Offset property to libbeat/reader.Message to store the total number of bytes read and discarded before generating the message. This enables inputs to accurately determine how much data has been read up to the message, using Message.Bytes + Message.Offset. 39873 39653

  • AWS CloudWatch Metrics record previous endTime to use for next collection period and change log.logger from cloudwatch to aws.cloudwatch. 40870

Added

  • Update watchers to be shared between metricsets in Kubernetes module. 37332

  • Add new metricset in Kubernetes module, state_namespace. 36406

  • Add configuration for APM instrumentation and expose the tracer trough the Beat object. 17938

  • Make the behavior of clientWorker and netClientWorker consistent when error is returned from publisher pipeline

  • Metricset generator generates beta modules by default now. 10657

  • The beat.Event accessor methods now support @metadata keys. 10761

  • Assertion for documented fields in tests fails if any of the fields in the tested event is documented as an alias. 10921

  • Support for Logger in the Metricset base instance. 11106

  • Filebeat modules can now use ingest pipelines in YAML format. 11209

  • Prometheus helper for metricbeat contains now Namespace field for prometheus.MetricsMappings 11424

  • Update Jinja2 version to 2.10.1. 11817

  • Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. 11777,https://github.com/elastic/beats/pull/12065[12065],https://github.com/elastic/beats/pull/12067[12067],https://github.com/elastic/beats/pull/12160[12160]

  • Update urllib3 version to 1.24.2 11930

  • Add libbeat/common/cleanup package. 12134

  • New helper to check for leaked goroutines on tests. 12106

  • Only Load minimal template if no fields are provided. 12103

  • Add new option IgnoreAllErrors to libbeat.common.schema for skipping fields that failed while converting. 12089

  • Deprecate setup cmds for template and ilm-policy. Add new setup cmd for index-management. 12132

  • Use the go-lookslike library for testing in heartbeat. Eventually the mapval package will be replaced with it. 12540

  • New ReporterV2 interfaces that can receive a context on Fetch(ctx, reporter), or Run(ctx, reporter). 11981

  • Generate configuration from mage for all Beats. 12618

  • Add ClientFactory to TCP input source to add SplitFunc/NetworkFuncs per client. 8543

  • Introduce beat.OutputChooses publisher mode. 12996

  • Ensure that beat.Processor, beat.ProcessorList, and processors.ProcessorList are compatible and can be composed more easily. 12996

  • Add support to close beat.Client via beat.CloseRef (a subset of context.Context). 13031

  • Add checks for types and formats used in fields definitions in fields.yml files. 13188

  • Makefile included in generator copies files from beats repository using git archive instead of cp. 13193

  • Strip debug symbols from binaries to reduce binary sizes. 12768

  • Compare event by event in testadata framework to avoid sorting problems 13747

  • Added a default_field option to fields in fields.yml to offer a way to exclude fields from the default_field list. 14262 14341

  • supported-versions.yml can be used in metricbeat python system tests to obtain the build args for docker compose builds. 14520

  • Fix dropped errors in the tests for the metricbeat Azure module. 13773

  • New mage target for Functionbeat: generate pkg folder to make manager easier. 15880

  • Add support for MODULE environment variable in mage goIntegTest in metricbeat to run integration tests for a single module. 17147

  • Add support for a TEST_TAGS environment variable to add tags for tests selection following go build tags semantics, this environment variable is used by mage test targets to add build tags. Python tests can also be tagged with a decorator (@beat.tag('sometag')). 16937 17075

  • Add fields validation for histogram subfields. 17759

  • Add IP* fields to fields.yml generator script in Filebeat. 17998 18256

  • Events intended for the Elasticsearch output can now take an op_type metadata field of type events.OpType or string to indicate the op_type to use for bulk indexing. 12606

  • Remove vendor folder from repository. 18655

  • Added SQL helper that can be used from any Metricbeat module 18955

  • Update Go version to 1.14.4. 19753

  • Update Go version to 1.14.7. 20508

  • Add packaging for docker image based on UBI minimal 8. 20576

  • Make the mage binary used by the build process in the docker container to be statically compiled. 20827

  • Add Pensando distributed firewall module. 21063

  • Update ecszap to v0.3.0 for using ECS 1.6.0 in logs 22267

  • Add support for customized monitoring API. 22605

  • Update Go version to 1.15.7. 22495

  • Update Go version to 1.15.8. 23955

  • Update Go version to 1.15.9. 24442

  • Update Go version to 1.15.10. 24606

  • Update Go version to 1.15.12. 25629

  • Update Go version to 1.16.4. 25346 25671

  • Add sorting to array fields for generated data files (*-generated.json) 25320

  • Update to go-concert 0.2.0 27162

  • Update Go version to 1.16.5. 26182 26186

  • Introduce libbeat/beat.Beat.OutputConfigReloader 28048

  • Update Go version to 1.17.1. 27543

  • Whitelist GCP_* environment variables in dev tools 28364

  • Add support for credentials_json in gcp module, all metricsets 29584

  • Add gcp firestore metricset. 29918

  • Added TESTING_FILEBEAT_FILEPATTERN option for filebeat module pytests 30103

  • Improve tests files with shorter statements. 35667

  • Add gcp dataproc metricset. 30008

  • Add Github action for linting

  • Add regex support for drop_fields processor.

  • Improve compatibility and reduce flakyness of Python tests 31588

  • Added .python-version file 32323

  • Use T.TempDir to create temporary test directory 33082

  • Add an option to disable event normalization when creating a beat.Client. 33657

  • Add the file path of the instance lock on the error when it’s is already locked 33788

  • Add DropFields processor to js API 33458

  • Add support for different folders when testing data 34467

  • Add logging of metric registration in inputmon. 35647

  • Add Okta API package for entity analytics. 35478

  • Add benchmarking to HTTPJSON input testing. 35138

  • Allow non-AWS endpoints for testing Filebeat awss3 input. 35496 35520

  • Add AUTH (username) and SSL/TLS support for Redis module 35240

  • Pin PyYAML version to 5.3.1 to avoid CI errors temporarily 36091

  • Skip dependabot updates for github.com/elastic/mito. 36158

  • Add device handling to Okta API package for entity analytics. 35980

  • Make Filebeat HTTPJSON input process responses sequentially. 36493

  • Add initial infrastructure for a caching enrichment processor. 36619

  • Add file-backed cache for cache enrichment processor. 36686 36696

  • Elide retryable HTTP client construction in Filebeat HTTPJSON and CEL inputs if not needed. 36916

  • Allow assignment of packetbeat protocols to interfaces. 36564 36852

  • Add Active Directory entity collector for Filebeat entity analytics. 37854

  • Make logs for empty and small files less noisy when using fingerprint file identity in filestream. 38421

  • Improve robustness and error reporting from packetbeat default route testing. 39757

  • Move x-pack/filebeat/input/salesforce jwt import to v5. 39823

  • Drop x-pack/filebeat/input dependency on github.com/lestrrat-go/jwx/v2. 39968

  • Added ignore_empty_values flag in decode_cef Filebeat processor. 40268

  • Bump version of elastic/toutoumomoma to remove internal forks of stdlib debug packages. 40325

  • Refactor x-pack/filebeat/input/websocket for generalisation. 40308

  • Add a configuration option for TCP/UDP network type. 40407 40623

  • Added debug logging to parquet reader in x-pack/libbeat/reader. 40651

  • Added filebeat debug histograms for s3 object size and events per processed s3 object. 40775

  • Simplified GCS input state checkpoint calculation logic. 40878 40937

  • Simplified Azure Blob Storage input state checkpoint calculation logic. 40674 40936

  • Add field redaction package. 40997

  • Add support for marked redaction to x-pack/filebeat/input/internal/private 41212

  • Add support for collecting Okta role and factor data for users with filebeat entityanalytics input. 41044

Deprecated

  • Deprecated the common.Float type. 28279 28280

  • Deprecate Beat generators. 28814

  • Remove garbled PE executable test from auditbeat FIM module testing. 35705 35724