Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

metrics_exporter: add support for filter metrics inside processors #9341

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

metrics_exporter: add support for filter metrics inside processors #9341

wants to merge 1 commit into from

Conversation

cw-Guo
Copy link

@cw-Guo cw-Guo commented Sep 4, 2024

Fixes #7391

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cw-Guo
Copy link
Author

cw-Guo commented Sep 4, 2024

test.yaml

service:
  daemon: false
  flush: 1
  grace: 30
  http_listen: 0.0.0.0
  http_port: 2020
  http_server: true

pipeline:
  inputs:
    - name: dummy
      dummy: '{"log": "test"}'
      rate: 10
      processors:
        logs:
          - name: kubernetes
            dummy_meta: true
          - name: modify
            add:
              - modify true
          - name: content_modifier
            action: insert
            key: content_modifier
            value: insertd
            
  outputs:
    - name: stdout
      match: '*'
      processors:
        logs:
          - name: modify
            alias: modify-out
            add:
              - modify-out true

valgrind check

fluent-bit ➜ /workspaces/fluent-bit/build (feat/processor-metrics) $ valgrind --leak-check=full ./bin/fluent-bit -c test.yaml
...
[8] dummy.0: [[1725433074.258322253, {}], {"log"=>"test", "kubernetes"=>{"dummy"=>"Wed Sep  4 06:57:54 2024"}, "modify"=>"true", "content_modifier"=>"insertd", "modify-out"=>"true"}]
[9] dummy.0: [[1725433074.358387212, {}], {"log"=>"test", "kubernetes"=>{"dummy"=>"Wed Sep  4 06:57:54 2024"}, "modify"=>"true", "content_modifier"=>"insertd", "modify-out"=>"true"}]
[2024/09/04 06:57:54] [ info] [engine] service has stopped (0 pending tasks)
[2024/09/04 06:57:54] [ info] [input] pausing dummy.0
[2024/09/04 06:57:54] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/09/04 06:57:54] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==60496== 
==60496== HEAP SUMMARY:
==60496==     in use at exit: 0 bytes in 0 blocks
==60496==   total heap usage: 333,063 allocs, 333,063 frees, 384,177,094 bytes allocated
==60496== 
==60496== All heap blocks were freed -- no leaks are possible
==60496== 
==60496== For lists of detected and suppressed errors, rerun with: -s
==60496== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

metrics output

fluent-bit ➜ /workspaces/fluent-bit/build (feat/processor-metrics) $ curl http://localhost:2020/api/v2/metrics/prometheus
# HELP fluentbit_uptime Number of seconds that Fluent Bit has been running.
# TYPE fluentbit_uptime counter
fluentbit_uptime{hostname="241383d3ee7f"} 74
# HELP fluentbit_input_bytes_total Number of input bytes.
# TYPE fluentbit_input_bytes_total counter
fluentbit_input_bytes_total{name="dummy.0"} 90036
# HELP fluentbit_input_records_total Number of input records.
# TYPE fluentbit_input_records_total counter
fluentbit_input_records_total{name="dummy.0"} 732
# HELP fluentbit_filter_records_total Total number of new records processed.
# TYPE fluentbit_filter_records_total counter
fluentbit_filter_records_total{name="kubernetes.0"} 732
# HELP fluentbit_filter_bytes_total Total number of new bytes processed.
# TYPE fluentbit_filter_bytes_total counter
fluentbit_filter_bytes_total{name="kubernetes.0"} 60024
# HELP fluentbit_filter_add_records_total Total number of new added records.
# TYPE fluentbit_filter_add_records_total counter
fluentbit_filter_add_records_total{name="kubernetes.0"} 0
# HELP fluentbit_filter_drop_records_total Total number of dropped records.
# TYPE fluentbit_filter_drop_records_total counter
fluentbit_filter_drop_records_total{name="kubernetes.0"} 0
# HELP fluentbit_filter_records_total Total number of new records processed.
# TYPE fluentbit_filter_records_total counter
fluentbit_filter_records_total{name="modify.0"} 732
# HELP fluentbit_filter_bytes_total Total number of new bytes processed.
# TYPE fluentbit_filter_bytes_total counter
fluentbit_filter_bytes_total{name="modify.0"} 62952
# HELP fluentbit_filter_add_records_total Total number of new added records.
# TYPE fluentbit_filter_add_records_total counter
fluentbit_filter_add_records_total{name="modify.0"} 0
# HELP fluentbit_filter_drop_records_total Total number of dropped records.
# TYPE fluentbit_filter_drop_records_total counter
fluentbit_filter_drop_records_total{name="modify.0"} 0
# HELP fluentbit_output_proc_records_total Number of processed output records.
# TYPE fluentbit_output_proc_records_total counter
fluentbit_output_proc_records_total{name="stdout.0"} 721
# HELP fluentbit_output_proc_bytes_total Number of processed output bytes.
# TYPE fluentbit_output_proc_bytes_total counter
fluentbit_output_proc_bytes_total{name="stdout.0"} 88683
# HELP fluentbit_output_errors_total Number of output errors.
# TYPE fluentbit_output_errors_total counter
fluentbit_output_errors_total{name="stdout.0"} 0
# HELP fluentbit_output_retries_total Number of output retries.
# TYPE fluentbit_output_retries_total counter
fluentbit_output_retries_total{name="stdout.0"} 0
# HELP fluentbit_output_retries_failed_total Number of abandoned batches because the maximum number of re-tries was reached.
# TYPE fluentbit_output_retries_failed_total counter
fluentbit_output_retries_failed_total{name="stdout.0"} 0
# HELP fluentbit_output_dropped_records_total Number of dropped records.
# TYPE fluentbit_output_dropped_records_total counter
fluentbit_output_dropped_records_total{name="stdout.0"} 0
# HELP fluentbit_output_retried_records_total Number of retried records.
# TYPE fluentbit_output_retried_records_total counter
fluentbit_output_retried_records_total{name="stdout.0"} 0
# HELP fluentbit_filter_records_total Total number of new records processed.
# TYPE fluentbit_filter_records_total counter
fluentbit_filter_records_total{name="modify-out"} 731
# HELP fluentbit_filter_bytes_total Total number of new bytes processed.
# TYPE fluentbit_filter_bytes_total counter
fluentbit_filter_bytes_total{name="modify-out"} 92837
# HELP fluentbit_filter_add_records_total Total number of new added records.
# TYPE fluentbit_filter_add_records_total counter
fluentbit_filter_add_records_total{name="modify-out"} 0
# HELP fluentbit_filter_drop_records_total Total number of dropped records.
# TYPE fluentbit_filter_drop_records_total counter
fluentbit_filter_drop_records_total{name="modify-out"} 0
# HELP fluentbit_process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE fluentbit_process_start_time_seconds gauge
fluentbit_process_start_time_seconds{hostname="241383d3ee7f"} 1725432955
# HELP fluentbit_build_info Build version information.
# TYPE fluentbit_build_info gauge
fluentbit_build_info{hostname="241383d3ee7f",version="3.1.7",os="linux"} 1725432955
# HELP fluentbit_hot_reloaded_times Collect the count of hot reloaded times.
# TYPE fluentbit_hot_reloaded_times gauge
fluentbit_hot_reloaded_times{hostname="241383d3ee7f"} 0
# HELP fluentbit_storage_chunks Total number of chunks in the storage layer.
# TYPE fluentbit_storage_chunks gauge
fluentbit_storage_chunks 2
# HELP fluentbit_storage_mem_chunks Total number of memory chunks.
# TYPE fluentbit_storage_mem_chunks gauge
fluentbit_storage_mem_chunks 2
# HELP fluentbit_storage_fs_chunks Total number of filesystem chunks.
# TYPE fluentbit_storage_fs_chunks gauge
fluentbit_storage_fs_chunks 0
# HELP fluentbit_storage_fs_chunks_up Total number of filesystem chunks up in memory.
# TYPE fluentbit_storage_fs_chunks_up gauge
fluentbit_storage_fs_chunks_up 0
# HELP fluentbit_storage_fs_chunks_down Total number of filesystem chunks down.
# TYPE fluentbit_storage_fs_chunks_down gauge
fluentbit_storage_fs_chunks_down 0
# HELP fluentbit_input_ingestion_paused Is the input paused or not?
# TYPE fluentbit_input_ingestion_paused gauge
fluentbit_input_ingestion_paused{name="dummy.0"} 0
# HELP fluentbit_input_storage_overlimit Is the input memory usage overlimit ?.
# TYPE fluentbit_input_storage_overlimit gauge
fluentbit_input_storage_overlimit{name="dummy.0"} 0
# HELP fluentbit_input_storage_memory_bytes Memory bytes used by the chunks.
# TYPE fluentbit_input_storage_memory_bytes gauge
fluentbit_input_storage_memory_bytes{name="dummy.0"} 1230
# HELP fluentbit_input_storage_chunks Total number of chunks.
# TYPE fluentbit_input_storage_chunks gauge
fluentbit_input_storage_chunks{name="dummy.0"} 1
# HELP fluentbit_input_storage_chunks_up Total number of chunks up in memory.
# TYPE fluentbit_input_storage_chunks_up gauge
fluentbit_input_storage_chunks_up{name="dummy.0"} 1
# HELP fluentbit_input_storage_chunks_down Total number of chunks down.
# TYPE fluentbit_input_storage_chunks_down gauge
fluentbit_input_storage_chunks_down{name="dummy.0"} 0
# HELP fluentbit_input_storage_chunks_busy Total number of chunks in a busy state.
# TYPE fluentbit_input_storage_chunks_busy gauge
fluentbit_input_storage_chunks_busy{name="dummy.0"} 0
# HELP fluentbit_input_storage_chunks_busy_bytes Total number of bytes used by chunks in a busy state.
# TYPE fluentbit_input_storage_chunks_busy_bytes gauge
fluentbit_input_storage_chunks_busy_bytes{name="dummy.0"} 0
# HELP fluentbit_output_upstream_total_connections Total Connection count.
# TYPE fluentbit_output_upstream_total_connections gauge
fluentbit_output_upstream_total_connections{name="stdout.0"} 0
# HELP fluentbit_output_upstream_busy_connections Busy Connection count.
# TYPE fluentbit_output_upstream_busy_connections gauge
fluentbit_output_upstream_busy_connections{name="stdout.0"} 0
# HELP fluentbit_output_chunk_available_capacity_percent Available chunk capacity (percent)
# TYPE fluentbit_output_chunk_available_capacity_percent gauge
fluentbit_output_chunk_available_capacity_percent{name="stdout.0"} 100

@edsiper edsiper added this to the Fluent Bit v3.2.0 milestone Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
Fluent Bit v3.2.0
Development

Successfully merging this pull request may close these issues.

Filters in Input Processors do not Emit Metrics
2 participants
@cw-Guo @edsiper