[bitnami/airflow] Metrics based on StatsD

[juan131]

Description of the change

⚠️ MAJOR VERSION

This PR version replaces exposing Prometheus metrics using the Airflow prometheus exporter, that exposes metrics based on the data retrieved from the database, by configuring Airflow components to send StatsD metrics to the StatsD exporter that transforms them into Prometheus metrics. Find more information about this approach in the Apache Airflow official documentation.

Benefits

Follow upstream approach for exposing metrics.

Possible drawbacks

Dashboards based on old metrics won't be valid anymore.

Applicable issues

• fixes Enabling statsd metrics on Airflow Hem Chart #9385
• fixes airflow-exporter providing limited metrics. #30185

Additional information

N/A

Checklist

• Chart version bumped in Chart.yaml according to semver. This is not necessary when the changes only affect README.md files.
• Variables are documented in the values.yaml and added to the README.md using readme-generator-for-helm
• Title of the pull request follows this pattern [bitnami/<name_of_the_chart>] Descriptive title
• All commits signed off and in agreement of Developer Certificate of Origin (DCO)

[javsalgar]

LGTM but the CI is failing

[dgomezleon]

LGTM

[carrodher]

Shouldn't this be docker.io/bitnami/statsd-exporter:0.28.0-debian-12-r0?
I though this is autogenerated by EAM, so not sure where this invalid airflow-statsd comes from

[juan131]

My fault! Manually edited, fixing it on a new PR

[sergiofteixeira]

got this enabled but still only get basic metrics such as :

garbage collection cycles. go_gc_duration_seconds{quantile="0"} 4.1798e-05 go_gc_duration_seconds{quantile="0.25"} 6.4768e-05 go_gc_duration_seconds{quantile="0.5"} 7.4839e-05 go_gc_duration_seconds{quantile="0.75"} 9.0782e-05 go_gc_duration_seconds{quantile="1"} 0.000147398 go_gc_duration_seconds_sum 0.000893869 go_gc_duration_seconds_count 11 statsd_metric_mapper_cache_length 0

etc, no airflow_ metrics, anything else im missing?

airflow.cfg has statsd enabled:

[metrics]
statsd_host=airflow-statsd-metrics
statsd_on=True
statsd_port=9125
statsd_prefix=airflow

Edit: found the issue, with network policies enabled, statsd cant collect metrics.

[JiffsMaverick]

@sergiofteixeira I also faced this issue. Disabling the Network Policy helped me as well. The problem is related to the Network Policy configuration specifying the TCP protocol instead of UDP. I created a pull request (#30990) to fix this.

[sergiofteixeira]

@sergiofteixeira I also faced this issue. Disabling the Network Policy helped me as well. The problem is related to the Network Policy configuration specifying the TCP protocol instead of UDP. I created a pull request (#30990) to fix this.

thanks