Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prometheus label "raft_cluster" to non-QQ raft metrics #12148

Merged
merged 1 commit into from
Sep 4, 2024

Conversation

the-mikedavis
Copy link
Member

@the-mikedavis the-mikedavis commented Aug 28, 2024

Adds a <<"raft_cluster">> tag for ra_metrics emitted the stream coordinator and Khepri. By default Ra sets the cluster name as the metrics_key (if not set in config, like it is for QQs). The prometheus plugin ignores atoms that aren't protocol names when rendering labels so these cluster name values disappear. We can convert them into a map #{<<"raft_cluster">> => atom_to_binary(ClusterName, utf8)} to render properly.

Before:

$ curl -s "localhost:15692/metrics/detailed?family=ra_metrics" | grep rabbitmq_raft_term_total
# TYPE rabbitmq_raft_term_total counter
# HELP rabbitmq_raft_term_total Current Raft term number
rabbitmq_raft_term_total{vhost="/",queue="qq"} 9
rabbitmq_raft_term_total 10

After:

$ curl -s "localhost:15692/metrics/detailed?family=ra_metrics" | grep rabbitmq_raft_term_total
# TYPE rabbitmq_raft_term_total counter
# HELP rabbitmq_raft_term_total Current Raft term number
rabbitmq_raft_term_total{vhost="/",queue="qq"} 9
rabbitmq_raft_term_total{raft_cluster="rabbitmq_metadata"} 10

Connects #9665
Connects #12142

@the-mikedavis the-mikedavis marked this pull request as ready for review August 28, 2024 17:50
@the-mikedavis the-mikedavis marked this pull request as draft August 28, 2024 20:03
By default Ra will use the cluster name as the metrics key. Currently
atom values are ignored by the prometheus plugin's tag rendering
functions, so if you have a QQ and Khepri running and request the
`/metrics/per-object` or `/metrics/detailed` endpoints you'll see values
that don't have labels set for the `ra_metrics` metrics:

    # TYPE rabbitmq_raft_term_total counter
    # HELP rabbitmq_raft_term_total Current Raft term number
    rabbitmq_raft_term_total{vhost="/",queue="qq"} 9
    rabbitmq_raft_term_total 10

With this change we map the name of the Ra cluster to a "raft_cluster"
tag, so instead an example metric might be:

    # TYPE rabbitmq_raft_term_total counter
    # HELP rabbitmq_raft_term_total Current Raft term number
    rabbitmq_raft_term_total{vhost="/",queue="qq"} 9
    rabbitmq_raft_term_total{raft_cluster="rabbitmq_metadata"} 10

This affects metrics for Khepri and the stream coordinator.
@the-mikedavis the-mikedavis changed the title Add a 'raft_cluster' prometheus tag for a Ra cluster name to ra_metrics Add prometheus tags "raft_cluster" to non-QQ raft metrics Aug 30, 2024
@the-mikedavis the-mikedavis marked this pull request as ready for review August 30, 2024 20:15
@the-mikedavis the-mikedavis marked this pull request as draft September 3, 2024 15:50
@the-mikedavis the-mikedavis added this to the 3.13.8 milestone Sep 4, 2024
@the-mikedavis
Copy link
Member Author

This doesn't quite solve #12142 but fixes the issue of these Raft metrics showing up without a label for Khepri and the stream coordinator. There are some larger changes @kjnilsson and I have been looking at for seshat, ra and here to improve how we expose Raft metrics: switching from the ra_metrics table to Ra counters, improving seshat's usability for Prometheus. In the meantime though this is a nearly trivial fix that we can make and easily backport so I will merge this down as-is and leave the rest for future work.

@the-mikedavis the-mikedavis marked this pull request as ready for review September 4, 2024 20:38
@the-mikedavis the-mikedavis merged commit c0a1c0c into main Sep 4, 2024
195 checks passed
@the-mikedavis the-mikedavis deleted the md/raft-cluster-prometheus-tag branch September 4, 2024 20:38
@the-mikedavis the-mikedavis changed the title Add prometheus tags "raft_cluster" to non-QQ raft metrics Add prometheus tag "raft_cluster" to non-QQ raft metrics Sep 4, 2024
@the-mikedavis the-mikedavis changed the title Add prometheus tag "raft_cluster" to non-QQ raft metrics Add prometheus label "raft_cluster" to non-QQ raft metrics Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant