Support consumer group and topic deletion on the fly

[mcinquin]

Hi,

The exporter is not able to update the list of consumer groups and topics after deletion without a restart.

Same type of issue as #17

Regards.

[omarsmak]

So to understand it correctly, if you delete a topic which consumer consume from that topic, its metrics will be still there, isn't? Which kinda metrics exactly (assuming you are using Prometheus)? Would be nice if you can add an example in order to clarify the issue :)

[mcinquin]

It's related to your PS

P/S: regarding the topic deletion, I haven't added it yet, please report it in a separate issue so I can take a look at it in the next release.

If it's needed, I give you more information and example tomorrow

[omarsmak]

would be great! thank you

[mcinquin]

A part of metrics present at the beginning of the procedure

# HELP kafka_consumer_group_total_lag The total lag of a consumer group behind the head of a topic. This gives the total lags over each partition, it provides good visibility but not a precise measurement since is not partition aware
# TYPE kafka_consumer_group_total_lag gauge
kafka_consumer_group_total_lag{group="toto2",topic="toto",} 0.0
kafka_consumer_group_total_lag{group="toto1",topic="toto",} 0.0

1. I create a new topic called titi and a consumer group called titi1

# HELP kafka_consumer_group_total_lag The total lag of a consumer group behind the head of a topic. This gives the total lags over each partition, it provides good visibility but not a precise measurement since is not partition aware
# TYPE kafka_consumer_group_total_lag gauge
kafka_consumer_group_total_lag{group="toto2",topic="toto",} 0.0
kafka_consumer_group_total_lag{group="titi1",topic="titi",} 0.0
kafka_consumer_group_total_lag{group="toto1",topic="toto",} 0.0

2. I stop the consumer group and delete the topic titi
   The topic titi is still present in the metrics

# HELP kafka_consumer_group_total_lag The total lag of a consumer group behind the head of a topic. This gives the total lags over each partition, it provides good visibility but not a precise measurement since is not partition aware
# TYPE kafka_consumer_group_total_lag gauge
kafka_consumer_group_total_lag{group="toto2",topic="toto",} 0.0
kafka_consumer_group_total_lag{group="titi1",topic="titi",} 0.0
kafka_consumer_group_total_lag{group="toto1",topic="toto",} 0.0

3. I restart the exporter and the topic titi is gone

# HELP kafka_consumer_group_total_lag The total lag of a consumer group behind the head of a topic. This gives the total lags over each partition, it provides good visibility but not a precise measurement since is not partition aware
# TYPE kafka_consumer_group_total_lag gauge
kafka_consumer_group_total_lag{group="toto2",topic="toto",} 0.0
kafka_consumer_group_total_lag{group="toto1",topic="toto",} 0.0

[omarsmak]

Ah I got it. I will dig and see if this could be fixable. Thanks for reporting this @Shini31 ! Appreciated 👍

[omarsmak]

It has been fixed and will be released with 0.0.8

[omarsmak]

@Shini31 0.0.8 has been released. Please give it a try and report any issues. Thanks

[mcinquin]

Hi,

Sorry for the delay.

I just tested the new version and there a small issue with the kafka_consumer_group_member_partition_lag metric.
The samples associated with a deleted topic are still present. If I restart the exporter, the samples are gone

It's the only metric with this issue. All the others have the right behavior.

Topic toto is present

kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="18",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="19",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="16",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="17",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="14",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="15",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="12",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="13",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="10",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="11",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="30",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="31",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="7",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="8",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="5",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="6",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="3",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="4",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="1",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="2",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="0",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="29",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="27",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="28",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="25",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="26",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="23",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="24",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="21",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="22",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="20",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="9",} 0.0

I delete the topic toto

kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="18",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="19",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="16",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="17",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="14",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="15",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="12",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="13",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="10",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="11",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="30",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="31",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="7",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="8",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="5",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="6",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="3",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="4",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="1",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="2",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="0",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="29",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="27",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="28",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="25",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="26",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="23",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="24",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="21",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="22",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="20",} 0.0
kafka_consumer_group_member_partition_lag{group="testtoto1",member="prometheus-exporter-44a33108-0f53-4bf5-ae76-a4c97586dc8d",topic="testoto",partition="9",} 0.0

I restart the exporter

# HELP kafka_consumer_group_member_partition_lag The lag of a consumer member within consumer group behind the head of a given partition of a topic
# TYPE kafka_consumer_group_member_partition_lag gauge

[omarsmak]

Hey @Shini31 sorry I may have missed your last message. Thanks for reporting this, I will need to test this and hopefully get it push to the next version.
As usual, please don't hesitate to open an issue if there is a bug or a new could be worth added.

[mcinquin]

Hey @omarsmak,

No prob :)
As usual, ping me to test the new release.

[omarsmak]

Awesome thanks a lot @Shini31 ! By the way, there is a new release 0.1.0, you can give it a try! (this bug is yet fixed but hopefully it will in version 0.1.1.