k8s-monitoring v2.0.7 broke the pod job label

[stefanandres]

Hi, after upgrading to 2.0.7 with the change of #1175 we don't get any job labels from pods anymore.

Before:

After:

It seems the changed rule is
from:

-    // set the job label from the k8s.grafana.com/logs.job annotation if it exists
-    rule {
-      source_labels = ["__meta_kubernetes_pod_annotation_k8s_grafana_com_logs_job"]
-      regex = "(.+)"
-      target_label = "job"
-    }

to

+    rule {
+      source_labels = ["__meta_kubernetes_pod_annotation_k8s_grafana_com_logs_job"]
+      target_label = "job"
+    }

According to the https://grafana.com/docs/alloy/latest/reference/components/loki/loki.relabel/#rule-block this implicitly sets regex = (.*), so I've tested it manually and well:
The job label is missing when set to regex = "(.*)" and the job label is there when regex = "(.+)" is set.

Now it get's funny:

The job label is already set by this rule

k8s-monitoring-helm/charts/k8s-monitoring/charts/feature-pod-logs/templates/_common_pod_discovery.alloy.tpl

Lines 26 to 32 in 326e18c

	rule {
	source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
	separator = "/"
	action = "replace"
	replacement = "$1"
	target_label = "job"
	}

So the second rule will will be evaluated last and will match with an empty string because of the default (.*) regex.

Proposal

I'd propose the behavior should be changed to use regex = (.+) again.

Workaround (unsuccessful)

I've tried to workaround the issue by using

podLogs:
  enabled: true
  annotations:
    # See https://github.com/grafana/k8s-monitoring-helm/issues/1220
    job: ~

But somehow, this is not deleting the job key from the map and it still ends up as

        rule {
          source_labels = ["__meta_kubernetes_pod_annotation_k8s_grafana_com_logs_job"]
          target_label = "job"
        }

Workaround (successfull, but stupid)

podLogs:
  enabled: true

  # Re-add job relabeling as last rule because the second job relabeling breaks the job label for us
  # See https://github.com/grafana/k8s-monitoring-helm/issues/1220 for more details
  extraDiscoveryRules: |
    rule {
      source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
      separator = "/"
      action = "replace"
      replacement = "$1"
      target_label = "job"
    }

[petewall]

Thanks for the catch! I reproduced this and I'll get this fixed!

[petewall]

fixed in 2.0.9