remove hardcoded job labels in prometheus queries

[ztzxt]

Feature Request

Is your feature request related to a problem? Please describe:

We deployed TiDb via tidb-operator. We are also configuring PodMonitor for prometheus-operator for external prometheus. When we do this, some of the panels in the dashboard are missing. This is due to hard-coded job labels in the prometheus queries such as this. If there were a way to configure these labels or some prefix to them, whole dashboard would work with prometheus-operator.

Describe the feature you'd like:

Ability to overwrite job label in Prometheus queries

Describe alternatives you've considered:

• Refactoring queries in a way they wouldn't need the job label. Since current metrics in the queries are too generic, probably some other metric should be used.

• Way to configure job label. Preferably it is templated when providing the config. If it can't be templated, it still would solve our problem.

• Using metricRelabelings in PodMonitor. Unfortunately, we can't do this as we plan to have more than one cluster scraped by a single Prometheus. We need a way to separate them by at least a namespace

This is how the dashboard looks now for us: