[ Documentation request ] Grafana datasource CA certificate

[froblesmartin]

Brief description

When configuring my Grafana instance with the datasource from the Prometheus endpoint you provide in the observability configuration, I created my own root CA certificate which I configured in the observability section, and then in the Grafana datasource configuration I stored the client certificate and key. Then TLS failed:

Post "https://*****.tmprl.cloud/prometheus/api/v1/query": tls: failed to verify certificate: x509: certificate signed by unknown authority - There was an error returned querying the Prometheus API.

I was then setting the root CA I created in the configuration for the CA certificate in Grafana, but I was still getting the same error.

Then I finally figured out that the certificate validation that fails is the HTTPS one and not the Prometheus TLS one. I then had to configure in there the Let's Encrypt ISRG Root X1 (https://letsencrypt.org/certificates/) pem certificate and it worked.

Your recommended content

I think it would be worth mentioning this in the docs, I guess this happened because where I run Grafana (in K8s) the truststore does not have the Let's Encrypt root CA certificate.

https://docs.temporal.io/cloud/metrics#data-sources-configuration-for-temporal-cloud-and-sdk-metrics-in-grafana