alias not matching prometheus

[Adel-Magebinary]

rate(net_agg_bytes_in{alias=~"sapi."}[5m]) is not matching anything for me atm?

Is this because of my Prometheus version?

Cheers,
Adel.

[isaacdavis]

Hi @Adel-Magebinary,

There could be many reasons for this. I have a few questions:

• Are other Joyent-specific queries working properly?
• Is your sapi service running and healthy?
• Do the prometheus logs show any errors?
• Do the grafana logs show any errors?
• Does this error show up when the query is executed from the prometheus interface itself, or only grafana?
• What does your Prometheus config file look like?

[Adel-Magebinary]

Hi Isaac,

Are other Joyent-specific queries working properly?

No. regular expression somehow not matching with sapi. or instances.

Once I remove the . in the alias regular express, everything shows up.

Is your sapi service running and healthy?

Yes. sdc-healthcheck and curl test all pass.

Do the prometheus logs show any errors?

I haven't checked my log file. But I don't think prometheus is throwing any error. Will check this soon.

Do the grafana logs show any errors?

No errors. Looks like prometheus query with alias doesn't return anything but success status.

Does this error show up when the query is executed from the prometheus interface itself, or only
grafana?

In both. There was no error but just can't find the report data.

What does your Prometheus config file look like?

global:
scrape_interval: 10s
evaluation_interval: 8s

scrape_timeout is set to the global default 10s

Note: you can create multiple stanzas starting with "job_name"

scrape_configs:

• job_name: triton
  scheme: https
  tls_config:
  cert_file: /root/.ssh/promcert.pem
  key_file: /root/.ssh/promkey.pem
  insecure_skip_verify: true
  triton_sd_configs:
  • account: 'admin'
    dns_suffix: 'cmon.magiclab.cns.MYDOMAIN.COM'
    endpoint: 'cmon.magiclab.cns.MYDOMAIN.COM'
    version: 1
    tls_config:
    cert_file: /root/.ssh/promcert.pem
    key_file: /root/.ssh/promkey.pem
    insecure_skip_verify: true
    relabel_configs:
  • source_labels: [__meta_triton_machine_alias]
    target_label: instance
    dns_sd_configs:
  • refresh_interval: 15s
    names:
    • 10.1.1.11
    • ns1.MYDOMAIN.COM
    • 8.8.8.8
• job_name: triton_compute_nodes
  scheme: https
  tls_config:
  cert_file: /root/.ssh/promcert.pem
  key_file: /root/.ssh/promkey.pem
  insecure_skip_verify: true
  #metrics_path: /v1/metrics
  static_configs:
  • targets: ['44454c4c-3600-104d-8038-c3c04f543932.cmon.magiclab.cns.MYDOMAIN.COM:9163','44454c4c-3600-104e-8031-c3c04f543932.cmon.magiclab.cns.MYDOMAIN.COM:9163','44454c4c-4c00-1042-804b-b9c04f383632.cmon.magiclab.cns.MYDOMAIN.COM:9163','44454c4c-5000-1053-8034-b1c04f433253.cmon.magiclab.cns.MYDOMAIN.COM:9163','44454c4c-5200-1031-8035-c6c04f563732.cmon.magiclab.cns.MYDOMAIN.COM:9163','44454c4c-5300-1032-8042-c8c04f423253.cmon.magiclab.cns.MYDOMAIN.COM:9163']
    labels:
    type: server
    relabel_configs:
  • source_labels: [__meta_triton_machine_alias]
    target_label: instance
    dns_sd_configs:
  • refresh_interval: 15s
    names:
    • 10.1.1.11
    • ns1.MYDOMAIN.COM
    • 8.8.8.8

[isaacdavis]

This all looks good. Let me know if you find anything in the prometheus logs.

I do have one question - what is the alias of your sapi service? The regex expects the aliases to be numbered - sapi0, sapi1, etc. I am curious if yours is just named "sapi" - this would explain why the data shows up when you delete the dot in the regex.

[Adel-Magebinary]

Hello Isaac,

It's sapi0. It's just not regx matching it. Will check Prometheus logs shortly.

Cheers,
Adel.

[Adel-Magebinary]

Hello Isaac,

What is your grafana version?

[isaacdavis]

I'm currently running 5.3.2.