Host metrics #293

mwiencek · 2025-01-29T06:06:43Z

This is based on #291.

The service definition was based on https://github.com/metabrainz/prometheus-exp/blob/main/node.sh

mwiencek · 2025-01-29T06:09:42Z

The only issue I have with this is the node-exporter logs being spammed with the following:

node-exporter  | ts=2025-01-29T06:07:03.163Z caller=stdlib.go:105 level=error msg="error gathering metrics: 21 error(s) occurred:\n* [from Gatherer #2] collected metric \"node_filesystem_device_error\" { label:{name:\"device\"  value:\"devpts\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"devpts\"}  label:{name:\"mountpoint\"  value:\"/dev/pts\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_readonly\" { label:{name:\"device\"  value:\"devpts\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"devpts\"}  label:{name:\"mountpoint\"  value:\"/dev/pts\"}  gauge:{value:1}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_size_bytes\" { label:{name:\"device\"  value:\"devpts\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"devpts\"}  label:{name:\"mountpoint\"  value:\"/dev/pts\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_free_bytes\" { label:{name:\"device\"  value:\"devpts\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"devpts\"}  label:{name:\"mountpoint\"  value:\"/dev/pts\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_avail_bytes\" { label:{name:\"device\"  value:\"devpts\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"devpts\"}  label:{name:\"mountpoint\"  value:\"/dev/pts\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_files\" { label:{name:\"device\"  value:\"devpts\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"devpts\"}  label:{name:\"mountpoint\"  value:\"/dev/pts\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_files_free\" { label:{name:\"device\"  value:\"devpts\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"devpts\"}  label:{name:\"mountpoint\"  value:\"/dev/pts\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_device_error\" { label:{name:\"device\"  value:\"mqueue\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"mqueue\"}  label:{name:\"mountpoint\"  value:\"/dev/mqueue\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_readonly\" { label:{name:\"device\"  value:\"mqueue\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"mqueue\"}  label:{name:\"mountpoint\"  value:\"/dev/mqueue\"}  gauge:{value:1}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_size_bytes\" { label:{name:\"device\"  value:\"mqueue\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"mqueue\"}  label:{name:\"mountpoint\"  value:\"/dev/mqueue\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_free_bytes\" { label:{name:\"device\"  value:\"mqueue\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"mqueue\"}  label:{name:\"mountpoint\"  value:\"/dev/mqueue\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_avail_bytes\" { label:{name:\"device\"  value:\"mqueue\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"mqueue\"}  label:{name:\"mountpoint\"  value:\"/dev/mqueue\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_files\" { label:{name:\"device\"  value:\"mqueue\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"mqueue\"}  label:{name:\"mountpoint\"  value:\"/dev/mqueue\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_files_free\" { label:{name:\"device\"  value:\"mqueue\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"mqueue\"}  label:{name:\"mountpoint\"  value:\"/dev/mqueue\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_device_error\" { label:{name:\"device\"  value:\"proc\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"proc\"}  label:{name:\"mountpoint\"  value:\"/proc\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_readonly\" { label:{name:\"device\"  value:\"proc\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"proc\"}  label:{name:\"mountpoint\"  value:\"/proc\"}  gauge:{value:1}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_size_bytes\" { label:{name:\"device\"  value:\"proc\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"proc\"}  label:{name:\"mountpoint\"  value:\"/proc\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_free_bytes\" { label:{name:\"device\"  value:\"proc\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"proc\"}  label:{name:\"mountpoint\"  value:\"/proc\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_avail_bytes\" { label:{name:\"device\"  value:\"proc\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"proc\"}  label:{name:\"mountpoint\"  value:\"/proc\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_files\" { label:{name:\"device\"  value:\"proc\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"proc\"}  label:{name:\"mountpoint\"  value:\"/proc\"}  gauge:{value:0}} was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_filesystem_files_free\" { label:{name:\"device\"  value:\"proc\"}  label:{name:\"device_error\"  value:\"\"}  label:{name:\"fstype\"  value:\"proc\"}  label:{name:\"mountpoint\"  value:\"/proc\"}  gauge:{value:0}} was collected before with the same name and label values"

I have no idea why this is happening or how to resolve it -- devpts for example is already listed in --collector.filesystem.ignored-fs-types, so I don't know why it's being collected.

reosarevok · 2025-01-31T10:26:38Z

Other than the same error you mentioned, and the fact that I had to rebuild the dev-search image every time because it failed at (re)creating existing collections (which I understand is being fixed already), this seems to work great!

yvanzo · 2025-01-31T16:28:16Z

I merged SIR dev stuff into master, rebased the target branch monitoring on it, and rebased the source branch host metrics on it to resolve conflicts.

zas · 2025-01-31T18:10:36Z

compose/monitoring.yml

+      - node-exporter-textfile-collector:/textfile-collector-directory
+    command:
+      - --path.rootfs=/host
+      - --collector.filesystem.ignored-fs-types="^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|tmpfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$"


FYI, not sure if it applies to the version we run yet, but:

According to https://github.com/prometheus/node_exporter/blob/810510e12b063690e6e52700a867676b93492e92/collector/filesystem_common.go#L262

"--collector.filesystem.ignored-fs-types is DEPRECATED and will be removed in 2.0.0, use --collector.filesystem.fs-types-exclude"

And according to https://github.com/prometheus/node_exporter/blob/810510e12b063690e6e52700a867676b93492e92/collector/filesystem_common.go#L233

"--collector.filesystem.ignored-mount-points is DEPRECATED and will be removed in 2.0.0, use --collector.filesystem.mount-points-exclude"

I checked the code, and I don't see a good reason for it to duplicate entries but if they are read twice (any symlink or multiple mounts?) plus it should be ignored.

I see these in /etc/mtab (inside the node-exporter container):

devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0 devpts /host/dev/pts devpts ro,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 devpts /host/mnt/hdd/docker/btrfs/subvolumes/c7cf67edc828a039fd362077c065f44799152030a7f79d252f2d274ad80f5460/dev/pts devpts ro,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0

The last one points to an empty dir. Not sure about symlinks. devpts wasn't the only filesystem type with issues, either...

For now I'll disable the filesystem collector since I don't think it's essential for monitoring Solr performance anyway.

Edit: Thanks for the hint about the deprecated flags -- updating them didn't help, but I'll rename them in the commented code.

It currently spams the logs with errors of this form: collected metric [...] was collected before with the same name and label values However, the metrics it references should already be ignored/excluded by the options that I'm disabling in this commit, so it's not clear how to resolve this.

mwiencek changed the base branch from master to monitoring January 29, 2025 06:06

yvanzo force-pushed the monitoring branch from 644fd4b to 0a398ce Compare January 31, 2025 16:20

yvanzo force-pushed the host-metrics branch from ea491a8 to 874e474 Compare January 31, 2025 16:27

zas reviewed Jan 31, 2025

View reviewed changes

mwiencek added 2 commits February 3, 2025 09:38

Add basic search healthcheck

8c74717

Solr metrics

c3249b5

mwiencek force-pushed the host-metrics branch from 874e474 to 28f9114 Compare February 4, 2025 00:18

mwiencek added 3 commits February 6, 2025 09:37

TMP: Use locally-built solr-9.7.0 image

e8933cd

Add host metrics (via node-exporter)

29372a4

mwiencek force-pushed the host-metrics branch from 28f9114 to 1c19d6e Compare February 6, 2025 15:58

mwiencek marked this pull request as ready for review February 6, 2025 16:00

mwiencek mentioned this pull request Feb 7, 2025

RabbitMQ metrics #294

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Host metrics #293

Host metrics #293

mwiencek commented Jan 29, 2025

mwiencek commented Jan 29, 2025

reosarevok commented Jan 31, 2025

yvanzo commented Jan 31, 2025

zas Jan 31, 2025

mwiencek Feb 6, 2025 •

edited

Loading

Host metrics #293

Are you sure you want to change the base?

Host metrics #293

Conversation

mwiencek commented Jan 29, 2025

mwiencek commented Jan 29, 2025

reosarevok commented Jan 31, 2025

yvanzo commented Jan 31, 2025

zas Jan 31, 2025

Choose a reason for hiding this comment

mwiencek Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

mwiencek Feb 6, 2025 •

edited

Loading