Need help with Kepler on Talos (no metrics)

[kaznak]

Hi all,

I'm new to Talos Linux and I'm really liking it so far! I'm running a Talos cluster at home and I'm trying to keep an eye on its power consumption.

I found Kepler( https://sustainable-computing.io/ ) , which is supposed to measure power consumption, but I'm having trouble getting it to work. I had to set pod-security.kubernetes.io/enforce: privileged, but that didn't fix the problem.

The logs show that it can't access these files:

1. /sys/devices/system/cpu/cpu0/online
2. /dev/cpu/0/msr
3. /sys/kernel/tracing/events/writeback/writeback_dirty_page/id

I'm not a Linux kernel expert, but it seems like these files should be accessible. Is there any way to get Kepler working on Talos?

Here is some informations about this problem.

➜ talosctl get cpu 
NODE            NAMESPACE   TYPE        ID        VERSION   MANUFACTURER           MODEL                                     CORES   THREADS
192.168.11.32   hardware    Processor   LGA1151   1         Intel(R) Corporation   Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz   6       12

➜ talosctl get ram                                                             
NODE            NAMESPACE   TYPE           ID               VERSION   MANUFACTURER   MODEL              SIZEMIB
192.168.11.32   hardware    MemoryModule   ChannelA-DIMM1   1         Samsung        M378A5244CB0-CTD   4096
192.168.11.32   hardware    MemoryModule   ChannelB-DIMM1   1         Samsung        M378A5244CB0-CTD   4096

➜ kubectl -n obs-prometheus logs kepler-pn57x   
WARNING: failed to read int from file: open /sys/devices/system/cpu/cpu0/online: no such file or directory
I0129 15:18:37.168036       1 exporter.go:103] Kepler running on version: v0.7.12-dirty
I0129 15:18:37.168126       1 config.go:293] using gCgroup ID in the BPF program: true
I0129 15:18:37.168159       1 config.go:295] kernel version: 6.12
I0129 15:18:37.168202       1 rapl_msr_util.go:129] failed to open path /dev/cpu/0/msr: no such file or directory
I0129 15:18:37.168537       1 power.go:78] Unable to obtain power, use estimate method
I0129 15:18:37.168551       1 redfish.go:169] failed to get redfish credential file path
I0129 15:18:37.169474       1 acpi.go:71] Could not find any ACPI power meter path. Is it a VM?
I0129 15:18:37.169480       1 power.go:79] using none to obtain power
E0129 15:18:37.169488       1 accelerator.go:154] [DUMMY] doesn't contain GPU
E0129 15:18:37.169506       1 exporter.go:154] failed to init GPU accelerators: no devices found
WARNING: failed to read int from file: open /sys/devices/system/cpu/cpu0/online: no such file or directory
I0129 15:18:37.171435       1 exporter.go:84] Number of CPUs: 12
W0129 15:18:37.390337       1 exporter.go:135] failed to attach tp/writeback/writeback_dirty_page: reading file "/sys/kernel/tracing/events/writeback/writeback_dirty_page/id": open /sys/kernel/tracing/events/writeback/writeback_dirty_page/id: no such file or directory. Kepler will not collect page cache write events. This will affect the DRAM power model estimation on VMs.
I0129 15:18:37.395382       1 watcher.go:83] Using in cluster k8s config
I0129 15:18:37.495932       1 watcher.go:229] k8s APIserver watcher was started
I0129 15:18:37.517744       1 process_energy.go:129] Using the Ratio Power Model to estimate PROCESS_TOTAL Power
I0129 15:18:37.517789       1 process_energy.go:130] Feature names: [bpf_cpu_time_ms]
I0129 15:18:37.522379       1 process_energy.go:129] Using the Ratio Power Model to estimate PROCESS_COMPONENTS Power
I0129 15:18:37.522419       1 process_energy.go:130] Feature names: [bpf_cpu_time_ms bpf_cpu_time_ms bpf_cpu_time_ms   gpu_compute_util]
I0129 15:18:37.541930       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.541968       1 model.go:125] Requesting for Machine Spec: &{genuineintel intel_core_i7_8700 12 1 7 4600 2}
I0129 15:18:37.542008       1 node_platform_energy.go:53] Using the Regressor/AbsPower Power Model to estimate Node Platform Power
I0129 15:18:37.560806       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.560839       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.560856       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.560881       1 regressor.go:276] Created predictor linear for trainer: "SGDRegressorTrainer"
I0129 15:18:37.560900       1 model.go:125] Requesting for Machine Spec: &{genuineintel intel_core_i7_8700 12 1 7 4600 2}
I0129 15:18:37.560952       1 node_component_energy.go:57] Using the Regressor/AbsPower Power Model to estimate Node Component Power
I0129 15:18:37.561225       1 prometheus_collector.go:95] Registered Container Prometheus metrics
I0129 15:18:37.561356       1 prometheus_collector.go:100] Registered VM Prometheus metrics
I0129 15:18:37.561403       1 prometheus_collector.go:104] Registered Node Prometheus metrics
I0129 15:18:37.563529       1 exporter.go:194] starting to listen on 0.0.0.0:9102
I0129 15:18:37.563745       1 exporter.go:208] Started Kepler in 395.842121ms
E0129 15:18:37.650029       1 watcher.go:189] parsing pod kepler-pn57x obs-prometheus ContainerStatuses issue : container kepler-exporter did not start yet status, InitContainerStatuses issue :<nil>, EphemeralContainerStatuses issue :<nil>

Any help would be appreciated!

Thanks,

[smira]

It's hard to guess, but it shouldn't be Talos-specific.

You can check presence of the files on the host with talosctl ls <path> or even talosctl read <path> to see the value.

If the container doesn't see these files, it's simply misconfigured, e.g. it might need to have privileged enabled, but that is better asked on Kepler side.

[kaznak]

Thank you for your answer.

I confirmed with talosctl ls that the above files are not present, so if it is not a Talos-specific issue, does that mean that these files should not be present originally?

I then booted the same machine from a ubuntu USB image and /dev/cpu/0/msr was present. This leads me to believe that this file may not exist due to differences in kernel configuration.

As for privileges, I assigned them to the namespace running pod-security.kubernetes.io/enforce: privileged. Is there anything else I should do?

In any case, Kepler seems to be the cause of this problem. I will contact them as well.

Thank you very much.

[smira]

I don't see anything immediately missing in Talos kernel config, for specifically cpu/online, but CONFIG_X86_MSR is disabled, as by default Talos Linux hardens the kernel configuration, and exposing this is considered insecure.