From 89b5f6048206f7ed2a10b3e5d768d1cfc468f8ce Mon Sep 17 00:00:00 2001 From: Antti Kervinen Date: Fri, 22 Dec 2023 11:52:49 +0200 Subject: [PATCH] doc: update balloons documentation to the CRD era Signed-off-by: Antti Kervinen --- docs/resource-policy/policy/balloons.md | 152 ++++++++++++++---------- 1 file changed, 88 insertions(+), 64 deletions(-) diff --git a/docs/resource-policy/policy/balloons.md b/docs/resource-policy/policy/balloons.md index e64760049..8986e72d4 100644 --- a/docs/resource-policy/policy/balloons.md +++ b/docs/resource-policy/policy/balloons.md @@ -61,20 +61,26 @@ more details on managing the configuration. Balloons policy parameters: -- `PinCPU` controls pinning a container to CPUs of its balloon. The +- `reservedResources`: + - `cpu` specifies cpuset or number of CPUs in the special `reserved` + balloon. By default all containers in the `kube-system` namespace + are assigned to the reserved balloon. Examples: `cpu: cpuset:0,48` + uses two logical CPUs: cpu0 and cpu48. `cpu: 2000m` uses + any two CPUs. +- `pinCPU` controls pinning a container to CPUs of its balloon. The default is `true`: the container cannot use other CPUs. -- `PinMemory` controls pinning a container to the memories that are +- `pinMemory` controls pinning a container to the memories that are closest to the CPUs of its balloon. The default is `true`: allow using memory only from the closest NUMA nodes. Warning: this may cause kernel to kill workloads due to out-of-memory error when closest NUMA nodes do not have enough memory. In this situation consider switching this option `false`. -- `IdleCPUClass` specifies the CPU class of those CPUs that do not +- `idleCPUClass` specifies the CPU class of those CPUs that do not belong to any balloon. -- `ReservedPoolNamespaces` is a list of namespaces (wildcards allowed) +- `reservedPoolNamespaces` is a list of namespaces (wildcards allowed) that are assigned to the special reserved balloon, that is, will run on reserved CPUs. This always includes the `kube-system` namespace. -- `AllocatorTopologyBalancing` affects selecting CPUs for new +- `allocatorTopologyBalancing` affects selecting CPUs for new balloons. If `true`, new balloons are created using CPUs on NUMA/die/package with most free CPUs, that is, balloons are spread across the hardware topology. This helps inflating balloons within @@ -83,7 +89,7 @@ Balloons policy parameters: pack new balloons tightly into the same NUMAs/dies/packages. This helps keeping large portions of hardware idle and entering into deep power saving states. -- `PreferSpreadOnPhysicalCores` prefers allocating logical CPUs +- `preferSpreadOnPhysicalCores` prefers allocating logical CPUs (possibly hyperthreads) for a balloon from separate physical CPU cores. This prevents workloads in the balloon from interfering with themselves as they do not compete on the resources of the same CPU @@ -93,46 +99,47 @@ Balloons policy parameters: value set here is the default for all balloon types, but it can be overridden with the balloon type specific setting with the same name. -- `BalloonTypes` is a list of balloon type definitions. Each type can +- `balloonTypes` is a list of balloon type definitions. Each type can be configured with the following parameters: - - `Name` of the balloon type. This is used in pod annotations to + - `name` of the balloon type. This is used in pod annotations to assign containers to balloons of this type. - - `Namespaces` is a list of namespaces (wildcards allowed) whose + - `namespaces` is a list of namespaces (wildcards allowed) whose pods should be assigned to this balloon type, unless overridden by pod annotations. - - `MinBalloons` is the minimum number of balloons of this type that + - `minBalloons` is the minimum number of balloons of this type that is always present, even if the balloons would not have any containers. The default is 0: if a balloon has no containers, it can be destroyed. - - `MaxBalloons` is the maximum number of balloons of this type that + - `maxBalloons` is the maximum number of balloons of this type that is allowed to co-exist. The default is 0: creating new balloons is not limited by the number of existing balloons. - - `MaxCPUs` specifies the maximum number of CPUs in any balloon of + - `maxCPUs` specifies the maximum number of CPUs in any balloon of this type. Balloons will not be inflated larger than this. 0 means unlimited. - - `MinCPUs` specifies the minimum number of CPUs in any balloon of + - `minCPUs` specifies the minimum number of CPUs in any balloon of this type. When a balloon is created or deflated, it will always have at least this many CPUs, even if containers in the balloon request less. - - `CpuClass` specifies the name of the CPU class according to which - CPUs of balloons are configured. - - `PreferCloseToDevices`: prefer creating new balloons close to + - `cpuClass` specifies the name of the CPU class according to which + CPUs of balloons are configured. Class properties are defined in + separate `cpu.classes` objects, see below. + - `preferCloseToDevices`: prefer creating new balloons close to listed devices. List of strings - - `PreferSpreadingPods`: if `true`, containers of the same pod + - `preferSpreadingPods`: if `true`, containers of the same pod should be spread to different balloons of this type. The default is `false`: prefer placing containers of the same pod to the same balloon(s). - - `PreferPerNamespaceBalloon`: if `true`, containers in the same + - `preferPerNamespaceBalloon`: if `true`, containers in the same namespace will be placed in the same balloon(s). On the other hand, containers in different namespaces are preferrably placed in different balloons. The default is `false`: namespace has no effect on choosing the balloon of this type. - - `PreferNewBalloons`: if `true`, prefer creating new balloons over + - `preferNewBalloons`: if `true`, prefer creating new balloons over placing containers to existing balloons. This results in preferring exclusive CPUs, as long as there are enough free CPUs. The default is `false`: prefer filling and inflating existing balloons over creating new ones. - - `ShareIdleCPUsInSame`: Whenever the number of or sizes of balloons + - `shareIdleCPUsInSame`: Whenever the number of or sizes of balloons change, idle CPUs (that do not belong to any balloon) are reshared as extra CPUs to workloads in balloons with this option. The value sets locality of allowed extra CPUs that will be common to these @@ -145,9 +152,9 @@ Balloons policy parameters: - `numa`: ...in the same numa node(s) as the balloon. - `core`: ...allowed to use idle CPU threads in the same cores with the balloon. - - `PreferSpreadOnPhysicalCores` overrides the policy level option + - `preferSpreadOnPhysicalCores` overrides the policy level option with the same name in the scope of this balloon type. - - `PreferCloseToDevices` prefers creating new balloons close to + - `preferCloseToDevices` prefers creating new balloons close to listed devices. If all preferences cannot be fulfilled, preference to first devices in the list override preferences to devices after them. Adding this preference to any balloon type automatically @@ -155,60 +162,77 @@ Balloons policy parameters: not prefer to be close to the same device: they prefer being created away from the device. Example: ``` - PreferCloseToDevices: + preferCloseToDevices: - /sys/class/net/eth0 - /sys/class/block/sda ``` - - `AllocatorPriority` (0: High, 1: Normal, 2: Low, 3: None). CPU + - `allocatorPriority` (0: High, 1: Normal, 2: Low, 3: None). CPU allocator parameter, used when creating new or resizing existing balloons. If there are balloon types with pre-created balloons - (`MinBalloons` > 0), balloons of the type with the highest - `AllocatorPriority` are created first. - -Related configuration parameters: - -- `policy.ReservedResources.CPU` specifies the (number of) CPUs in the - special `reserved` balloon. By default all containers in the - `kube-system` namespace are assigned to the reserved balloon. -- `cpu.classes` defines CPU classes and their parameters (such as - `minFreq`, `maxFreq`, `uncoreMinFreq` and `uncoreMaxFreq`). + (`minBalloons` > 0), balloons of the type with the highest + `allocatorPriority` are created first. +- `control.cpu.classes`: defines CPU classes and their + properties. Class names are keys followed by properties: + - `minFreq` minimum frequency for CPUs in this class (kHz). + - `maxFreq` maximum frequency for CPUs in this class (kHz). + - `uncoreMinFreq` minimum uncore frequency for CPUs in this + class (kHz). If there are differences in `uncoreMinFreq`s in + CPUs within the same uncore frequency zone, the maximum value + of all `uncoreMinFreq`s is used. + - `uncoreMaxFreq` maximum uncore frequency for CPUs in this + class (kHz). +- `instrumentation`: configures interface for runtime instrumentation. + - `httpEndpoint`: the address the HTTP server listens on. Example: + `:8891`. + - `prometheusExport`: if set to True, balloon instances, their CPUs + and assigned workloads are readable through `/metrics` from the + httpEndpoint. + - `reportPeriod`: `/metrics` aggregation interval. ### Example -Example configuration that runs all pods in balloons of 1-4 CPUs. +Example configuration that runs all pods in balloons of 1-4 +CPUs. Instrumentation enables reading CPUs and workloads in balloons +from `http://localhost:8891/metrics`. ```yaml - ReservedResources: - CPU: 1 - balloons: - PinCPU: true - PinMemory: true - IdleCPUClass: lowpower - BalloonTypes: - - Name: "quad" - MinCpus: 1 - MaxCPUs: 4 - CPUClass: dynamic - Namespaces: - - "*" -cpu: - classes: - lowpower: - minFreq: 800 - maxFreq: 800 - dynamic: - minFreq: 800 - maxFreq: 3600 - turbo: - minFreq: 3000 - maxFreq: 3600 - uncoreMinFreq: 2000 - uncoreMaxFreq: 2400 +apiVersion: config.nri/v1alpha1 +kind: BalloonsPolicy +metadata: + name: default + namespace: kube-system +spec: + reservedResources: + cpu: 1000m + pinCPU: true + pinMemory: true + allocatorTopologyBalancing: true + idleCPUClass: lowpower + balloonTypes: + - name: "quad" + maxCPUs: 4 + cpuClass: dynamic + namespaces: + - "*" + control: + cpu: + classes: + lowpower: + minFreq: 800000 + maxFreq: 800000 + dynamic: + minFreq: 800000 + maxFreq: 3600000 + turbo: + minFreq: 3000000 + maxFreq: 3600000 + uncoreMinFreq: 2000000 + uncoreMaxFreq: 2400000 + instrumentation: + httpEndpoint: :8891 + prometheusExport: true ``` -See the [sample configuration custom resource](/sample-configs/balloons-config.yaml) for a -more complete example. - ## Assigning a Container to a Balloon The balloon type of a container can be defined in pod annotations. In