From 338a2b6d26d4e2e93736ec5b0ca34bf0d4739e87 Mon Sep 17 00:00:00 2001 From: Lee Yarwood Date: Tue, 24 Sep 2024 12:04:43 +0100 Subject: [PATCH] design-proposal: Expand during initial submission Signed-off-by: Lee Yarwood --- .../expand-during-submission.md | 248 ++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 design-proposals/instancetype.kubevirt.io/expand-during-submission.md diff --git a/design-proposals/instancetype.kubevirt.io/expand-during-submission.md b/design-proposals/instancetype.kubevirt.io/expand-during-submission.md new file mode 100644 index 00000000..e5f9e064 --- /dev/null +++ b/design-proposals/instancetype.kubevirt.io/expand-during-submission.md @@ -0,0 +1,248 @@ +# Overview + +The initial design of instance types and preferences captured a point in time +copy of the original resource for future use by the VirtualMachine to ensure we +always generate the same VirtualMachineInstance at runtime. + +This additionally allows users to easily switch between classes, sizes and +generations of these resources at some point in the future. However this +flexibility comes at the cost of complexity within KubeVirt itself when +handling more advanced VirtualMachine lifecycle flows. + +This complexity is also exposed to users and any third party integrations such +as back-up tooling or user-interfaces built on top of KubeVirt. Such tooling +needing to have knowledge of these revisions and even how they map to the +eventual `VirtualMachineInstance` at runtime. + +This design proposal aims to set out new configurable behavior where any +referenced instance types and/or preferences are expanded into the +VirtualMachine during submission. + +## Motivation + +TBD + +## Goals + +* Provide simple configurables at the cluster and VirtualMachine level to expand + instance types and preferences during submission + +## Non Goals + +* The default behavior will not change as part of this work + +## User Stories + +* As a cluster owner I want to configure the expansion of all new + `VirtualMachines` referencing instance types and preferences during submission +* As a VirtualMachine owner I want to optionally enable or disable the expansion + my VirtualMachine referencing an instance type or preference + +## Repos + +* kubevirt/kubevirt + +# Design + +A new `KubeVirt` configurable will be introduced to have newly submitted +`VirtualMachines` referencing instance types and/or preferences expanded. + +Additional `VirtualMachine` configurables will also be introduced to allow a +specific newly submitted VirtualMachine referencing instance types and/or +preferences to be expanded. + +Both sets of configurables will default to `reference`. + +The `VirtualMachine` configurable will override the cluster-wide `KubeVirt` +configurable in situations where they are not equal. + +When requested the `VirtualMachine` controller will expand any referenced +instance type or preference when the `VirtualMachine` is first seen. + +## Concerns + +### Exposing users to API complexity within VirtualMachines + +One of the original design goals with instance types and preferences was to +simplify creation by reducing a users exposure to the core +`VirtualMachineInstanceSpec` API and thus their overall decision matrix. + +While this proposal doesn't change the initial creation behaviour it will result +in a fully flattened `VirtualMachine` exposing all of this complexity at that +level once again once submission. + +### Breaking declarative management of VirtualMachines using instance types + +This expansion behavior will break any declarative management of these +`VirtualMachines` as they will substantially change after initial submission. +Users will need to explicitly request to not expand their `VirtualMachines` +referencing instance types and/or preferences to avoid this. + +## Alternatives + +### Eventual expansion orchestrated by the VM controller + +A possible alternative to this design proposal is to have the VirtualMachine +controller eventually expand all VirtualMachines referencing instance types +and/or preferences as part of the core reconcile loop. + +This behaviour could be controlled by the same cluster-wide and VirtualMachine +specific configurables as suggested by this proposal. + +This could however result in drastic changes to existing VirtualMachine objects +without user interaction. + +We would also still need to retain webhook validation logic to ensure that a +given `VirtualMachine` using an instance type and/or preference is valid during +submission. + +This would also put additional load on the controller if the cluster-wide +configurable was enabled in an environment with a large number of existing +`VirtualMachines` using instance types and preferences. + +### Immutable instance types and preferences + +The need to retain point in time copies of instance types and preferences is due +to the simple fact that in the current implementation these resources are +mutable and can change over time. Thus to ensure we always get the same +`VirtualMachineInstance` at runtime copies need to be taken and referenced from +the `VirtualMachine`. + +We could possibly remove this requirement by making these object immutable and +thus dropping the need capture and reference `ControllerRevisions` from the +`VirtualMachines` at all. + +This however still retains the need for additional logic in more complex +`VirtualMachine` lifecycle operations where we need to expand these now +immutable resources in the `VirtualMachine`. For example during hot plug. + +### Keep existing behaviour + +Ultimately we can also decide not to implement the core proposal or any of the +above alternatives and continue to support the original `ControllerRevision` +based flow. + +## API Examples + +The default `policy` will be `reference`. + +A cluster admin can default the policy of all new `VirtualMachines` by setting +`expand` within the `KubeVirt` `CR`: + +```yaml +apiVersion: kubevirt.io/v1 +kind: KubeVirt +metadata: + name: kv +spec: + configuration: + instancetype: + policy: expand + preference: + policy: expand +``` + +Additionally a VM owner can explicitly request `expand` by setting the policy +directly on the `VirtualMachine`: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: cirros +spec: + instancetype: + name: foo + policy: expand + preference: + name: bar + policy: expand +``` + +Likewise users can explicitly request the `reference` policy to counter a +different default being defined in the `KubeVirt` `CR`: + +```yaml +apiVersion: kubevirt.io/v1 +kind: VirtualMachine +metadata: + name: cirros +spec: + instancetype: + name: foo + policy: reference +``` + +```go +// KubeVirtConfiguration holds all kubevirt configurations +type KubeVirtConfiguration struct { +[..] + // Instancetype configuration + Instancetype *InstancetypeConfiguration `json:"instancetype,omitempty"` + + // Preference configuration + Preference *PreferenceConfiguration `json:"preference,omitempty"` +} + +type ExpansionPolicy string + +const ( + Expand ExpansionPolicy = "expand" + Reference ExpansionPolicy = "reference" +) + +type InstancetypeConfiguration struct { + Policy *ExpansionPolicy `json:"policy,omitempty"` +} + +type PreferenceConfiguration struct { + Policy *ExpansionPolicy `json:"policy,omitempty"` +} + +[..] + +// InstancetypeMatcher references a instancetype that is used to fill fields in the VMI template. +type InstancetypeMatcher struct { + [..] + Policy *ExpansionPolicy `json:"policy,omitempty"` +} + +[..] + +// PreferenceMatcher references a set of preference that is used to fill fields in the VMI template. +type PreferenceMatcher struct { + [..] + Policy *ExpansionPolicy `json:"policy,omitempty"` +} +``` + +## Scalability + +There is already limited in-memory expansion of instance types and preferences +during submission to allow for validations etc. + +However the resulting mutation of the `VirtualMachine` with this proposal will +cause an additional reconciliation loop to trigger for the VM. + +Work should be carried out to ensure that the substantial mutation of the +`VirtualMachine` during submission doesn't negatively impact the control plane. + +## Update/Rollback Compatibility + +Existing `VirtualMachines` referencing instance types and/or preferences will +not be changed as a result of this behaviour. + +There will be no ability to automatically rollback new `VirtualMachines` once +they have their instance type and/or preference expanded by this new +functionality. + +Users will also be unable to resize their `VirtualMachines` by making a singular +choice of instance type in the future. + +## Functional Testing Approach + +TBD + +# Implementation Phases + +TBD \ No newline at end of file