design-proposal: Expand during initial submission

Signed-off-by: Lee Yarwood <[email protected]>

design-proposals/instancetype.kubevirt.io/expand-during-submission.md

@@ -0,0 +1,248 @@
+# Overview
+
+The initial design of instance types and preferences captured a point in time
+copy of the original resource for future use by the VirtualMachine to ensure we
+always generate the same VirtualMachineInstance at runtime.
+
+This additionally allows users to easily switch between classes, sizes and
+generations of these resources at some point in the future. However this
+flexibility comes at the cost of complexity within KubeVirt itself when
+handling more advanced VirtualMachine lifecycle flows.
+
+This complexity is also exposed to users and any third party integrations such
+as back-up tooling or user-interfaces built on top of KubeVirt. Such tooling
+needing to have knowledge of these revisions and even how they map to the
+eventual `VirtualMachineInstance` at runtime.
+
+This design proposal aims to set out new configurable behavior where any
+referenced instance types and/or preferences are expanded into the
+VirtualMachine during submission.
+
+## Motivation
+
+TBD
+
+## Goals
+
+* Provide simple configurables at the cluster and VirtualMachine level to expand
+  instance types and preferences during submission
+
+## Non Goals
+
+* The default behavior will not change as part of this work
+
+## User Stories
+
+* As a cluster owner I want to configure the expansion of all new
+  `VirtualMachines` referencing instance types and preferences during submission
+* As a VirtualMachine owner I want to optionally enable or disable the expansion
+  my VirtualMachine referencing an instance type or preference
+
+## Repos
+
+* kubevirt/kubevirt
+
+# Design
+
+A new `KubeVirt` configurable will be introduced to have newly submitted
+`VirtualMachines` referencing instance types and/or preferences expanded.
+
+Additional `VirtualMachine` configurables will also be introduced to allow a
+specific newly submitted VirtualMachine referencing instance types and/or
+preferences to be expanded.
+
+Both sets of configurables will default to `reference`.
+
+The `VirtualMachine` configurable will override the cluster-wide `KubeVirt`
+configurable in situations where they are not equal.
+
+When requested the `VirtualMachine` controller will expand any referenced
+instance type or preference when the `VirtualMachine` is first seen.
+
+## Concerns
+
+### Exposing users to API complexity within VirtualMachines
+
+One of the original design goals with instance types and preferences was to
+simplify creation by reducing a users exposure to the core
+`VirtualMachineInstanceSpec` API and thus their overall decision matrix.
+
+While this proposal doesn't change the initial creation behaviour it will result
+in a fully flattened `VirtualMachine` exposing all of this complexity at that
+level once again once submission.
+
+### Breaking declarative management of VirtualMachines using instance types
+
+This expansion behavior will break any declarative management of these
+`VirtualMachines` as they will substantially change after initial submission.
+Users will need to explicitly request to not expand their `VirtualMachines`
+referencing instance types and/or preferences to avoid this.
+
+## Alternatives
+
+### Eventual expansion orchestrated by the VM controller
+
+A possible alternative to this design proposal is to have the VirtualMachine
+controller eventually expand all VirtualMachines referencing instance types
+and/or preferences as part of the core reconcile loop.
+
+This behaviour could be controlled by the same cluster-wide and VirtualMachine
+specific configurables as suggested by this proposal.
+
+This could however result in drastic changes to existing VirtualMachine objects
+without user interaction.
+
+We would also still need to retain webhook validation logic to ensure that a
+given `VirtualMachine` using an instance type and/or preference is valid during
+submission.
+
+This would also put additional load on the controller if the cluster-wide
+configurable was enabled in an environment with a large number of existing
+`VirtualMachines` using instance types and preferences.
+
+### Immutable instance types and preferences
+
+The need to retain point in time copies of instance types and preferences is due
+to the simple fact that in the current implementation these resources are
+mutable and can change over time. Thus to ensure we always get the same
+`VirtualMachineInstance` at runtime copies need to be taken and referenced from
+the `VirtualMachine`.
+
+We could possibly remove this requirement by making these object immutable and
+thus dropping the need capture and reference `ControllerRevisions` from the
+`VirtualMachines` at all.
+
+This however still retains the need for additional logic in more complex
+`VirtualMachine` lifecycle operations where we need to expand these now
+immutable resources in the `VirtualMachine`. For example during hot plug.
+
+### Keep existing behaviour
+
+Ultimately we can also decide not to implement the core proposal or any of the
+above alternatives and continue to support the original `ControllerRevision`
+based flow.
+
+## API Examples
+
+The default `policy` will be `reference`.
+
+A cluster admin can default the policy of all new `VirtualMachines` by setting
+`expand` within the `KubeVirt` `CR`:
+
+```yaml
+apiVersion: kubevirt.io/v1
+kind: KubeVirt
+metadata:
+  name: kv
+spec:
+  configuration:
+    instancetype:
+      policy: expand
+    preference:
+      policy: expand
+```
+
+Additionally a VM owner can explicitly request `expand` by setting the policy
+directly on the `VirtualMachine`:
+
+```yaml
+apiVersion: kubevirt.io/v1
+kind: VirtualMachine
+metadata:
+  name: cirros
+spec:
+  instancetype:
+    name: foo
+    policy: expand
+  preference:
+    name: bar
+    policy: expand
+```
+
+Likewise users can explicitly request the `reference` policy to counter a
+different default being defined in the `KubeVirt` `CR`:
+
+```yaml
+apiVersion: kubevirt.io/v1
+kind: VirtualMachine
+metadata:
+  name: cirros
+spec:
+  instancetype:
+    name: foo
+    policy: reference
+```
+
+```go
+// KubeVirtConfiguration holds all kubevirt configurations
+type KubeVirtConfiguration struct {
+[..]
+    // Instancetype configuration
+    Instancetype *InstancetypeConfiguration `json:"instancetype,omitempty"`
+
+    // Preference configuration
+    Preference *PreferenceConfiguration `json:"preference,omitempty"`
+}
+
+type ExpansionPolicy string
+
+const (
+    Expand    ExpansionPolicy = "expand"
+    Reference ExpansionPolicy = "reference"
+)
+
+type InstancetypeConfiguration struct {
+    Policy *ExpansionPolicy `json:"policy,omitempty"`
+}
+
+type PreferenceConfiguration struct {
+    Policy *ExpansionPolicy `json:"policy,omitempty"`
+}
+
+[..]
+
+// InstancetypeMatcher references a instancetype that is used to fill fields in the VMI template.
+type InstancetypeMatcher struct {
+    [..]
+    Policy *ExpansionPolicy `json:"policy,omitempty"`
+}
+
+[..]
+
+// PreferenceMatcher references a set of preference that is used to fill fields in the VMI template.
+type PreferenceMatcher struct {
+    [..]
+    Policy *ExpansionPolicy `json:"policy,omitempty"`
+}
+```
+
+## Scalability
+
+There is already limited in-memory expansion of instance types and preferences
+during submission to allow for validations etc.
+
+However the resulting mutation of the `VirtualMachine` with this proposal will
+cause an additional reconciliation loop to trigger for the VM.
+
+Work should be carried out to ensure that the substantial mutation of the
+`VirtualMachine` during submission doesn't negatively impact the control plane.
+
+## Update/Rollback Compatibility
+
+Existing `VirtualMachines` referencing instance types and/or preferences will
+not be changed as a result of this behaviour.
+
+There will be no ability to automatically rollback new `VirtualMachines` once
+they have their instance type and/or preference expanded by this new
+functionality.
+
+Users will also be unable to resize their `VirtualMachines` by making a singular
+choice of instance type in the future.
+
+## Functional Testing Approach
+
+TBD
+
+# Implementation Phases
+
+TBD