GKE Autopilot: DataPlane and ControlPlane never get marked as ready

[mlavacca]

Current Behavior

When creating a DataPlane with a KonnectExtension all the expected resources are provisioned and properly works, but the DataPlane CR never gets marked as ready by KGO

Expected Behavior

When the DataPlane is properly configured and running, it should be marked as Ready by KGO

Additional Information

This issue has been observed in GKE. It should be tested if that happens in Kind or other environments as well.

[programmer04]

It occurs on a GKE Autopilot cluster. You can use config/samples/dataplane-konnect-extension.yaml to reproduce it.

The root cause is mutation of DataPlane's respective K8s Deployment by K8s GKE Autopilot cluster, see

...
- 				Resources: v1.ResourceRequirements{
- 					Limits: v1.ResourceList{
- 						s"cpu":               {i: resource.int64Amount{...}, s: "1", Format: "DecimalSI"},
- 						s"ephemeral-storage": {i: resource.int64Amount{...}, s: "1Gi", Format: "BinarySI"},
- 						s"memory":            {i: resource.int64Amount{...}, Format: "BinarySI"},
- 					},
- 					Requests: v1.ResourceList{
- 						s"cpu":               {i: resource.int64Amount{...}, s: "100m", Format: "DecimalSI"},
- 						s"ephemeral-storage": {i: resource.int64Amount{...}, s: "1Gi", Format: "BinarySI"},
- 						s"memory":            {i: resource.int64Amount{...}, s: "103Mi", Format: "BinarySI"},
- 					},
- 				},
+ 				Resources: v1.ResourceRequirements{
+ 					Limits: v1.ResourceList{
+ 						s"cpu":    {i: resource.int64Amount{...}, s: "1", Format: "DecimalSI"},
+ 						s"memory": {i: resource.int64Amount{...}, Format: "BinarySI"},
+ 					},
+ 					Requests: v1.ResourceList{
+ 						s"cpu":    {i: resource.int64Amount{...}, s: "100m", Format: "DecimalSI"},
+ 						s"memory": {i: resource.int64Amount{...}, s: "20Mi", Format: "BinarySI"},
+ 					},
+ 				},
...
- 				SecurityContext:          s"&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[NET_RAW],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,"...,
+ 				SecurityContext:          nil,

where those marked with - are from GKE and those with + are prepared by KGO.

Modified fields

• SecurityContext adjustment is the result of Security measures in Autopilot
• Resources adjustment is the result of Resource requests in Autopilot

that leads to getting stuck on

gateway-operator/controller/dataplane/controller.go

Lines 213 to 218 in 52f9b09

	if err != nil {
	return ctrl.Result{}, fmt.Errorf("could not build Deployment for DataPlane %s: %w", dpNn, err)
	}
	if res != op.Noop {
	return ctrl.Result{}, nil
	}

because res is always different than op.Noop and reconciliation doesn't progress, hence code responsible for setting proper status is never reached

❗ Exactly the same happens for ControlPlane

[programmer04]

❗ Blocked by #500 - resolving it will resolve this issue too in a robust way.

Rationale, it can be resolved by special treatment of fields mentioned in #888 (comment) like adding

		for i := range existingDeployment.Spec.Template.Spec.Containers {
			generatedDeployment.Spec.Template.Spec.Containers[i].SecurityContext = oldExistingDeployment.Spec.Template.Spec.Containers[i].SecurityContext
			generatedDeployment.Spec.Template.Spec.Containers[i].Resources = oldExistingDeployment.Spec.Template.Spec.Containers[i].Resources
		}
		generatedDeployment.Spec.Template.Spec.SecurityContext = oldExistingDeployment.Spec.Template.Spec.SecurityContext
		generatedDeployment.Spec.Template.Spec.Tolerations = oldExistingDeployment.Spec.Template.Spec.Tolerations

above

gateway-operator/controller/controlplane/controller_reconciler_utils.go

Lines 216 to 220 in 3607dde

	switch {
	case !dataplaneIsSet && (replicas == nil || *replicas != numReplicasWhenNoDataPlane):
	// DataPlane was just unset, so we need to scale down the Deployment.
	if !cmp.Equal(existingDeployment.Spec.Replicas, lo.ToPtr(int32(numReplicasWhenNoDataPlane))) {
	existingDeployment.Spec.Replicas = lo.ToPtr(int32(numReplicasWhenNoDataPlane))

and adding

		for i := range desired.Spec.Template.Spec.Containers {
			desired.Spec.Template.Spec.Containers[i].SecurityContext = existing.Spec.Template.Spec.Containers[i].SecurityContext
			desired.Spec.Template.Spec.Containers[i].Resources = existing.Spec.Template.Spec.Containers[i].Resources
		}
		desired.Spec.Template.Spec.SecurityContext = existing.Spec.Template.Spec.SecurityContext
		desired.Spec.Template.Spec.Tolerations = existing.Spec.Template.Spec.Tolerations

above

gateway-operator/controller/dataplane/owned_deployment.go

Lines 305 to 308 in b3e22cc

	if scaling := dataplane.Spec.Deployment.DeploymentOptions.Scaling; false ||
	// If the scaling strategy is not specified, we compare the replicas.
	(scaling == nil || scaling.HorizontalScaling == nil) ||

this will resolve those specific problems in GKE Autopilot, but it is still not a general solution.

A robust solution that will not require those changes and protect from further adjustment like the aforementioned in the future is proposed in #500 (comment) and it should be implemented