v0.16 -> v.017 upgrade, cannot establish control of object

[patjones]

What happened?

I was trying to upgrade my aws provider from v0.16 -> v0.17. I re-applied my provider file after bumping the version

  
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  package: crossplane/provider-aws:v0.16.0 -> crossplane/provider-aws:v0.17.0

Unfortunately, the providerrevision for v0.17 is showing up unhealthy

$ k get providerrevision
NAME                        HEALTHY   REVISION   IMAGE                             STATE      DEP-FOUND   DEP-INSTALLED   AGE
provider-aws-4ebd5d8c7cb7   False     2          crossplane/provider-aws:v0.17.0   Active     0           0               146m
provider-aws-6b36ca882d8e   True      1          crossplane/provider-aws:v0.16.0   Inactive   0           0               9d

If I describe the troublesome revision I get a few events that seen problematic

Events:
  Type     Reason             Age                      From                                         Message
  ----     ------             ----                     ----                                         -------
  Normal   BindClusterRole    143m (x4 over 143m)      rbac/providerrevision.pkg.crossplane.io      Bound system ClusterRole to provider ServiceAccount(s)
  Warning  SyncPackage        143m                     packages/providerrevision.pkg.crossplane.io  cannot establish control of object: elbs.elasticloadbalancing.aws.crossplane.io is already controlled by ProviderRevision provider-aws-6b36ca882d8e (UID 42a128bd-91a3-42ce-b1bc-1ca50bd97e84)
  Normal   ApplyClusterRoles  3m17s (x1383 over 143m)  rbac/providerrevision.pkg.crossplane.io      Applied RBAC ClusterRoles
  Warning  SyncPackage        2m59s (x255 over 142m)   packages/providerrevision.pkg.crossplane.io  cannot establish control of object: CustomResourceDefinition.apiextensions.k8s.io "natgateways.ec2.aws.crossplane.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

How can we reproduce it?

Install the aws provider v0.16 with Provider and ProviderConfig yaml files, bump the provider version to v0.17.

What environment did it happen in?

Crossplane version: 1.10.0
Openshift 4.6.19 (k8s 1.19)

Thanks for any insight!

[hasheddan]

Hey @patjones 👋 This is due to dropping support for those alpha level CRDs. You can fix the issue by just deleting those CRDs (i.e. kubectl delete crd natgateways.ec2.aws.crossplane.io) and the new version will automatically be installed. You can see more discussion on this issue here: crossplane/crossplane#2165

Feel free to weigh in on that issue or follow up here :)

[patjones]

Thanks for the response @hasheddan! Well that cleared one of those away however I'm still having issues with the natgatway crd even after deleting it a few times.

Events:
  Type     Reason             Age                    From                                         Message
  ----     ------             ----                   ----                                         -------
  Normal   ApplyClusterRoles  73s (x2325 over 4h1m)  rbac/providerrevision.pkg.crossplane.io      Applied RBAC ClusterRoles
  Warning  SyncPackage        32s (x435 over 4h)     packages/providerrevision.pkg.crossplane.io  cannot establish control of object: CustomResourceDefinition.apiextensions.k8s.io "natgateways.ec2.aws.crossplane.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

[hasheddan]

@patjones do you have any NATGateway instances in the cluster? That would block deletion of the CRD

[patjones]

$ k get NATGateway -A
No resources found

I do not actually

[hasheddan]

Issue here was that the old revision was re-creating the CRD before the new revision. We solved the issue by deleting the old revision, but the buggy behavior is being tracked in crossplane/crossplane#2197.

@patjones thanks for bringing this up! I am going to close this out in favor of the linked issue, but please feel free to re-open if you have additional questions.

[lanwen]

Perhaps unrelated, but if someone finds the issue by the error similar to this (as I did):

Warning  SyncPackage        6m40s (x59 over 96m)  packages/providerrevision.pkg.crossplane.io  cannot establish control of object: certificates.acm.aws.crossplane.io is already controlled by ProviderRevision crossplane-provider-aws-245ce7f │
│ b587d (UID 271bbc59-ff45-4ca3-b0d6-f7a60faa7deb)

Changing the owner could help with the script that looks like:

#!/bin/bash

# Variables
old_owner_name="crossplane-provider-aws-245ce7fb587d"
new_owner_name="crossplanecontrib-provider-aws-6707d06fe75f"
# Get the new UID from providerrevisions.pkg.crossplane.io
new_uid=$(kubectl get providerrevisions.pkg.crossplane.io $new_owner_name -o jsonpath='{.metadata.uid}')
echo "Replacing with: $(kubectl get providerrevisions.pkg.crossplane.io $new_owner_name -o jsonpath='{.spec.image}')"


# Get all CRDs with the old owner name
crds=$(kubectl get crds -o json | jq -r --arg old_owner_name "$old_owner_name" '.items[] | select(.metadata.ownerReferences[]?.name == $old_owner_name) | .metadata.name')

# Loop through each CRD and patch it
# Loop through each CRD and patch it
for crd in $crds; do
  echo "Patching CRD: $crd"

  # Get the index of the old owner reference
  index=$(kubectl get crd $crd -o json | jq -r --arg old_owner_name "$old_owner_name" '
    .metadata.ownerReferences | to_entries[] | select(.value.name == $old_owner_name) | .key')

  # Check if index was found
  if [ -z "$index" ]; then
    echo "Old owner reference not found for CRD: $crd"
    continue
  fi

  # Patch the CRD
  kubectl patch crd $crd --type='json' -p='[
    {"op": "remove", "path": "/metadata/ownerReferences/'$index'"},
    {"op": "add", "path": "/metadata/ownerReferences/-", "value": {"apiVersion": "pkg.crossplane.io/v1", "kind": "ProviderRevision", "name": "'$new_owner_name'", "uid": "'$new_uid'", "controller": true}}
  ]'
done