Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: PodIdentityAssociation errors with ResourceInUseException: Association already exists #1437

Open
1 task done
lajchon opened this issue Aug 2, 2024 · 8 comments
Open
1 task done

[Bug]: PodIdentityAssociation errors with ResourceInUseException: Association already exists #1437

lajchon opened this issue Aug 2, 2024 · 8 comments
Labels
bug Something isn't working is:triaged Indicates that an issue has been reviewed.

Comments

@lajchon
Copy link

lajchon commented Aug 2, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Affected Resource(s)

PodIdentityAssociation.eks.aws.upbound.io/v1beta1

Resource MRs required to reproduce the bug

No response

Steps to Reproduce

This issue appear to randomly affect newly created PodIdentityAssociations.

What happened?

New PodIdentityAssociation is applied and confirm creation in AWS EKS console. But PodIdentityAssociation Synced and Ready states immediately turn to False and the following conditions are returned.

status:
  atProvider:
    associationArn: >-
      arn:aws:eks:us-east-1:REMOVED:podidentityassociation/test-cluster/a-kyevsbqhrnchy79jw
    associationId: a-stubassocid123456
    clusterName: test-cluster
    id: ''
    namespace: crossplane-system
    roleArn: >-
      arn:aws:iam::REMOVED:role/test-podidentityassociation-hfvll-gzjnw
    serviceAccount: upbound-provider-aws
    tags:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: test-podidentityassociation-hfvll-6qwj7
      crossplane-providerconfig: provider-aws-REMOVED
    tagsAll:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: test-podidentityassociation-hfvll-6qwj7
      crossplane-providerconfig: provider-aws-REMOVED
  conditions:
    - lastTransitionTime: '2024-08-02T15:18:35Z'
      reason: Creating
      status: 'False'
      type: Ready
    - lastTransitionTime: '2024-08-02T15:19:08Z'
      message: >-
        create failed: async create failed: resource creation call returned
        error diags: creating AWS EKS (Elastic Kubernetes) Pod Identity
        Association ("a-kyevsbqhrnchy79jw"): operation error EKS:
        CreatePodIdentityAssociation, https response error StatusCode: 409,
        RequestID: 93f3bb02-6a0c-4d04-b1bc-0a5f7e54cb6e, ResourceInUseException:
        Association already exists: a-kyevsbqhrnchy79jw: operation error EKS:
        CreatePodIdentityAssociation, https response error StatusCode: 409,
        RequestID: 93f3bb02-6a0c-4d04-b1bc-0a5f7e54cb6e, ResourceInUseException:
        Association already exists: a-kyevsbqhrnchy79jw
      reason: ReconcileError
      status: 'False'
      type: Synced
    - lastTransitionTime: '2024-08-02T15:19:08Z'
      message: >-
        async create failed: resource creation call returned error diags:
        creating AWS EKS (Elastic Kubernetes) Pod Identity Association
        ("a-kyevsbqhrnchy79jw"): operation error EKS:
        CreatePodIdentityAssociation, https response error StatusCode: 409,
        RequestID: 93f3bb02-6a0c-4d04-b1bc-0a5f7e54cb6e, ResourceInUseException:
        Association already exists: a-kyevsbqhrnchy79jw: operation error EKS:
        CreatePodIdentityAssociation, https response error StatusCode: 409,
        RequestID: 93f3bb02-6a0c-4d04-b1bc-0a5f7e54cb6e, ResourceInUseException:
        Association already exists: a-kyevsbqhrnchy79jw
      reason: AsyncCreateFailure
      status: 'False'
      type: LastAsyncOperation
spec:
  deletionPolicy: Delete
  forProvider:
    clusterName: test-cluster
    clusterNameRef:
      name: test-cluster-jpq8h-ncz7l
    clusterNameSelector:
      matchLabels:
        name: test-cluster
    namespace: crossplane-system
    region: us-east-1
    roleArn: >-
      arn:aws:iam::REMOVED:role/test-podidentityassociation-hfvll-gzjnw
    roleArnRef:
      name: test-podidentityassociation-hfvll-gzjnw
    roleArnSelector:
      matchControllerRef: true
    serviceAccount: upbound-provider-aws
    tags:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: test-podidentityassociation-hfvll-6qwj7
      crossplane-providerconfig: provider-aws-REMOVED
  initProvider: {}
  managementPolicies:
    - '*'
  providerConfigRef:
    name: provider-aws-REMOVED

One observation with these failed PodIdentityAssociations is, the status.atProvider.associationId, ie a-stubassocid123456, is set to a value that does not correspond to any Identity Associations with the AWS console or within any provider-aws-eks Pod logs. Also, when I have more than one failed PodIdentityAssociation, the status.atProvider.associationId all match, a-stubassocid123456.

Searching the codebase, this value only appears in one location:

provider-upjet-aws/config/externalname.go

Line 3171 in 779097a

base["association_id"] = "a-stubassocid123456"

If I delete the Identity Association within the AWS console, the resource is eventually reconciled, recreated within the console, but PodIdentityAssociation enters the same failed Synced and Ready states. Only after deleting the Identity Association from AWS console and PodIdentityAssociation from Kubernetes, Composite Resource creates a new PodIdentityAssociation managed resource, the Identity Association is created in AWS console and PodIdentityAssociation enters a Synced and Ready state of True.

Relevant Error Output Snippet

2024-08-02T15:18:35Z    DEBUG   provider-aws    Calling the inner handler for Create event.     {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "queueLength": 0}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Calling the inner handler for Update event.     {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "queueLength": 0}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Calling the inner handler for Update event.     {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "queueLength": 0}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Async create starting...        {"trackerUID": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "resourceName": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Creating the external resource  {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Calling the inner handler for Update event.     {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "queueLength": 0}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Successfully requested creation of external resource    {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21100", "external-name": "", "external-name": ""}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:18:35Z    DEBUG   provider-aws    Cannot initialize managed resource      {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21130", "external-name": "", "error": "Operation cannot be fulfilled on podidentityassociations.eks.aws.upbound.io \"test-podidentityassociation-hfvll-6qwj7\": the object has been modified; please apply your changes to the latest version and try again"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Async create ended.     {"trackerUID": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "resourceName": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "error": null}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Reconcile request has been requeued.    {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "rateLimiterName": "", "when": "0s"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Calling the inner handler for Update event.     {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "queueLength": 0}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Successfully requested update of external resource      {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21187", "external-name": "", "requeue-after": "2024-08-02T15:28:33Z"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Async update starting...        {"trackerUID": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "resourceName": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Updating the external resource  {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Async update ended.     {"trackerUID": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "resourceName": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "error": null}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Reconcile request has been requeued.    {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "rateLimiterName": "", "when": "0s"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:36Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:37Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21208", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:18:37Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:18:37Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:37Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:37Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:37Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21208", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:18:38Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:18:38Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:38Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:38Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:38Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21208", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:18:42Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:18:42Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:42Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:42Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:43Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21208", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:18:51Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:18:51Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:51Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:51Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:18:51Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21208", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Async create starting...        {"trackerUID": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "resourceName": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Creating the external resource  {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Successfully requested creation of external resource    {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21208", "external-name": "a-kyevsbqhrnchy79jw", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Calling the inner handler for Update event.     {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "queueLength": 0}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    ongoing async operation {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "opType": "create"}

2024-08-02T15:19:07Z    DEBUG   provider-aws    External resource is up to date {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21655", "external-name": "a-kyevsbqhrnchy79jw", "requeue-after": "2024-08-02T15:29:37Z"}

2024-08-02T15:19:08Z    DEBUG   provider-aws    Async create ended.     {"trackerUID": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "resourceName": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "error": "async create failed: resource creation call returned error diags: creating AWS EKS (Elastic Kubernetes) Pod Identity Association (\"a-kyevsbqhrnchy79jw\"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 93f3bb02-6a0c-4d04-b1bc-0a5f7e54cb6e, ResourceInUseException: Association already exists: a-kyevsbqhrnchy79jw: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 93f3bb02-6a0c-4d04-b1bc-0a5f7e54cb6e, ResourceInUseException: Association already exists: a-kyevsbqhrnchy79jw"}

2024-08-02T15:19:08Z    DEBUG   provider-aws    Reconcile request has been requeued.    {"gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation", "name": "test-podidentityassociation-hfvll-6qwj7", "rateLimiterName": "asyncCallback", "when": "5ms"}

2024-08-02T15:19:08Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:19:08Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:08Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:08Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21663", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:19:09Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:19:09Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:09Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:09Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:10Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21663", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:19:12Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:19:12Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:12Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:12Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:12Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21663", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:19:16Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:19:16Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:16Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:16Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:16Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21663", "external-name": "a-kyevsbqhrnchy79jw"}

2024-08-02T15:19:24Z    DEBUG   provider-aws    Reconciling     {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}}

2024-08-02T15:19:24Z    DEBUG   provider-aws    Connecting to the service provider      {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:24Z    DEBUG   provider-aws    Instance state not found in cache, reconstructing...    {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:24Z    DEBUG   provider-aws    Observing the external resource {"uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "name": "test-podidentityassociation-hfvll-6qwj7", "gvk": "eks.aws.upbound.io/v1beta1, Kind=PodIdentityAssociation"}

2024-08-02T15:19:25Z    DEBUG   provider-aws    Waiting for external resource existence to be confirmed {"controller": "managed/eks.aws.upbound.io/v1beta1, kind=podidentityassociation", "request": {"name":"test-podidentityassociation-hfvll-6qwj7"}, "uid": "497cab7f-15b8-4442-8fb4-a09e03e62d7a", "version": "21663", "external-name": "a-kyevsbqhrnchy79jw"}

Crossplane Version

1.16.0

Provider Version

1.10.0

Kubernetes Version

No response

Kubernetes Distribution

EKS

Additional Info

No response
@lajchon lajchon added bug Something isn't working needs:triage labels Aug 2, 2024
@haarchri
Copy link
Member

i tested v1.11.0 without any issues:

apiVersion: eks.aws.upbound.io/v1beta1
kind: PodIdentityAssociation
metadata:
  annotations:
    crossplane.io/composition-resource-name: providerAws
    crossplane.io/external-create-pending: "2024-08-13T14:51:39Z"
    crossplane.io/external-create-succeeded: "2024-08-13T14:51:39Z"
    crossplane.io/external-name: a-zl1jvb20aktgc8boy
  creationTimestamp: "2024-08-13T14:51:33Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generateName: configuration-aws-eks-
  generation: 3
  labels:
    crossplane.io/claim-name: ""
    crossplane.io/claim-namespace: ""
    crossplane.io/composite: configuration-aws-eks
  name: configuration-aws-eks-8r5mb
  ownerReferences:
  - apiVersion: aws.platform.upbound.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: XEKS
    name: configuration-aws-eks
    uid: d1e95a6c-e13a-4753-a653-7ae264717e66
  resourceVersion: "6699"
  uid: 602c67fe-7873-4311-9055-2843e4bf6c0c
spec:
  deletionPolicy: Delete
  forProvider:
    clusterName: configuration-aws-eks-v2lbr
    clusterNameRef:
      name: configuration-aws-eks-v2lbr
    clusterNameSelector:
      matchControllerRef: true
    namespace: upbound-system
    region: us-west-2
    roleArn: arn:aws:iam::123456789101:role/configuration-aws-eks-x7nmh
    roleArnRef:
      name: configuration-aws-eks-x7nmh
    roleArnSelector:
      matchLabels:
        role: provider
    serviceAccount: provider-aws
    tags:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: configuration-aws-eks-8r5mb
      crossplane-providerconfig: default
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
status:
  atProvider:
    associationArn: arn:aws:eks:us-west-2:123456789101:podidentityassociation/configuration-aws-eks-v2lbr/a-zl1jvb20aktgc8boy
    associationId: a-zl1jvb20aktgc8boy
    clusterName: configuration-aws-eks-v2lbr
    id: a-zl1jvb20aktgc8boy
    namespace: upbound-system
    roleArn: arn:aws:iam::123456789101:role/configuration-aws-eks-x7nmh
    serviceAccount: provider-aws
    tags:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: configuration-aws-eks-8r5mb
      crossplane-providerconfig: default
    tagsAll:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: configuration-aws-eks-8r5mb
      crossplane-providerconfig: default
  conditions:
  - lastTransitionTime: "2024-08-13T14:51:39Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced
  - lastTransitionTime: "2024-08-13T14:51:42Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-08-13T14:51:40Z"
    reason: Success
    status: "True"
    type: LastAsyncOperation

and i can see the associations with the correct ID in aws console

image

@haarchri haarchri added is:triaged Indicates that an issue has been reviewed. and removed needs:triage labels Aug 13, 2024
@lajchon
Copy link
Author

lajchon commented Aug 15, 2024

After updating EKS Provider to v1.11.0, I've yet to reproduce this issue as well.

@lajchon
Copy link
Author

lajchon commented Aug 20, 2024

After several days of testing, ran into this issue again while using v1.11.0.

status:
  atProvider:
    associationArn: >-
      arn:aws:eks:us-east-1:REMOVED:podidentityassociation/REMOVED/a-7t2bfjb6skkxc4dbn
    associationId: a-stubassocid123456
    clusterName: REMOVED
    id: ''
    namespace: external-secrets
    roleArn: >-
      arn:aws:iam::REMOVED:role/test-podidentityassociation-gk4x2-dm22k
    serviceAccount: external-secrets
    tags:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: test-podidentityassociation-gk4x2-dzll7
      crossplane-providerconfig: provider-aws-REMOVED
    tagsAll:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: test-podidentityassociation-gk4x2-dzll7
      crossplane-providerconfig: provider-aws-REMOVED
  conditions:
    - lastTransitionTime: '2024-08-20T13:28:10Z'
      reason: Creating
      status: 'False'
      type: Ready
    - lastTransitionTime: '2024-08-20T14:00:06Z'
      message: >-
        create failed: async create failed: resource creation call returned
        error diags: creating AWS EKS (Elastic Kubernetes) Pod Identity
        Association ("a-7t2bfjb6skkxc4dbn"): operation error EKS:
        CreatePodIdentityAssociation, https response error StatusCode: 409,
        RequestID: 443612a9-5765-4f3c-a47d-5285dc6147e9, ResourceInUseException:
        Association already exists: a-7t2bfjb6skkxc4dbn: operation error EKS:
        CreatePodIdentityAssociation, https response error StatusCode: 409,
        RequestID: 443612a9-5765-4f3c-a47d-5285dc6147e9, ResourceInUseException:
        Association already exists: a-7t2bfjb6skkxc4dbn
      reason: ReconcileError
      status: 'False'
      type: Synced
    - lastTransitionTime: '2024-08-20T14:00:06Z'
      message: >-
        async create failed: resource creation call returned error diags:
        creating AWS EKS (Elastic Kubernetes) Pod Identity Association
        ("a-7t2bfjb6skkxc4dbn"): operation error EKS:
        CreatePodIdentityAssociation, https response error StatusCode: 409,
        RequestID: 443612a9-5765-4f3c-a47d-5285dc6147e9, ResourceInUseException:
        Association already exists: a-7t2bfjb6skkxc4dbn: operation error EKS:
        CreatePodIdentityAssociation, https response error StatusCode: 409,
        RequestID: 443612a9-5765-4f3c-a47d-5285dc6147e9, ResourceInUseException:
        Association already exists: a-7t2bfjb6skkxc4dbn
      reason: AsyncCreateFailure
      status: 'False'
      type: LastAsyncOperation
spec:
  deletionPolicy: Delete
  forProvider:
    clusterName: REMOVED
    clusterNameRef:
      name: REMOVED-4bxk8-9567c
    clusterNameSelector:
      matchLabels:
        name: REMOVED
    namespace: external-secrets
    region: us-east-1
    roleArn: >-
      arn:aws:iam::REMOVED:role/test-podidentityassociation-gk4x2-dm22k
    roleArnRef:
      name: test-podidentityassociation-gk4x2-dm22k
    roleArnSelector:
      matchControllerRef: true
    serviceAccount: external-secrets
    tags:
      crossplane-kind: podidentityassociation.eks.aws.upbound.io
      crossplane-name: test-podidentityassociation-gk4x2-dzll7
      crossplane-providerconfig: provider-aws-REMOVED
  initProvider: {}
  managementPolicies:
    - '*'
  providerConfigRef:
    name: provider-aws-REMOVED

@lajchon
Copy link
Author

lajchon commented Oct 2, 2024

Still experiencing this issue with v1.14.0 release

@fe-ax
Copy link

fe-ax commented Oct 28, 2024 •
edited
Loading

I'm experiencing this, too. I'm running crossplane 1.16.2, eks provider 1.11 and aws provider family 1.16.0.

Deleting the podidentityassociation in AWS fixes the issue. However, I need to delete the resource in Kubernetes first, then quickly trigger the delete in AWS.

As extra information: I'm running all pods with two replicas as only customization. It's also not with every pia, but when creating ten or so, it happens.

@github-actions GitHub Actions
Copy link

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

@github-actions github-actions bot added the stale label Jan 27, 2025
@lajchon
Copy link
Author

lajchon commented Jan 27, 2025

/fresh

@github-actions github-actions bot removed the stale label Jan 28, 2025
@fernandoalexandre
Copy link

I'm experiencing this issue as well in v1.19.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working is:triaged Indicates that an issue has been reviewed.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@fernandoalexandre @fe-ax @haarchri @lajchon