IAM resources fail in isolated AWS partitions (e.g. AWS GovCloud) unless AWS_DEFAULT_REGION set

[sidpalas]

What happened?

I attempted to create IAM resources in us-gov but the resources were never created and resulted in the following STS error:

managed/iam.aws.upbound.io/v1beta1, kind=role  cannot run refresh: refresh failed: configuring Terraform AWS Provider: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 638c6984-99a6-45d6-b2d3-1edc0ea00017, api error InvalidClientTokenId: The security token included in the request is invalid.:

I believe this has to do with the fact that IAM resources do not have a region field in their forProvider configuration and it ends up falling back to a hardcoded default of us-east-1

I found this issue: crossplane-contrib/provider-aws#596 and attempted to use the custom endpoint configuration for the ProviderConfig to specify a specific endpoint signing region with the following configuration but the STS error remained.

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: creds
  endpoint:
    partitionId: aws-us-gov
    signingRegion: us-gov-west-1
    source: Custom
    url:
      dynamic:
        host: 'iam.us-gov.amazonaws.com'
        protocol: https
      type: Dynamic

Finally, I used a ControllerConfig to set AWS_DEFAULT_REGION in the AWS provider deployment at which point I was able to successfully provision the resources.

I would have expected the custom endpoint to be sufficient, but I think the STS error was being thrown before the custom endpoint could be used.

How can we reproduce it?

1. Install the AWS provider

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-iam
spec:
  package: xpkg.upbound.io/upbound/provider-aws-iam:v0.36.0

3. Create a ProviderConfig with AWS credentials for a user with permissions in non aws partition (e.g. aws-gov)

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-secret
      key: creds

3. Attempt to create an IAM role

apiVersion: iam.aws.upbound.io/v1beta1
kind: Role
metadata:
  name: crossplane-cluster-role
spec:
  forProvider:
    assumeRolePolicy: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Service": "eks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
        ]
      }

What environment did it happen in?

• Crossplane Version: v1.12.2
• Provider Version: v0.36.0 (using static credentials in k8s secret)
• Kubernetes Version: v1.27.3
• Kubernetes Distribution: KinD (Kubernetes in Docker)

[jeanduplessis]

Possible area of investigation:

• the region falls back to us-east-1 if not set: https://github.com/upbound/provider-aws/blob/b09b073c081a48c9b4411ebfbd283387f6c722b9/internal/clients/aws.go#L227
• here is where the provider pulls the endpoint config https://github.com/upbound/provider-aws/blob/b09b073c081a48c9b4411ebfbd283387f6c722b9/internal/clients/provider_config.go#L186

[yebolenko]

any update on this issue? the same is also applicable to china (cn-north-1)

[torg28]

Any updates or workaround for this issue? @sidpalas your workaround does not seem to work for me.

[torg28]

I did find a workaround. Essentially I did what was mentioned above, but instead of putting the endpoint config in the default providerconfig I created a new one called "gov" and used that instead of default for any resources that needed it.

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!

[sidpalas]

I did find a workaround. Essentially I did what was mentioned above, but instead of putting the endpoint config in the default providerconfig I created a new one called "gov" and used that instead of default for any resources that needed it.

@torg28 -- Would you be able to share the endpoint config used for your govcloud ProviderConfig? I have been unable to get it to work without setting AWS_DEFAULT_REGION in the provider pod (via DeploymentRuntimeConfig) but that prevents a single crossplane cluster from managing resources across partitions.

[jeanduplessis]

Note the next release will have updates for accessing non-default partitions: #1554

[sidpalas]

Note the next release will have updates for accessing non-default partitions: #1554

What amazing timing! I built the latest from main and confirmed that using a separate ProviderConfig with the new Auto type endpoint allowed me to have a single crossplane instance create/manage resources across partitions! 🎉

apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: govcloud
spec:
  credentials:
    secretRef:
      key: creds
      name: aws-secret
      namespace: my-namespace
    source: Secret
  endpoint:
    partitionId: aws-us-gov
    signingRegion: us-gov-west-1
    source: Custom
    url:
      type: Auto