cdk-cli deploy: fail: docker push to ecr unexpected status from PUT request 400 Bad Request

[james-g-stream]

Describe the bug

when trying to run cdk deploy with an ecs.ContainerImage.fromAsset() I'm getting a fail: docker push to ecr unexpected status from PUT request 400 Bad Request. I've uninstalled and reinstalled docker, deleted the image asset it references from ecr and nothing seems to be working. cdk version 2.177.0 (build b396961)

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

ckd successfully push the docker image to ecr

Current Behavior

the docker push is returning 400 bad request and failing cdk deploy

Reproduction Steps

yarn cdk deploy

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.177.0 (build b396961)

Framework Version

No response

Node.js Version

v20.15.0

OS

macOS 14.5

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Was it working prior to 2.177.0 but failing in 2.177.0 ?

400 bad request generally indicates bad requests and it could be many different causes.

Are you able to simply push a local image to your ECR using your current AWS identity from local?

What's happening in CDK is essentially using $CDK_DOCKER or docker executive to docker build and push to ECR assets and handle required get-login-password for you. The image assets uploading process could fail due to network issue as well.

Let me know if you are able to manually push an random image to ecr?

[james-g-stream]

Hi @pahud thanks for the response, I reverted to version 2.173.1 and it still fails with 400 bad request on ecr push.

I was able to docker login with aws ecr get-login-password --region YOUR_AWS_REGION | docker login --username AWS --password-stdin YOUR_AWS_ACCOUNT_ID.dkr.ecr.YOUR_AWS_REGION.amazonaws.com
docker build, tag and push to my ecr account.

I then tried to cdk deploy and it failed again with 400 Bad Request.

What I did notice was this. When i check my docker AWS credentials with docker-credential-desktop get <<< "https://{AWS_ACCOUNT}.dkr.ecr.us-east-1.amazonaws.com"

I checked the JWT in the "Secret" property of that credential JSON in jwt.io. when i manually login with ecr, the jwt was valid in jwt.io. But the JWT from "Secret" that gets set when I run cdk deploy comes back as an invalid JWT in jwt.io as it ends in "==".
The JWT from the manual login does not end in "==".

Not sure if this maybe causing issues.

[james-g-stream]

I ended up deleting docker desktop and installing docker and colima and it's working fine with colima with both buildkit enabled and disabled.

[wbeardall]

+1 to this with Docker Desktop on MacOS 15.2, CDK 2.177.0 (build b396961). Same error comes up with ecs.AssetImage.fromTarball.

[pahud]

Thank you for the report. @james-g-stream and @wbeardall. While we'll investigate this issue, can you help us check is it only happening in 2.177.0 ? What about 2.176 or even earlier versions?

[james-g-stream]

@pahud i tried it on 2.173.1 and still happening only for docker desktop

[kaizencc]

Possibly related to these issues, I'm looking into it: #30258 (comment)

Need to turn off containerd or set --provenance=false. CDK fix is likely to add that as default.

[tmokmss]

@kaizencc yes they are related. In our cases, we get the 400 error for each image asset when cdk tries to push attestations and images, which sometimes results in 0mb images. (The behavior seems indeterministic. Sometimes attestation wins and sometimes the actual image wins. They share the same image tag and the first one is pushed but the second one throws an error because of ecr tag immutability, it seems.)

Then if we retry cdk deploy, it continues to deploy with those invalid images, because cdk does not have to push any images due to cache, which finally results in ecs or lambda deployment errors.