(aws_ecs_patterns): ApplicationLoadBalancedFargateService fails to update when switched from public to private - Fails due to target group

[hqv-divya]

Describe the bug

When the ApplicationLoadBalancedFargateService pattern is used to create an ALB along with a target group, it creates a public facing load balancer. Once this is created, if I try to switch it to a private ALB, using the parameter publicLoadBalancer: false, it fails with error:

Resource handler returned message: "The following target groups cannot be associated with more than one load balancer: arn:aws:elasticloadbalancing:eu-west-1:318627068478:targetgroup/EcsCon-Sampl-HV24A4ZHHGG1/829e923a9abe6705 (Service: ElasticLoadBalancingV2, Status Code: 400, Request ID: dbe30509-a4b7-42fa-9087-bcff22fdb55b)" (RequestToken: 544175ba-248a-5a75-3458-50b0f5d80582, HandlerErrorCode: ServiceLimitExceeded)

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The target group should be disassociated from the previous load balancer and then attempted to be attached to the new load balancer or a new target group could be created to attach to the new load balancer.

Here's the CDK diff:

Resources
[~] AWS::ElasticLoadBalancingV2::LoadBalancer SampleNetworkLoadBalancedFargateService/LB SampleNetworkLoadBalancedFargateServiceLB3B04F5EA replace
 ├─ [~] Scheme (requires replacement)
 │   ├─ [-] internet-facing
 │   └─ [+] internal
 └─ [~] Subnets
     └─ @@ -1,4 +1,4 @@
        [ ] [
        [-]   "subnet-08d2e5e9a5526d5cd",
        [-]   "subnet-03639cc2fb087e322"
        [+]   "subnet-0be5b8bc6890cdfd1",
        [+]   "subnet-002110f783fd87957"
        [ ] ]
[~] AWS::ElasticLoadBalancingV2::Listener SampleNetworkLoadBalancedFargateService/LB/PublicListener SampleNetworkLoadBalancedFargateServiceLBPublicListener8ED1C5F0 replace
 └─ [~] LoadBalancerArn (requires replacement)
     └─ [~] .Ref:
         ├─ [-] SampleNetworkLoadBalancedFargateServiceLB3B04F5EA
         └─ [+] SampleNetworkLoadBalancedFargateServiceLB3B04F5EA (replaced)


✨  Number of stacks with differences: 1

Current Behavior

Stack deployment fails because the target group is not attached to the private load balancer since it is already attached to the public load balancer

Reproduction Steps

Create an application load balanced fargate service using the construct as follows:

const fargateService = new ApplicationLoadBalancedFargateService(this, "SampleNetworkLoadBalancedFargateService", {
  vpc,
  taskDefinition: taskDefinition,
  minHealthyPercent: 100
});

// fargateService.service.connections.allowFromAnyIpv4(Port.tcp(80), "Allow inbound HTTP traffic");

fargateService.targetGroup.healthCheck = {
  port: "80",
  protocol: Protocol.HTTP,
  path: "/alive",
  interval: Duration.seconds(20),
}

Simply add the publicLoadBalancer: false to the construct ApplicationLoadBalancedFargateService as follows:

const fargateService = new ApplicationLoadBalancedFargateService(this, "SampleNetworkLoadBalancedFargateService", {
      vpc,
      taskDefinition: taskDefinition,
      minHealthyPercent: 100,
      publicLoadBalancer: false
});

// fargateService.service.connections.allowFromAnyIpv4(Port.tcp(80), "Allow inbound HTTP traffic");

fargateService.targetGroup.healthCheck = {
  port: "80",
  protocol: Protocol.HTTP,
  path: "/alive",
  interval: Duration.seconds(20),
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

v2.177.0

Framework Version

No response

Node.js Version

v20.17.0

OS

Windows

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

I am still trying to reproduce this issue but off the top off my head:

Resources
[~] AWS::ElasticLoadBalancingV2::LoadBalancer DummyService/LB DummyServiceLBEC880CC4 replace
 ├─ [~] Scheme (requires replacement)
 │   ├─ [-] internet-facing
 │   └─ [+] internal
 └─ [~] Subnets
     └─ @@ -1,5 +1,5 @@
        [ ] [
        [-]   "subnet-0564da5939b9f37e8",
        [-]   "subnet-0a2ae97b17b9f6820",
        [-]   "subnet-0f1bbb27b1f58cca7"
        [+]   "subnet-071c85610846aa9c0",
        [+]   "subnet-0ef7ac49e1edb06e4",
        [+]   "subnet-0e2177a10a166f87d"
        [ ] ]
[~] AWS::ElasticLoadBalancingV2::Listener DummyService/LB/PublicListener DummyServiceLBPublicListener47F599C1 replace
 └─ [~] LoadBalancerArn (requires replacement)
     └─ [~] .Ref:
         ├─ [-] DummyServiceLBEC880CC4
         └─ [+] DummyServiceLBEC880CC4 (replaced)

DummyStack6: deploying... [1/1]
DummyStack6: creating CloudFormation changeset...
DummyStack6 | 0/4 | 9:42:37 AM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack                | DummyStack6 User Initiated
DummyStack6 | 0/4 | 9:42:41 AM | UPDATE_IN_PROGRESS   | AWS::ElasticLoadBalancingV2::LoadBalancer | DummyService/LB (DummyServiceLBEC880CC4) Requested update requires the creation of a new physical resource; hence creating one.
DummyStack6 | 0/4 | 9:42:44 AM | UPDATE_IN_PROGRESS   | AWS::ElasticLoadBalancingV2::LoadBalancer | DummyService/LB (DummyServiceLBEC880CC4) Resource creation Initiated
0/4 Currently in progress: DummyStack6, DummyServiceLBEC880CC4
DummyStack6 | 1/4 | 9:45:47 AM | UPDATE_COMPLETE      | AWS::ElasticLoadBalancingV2::LoadBalancer | DummyService/LB (DummyServiceLBEC880CC4) 
DummyStack6 | 1/4 | 9:45:49 AM | UPDATE_IN_PROGRESS   | AWS::ElasticLoadBalancingV2::Listener     | DummyService/LB/PublicListener (DummyServiceLBPublicListener47F599C1) Requested update requires the creation of a new physical resource; hence creating one.
DummyStack6 | 1/4 | 9:45:50 AM | UPDATE_FAILED        | AWS::ElasticLoadBalancingV2::Listener     | DummyService/LB/PublicListener (DummyServiceLBPublicListener47F599C1) Resource handler returned message: "The following target groups cannot be associated with more than one load balancer: arn:aws:elasticloadbalancing:us-east-1:903779448426:targetgroup/DummyS-Dummy-4AU3AQD6FKVD/11f45c93e6a4f438 (Service: ElasticLoadBalancingV2, Status Code: 400, Request ID: 46e07555-a1fe-4531-af3c-4047667fc5fc)" (RequestToken: 34a3fb0e-193e-6c55-82e2-00fb06b90d77, HandlerErrorCode: ServiceLimitExceeded)
DummyStack6 | 1/4 | 9:45:50 AM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack                | DummyStack6 The following resource(s) failed to update: [DummyServiceLBPublicListener47F599C1]. 
DummyStack6 | 0/4 | 9:45:52 AM | UPDATE_COMPLETE      | AWS::ElasticLoadBalancingV2::LoadBalancer | DummyService/LB (DummyServiceLBEC880CC4) 
DummyStack6 | 1/4 | 9:45:53 AM | UPDATE_COMPLETE      | AWS::ElasticLoadBalancingV2::Listener     | DummyService/LB/PublicListener (DummyServiceLBPublicListener47F599C1) 
DummyStack6 | 2/4 | 9:45:54 AM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack                | DummyStack6 
DummyStack6 | 1/4 | 9:45:56 AM | DELETE_COMPLETE      | AWS::ElasticLoadBalancingV2::Listener     | DummyService/LB/PublicListener (DummyServiceLBPublicListener47F599C1) 
DummyStack6 | 1/4 | 9:45:56 AM | DELETE_IN_PROGRESS   | AWS::ElasticLoadBalancingV2::LoadBalancer | DummyService/LB (DummyServiceLBEC880CC4) 
1/4 Currently in progress: DummyStack6, DummyServiceLBEC880CC4
DummyStack6 | 0/4 | 9:46:57 AM | DELETE_COMPLETE      | AWS::ElasticLoadBalancingV2::LoadBalancer | DummyService/LB (DummyServiceLBEC880CC4) 
DummyStack6 | 1/4 | 9:46:57 AM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack                | DummyStack6 

Failed resources:
DummyStack6 | 9:45:50 AM | UPDATE_FAILED        | AWS::ElasticLoadBalancingV2::Listener     | DummyService/LB/PublicListener (DummyServiceLBPublicListener47F599C1) Resource handler returned message: "The following target groups cannot be associated with more than one load balancer: arn:aws:elasticloadbalancing:us-east-1:903779448426:targetgroup/DummyS-Dummy-4AU3AQD6FKVD/11f45c93e6a4f438 (Service: ElasticLoadBalancingV2, Status Code: 400, Request ID: 46e07555-a1fe-4531-af3c-4047667fc5fc)" (RequestToken: 34a3fb0e-193e-6c55-82e2-00fb06b90d77, HandlerErrorCode: ServiceLimitExceeded)
❌  DummyStack6 failed: _ToolkitError: The stack named DummyStack6 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The following target groups cannot be associated with more than one load balancer: arn:aws:elasticloadbalancing:us-east-1:903779448426:targetgroup/DummyS-Dummy-4AU3AQD6FKVD/11f45c93e6a4f438 (Service: ElasticLoadBalancingV2, Status Code: 400, Request ID: 46e07555-a1fe-4531-af3c-4047667fc5fc)" (RequestToken: 34a3fb0e-193e-6c55-82e2-00fb06b90d77, HandlerErrorCode: ServiceLimitExceeded)

what's happening here is:

1. CFN would try to create a new LoadBalancer in replace of the existing one
2. a new Listener resource would be created for replacement as well and its Action will link to the existing target group, hence the error.

I guess we probably can't simply replace that attribute from "true" to "false" as this seems a restriction from CFN that CDK can't just easily workaround. I guess we might need to target group replacement. If we don't have a workaround, we should note the limit in the document. Making this a p1.