Rewrite Rule Set CRD issue in shared AppGw setup

[vedadsma]

Describe the bug
Issue occurs in a setup with a shared Application Gateway resource that is used between two Kubernetes clusters (DEVCD & DEV environments). While the shared Application Gateway setup with AzureIngressProhibitedTarget has been working fine for a long time, we encountered an issue when implementing Rewrite Rule Set Custom Resource:

• Both clusters are configured to use the same shared Application Gateway resource
• Rewrite rule set works successfully on the DEVCD cluster (first cluster)
• When deploying to the DEV cluster (second cluster), an error appears indicating that the referenced rewrite rule set resource was not found in the shared Application Gateway
• AzureIngressProhibitedTarget is configured on both clusters to manage the shared resource

To Reproduce
Steps to reproduce the behavior:

1. Configure shared Application Gateway resource to be used by both DEVCD and DEV clusters
2. Deploy Rewrite Rule Set Custom Resource to DEVCD cluster (works successfully)
3. Deploy Rewrite Rule Set Custom Resource to DEV cluster
4. Check for errors in AGIC logs on the DEV cluster.

Ingress Controller details

• Output of kubectl describe pod from DEVCD cluster:

I0116 08:04:49.054483       1 mutate_app_gateway.go:166] BEGIN AppGateway deployment
I0116 08:04:55.664506       1 client.go:220] OperationID='ad365204-07d8-4b94-8e6a-d71ce96cdc85'
I0116 08:05:23.113274       1 mutate_app_gateway.go:174] Applied generated Application Gateway configuration
I0116 08:05:23.113297       1 mutate_app_gateway.go:189] cache: Updated with latest applied config.
I0116 08:05:23.142408       1 mutate_app_gateway.go:193] END AppGateway deployment
I0116 08:05:23.142432       1 controller.go:152] Completed last event loop run in: 40.081719092

• Output of kubectl logs from DEV cluster showing the error:

I0116 08:08:20.414164       1 mutate_app_gateway.go:166] BEGIN AppGateway deployment
I0116 08:08:23.409601       1 mutate_app_gateway.go:172] END AppGateway deployment
E0116 08:08:23.409678       1 controller.go:142] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidResourceReference" Message="Resource /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myproj-dev-base-rg/providers/Microsoft.Network/applicationGateways/myproj-dev-gateway/rewriteRuleSets/crd-myproj-app-devcd-rewrite-rule-set referenced by resource /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myproj-dev-base-rg/providers/Microsoft.Network/applicationGateways/myproj-dev-gateway/urlPathMaps/devcd-url-xxxxxxxxxxxxxxxxxxxxxxxxx was not found. Please make sure that the referenced resource exists, and that both resources are in the same region." Details=[]
E0116 08:08:23.409697       1 worker.go:72] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidResourceReference" Message="Resource /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myproj-dev-base-rg/providers/Microsoft.Network/applicationGateways/myproj-dev-gateway/rewriteRuleSets/crd-myproj-app-devcd-rewrite-rule-set referenced by resource /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myproj-dev-base-rg/providers/Microsoft.Network/applicationGateways/myproj-dev-gateway/urlPathMaps/devcd-url-xxxxxxxxxxxxxxxxxxxxxxxxx was not found. Please make sure that the referenced resource exists, and that both resources are in the same region."