Unable to get metrics data from Cloudwach when using more than one dimension

[abanfi-nozomi]

Report

We are using AWS Cloudwatch scaler.
When the metric we want to retrieve data for has one dimension, everything works as expected
When the metric has more than one diimension, the scaler seems not able to retrieve metrics data and default values is returned.

Expected Behavior

By defining this trigger

{{ if .Values.application.AutoscalingEnabled }}
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {{ .Values.application.name }}
  labels:
    deploymentName: {{ .Values.application.name }}
    {{- include "n2eos.labels" . | nindent 4 }}
spec:
  minReplicaCount: {{ .Values.application.minReplicas }}
  maxReplicaCount: {{ .Values.application.maxReplicas }}
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: {{ .Values.application.name }}
  triggers:
  - type: aws-cloudwatch
    metricType: Value
    metadata:
      identityOwner: "operator"
      awsRegion: {{ .Values.application.env.AWS_REGION }}
      namespace: "AWS/ApplicationELB"
      dimensionName: "TargetGroup;LoadBalancer"
      dimensionValue: "{{ .Values.application.env.TG_NAME }};{{ .Values.application.env.ALB_NAME }}"
      metricName: "TargetResponseTime"
      metricStat: "Average"
      metricCollectionTime: "180"
      metricStatPeriod: "60"
      targetMetricValue: {{ .Values.application.HighExternalRequestsLatencyMetrictarget | quote }}
      minMetricValue: "0"
  - type: cpu
    metadata:
      type: Utilization
      value: {{ .Values.application.targetAverageUtilization | quote }}
{{ end }}

we expect to retrieve data from cloudwatch, but this is not happening

Actual Behavior

The keda operator logs says that the metrics is empty and the default value (0) is returned.
Running
``kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/***/s0-aws-cloudwatch?labelSelector=scaledobject.keda.sh%2Fname%3Dapplication"`
zero value (which is the default is returned)
The workaround we found to make this work is to use expression.
If we define the trigger like the following

{{ if .Values.application.AutoscalingEnabled }}
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {{ .Values.application.name }}
  labels:
    deploymentName: {{ .Values.application.name }}
    {{- include "n2eos.labels" . | nindent 4 }}
spec:
  minReplicaCount: {{ .Values.application.minReplicas }}
  maxReplicaCount: {{ .Values.application.maxReplicas }}
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: {{ .Values.application.name }}
  triggers:
  - type: aws-cloudwatch
    metricType: Value
    metadata:
      identityOwner: "operator"
      awsRegion: {{ .Values.application.env.AWS_REGION }}
      expression: SELECT AVG(TargetResponseTime) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer,TargetGroup) WHERE TargetGroup = '{{ .Values.application.env.TG_NAME }}' AND LoadBalancer = '{{ .Values.application.env.ALB_NAME }}'
      metricStat: "Average"
      metricCollectionTime: "180"
      metricStatPeriod: "60"
      targetMetricValue: {{ .Values.application.HighExternalRequestsLatencyMetrictarget | quote }}
      minMetricValue: "0"
  - type: cpu
    metadata:
      type: Utilization
      value: {{ .Values.application.targetAverageUtilization | quote }}
{{ end }}

the actual value for the metric is returned

Steps to Reproduce the Problem

1. Create the trigger
2. See that metrics value is not returned from keda

Logs from KEDA operator

│ 2024-10-14T05:54:18Z    INFO    aws_cloudwatch_scaler    empty metric data received, returning minMetricValue    {"type": "ScaledObject", "namespace": "***", "name": "***"}                         

KEDA Version

2.15.1

Kubernetes Version

1.30

Platform

Amazon Web Services

Scaler Details

aws-cloudwatch

Anything else?

Another trigger definition we have, with only one dimension, is working as expected

{{- if .Values.applications.rq.enabled }}
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {{ .Values.applications.rq.name }}-scaling
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: {{ .Values.applications.rq.name }}
  minReplicaCount: {{ .Values.applications.rq.deployment.minReplicas }}
  maxReplicaCount: {{ .Values.applications.rq.deployment.maxReplicas }}
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 30
  triggers:
  - type: aws-cloudwatch
    metricType: Value
    metadata:
      identityOwner: "operator"
      awsRegion: {{ $.Values.applications.rq.env.AWS_REGION }}
      namespace: "AWS/ElastiCache"
      dimensionName: "CacheClusterId"
      dimensionValue: {{ $.Values.applications.rq.dimensionValue }}
      metricName: "DatabaseMemoryUsagePercentage"
      metricStat: "Maximum"
      metricCollectionTime: "180"
      metricStatPeriod: "60"
      targetMetricValue: {{ .Values.applications.rq.memoryPercent | quote }}
      minMetricValue: "0"
{{- end }}

[JorTurFer]

Hello
I missed this message :(
AFAIK, the scaler only supports a single dimension query atm because there isn't any aggregation mechanism to aggregate multiple dimensions and currently the first one is used. I'd say that the expression works because the operation is done on AWS side and KEDA only receives a single value (but I'm not 100% about this).
Are you willing to open a PR adding the aggregation support?

[mannbiher]

I just encountered this bug. It is regression.

Version 2.15.1

This is the API call that is sent to AWS which doesn't have dimensions correctly split. So AWS returns an empty result.

      "MetricStat": {
        "Metric": {
          "Dimensions": [
            {
              "Name": "LoadBalancer;TargetGroup",
              "Value": "app/lb1;targetgroup/tg1"
            }
          ],
          "MetricName": "RequestCount",
          "Namespace": "AWS/ApplicationELB"
        },

Version 2.14.1

The API call working correctly in version 2.14.1 which we are currently running.

      "MetricStat": {
        "Metric": {
          "Dimensions": [
            {
              "Name": "LoadBalancer",
              "Value": "app/lb1"
            },
            {
              "Name": "TargetGroup",
              "Value": "targetgroup/tg1"
            }
          ],
          "MetricName": "RequestCount",
          "Namespace": "AWS/ApplicationELB"
        },

I think this is caused by recent refactoring of cloudwatch scaler which removed logic for split of multiple dimensions.
https://github.com/dttung2905/keda/blame/4f963d8ef1051603b9b275f2d4bd79dce4943ea9/pkg/scalers/aws_cloudwatch_scaler.go#L151-L165

[zroubalik]

Thanks for pointing this out, seems like this is a bug indeed.

[JorTurFer]

Nice catch! I was wrong AFAICS, Could you open a PR fixing the issue?