[BUG] Edge Lambda scraping throw error when adding multiple regions in config

[tyagian]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

We have EKS cluster deployed in eu-central-1 so
openID is https://oidc.eks.eu-central-1.amazonaws.com/id/xxxxxxxxxxxx

and IAM role policy have these permissions.

{ "Statement": [ { "Action": [ "tag:GetResources", "cloudwatch:*", "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "lambda:Get*", "lambda:List*", "lambda:Describe*", "lambda:TagResource", ], "Effect": "Allow", "Resource": "*", "Sid": "yacemetricsExporterPolicy" } ], "Version": "2012-10-17" }

Trust relationship in IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::xxxxxx:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/xxxxxxx1"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-central-1.amazonaws.com/id/1xxxxxxxxx1:sub": "system:serviceaccount:yace-exporter:yace-exporter"
                }

            }
        }
    ]
}

In yace helm value file, service account is defined as:

serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::00xxxxxxx:role/yace_exporter_role
  labels: {}
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name: yace

Now, when I am adding Lambda metrics in config section of helm values.yaml file:

config: |-
  apiVersion: v1alpha1
  discovery:
    jobs:
    - regions: 
        - eu-central-1
        - us-west-1
        - us-west-2
        - us-east-1
        - us-east-2
      type: lambda
      enableMetricData: true
      metrics:
        - name: Duration
          statistics:
          - Maximum
          - Minimum
          - Average
          period: 300
          length: 3600
        - name: Invocations
          statistics:
          - Sum
          period: 300
          length: 3600
        - name: Throttles
          statistics:
          - Sum
          period: 300
          length: 3600
        - name: Errors
          statistics:
          - Sum
          period: 300
          length: 3600
        - name: ConcurrentExecutions
          statistics:
          - Sum
          period: 300
          length: 3600

I get error:

{"level":"error","msg":"Resource tag list is empty (in 0xxxxxxxxx). Tags must be defined for lambda to be discovered.","time":"2023-03-22T21:56:10Z"}

but when I keep only - eu-central-1 under regions in config, I don't get any such error.
I have Prometheus running in EKS cluster on eu-central-1.

This is the IAM policy for OIDC.

 {
  name = "rcw_yace_exporter_role"

  assume_role_policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::xxxx:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/xxxx"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-central-1.amazonaws.com/id/xxxx:sub": "system:serviceaccount:yace-exporter:yace-exporter"
                }
            }
        }
    ]
  })

How do I know where I am lacking permissions to get rid from this error? I am also using edge Lambda.

• User permissions or IAM role?

Expected Behavior

Expected not to get any error.

Steps To Reproduce

explained above

Anything else?

No response

[cristiangreco]

Hi @tyagian, what version of YACE are you running? Can you please make sure to update to latest? (v0.50.0)

Your config file contains an unsupported field enableMetricData: true. This should be logged as a warning. I'd suggest you look at all log lines to check for other errors.

[tyagian]

I have removed enableMetricData: true and now I see error after upgrading version v0.50.0

{"account":"xxx","arn":"","error":"expected to discover resources but none were found","job_type":"lambda","level":"error","msg":"No tagged resources made it through filtering","region":"us-west-1","time":"2023-04-11T21:43:46Z"}
2
{"account":"xxx","arn":"","error":"expected to discover resources but none were found","job_type":"lambda","level":"error","msg":"No tagged resources made it through filtering","region":"us-west-2","time":"2023-04-11T21:43:46Z"}
1
{"account":"xxx","arn":"","error":"expected to discover resources but none were found","job_type":"lambda","level":"error","msg":"No tagged resources made it through filtering","region":"us-east-2","time":"2023-04-11T21:43:46Z"}

[tyagian]

I have 2 queries:

• If there are no lambda functions in above regions, will it throw this error? Can I stop it from flooding the logger. So, since it can't find any lambda function in those regions, it's assuming, this is an error.
• How can I disable storing for metrics which have value Nan ?

[cristiangreco]

If there are no lambda functions in above regions, will it throw this error? Can I stop it from flooding the logger. So, since it can't find any lambda function in those regions, it's assuming, this is an error.

Yes the error indicates that you're scraping a region where there are no Lambda functions with any tag attached. We might consider changing the log message to "info" rather than "error", but as of today it needs to be logged somehow to signal that there's possibly something wrong with the configuration. In fact, you can just stop scraping the regions where there are no Lambdas maybe?

How can I disable storing for metrics which have value Nan ?

You can set nilToZero: true at the metric level.

[tyagian]

@cristiangreco
The issue happen in case of Lambda @edge functions. Since it use CloudFront and thus Lambda metrics are available in different regions based on from where traffic is coming.

In that case, when I go to CloudWatch and check, I see metrics data in multiple regions where we actually don't have metrics available.

How can I make changes in YACE configuration to allow it to scrape metrics in those regions?

Example, if we have lambda functions in us-west-2 but on Lambda@edge traffic comes from us-east-2, CloudWatch shows metrics in us-east-2 but YACE throw error.

[cristiangreco]

@tyagian can you run yace with -debug and paste the output here?

Additionally, can you please provide the output of:

• aws cloudwatch list-metrics --namespace "AWS/Lambda" --region us-east-2 --recently-active PT3H
• aws cloudwatch list-metrics --namespace "AWS/Lambda" --region us-west-2 --recently-active PT3H
  and:
• aws cloudwatch list-metrics --namespace "AWS/CloudFront" --region us-east-2 --recently-active PT3H
• aws cloudwatch list-metrics --namespace "AWS/CloudFront" --region us-west-2 --recently-active PT3H

[tyagian]

Sure. May I know where do we have option in yace helm chart to enable debug mode: https://github.com/nerdswords/helm-charts/blob/main/charts/yet-another-cloudwatch-exporter/values.yaml ?

[cristiangreco]

Sure. May I know where do we have option in yace helm chart to enable debug mode: https://github.com/nerdswords/helm-charts/blob/main/charts/yet-another-cloudwatch-exporter/values.yaml ?

In the extraArgs field you can set debug: true.

[tyagian]

Okay, I will try to test in Lab.

Meanwhile I also found similar another issue in past asked same query to collect metrics from different regions of edge lambda function : #214
I will try with static config too.

In my case other regions (in which lambda functions don't exist), I have added to the lambda discovery configuration don't have consistent traffic.
So, when yace exporter check those regions like us-west which don't have any lambda function and no traffic, it throw the error.

[tevinjoseph]

I'm also trying to make Lambda@edge work. However, I'm only getting aws_lambda_info results and all other metrics such as Invocations, Errors etc are not showing up. I'm not getting any results when I try

aws cloudwatch list-metrics --namespace "AWS/Lambda" --region us-east-1 --recently-active PT3H

However, I get results when I run the above command without --recently-active PT3H flag

Also, I ran aws cloudwatch list-metrics --namespace "AWS/CloudFront" --region us-east-1 --recently-active PT3H and I'm seeing the default CloudFront metrics such as Requests, BytesDownloaded etc. I can see them in Yace as well.

Is there anything else to add other than Lambda and CloudFront configuration in the yace config.yml?

[tevinjoseph]

Any update on this? @cristiangreco

[cristiangreco]

Hi, can any of you provide a Terraform/CloudFormation example to create necessary test resources? I'm happy to help here.

[cristiangreco]

I've also realised that Lambda@edge functions don't support tags, so they can't be discovered by yace discovery jobs.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions-restrictions.html#lambda-at-edge-function-restrictions

[tyagian]

@cristiangreco The workaround is to use lambda@edge as static job instead of discovery job.
I have added PR to add in example how I fixed it but I recall it today. This can help others.
#1628

Now, we can close this issue after merging this PR. You can make edits in PR.