Kibana UI seems to enforce "notifyWhen" under alerts actions in "frequency", but Terraform does not import it

[ASH895-N]

Describe the bug
When we started automating the Kibana alerts using Terraform provider: https://registry.terraform.io/providers/elastic/elasticstack/latest/docs, we started checking the basic skeleton of the resource "elasticstack_kibana_alerting_rule" across multiple Kibana alert types like "ElasticSearch Query rule", "Log threshold", "Index threshold" etc. which were created in our ELK manually via UI.

We are now managing alerts using this automation with the Terraform configuration for the resources exactly as specified in the sample here: https://github.com/elastic/terraform-provider-elasticstack/blob/main/examples/resources/elasticstack_kibana_alerting_rule/resource-index-rule.tf

For the above alert types, they required "notifyWhen" field at the rule level: https://github.com/elastic/terraform-provider-elasticstack/blob/main/examples/resources/elasticstack_kibana_alerting_rule/resource-index-rule.tf#L7

This automation was working fine until we upgraded our ELK clusters from ~v8.9 to v8.12.

For the alerts created with "notifyWhen" set on "rule-level", I am currently observing that manual updates to these alerts results in UI enforcing it at "action-level" in the form of "frequency" map as follows -

actions {
         ...
        frequency = {
            notify_when = "onActionGroupChange"
            summary     = "false"
            throttle    = //not needed 
        }
    }

For context, we have all rules imported to have source of truth, but also allow our teams to test feature via UI.

My apologies for such lengthy issue description, but I have tried to add as much details as possible.

To Reproduce
Steps to reproduce the behavior:

1. Terraform resource configuration -

• Terraform resource -

  # module.kibana_alert_rules["abc copy.yaml"].elasticstack_kibana_alerting_rule.elasticsearch_query_rules[0] will be created
  resource "elasticstack_kibana_alerting_rule" "elasticsearch_query_rules" {
      consumer              = "discover"
      enabled               = true
      interval              = "1m"
      name                  = "Test-frequency"
      notify_when           = "onActionGroupChange" --> Set here 
      params                = jsonencode(
            {
              aggType                    = "count"
              excludeHitsFromPreviousRun = true
              groupBy                    = "all"
              searchConfiguration        = {
                  filter = [
                      {
                          meta  = {
                              index = "index ID"
                            }
                          query = {
                              query_string = {
                                  query = "service.name: abc"
                                }
                            }
                        },
                    ]
                  index  = "index ID"
                  query  = {
                      language = "kuery"
                      query    = ""
                    }
                }
              searchType                 = "searchSource"
              size                       = 5
              termSize                   = 5
              threshold                  = [
                  100,
                ]
              thresholdComparator        = ">="
              timeField                  = "@timestamp"
              timeWindowSize             = 5
              timeWindowUnit             = "m"
            }
        )
      rule_type_id          = ".es-query"
      space_id              = "default"
      tags                  = [
          "test",
        ]
      actions {
          group  = "query matched"
          id     = "Slack webapi connector ID"
          params = jsonencode(
                {
                  subAction       = "postMessage"
                  subActionParams = {
                      channels = [
                          "channel_ID",
                        ]
                      text     = <<-EOT
                            This is a sample - test frequency
                            
                            '{{alertName}}' is active:
                            
                            Hello team, please check :point_up:
                            Details:
                            - Timestamp: {{context.date}}
                            - Conditions Met: {{context.conditions}} over {{params.timeWindowSize}}{{params.timeWindowUnit}}
                            - Matches: {{context.value}}
                        EOT
                    }
                }
            )
        }
    }

• API call made by Terraform to create alert rule (POST) -

{
    "actions": [
        {
            "group": "query matched",
            "id": "slack_web_api_connector_ID",
            "params": {
                "subAction": "postMessage",
                "subActionParams": {
                    "channels": [
                        "channel_ID"
                    ],
                    "text": "This is a sample - test frequency\n\n'{{alertName}}' is active:\n\nHello team, please check :point_up:\nDetails:\n- Timestamp: {{context.date}}\n- Conditions Met: {{context.conditions}} over {{params.timeWindowSize}}{{params.timeWindowUnit}}\n- Matches: {{context.value}}"
                }
            }
        }
    ],
    "consumer": "discover",
    "enabled": true,
    "name": "Test-frequency",
    "notify_when": "onActionGroupChange", --> Set here 
    "params": {
        "aggType": "count",
        "excludeHitsFromPreviousRun": true,
        "groupBy": "all",
        "searchConfiguration": {
            "filter": [
                {
                    "meta": {
                        "index": "d3d7af60-4c81-11e8-b3d7-01146121b73d"
                    },
                    "query": {
                        "query_string": {
                            "query": "service.name: abc"
                        }
                    }
                }
            ],
            "index": "d3d7af60-4c81-11e8-b3d7-01146121b73d",
            "query": {
                "language": "kuery",
                "query": ""
            }
        },
        "searchType": "searchSource",
        "size": 5,
        "termSize": 5,
        "threshold": [
            100
        ],
        "thresholdComparator": ">=",
        "timeField": "@timestamp",
        "timeWindowSize": 5,
        "timeWindowUnit": "m"
    },
    "rule_type_id": ".es-query",
    "schedule": {
        "interval": "1m"
    },
    "tags": [
        "IAAC",
        "test"
    ],
    "throttle": null
}

• Resource after apply - in local state

# module.kibana_alert_rules["abc copy.yaml"].elasticstack_kibana_alerting_rule.elasticsearch_query_rules[0]:
resource "elasticstack_kibana_alerting_rule" "elasticsearch_query_rules" {
    consumer              = "discover"
    enabled               = true
    id                    = "default/88a8958f-a58f-4195-afb1-604e713c79a9"
    interval              = "1m"
    last_execution_date   = "2024-03-19 11:31:09.26 +0000 UTC"
    last_execution_status = "pending"
    name                  = "Test-frequency"
    notify_when           = "onActionGroupChange" ---> set here 
    params                = jsonencode(
        {....}
     )
    rule_id               = "88a8958f-a58f-4195-afb1-604e713c79a9"
    rule_type_id          = ".es-query"
    scheduled_task_id     = "88a8958f-a58f-4195-afb1-604e713c79a9"
    space_id              = "default"
    tags                  = [
        "IAAC",
        "test",
    ]

    actions {
        group  = "query matched"
        id     = "64956021-6980-4958-8469-afdba75baee0"
        params = jsonencode(
            {
                subAction       = "postMessage"
                subActionParams = {
                    channels = [
                        "channel_ID",
                    ]
                    text     = <<-EOT
                        This is a sample - test frequency
                        
                        '{{alertName}}' is active:
                        
                        Hello team, please check :point_up:
                        Details:
                        - Timestamp: {{context.date}}
                        - Conditions Met: {{context.conditions}} over {{params.timeWindowSize}}{{params.timeWindowUnit}}
                        - Matches: {{context.value}}
                    EOT
                }
            }
        )
    }
}

When I try to update from UI, I also tracked the API call it makes to update the alert - PUT API call with following payload :

{
    "name": "Test-frequency",
    "tags": [
        "test",
        "abc"
    ],
    --> Not here 
    "schedule": {
        "interval": "1m"
    },
    "params": {
        ...
        },
        "timeField": "timestamp",
        "searchType": "searchSource",
        "timeWindowSize": 5,
        "timeWindowUnit": "m",
        "threshold": [
            100
        ],
        "thresholdComparator": ">=",
        "size": 5,
        "aggType": "count",
        "groupBy": "all",
        "termSize": 5,
        "excludeHitsFromPreviousRun": true
    },
    "actions": [
        {
            "group": "query matched",
            "id": "64956021-6980-4958-8469-afdba75baee0",
            "params": {
                "subAction": "postMessage",
                "subActionParams": {
                    "channels": [
                        "channel_ID"
                    ],
                    "text": "some text"
                }
            },
            "frequency": {
                "notify_when": "onActionGroupChange", ----> Moved here 
                "throttle": null,
                "summary": false
            },
            "uuid": "07fe9cf3-0343-44aa-a946-953d00f18e04"
        }
    ]
}

If I remove the resource from Terraform state just to import it , the following is from the GET API call made by Terraform:

We can see that "notifyWhen" setting is moved to "frequency" map under alert "actions" :

{
    "id": "88a8958f-a58f-4195-afb1-604e713c79a9",
    "name": "Test-frequency",
    "tags": [
        "test",
        "abc"
    ],
    "enabled": true,
    "consumer": "discover",
    "throttle": null,
    "revision": 2,
    "running": false,
    "schedule": {
        "interval": "1m"
    },
    "params": {
        ...
            },
            "index": "d3d7af60-4c81-11e8-b3d7-01146121b73d"
        },
        "timeField": "timestamp",
        "searchType": "searchSource",
        "timeWindowSize": 5,
        "timeWindowUnit": "m",
        "threshold": [
            100
        ],
        "thresholdComparator": ">=",
        "size": 5,
        "aggType": "count",
        "groupBy": "all",
        "termSize": 5,
        "excludeHitsFromPreviousRun": true
    },
    "rule_type_id": ".es-query",
    ...
    "notify_when": null, --> Here it's "null"
    ...
    "actions": [
        {
            "group": "query matched",
            "params": {
                "subAction": "postMessage",
                "subActionParams": {
                    "channels": [
                        "channel_ID"
                    ],
                    "text": "some Text"
                }
            },
            "uuid": "07fe9cf3-0343-44aa-a946-953d00f18e04",
            "id": "64956021-6980-4958-8469-afdba75baee0",
            "connector_type_id": ".slack_api",
            "frequency": { ----> But it's moved here
                "throttle": null,
                "summary": false,
                "notify_when": "onActionGroupChange"
            }
        }
    ],
    ...
}

But it is not imported in Terraform resource configuration - It's not present at either places

# module.kibana_alert_rules["abc-copy.yaml"].elasticstack_kibana_alerting_rule.elasticsearch_query_rules[0]:
resource "elasticstack_kibana_alerting_rule" "elasticsearch_query_rules" {
    consumer              = "discover"
    enabled               = true
    id                    = "default/88a8958f-a58f-4195-afb1-604e713c79a9"
    interval              = "1m"
    last_execution_date   = "2024-03-19 11:56:39.675 +0000 UTC"
    last_execution_status = "ok"
    name                  = "Test-frequency"
    params                = jsonencode(
        {
        ...
        }
    )
    rule_id               = "88a8958f-a58f-4195-afb1-604e713c79a9"
    rule_type_id          = ".es-query"
    scheduled_task_id     = "88a8958f-a58f-4195-afb1-604e713c79a9"
    space_id              = "default"
    tags                  = [
        "test",
        "abc",
    ]

    actions {
        group  = "query matched"
        id     = "64956021-6980-4958-8469-afdba75baee0"
        params = jsonencode(
            {
                subAction       = "postMessage"
                subActionParams = {
                    channels = [
                        "channel_ID",
                    ]
                    text     = <<-EOT
                        someText
                    EOT
                }
            }
        )
    }
}

Additionally, if an alert is created from UI, the "notifyWhen" is set at "action-level" - The following is from "Inspect" browser tool to get the request payload while the UI performs POST API call to create a new rule :

{
    "params": {
       "searchConfiguration": {
            ...
    }
}
    "consumer": "stackAlerts",
    "schedule": {
        "interval": "1m"
    },
    "tags": [],
    "name": "test-2",
    "rule_type_id": ".es-query",
    "actions": [
        {
            "group": "query matched",
            "id": "64956021-6980-4958-8469-afdba75baee0",
            "params": {
                "subAction": "postMessage",
                "subActionParams": {
                    "channelIds": [
                        "channel_ID"
                    ],
                    "text": "some text"
                }
            },
            "frequency": { --> here 
                "notify_when": "onActionGroupChange",
                "throttle": null,
                "summary": false
            }
        }
    ]
}

So, our expectation is that for alerts created with UI, we need a means to support the "frequency" setting at "action-level" as well in the Terraform provider. This is also because we would like to use the newer features like "on custom action intervals" which internally uses the "frequency" setting which we are unable as the provider does not support it :

"frequency": {
                "notify_when": "onThrottleInterval",
                "throttle": "1h",
                "summary": false
            }

Expected behavior
The provider should import the frequency setting under "frequency" in alert actions as the API call returns it.

Versions (please complete the following information):

• Provider version - 0.11.1
• Elasticsearch Version - 8.12.1 in my local cluster and 8.12.2 in our ELK cluster

[ASH895-N]

Also, I tried adding the "frequency" setting in Terraform -

 actions {
         ...
        frequency = {
            notify_when = "onActionGroupChange"
            summary     = "false"
            throttle    = //not needed 
        }
    }

But Terraform plan resulted in this error - An argument named "frequency" is not expected here.

The documentation here : https://registry.terraform.io/providers/elastic/elasticstack/latest/docs/resources/kibana_alerting_rule#notify_when specifies this

NOTE: This is a rule level property; if you update the rule in Kibana, it is automatically changed to use action-specific notify_when values.

But the action schema itself does not support frequency setting under "actions".

[tobio]

Duplicate of #524

[cnasikas]

Fixed by elastic/kibana#186963.