Unable to upgrade cluster version due to APM issue

[koushikgongireddy]

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

Terraform apply should return success.

Current Behavior

Trying to update Elastic Cluster from 7.17.6 to 7.17.24 and am seeing issues with apm, below is the Terraform error

╷
│ Error: failed updating deployment: 1 error occurred:
│   * api error: deployments.topology_element_using_legacy_ic: The referenced instance configuration [gcp.apm.1] in topology element [<unknown>] is no longer available .
│ 
│ 
│ 
│   with module.ec.ec_deployment.cluster,
│   on ../../gcp-modules/elastic-cloud/cluster.tf line 3, in resource "ec_deployment" "cluster":
│    3: resource "ec_deployment" "cluster" {
│ 

## Terraform definition
We are not specifying anything for APM and directly reading what provider offers
example

resource "ec_deployment" "cluster" {
  # Optional name.
  name = "${var.environment}-${var.cluster_name_suffix}${var.hash_suffix}"

  # Mandatory fields
  region                 = "gcp-${var.location}"
  version                = var.es_version
   
   apm {}
}

Context

1. Terragrunt apply
2. I tried to remove apm block from terraform and its removing the below which we don't want to

  # module.ec.ec_deployment.cluster will be updated in-place
  ~ resource "ec_deployment" "cluster" {
        id                     = "xxxxx"
        name                   = "testec"
        tags                   = {}
      ~ version                = "7.17.6" -> "7.17.24"
        # (7 unchanged attributes hidden)

      - apm {
          - elasticsearch_cluster_ref_id = "main-elasticsearch" -> null
          - http_endpoint                = "xxxxx" -> null
          - https_endpoint               = "xxxxxx" -> null
          - ref_id                       = "main-apm" -> null
          - region                       = "xxx-us-central1" -> null
          - resource_id                  = "xxxxxxx" -> null

          - topology {
              - instance_configuration_id = "gcp.apm.1" -> null
              - size                      = "0.5g" -> null
              - size_resource             = "memory" -> null
              - zone_count                = 1 -> null
            }
        }

        # (2 unchanged blocks hidden)
    }

Possible Solution

Need to apply TF changes without any issues when there is no change in TF version or elastic version

Your Environment

• Version used:

terraform {
  required_version = "1.5.5"

  required_providers {
    google = "5.12.0"
    ec = {
      source  = "elastic/ec"
      version = "0.5.1"
    }
  }
}

[koushikgongireddy]

@tobio Can you please check on this issue!!
We are facing issue with elastic cluster upgrade from 7.17.6 to 7.17.24

I see in documentation its deprecated - https://registry.terraform.io/providers/elastic/ec/0.5.1/docs/resources/ec_deployment

apm DEPRECATED (Optional) APM instance definition, can only be specified once. It should only be used with deployments with a version prior to 8.0.0.

Does that means we need to remove it from our config and removing it will cause any issues on the cluster?

Also we are using gcp-io-optimized for deployment_template_id is that causing this issue?

[tobio]

Does that means we need to remove it from our config and removing it will cause any issues on the cluster?

Nope, continuing to use APM won't cause issues with you deployment. APM has been essentially replaced by the integrations_server component in deployments running Elastic Stack 8+, but will continue to be supported in 7.x deployments.

Also we are using gcp-io-optimized for deployment_template_id is that causing this issue?

Looking at the error, I believe this is causing the issue. I'd expect migrating to a current deployment template will rectify the problem for you. gcp-general-purpose looks like the most similar option.

Changing the deployment template may cause changes to the allocated CPU/storage resources, and the hourly cost. You can view the potential changes in the Cloud console, however we don't surface this data in the provider at the moment.

Hit the Edit button on the current hardware profile in the deployment overview:

Then when you select a new hardware profile, you'll see a breakdown of any changes that apply:

You can just change the deployment_template_id attribute and apply the change through Terraform as well though.

If there continues to be issues, or you'd like us to look at your deployment specifically then reach out to Support either through the Cloud Console or via [email protected] and we can work through the steps.

[koushikgongireddy]

Thanks @tobio
Let me check on this and get back!!