Volume attach fails with “unexpected state ‘reserved’” error.

[ksawai996]

Terraform Version

terraform:1.9.5
ecl provider:2.9.0

Affected Resource(s)

• ecl_compute_volume_attach_v2

Terraform Configuration Files

main.tf

variable "api_key" {}
variable "api_secret_key" {}
variable "tenant_id" {}
variable "tenant_region" {}

terraform {
  required_providers {
    ecl = {
      source  = "nttcom/ecl"
      version = "2.9.0"
    }
  }
}

provider "ecl" {
  auth_url          = "https://keystone-${var.tenant_region}-ecl.api.ntt.com/v3/"
  user_name         = var.api_key
  password          = var.api_secret_key
  tenant_id         = var.tenant_id
  user_domain_id    = "default"
  project_domain_id = "default"
}

compute_instance.tf

data "ecl_imagestorages_image_v2" "***"{
  name = "***"
}

data "ecl_network_network_v2" "tsdzln02" {
  name = "tsdzln02"
}

data "ecl_network_network_v2" "tsmzln02" {
  name = "tsmzln02"
}

data "ecl_network_network_v2" "tsinln02" {
  name = "tsinln02"
}

data "ecl_network_common_function_gateway_v2" "tscmgw01" {
  name = "tscmgw01"
}

resource "ecl_compute_instance_v2" "tsbastion01" {
  name              = "tsbastion01"
  flavor_id         = "1CPU-2GB"
  availability_zone = "zone1-groupb"
  user_data = file("./user-data/tsbastion01.yml")
  image_id          = data.ecl_imagestorages_image_v2.***.id
  network {
    uuid        = data.ecl_network_network_v2.tsdzln02.id
    fixed_ip_v4 = "***"
  }
}

resource "ecl_compute_instance_v2" "tsmonitor01" {
  name              = "tsmonitor01"
  flavor_id         = "1CPU-2GB"
  availability_zone = "zone1-groupb"
  user_data = file("./user-data/tsmonitor01.yml")
  image_id          = data.ecl_imagestorages_image_v2.***.id
  network {
    uuid        = data.ecl_network_network_v2.tsdzln02.id
    fixed_ip_v4 = "***"
  }
  network {
    uuid        = data.ecl_network_network_v2.tsmzln02.id
    fixed_ip_v4 = "***"
  }
  network {
    uuid        = data.ecl_network_network_v2.tsinln02.id
    fixed_ip_v4 = "***"
  }
  network {
    uuid        = data.ecl_network_common_function_gateway_v2.tscmgw01.network_id
  }
  lifecycle {
    ignore_changes = [user_data]
  }
}

resource "ecl_compute_volume_attach_v2" "volume_attach_tsmonitor01_data" {
  volume_id = "${ecl_compute_volume_v2.tsmonitor01_data_vol.id}"
  server_id = "${ecl_compute_instance_v2.tsmonitor01.id}"
  device    = "/dev/vdb"
}

resource "ecl_compute_volume_attach_v2" "volume_attach_tsmonitor01_backup" {
  volume_id = "${ecl_compute_volume_v2.tsmonitor01_backup_vol.id}"
  server_id = "${ecl_compute_instance_v2.tsmonitor01.id}"
  device    = "/dev/vdc"
  depends_on = [
    "ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data"
  ]
}

storage.tf

resource "ecl_compute_volume_v2" "tsmonitor01_data_vol" {
  name              = "tsmonitor01_data_vol"
  availability_zone = "zone1-groupb"
  size              = "15"
}

resource "ecl_compute_volume_v2" "tsmonitor01_backup_vol" {
  name              = "tsmonitor01_backup_vol"
  availability_zone = "zone1-groupb"
  size              = "15"
}

Debug Output

The debug output is difficult to upload to Gist because it contains confidential information including the api secret key.
Please let us know how to do this so that we can send you the logs encrypted or masked if necessary.

Here is an excerpt from the relevant part of the debug output(Some information is masked).

ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data: Creating...
2024-09-17T07:32:27.453Z [INFO]  Starting apply for ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data
2024-09-17T07:32:27.454Z [DEBUG] ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data: applying the planned Create change
2024-09-17T07:32:27.456Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:27 [DEBUG] ECL Region is: 
2024-09-17T07:32:27.456Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:27 [DEBUG] ECL Region is: 
2024-09-17T07:32:27.457Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:27 [DEBUG] Request: POST https://nova-jp7-ecl.api.ntt.com/v2/***/servers/7980d759-ac7a-4994-87a3-697db21a5ecf/os-volume_attachments
2024-09-17T07:32:27.457Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:27 [DEBUG] Request body: &{{"volumeAttachment":{"device":"/dev/vdb","volumeId":"63bbf49d-b142-4905-90db-2c6eb9fe292f"}} %!s(int64=0) %!s(int=-1)}
2024-09-17T07:32:27.457Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:27 [DEBUG] ECL Request URL: POST https://nova-jp7-ecl.api.ntt.com/v2/***/servers/7980d759-ac7a-4994-87a3-697db21a5ecf/os-volume_attachments
2024-09-17T07:32:27.457Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:27 [DEBUG] ECL Request Headers:
2024-09-17T07:32:27.457Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Accept: application/json
2024-09-17T07:32:27.458Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Content-Type: application/json
2024-09-17T07:32:27.458Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: User-Agent: Terraform/0.12.6 eclcloud/1.0.0
2024-09-17T07:32:27.458Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: X-Auth-Token: ***
2024-09-17T07:32:27.458Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:27 [DEBUG] ECL Request Body: {
2024-09-17T07:32:27.458Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:   "volumeAttachment": {
2024-09-17T07:32:27.459Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "device": "/dev/vdb",
2024-09-17T07:32:27.459Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "volumeId": "63bbf49d-b142-4905-90db-2c6eb9fe292f"
2024-09-17T07:32:27.459Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:   }
2024-09-17T07:32:27.459Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: }
ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data: Still creating... [10s elapsed]
2024-09-17T07:32:38.507Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:38 [DEBUG] ECL Response Code: 200
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:38 [DEBUG] ECL Response Headers:
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Content-Length: 194
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Content-Type: application/json
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Date: Tue, 17 Sep 2024 07:32:38 GMT
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Server: nginx
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: X-Compute-Request-Id: req-38426631-cea9-48db-8665-86f2e8ab6b1e
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: X-Openstack-Request-Id: req-38426631-cea9-48db-8665-86f2e8ab6b1e
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: X-Request-Id: 0b158dda-74c4-456d-badf-84c94512b756
2024-09-17T07:32:38.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:38 [DEBUG] ECL Response Body: {
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:   "volumeAttachment": {
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "device": "/dev/vdb",
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "id": "63bbf49d-b142-4905-90db-2c6eb9fe292f",
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "serverId": "7980d759-ac7a-4994-87a3-697db21a5ecf",
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "volumeId": "63bbf49d-b142-4905-90db-2c6eb9fe292f"
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:   }
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: }
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:38 [DEBUG] Response body: {"volumeAttachment":{"device":"/dev/vdb","id":"63bbf49d-b142-4905-90db-2c6eb9fe292f","serverId":"7980d759-ac7a-4994-87a3-697db21a5ecf","volumeId":"63bbf49d-b142-4905-90db-2c6eb9fe292f"}}
2024-09-17T07:32:38.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:38 [DEBUG] Waiting for state to become: [in-use]
ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data: Still creating... [20s elapsed]
2024-09-17T07:32:48.508Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:48 [DEBUG] Request: GET https://cinder-jp7-ecl.api.ntt.com/v2/***/volumes/63bbf49d-b142-4905-90db-2c6eb9fe292f
2024-09-17T07:32:48.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:48 [DEBUG] ECL Request URL: GET https://cinder-jp7-ecl.api.ntt.com/v2/***/volumes/63bbf49d-b142-4905-90db-2c6eb9fe292f
2024-09-17T07:32:48.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:48 [DEBUG] ECL Request Headers:
2024-09-17T07:32:48.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Accept: application/json
2024-09-17T07:32:48.509Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: User-Agent: Terraform/0.12.6 eclcloud/1.0.0
2024-09-17T07:32:48.510Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: X-Auth-Token: ***
2024-09-17T07:32:48.733Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:48 [DEBUG] ECL Response Code: 200
2024-09-17T07:32:48.734Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:48 [DEBUG] ECL Response Headers:
2024-09-17T07:32:48.734Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Content-Length: 911
2024-09-17T07:32:48.734Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Content-Type: application/json
2024-09-17T07:32:48.734Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Date: Tue, 17 Sep 2024 07:32:48 GMT
2024-09-17T07:32:48.734Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: Server: nginx
2024-09-17T07:32:48.734Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: X-Compute-Request-Id: req-e770a5cc-6b55-40a2-9e89-5bf164ddcff7
2024-09-17T07:32:48.734Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: X-Openstack-Request-Id: req-e770a5cc-6b55-40a2-9e89-5bf164ddcff7
2024-09-17T07:32:48.734Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: X-Request-Id: 000685dc-a0ba-4ac3-a014-55b93b9db5d2
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:48 [DEBUG] ECL Response Body: {
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:   "volume": {
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "attachments": [],
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "availability_zone": "zone1-groupb",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "bootable": "false",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "consistencygroup_id": null,
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "created_at": "2024-09-17T07:31:53.000000",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "description": null,
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "encrypted": false,
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "id": "63bbf49d-b142-4905-90db-2c6eb9fe292f",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "links": [
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:       {
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:         "href": "https://cinder-jp7-ecl.api.ntt.com/v2/***/volumes/63bbf49d-b142-4905-90db-2c6eb9fe292f",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:         "rel": "self"
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:       },
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:       {
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:         "href": "https://cinder-jp7-ecl.api.ntt.com/***/volumes/63bbf49d-b142-4905-90db-2c6eb9fe292f",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:         "rel": "bookmark"
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:       }
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     ],
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "metadata": {},
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "multiattach": false,
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "name": "tsmonitor01_data_vol",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "os-vol-tenant-attr:tenant_id": "***",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "replication_status": null,
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "size": 15,
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "snapshot_id": null,
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "source_volid": null,
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "status": "reserved",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "updated_at": "2024-09-17T07:32:38.000000",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "user_id": "***",
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:     "volume_type": "nfsdriver"
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0:   }
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: }
2024-09-17T07:32:48.735Z [DEBUG] provider.terraform-provider-ecl_v2.9.0: 2024/09/17 07:32:48 [DEBUG] Response body: {"volume":{"attachments":[],"availability_zone":"zone1-groupb","bootable":"false","consistencygroup_id":null,"created_at":"2024-09-17T07:31:53.000000","description":null,"encrypted":false,"id":"63bbf49d-b142-4905-90db-2c6eb9fe292f","links":[{"href":"https://cinder-jp7-ecl.api.ntt.com/v2/***/volumes/63bbf49d-b142-4905-90db-2c6eb9fe292f","rel":"self"},{"href":"https://cinder-jp7-ecl.api.ntt.com/***/volumes/63bbf49d-b142-4905-90db-2c6eb9fe292f","rel":"bookmark"}],"metadata":{},"multiattach":false,"name":"tsmonitor01_data_vol","os-vol-tenant-attr:tenant_id":"***","replication_status":null,"size":15,"snapshot_id":null,"source_volid":null,"status":"reserved","updated_at":"2024-09-17T07:32:38.000000","user_id":"***","volume_type":"nfsdriver"}}
2024-09-17T07:32:48.744Z [ERROR] vertex "ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data" error: Error waiting for volume (63bbf49d-b142-4905-90db-2c6eb9fe292f) be attached: unexpected state 'reserved', wanted target 'in-use'. last error: %!s(<nil>)

Warning: Quoted references are deprecated

  on compute_instance.tf line 84, in resource "ecl_compute_volume_attach_v2" "volume_attach_tsmonitor01_backup":
  "ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data"

  In this context, references are expected literally rather than in quotes.
  Terraform 0.11 and earlier required quotes, but quoted references are now
  deprecated and will be removed in a future version of Terraform. Remove the
  quotes surrounding this reference to silence this warning.

Error: Error waiting for volume (63bbf49d-b142-4905-90db-2c6eb9fe292f) be attached: unexpected state 'reserved', wanted target 'in-use'. last error: %!s(<nil>)

  with ecl_compute_volume_attach_v2.volume_attach_tsmonitor01_data,
  on compute_instance.tf line 71, in resource "ecl_compute_volume_attach_v2" "volume_attach_tsmonitor01_data":
  71: resource "ecl_compute_volume_attach_v2" "volume_attach_tsmonitor01_data" 

2024-09-17T07:32:48.757Z [INFO]  provider: plugin process exited: plugin=.terraform/providers/registry.terraform.io/nttcom/ecl/2.9.0/linux_amd64/terraform-provider-ecl_v2.9.0 id=36
2024-09-17T07:32:48.757Z [DEBUG] provider: plugin exited

Panic Output

N/A

Expected Behavior

Successful attachment of the volume to the instance.

Actual Behavior

The following error occurs.

Error: Error waiting for volume (63bbf49d-b142-4905-90db-2c6eb9fe292f) be attached: unexpected state 'reserved', wanted target 'in-use'. last error: %!s(<nil>)

After the error occurs, checking the status on the portal site, it appears to be attached normally. However, there is a discrepancy between the tfstate and the actual state, which cannot be managed as terraform.

Steps to Reproduce

Just run terraform apply.
It does not occur every time, but rather once every 3 to 4 times, according to our experience.

Looking at line 99 of the code (https://github.com/nttcom/terraform-provider-ecl/blob/master/ecl/resource_ecl_compute_volume_attach_v2.go), The volume attach create seems to expect a state transition from “available” to “attaching” to “in-use”, but in fact there is a timing when the state becomes “reserved”, which seems to be the cause of this issue.

I cannot suggest a code fix because I do not know the exact state transition at volume attach, but I would very much appreciate a fix.

Important Factoids

N/A

References

N/A