Feature Request: Allow arrays from external datasource result

[Linuturk]

Terraform Version

Terraform v0.13.4
+ provider registry.terraform.io/hashicorp/aws v2.68.0
+ provider registry.terraform.io/hashicorp/external v2.0.0
+ provider registry.terraform.io/hashicorp/github v3.1.0
+ provider registry.terraform.io/hashicorp/random v2.2.1

Affected Resource(s)

• data external

Terraform Configuration Files

Given the following Terraform:

data "external" "bastions" {
    program = ["bash", "${path.module}/external/bastion-ips.sh"]
}


# Create our security group referencing the IP ranges
resource "aws_security_group" "ssh" {
  name        = "ssh"
  description = "Allow SSH ingress from bastions"
  vpc_id      = var.vpc_id

  ingress {
    description      = "SSH Access"
    from_port        = 22
    to_port          = 22
    protocol         = "-1"
    cidr_blocks      = data.external.bastions.result.ipv4
    ipv6_cidr_blocks = data.external.bastions.result.ipv6
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

}

and the following redacted script snippet:

# Query for the addresses
IPV4_ADDRESSES=$(curl -s "${URL_IPV4}" | jq '[.SomeKey[] | .Address]')
IPV6_ADDRESSES=$(curl -s "${URL_IPV6}" | jq '[.SomeKey[] | .Address]')

# Output the addresses for terraform
jq -n \
    --argjson ipv4 "${IPV4_ADDRESSES}" \
    --argjson ipv6 "${IPV6_ADDRESSES}" \
    '{"ipv4":$ipv4,"ipv6":$ipv6}'

And the script's output (with actual ranges redacted):

{
  "ipv4": [
    "192.168.1.1/32",
    "192.168.1.1/32",
    "192.168.1.1/32"
  ],
  "ipv6": [
    "dead:beef::/56"
  ]
}

Expected Behavior

The expected behavior for a terraform plan is to see the IP ranges that are pulled from the bash script.

Actual Behavior

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.external.bless-bastions: Refreshing state...

Error: command "bash" produced invalid JSON: json: cannot unmarshal array into Go value of type string

Steps to Reproduce

1. terraform apply

References

I've seen a couple of closed issues similar to this, but none specially cover the use case of ingesting a JSON array as the data source's result. Instead I've seen people trying to provide an array to the external script.

[jonmaestas]

I have a very similar use case. I needed to return a list of objects. Here was my solution to this problem; utilizing a base64 result.

The first thing I needed to do was return my result from the script in a format acceptable to the external data source.
i.e.

{"key": "value"}

For my solution, I returned the json object base64 encoded as the value of the external data source result. Giving me something like this.

{"base64": "eyJuYW1lIjogImpvbiJ9Cg=="}

And then in terraform we can use base64decode() and jsondecode() on the result with a local variable and use our json result without a problem.

Using your example, it would look something like this:

# bastion-ips.sh
# Query for the addresses
IPV4_ADDRESSES=$(curl -s "${URL_IPV4}" | jq '[.SomeKey[] | .Address]')
IPV6_ADDRESSES=$(curl -s "${URL_IPV6}" | jq '[.SomeKey[] | .Address]')

# Output the addresses for terraform
# get the raw output from jq and base64 encode it; 
base64_result=$(jq -r -n \
    --argjson ipv4 "${IPV4_ADDRESSES}" \
    --argjson ipv6 "${IPV6_ADDRESSES}" \
    '{"ipv4":$ipv4,"ipv6":$ipv6}' | base64 --wrap=0) # We need --wrap=0 here to prevent multi-line base64 output

# Return KV output with value as base64 result of our json.
jq -c -n --arg base64_result "$base64_result" '{"base64": $base64_result}'

Returns:

{"base64": "ewogICJpcHY0IjogIiIsCiAgImlwdjYiOiAiIgp9Cg=="}

And then in Terraform:

1. Use base64decode()
2. Use jsondecode()

data "external" "bastions" {
    program = ["bash", "${path.module}/external/bastion-ips.sh"]
}

# 1. base64decode() - decode the result; 
# 2. jsondecode() - decode the json returned from base64decode; 
# 3. try() - return default value if result isn't decodable
locals {
  bastions = try(jsondecode(base64decode(data.external.bastions.result.base64), {"ipv4":[],"ipv6":[]})
}

# Create our security group referencing the IP ranges
resource "aws_security_group" "ssh" {
  name        = "ssh"
  description = "Allow SSH ingress from bastions"
  vpc_id      = var.vpc_id

  ingress {
    description      = "SSH Access"
    from_port        = 22
    to_port          = 22
    protocol         = "-1"
    cidr_blocks      = local.bastions.ipv4
    ipv6_cidr_blocks = local.bastions.ipv6
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

}

Hope this helps.

Note: I am on Terraform v0.15.1