feat: Add data sources

[alexluong]

Implements #84

Overview

This PR is to add support for Terraform Data Sources for Hookdeck's resources

• Source
• Destination
• Connection
• Transformation
• Source verification (optional - not sure if this is necessary given it's not a dedicated Hookdeck resource)

Implementation

Within our provider, we need to implement Terraform's data source logic for each of the above resources. Currently, only source is implemented here.

The logic for each data source is relatively straightforward, so we should be able to finish a naive implementation within an hour or two.

Some considerations:

1. Improved identification DX

The most straightforward way to use data source is via id. For example, this would be a valid way to utilize data source:

+ data "hookdeck_source" "source" {
+   id = "src_abcdef"
+ }

resource "hookdeck_destination" "destination" {
  name = "destination"
}

resource "hookdeck_connection" "connection" {
  name           = "connection"
  source_id      = data.hookdeck_source.source.id
  destination_id = hookdeck_destination.destination.id
}

With this, the source src_abcdef is not managed by Terraform but we can use this source to create connections and other resources that are managed by Terraform.

Since Hookdeck source is identifiable via name as well, we can potentially replace id with name for better DX:

data "hookdeck_source" "source" {
-  id = "src_abcdef"
+  name = "human_readable_name"
}

With that said, this may worth a bigger discussion because this can improve DX but it may not be viable because other resources don't have the same identification patterns:

• Connections & destinations can have the same name
• There can be duplicate connections pointing from the same source & destination

So all in all, it seems Hookdeck source is the only one that can be identified with something else but id. Other resources must always be identified by id.

2. Copied-pasted schema

Currently, every resource already has a dedicated schema.go file that works with Terraform Resources. When implementing Data Sources, we need to also provide a schema that works for Terraform Data Sources. These schemas are mostly the same shape with typings from different Terraform's packages. We have 2 ways to go about this:

1. Copy and paste resource schemas & manually update them to work with data source typings
2. Write a generic schema that can be parsed into Resource schema & Data Source schema

With option 1, it's much simpler to implement now but we will have to maintain 2 different data structure that looks mostly the same.

With option 2, we need to figure out a good way to implement this logic. It doesn't seem very straightforward at the moment, especially since Go typing is quite strict, so it can be a bit of a challenge.

Personally, I lean towards option 1 to ship quickly and we can consider the refactor later if the maintenance has proved to be a problem.

[leggetter]

@alexluong - my take:

1. Go with id for consistency
2. Go with your recommendation, option 1.

[alexluong]

@leggetter this is ready for review, can you help take a look?

Here's what I've found to be the most straightforward way for me to test that the data sources are working. First, you set up a source, destination, and connection. From then, using this snippet:

data "hookdeck_source" "source" {
  id = "src_6b08g7otq5hq6f"
}

data "hookdeck_destination" "destination" {
  id = "des_ri89vlqn4YL7"
}

data "hookdeck_connection" "connection" {
  id = "web_BNzui97F7cH7"
}

resource "hookdeck_connection" "connection_2" {
  name           = "connection_2"
  source_id      = data.hookdeck_source.source.id
  destination_id = data.hookdeck_destination.destination.id
}

resource "hookdeck_connection" "connection_3" {
  name           = "connection_3"
  source_id      = data.hookdeck_connection.connection.source_id
  destination_id = data.hookdeck_connection.connection.destination_id
}

What you're doing here is importing your source, destination, and connection, and using these data, you're setting up 2 new connections which are similar to your existing connection. This should show that all the data sources are working correctly.

From then, feel free to tweak things around, test out other properties, rules, etc., whichever you think people may use this for, and go from there.

[alexluong]

Regarding the schema duplication issue, I tried to just duplicate things at first but turns out there is a lot of schema to be duplicated. Particularly, all the destination authentication methods mechanism, each requiring its own schema that must be duplicated.

After some further exploration, I was able to write a conversion function where it takes the resource schema and converts to data source schema. I found a snippet online from another provider (MIT license of course) and expanded upon it to make it works for our use case.

From there, everything is pretty straightforward. I refactored each service's resource.go, datasource.go, and schema.go files to make things cleaner and follow a particular template. Hope everything makes sense and please let me know if you have any input here!