Include Terraform Plugins in build

[tamirkamara]

Hi,

Do you have some guidance on how one can include terraform plugins in the bundle (during build time)?

I found this doc from Terraform and it might work if include in the bundle's docker file.
However, I see an issue since terraform providers mirror requires an init beforehand to discover local modules. Should I also run terraform init in the bundle's docker file, or do you run it automatically?

Any guidance would be appreciated.

[tamirkamara]

I think I can run the following in the bundle dockerfile, but is that the right thing todo given the mixin also runs terraform init?
RUN cd $BUNDLE_DIR/terraform && terraform init -backend=false && terraform providers mirror /usr/local/share/terraform/plugins

[carolynvs]

Disclaimer: I don't use terraform 😀

From Terraform's docs for terraform init

When terraform init is run, Terraform reads configuration files in the working directory to determine which plugins are necessary, searches for installed plugins in several locations, sometimes downloads additional plugins, decides which plugin versions to use, and writes a lock file to ensure Terraform will use the same plugin versions in this directory until terraform init runs again.

Based on feedback from other people using terraform in bundles, I have always thought that was sufficient and that you don't need to do extra commands to ensure that the bundle has all the necessary files locally in order to execute your terraform plans.

When you run your bundle using just the default behavior of the terraform mixin (which calls terraform init in the configured directory and optionally using a custom providers file), what is the error that you are seeing that prompted you to look into that other terraform providers mirror command?

[tamirkamara]

@carolynvs so my theory didn't work :-( still seeing the error.

Error: could not init terraform, exit status 1
Error: mixin execution failed: exit status 1
Error: 2 errors occurred:
* container exit code: 1, message: <nil>. fetching outputs failed: error copying outputs from container: Error: No such container:path: d776b1d1ee4a935d0c77922d1fd4fdbfecdd7dcc996b97795fe1a2ade260a1f2:/cnab/app/outputs
* required output app_role_id_workspace_airlock_manager is missing and has no default

Is there another init while executing install?

[tamirkamara]

So it does look like it runs init again. Possible this is it?

Initializing Terraform...
/usr/bin/terraform terraform init -backend=true -backend-config=container_name=tfstate -backend-config=key=42731aab-f352-40e0-a726-4fc02ef7b59f -backend-config=resource_group_name=rg-foo -backend-config=storage_account_name=foo -reconfigure
Initializing modules...

Initializing the backend...

[carolynvs]

Thanks for sticking with me, and reporting this problem! Again I don't use terraform so I'm trying to learn as I go! 😅

Yup, that's why I asked about the remote backend. Here's the issue where we have been collecting info on how to better handle that remote backends require that terraform init is called when the bundle runs.

getporter/terraform-mixin#80

I wasn't aware of the mirror functionality but after reading a few docs on it and the old bundle functionality, I understand now that is Hashicorp's official solution to this problem.

Long term, I think(?) that the terraform mixin should call mirror by default so that we don't re-download everything when it runs. Short term, yes that command that you are running seems like the right workaround. I believe the custom dockerfile should give you everything you need to do that, and you can move around the special comments like # PORTER_MIXINS if you need to change the order of when certain commands are run. If not, let me know and I promise I am more helpful about how to make porter do stuff than how terraform works! 🤦‍♀️

If you can help me figure out which command that the mixin should be calling by default when we build the bundle, or if we need additional configuration, let me know and we can get the terraform mixin updated to do the right thing!

[tamirkamara]

Thanks @carolynvs!
I did a couple of tests and looks like the mirror one works just fine. We have a large solution which is deployed multiple times each day (part of cicd), so we might be able to give this a try for a while and see how it goes.

As a side note, I will offer a PR on the mixin readme to note that the initFile property should be used carefully when having submodules.

[carolynvs]

I created getporter/terraform-mixin#90 to track adding the mirror command to the default commands run by the mixin during porter build.