feat: support sops decryption

[js-timbirkett]

This PR is an initial attempt at #1179 - it enables users of sops to decrypt YAML or JSON files for use as inputs.

The issue of those initial secrets you need for bootstrapping and getting your infrastructure running is a contentious one. I've had engineers remark: "use vault", "use SSM", "use kubectl create secret..." and so on. But there is, a time when you don't have Vault, you don't want to manually manage the secret in SSM or Kubernetes. All you have is you, a secret or 2, Terraform, and an AWS account. sops_decrypt_file attempts to help solve that problem.

I took a fairly simplistic approach and return the contents of the entire file for the user to make use of jsondecode or yamledecode rather than creating a function to return a single secret as once the file contents are decoded, you can access individual items specifically if you need to.

I'm not precious over any naming, I haven't written any unit tests (yet) but I do have an integration test which works (after importing the dev / test key) and the docs are updated.

Any feedback would be greatly appreciated.

Thanks! ❤️

[yorinasub17]

Thanks for your contribution! This is really neat and thanks for being comprehensive with docs and tests! Will kick off the build and if it passes, will merge.

[yorinasub17]

Looks like the build failed with

 integration_test.go:2908: Copying fixture-sops to /tmp/terragrunt-test716930863
    integration_test.go:2898: Failed to run Terragrunt command 'terragrunt apply --terragrunt-non-interactive --terragrunt-working-dir /tmp/terragrunt-test716930863/fixture-sops' due to error: /tmp/terragrunt-test716930863/fixture-sops/terragrunt.hcl:2,21-39: Error in function call; Call to function "sops_decrypt_file" failed: Error getting data key: 0 successful groups required, got 0.
        
        Stdout: (see log output above)
        
        Stderr: (see log output above)

This looks like an error in sops?

[js-timbirkett]

Thanks @yorinasub17
How interesting... I'm wondering if the key import failed 🤔

Are you able to post the whole build log to a gist and link to it in here? (redact anything "sensitive").

[yorinasub17]

Since this is an open source repo, I think you should be able to see the build logs when you loginto circleci with your github account and click through to https://circleci.com/gh/gruntwork-io/terragrunt/4704

If not, here's the logs for the gpg key import step:

gpg: directory '/home/circleci/.gnupg' created
gpg: keybox '/home/circleci/.gnupg/pubring.kbx' created
gpg: /home/circleci/.gnupg/trustdb.gpg: trustdb created
gpg: key 52B83528AD5BB476: public key "Terragrunt DEV" imported
gpg: key 52B83528AD5BB476: secret key imported
gpg: key 52B83528AD5BB476: "Terragrunt DEV" not changed
gpg: key 52B83528AD5BB476: "Terragrunt DEV" 1 new signature
gpg: key 52B83528AD5BB476: "Terragrunt DEV" 1 new subkey
gpg: Total number processed: 3
gpg:               imported: 1
gpg:              unchanged: 1
gpg:            new subkeys: 1
gpg:         new signatures: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

And the original logs I pasted is actually the only logs for the test.

[js-timbirkett]

@yorinasub17 I have a strong suspicion I got bitten by the case explained here: getsops/sops#489 (comment) as I had generated a key / sub-key originally.

I was able to reproduce the issue locally and get sops to fail decryption, but I did have to kill my GPG agent first. Strange. It's the first time using a gpg key with sops so I wasn't aware of that, typically I'd make use of KMS.

I've updated the test GPG key and the encrypted files.

🤞

[yorinasub17]

Thanks for digging in! I kicked off a build after I saw your commit, so hopefully this one works!

[js-timbirkett]

That ✅is a beautiful sight 😄

[yorinasub17]

Build passed, so will go ahead and merge + release this. Thanks again for your contribution!

[sc250024]

Wow! Thank you @js-timbirkett , this is a huge time-saving feature!

[barryib]

Awesome. Falling in love with this feature.

Thank you so much @js-timbirkett ❤️❤️❤️

[brikis98]

Released in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.23.18. Thank you @js-timbirkett!