Pipelines v3 Release Candidate #83

ZachGoldberg · 2024-08-29T16:10:14Z

🎈 Gruntwork Pipelines v3.0.0 Release Notes

Pipelines version 3 introduces several new features which include a number of breaking changes. The migration guide below should take less than 5 minutes for most teams to complete, but please do make sure to follow it closely. We're providing a checklist here to help teams ensure that steps are not skipped.

Add actions:read permission to PIPELINES_READ_TOKEN
Update to Terragrunt v0.67.0 or later
(If necessary) Add new actions to GitHub Organization allow list settings
(Enterprise Only) Install and activate Drift Detection

Migration Guide

Update Pipeline Permissions to add `actions: read` permissions

Customers explicitly list permissions that Gruntwork Pipelines workflows has by default in the pipelines workflow files in .github/workflows inside their infrastructure repositories. Pipelines v3 now requires actions: read permissions in order to introspect its own runs and provide more helpful logging capabilities.

Customers should make the one-line change to add the actions: read permission in the following files in every repository that uses Gruntwork pipelines (including root, access control and delegated repos). Note, most repositories will have only 1 or two of these workflow files, in which case update what is present and don't worry about the others.

.github/workflows/pipelines.yml
.github/workflows/pipelines-drift-detection.yml
.github/workflows/pipelines-root.yml
.github/workflows/pipelines-unlock.yml

Old Permissions

permissions:
  id-token: write
  contents: write
  pull-requests: write

New Permissions

permissions:
  id-token: write
  contents: write
  pull-requests: write
  actions: read

Update to Terragrunt v0.67.0

The latest pipelines works best with the latest version of Terragrunt. Recent versions of Terragrunt have much improved logging, performance and correctness improvements in run-all scenarios with pipelines.

Allowlist Actions

This is only for customers who only allow GitHub actions to run if they are on an explicit allowlist

New actions to add

gruntwork-io/pipelines-actions/.github/actions/pipelines-drift-detection-consolidate-jobs
gruntwork-io/pipelines-actions/.github/actions/pipelines-drift-detection-determine-units
gruntwork-io/pipelines-actions/.github/actions/pipelines-drift-detection-determine-drift
gruntwork-io/pipelines-actions/.github/actions/pipelines-new-pr-action
gruntwork-io/pipelines-actions/.github/actions/pipelines-get-job-logs-url

(Optional) Replace Machine Users with a GitHub App

Install the app
Leave the tokens

New Features

Pipelines Drift Detection (Enterprise Only)

Pipelines as a GitHub App

(Maybe? We might need to hold off for e.g. v3.1 for this)

Improved Account Factory Bootstrap Customization Hooks

Pipelines Log Link Improvements

…epos (#69) * chore: start work on dev-378 * chore: change the definition of post-provision new account to be more narrowly tailored to creating the new account, not the PR * chore: move around some values * chore: wire up data flows * chor: update for api changes * chore: more wiring cleanups

* Update pipelines-root.yml * Update pipelines-root.yml * debugging * Update pipelines-root.yml * JOB_NAME * add job_id * fix missing bracket * fix * Run logs url action * Use 2024-07-19_get-job-id actions * Add actions: read permission * permissions * pass var into next step * permissions * fix input * Dynamic step name * typo * test syntax * test syntax * colons are broken * try other quoting * Add get logs url to other jobs --------- Co-authored-by: Zach Goldberg <[email protected]>

* WIP add drift detection * Fix broken if statement * Pass root as working directory to bootstrap * Fix exclude root dir from run-all plan * Fix git change detection * Working dir for later steps * Add parallelism limit 6 * Use 0.26.0-rc4. Remove paralllelism limit * Fix role used for run-all plan * feat: Adding cross runner cache persistence * fix whitespace * Fix missing GH_TOKEN * Use -n for git status check * Pipelines CLI v0.26.1 * Fix missing author for create pr * Pipelines CLI v0.26.2-rc1 * Drift detection 2.0 * Fix command * GH_TOKEN * MACHINE_USER_NAME * Use org repo admin for pr creation * Add org admin token * Allow erroring modules. Always cache providers * Cache auth on disk. Only run first 10. * Pipelines CLI v0.28.0-rc2 * Pipelines CLI v0.28.0-rc3 * Switch to just plan * mkdir planfolder * fix plan folder * fix plan folder for parse plans * Pipelines CLI v0.28.0-rc4 * Pipelines CLI v0.28.0-rc5 * Matrixed drift detection jobs * Inline repo dir * Fix * Fix typo * debug mktemp missing * fix path var * dont override path * pathing * Update pipelines-drift-detection.yml * Update pipelines-root.yml * fix json escape string * fix array slicing * fix json escaping * property quotes * Add first pass pr body content * Fix missing pipelines actions * avoid backtick expansion * avoid backtick expansion * echo pr body * move arg * bash escape newline * try other escape * Use INFRA_ROOT_WRITE_TOKEN * Use INFRA_ROOT_WRITE_TOKEN * Add branch link, switch back to drift-detection branch * Actions @ main * actions @ main * Add error detection * tee to file * EOF * multiline github output * more newlines * fix job url, path * debugging * debugging * debugging * debugging * debugging * debugging * add log url link to top of pr body * remove debug limit of 10 units * Add step summary * Fix json output * try without cred caching * try without color removal * restore color removal, remove echos * Increase line height of drifted items * Remove debug limit * Reset changes to other workflows * Use matching pipelines versions * Use action for determine * Pipelines CLI v0.28.0 * Extract drift detection * Pass args * typo * Pass JOB_NAME and STEP_NAME through * Use merged determine units * Var rename. Remove terraform syntax highlight * Test sort units fix * consolidate jobs action. rename secret * Add branch_name input * Fix actions version * Fix actions version * Use merged action * Update pipelines-drift-detection.yml --------- Co-authored-by: Yousif Akbar <[email protected]> Co-authored-by: Oreoluwa Agunbiade <[email protected]>

* Use actions@2024-08-27_gruntcon_githubapp * Use [email protected] * Use pipelines-credentials for downloading actions * uses format * Update pipelines-root.yml * Fix correctly use outputs * typo * Try concatenate env * Try github var * Pass tokens to execute * Fix typo * debug cloning * chars * interp * less args * remove debugging * add new tokens to preflight * rename token * chore: baseline needs both tokens * chore: use the new tokens everywhere * chore: fix token thing * Fetch infra root write and org admin, pass to preflight * Switch out admin tokens in root * Disable provisioning temporarily * Fix read token ref * Update unlock to use github app tokens * Bump pipelines CLI to v0.29.0-rc2 * Use top level env GH_TOKEN * Use github app in delegated workflow * Bump pipelines CLI to v0.29.0-rc3 * Try rc3 * Try v0.28.2 * Trace log * v0.28.3-rc2 * v0.28.3-rc3 * v0.28.3-rc5 * v0.29.0-rc5 * fix: Set `api_base_url` * fix: Use `gruntwork-io` for `pipelines-credentials` * feat: Adding dynamicity to API URL * Use customer org token * DEV-519 Integerate drift detection with GitHub App (#86) * Integrate app * Add org read token * Add create pr token * Make secrets not required * Dynamic api_base_url * Use moved action --------- Co-authored-by: Zach Goldberg <[email protected]> Co-authored-by: Yousif Akbar <[email protected]>

yhakbar · 2024-09-27T15:40:02Z

.github/workflows/pipelines-drift-detection.yml

+    steps:
+      - name: Fetch Gruntwork Read Token
+        id: pipelines-gruntwork-read-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

yhakbar · 2024-09-27T15:40:10Z

.github/workflows/pipelines-drift-detection.yml

+
+      - name: Fetch Org Read Token
+        id: pipelines-customer-org-read-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

yhakbar · 2024-09-27T15:40:20Z

.github/workflows/pipelines-drift-detection.yml

+    steps:
+      - name: Fetch Gruntwork Read Token
+        id: pipelines-gruntwork-read-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

yhakbar · 2024-09-27T15:40:27Z

.github/workflows/pipelines-drift-detection.yml

+
+      - name: Fetch Org Read Token
+        id: pipelines-customer-org-read-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

yhakbar · 2024-09-27T15:40:34Z

.github/workflows/pipelines-drift-detection.yml

+    steps:
+      - name: Fetch Gruntwork Read Token
+        id: pipelines-gruntwork-read-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

yhakbar · 2024-09-27T15:41:36Z

.github/workflows/pipelines-root.yml

+
+      - name: Fetch Infra Root Write Token
+        id: pipelines-infra-root-write-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

yhakbar · 2024-09-27T15:41:43Z

.github/workflows/pipelines-root.yml

@@ -238,32 +342,59 @@ jobs:
      matrix:
        jobs: ${{ fromJson(needs.pipelines_orchestrate.outputs.pipelines_jobs)[0].NewAccounts }}
    steps:
+      - name: Fetch Gruntwork Read Token
+        id: pipelines-gruntwork-read-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

yhakbar · 2024-09-27T15:41:51Z

.github/workflows/pipelines-root.yml

+
+      - name: Fetch Org Read Token
+        id: pipelines-customer-org-read-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

yhakbar · 2024-09-27T15:44:02Z

.github/workflows/pipelines-drift-detection.yml

+          JOB_NAME: ${{ env.JOB_NAME }}
+          STEP_NAME: "Execute Drift Detection"
+
+  consolidate_jobs:


Suggested change

consolidate_jobs:

consolidate_jobs:

name: Consolidate Jobs

yhakbar · 2024-09-27T15:49:22Z

.github/workflows/pipelines-unlock.yml

+
+      - name: Fetch Org Read Token
+        id: pipelines-customer-org-read-token
+        uses: gruntwork-io/pipelines-credentials@main


Suggested change

uses: gruntwork-io/pipelines-credentials@main

uses: gruntwork-io/pipelines-credentials@v1

ZachGoldberg and others added 3 commits August 29, 2024 08:28

ZachGoldberg changed the title ~~Pipelines v3 Release Candidate Testing~~ Pipelines v3 Release Candidate Aug 29, 2024

ZachGoldberg requested review from Resonance1584, oredavids and yhakbar August 29, 2024 16:39

ZachGoldberg added the breaking-change Changes that require a major version increment label Aug 29, 2024

Resonance1584 and others added 9 commits September 17, 2024 15:04

Use github app token when creating comments

5499fd1

Fix checkout repo not using github app token

27fcfb7

Fix token not passed to preflight

450e01f

Fix tokens not used

09f6e30

Fix tokens not used

fb8d0f6

Fix tokens not used

d2d00ef

Fix tokens not used

828ab0d

Fix tokens not used

5fb87ca

yhakbar requested changes Sep 27, 2024

View reviewed changes

Resonance1584 and others added 8 commits September 27, 2024 14:32

pipelines-credentials@v1

6e554d6

Merge branch 'main' into v3-rc1

ab8a6f6

Pipelines v0.29.0-rc15

ef6bf9d

Add path filter

4a47179

Add path to PR title

852fad6

Dont use PATH as an env var

95f055d

Pipelines v0.29.0-rc16

854afe6

Pipelines v0.29.0-rc17

41c0d02

Resonance1584 force-pushed the v3-rc1 branch from 84bcc09 to 41c0d02 Compare October 4, 2024 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipelines v3 Release Candidate #83

Pipelines v3 Release Candidate #83

ZachGoldberg commented Aug 29, 2024 •

edited by Resonance1584

Loading

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

yhakbar Sep 27, 2024

	uses: gruntwork-io/pipelines-credentials@main
	uses: gruntwork-io/pipelines-credentials@v1

Pipelines v3 Release Candidate #83

Are you sure you want to change the base?

Pipelines v3 Release Candidate #83

Conversation

ZachGoldberg commented Aug 29, 2024 • edited by Resonance1584 Loading

🎈 Gruntwork Pipelines v3.0.0 Release Notes

Migration Guide

Update Pipeline Permissions to add actions: read permissions

Old Permissions

New Permissions

Update to Terragrunt v0.67.0

Allowlist Actions

New actions to add

(Optional) Replace Machine Users with a GitHub App

New Features

Pipelines Drift Detection (Enterprise Only)

Pipelines as a GitHub App

Improved Account Factory Bootstrap Customization Hooks

Pipelines Log Link Improvements

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZachGoldberg commented Aug 29, 2024 •

edited by Resonance1584

Loading

Update Pipeline Permissions to add `actions: read` permissions