Modular deployments

[HendrikSchmidt]

This PR introduces several changes to make the pipeline modular and dependent on which parts of the code were changed:

Preparatory changes

• Merge test and audit-licenses into singular reusable check-and-test.yml workflow
  • Both were using a very similar node setup including caching
• Reduce code duplication by reusing scan.yml instead of recreating the steps in vulnerability-scan

Modularity

• Use dorny/paths-filter@v3 to create list of changed packages
  • Run check-and-test with matrix of changed packages to only run tests of changed packages
  • Run build-and-deploy with matrix of changed packages and set max-parallel to 1 to prevent git push conflicts
  • Use package specific concurrency group in deploy step to only deploy a package once concurrently
• Update README with instructions on how to create new pipeline

---

Archive

Modularity

• Create pipeline-shared.yml
  • Runs on all changes expect changes to application code or application pipelines (❓Alternatively this could be changed to explicitly include all files that this pipeline runs on, but seems more cumbersome )
  • It reuses check-and-test.yml and scan.yml
  • It runs build-and-deploy.yml using a matrix strategy for every app
• On changes to app specific code, pipelines or the reusable workflows the respective pipeline-<app>.yml is run
  • Runs check-and-test.yml with a --workspace=packages/<app> flag for the applicable npm actions
  • Reuses global scan.yml
  • Runs build-and-deploy.yml for the respective app only
• Concurrency is set globally for the deployments
  • While runs in progress aren't canceled, pending runs are, leading to the problem described below
• Update README with instructions on how to create new pipeline

⚠️ Concurrency problem

When pushing multiple commits containing changes to the shared code and multiple apps this could lead to runs being canceled wrongly and leading to apps not being deployed correctly:

1. All three pipelines are started simultaneously
2. The first pipeline starts deployment of app1
3. The second pipeline waits with deployment of app2 as deployment of app1 is still running
4. The shared pipeline starts deployment of app2 first, cancelling the first deployment of app2
5. The shared pipeline starts deployment of app1, cancelling its own deployment of app2

I think this is a minor edge case but something we should look out for.

[mpanne]

First of all: well done! I like the refactoring into reusable workflows and the code de-duplication.

About the edge case regarding concurrency: I'm not sure this really is an edge case. Imagine re-naming a shared component or adding a property to a shared components. It's a simple and not too unusual change which would already trigger the case and might lead to problems.

Another minor thing I would like to have improved is the code duplication between the pipeline-<package>.ymls and the need to duplicate this workflow for all new apps.

I have a solution in mind which I didn't try out yet but which might solve both of the issues above:

Instead of having multiple pipeline-* workflows triggered depending on on.push.paths, we only keep the shared pipeline. This pipeline implements a similar functionality as on.push.paths to check which paths have been touched and runs the jobs for all packages depending on the same paths patterns (e.g. using the paths-filter GHA). By doing that we solve the concurrency problem since there won't be multiple workflows trying to deploy the same application and there is no code duplication or explicit workflows for apps since all apps share the same pipeline.

[mpanne]

Why is the input the whole flag instead of just the package name (similar to the build-and-deploy.yml workflow? Are there current use cases where this is needed? Otherwise I would say YAGNI and KISS 🤪 and streamline aka. use only package name as input.

[HendrikSchmidt]

Finde die Varianten alle nicht besonders schön: https://github.com/orgs/community/discussions/25725

[mpanne]

good job!