Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quality: Parameterize SDK and test-data branches in CI workflow #73

Merged
merged 3 commits into from
Dec 3, 2024

Conversation

typotter
Copy link
Collaborator

@typotter typotter commented Nov 22, 2024

🎟️ Fixes FF-3141
🎟️ Fixes FF-3095
towards FF-3085

πŸ‘―β€β™‚οΈ Related PRs

Motivation

Changes are often made to the sdk-test-data repository to capture new behaviours, bugs and edge cases. When these changes are pushed to main, the SDKs are cloned locally (locally to the github action running) and their respective tests are run. These tests are set up and run by copies of the SDK test workflows - see sdk-test-data workflow. There are a number of limitations to this setup:

  • Test steps are copied from the SDK’s respective workflows; changes to the SDK test workflows need to be replicated in sdk-test-data and this is not obvious to devs
  • The SDK tests are not able to run against in-flight changes to sdk-test-data
  • When new test-data is committed that breaks an SDK, that breakage is not surfaced in the SDK’s repository until some action triggers its testing workflow (on demand, on pull-request, etc.).

Description of Changes

This change
πŸš€ - the CI workflow is enhanced into a reusable workflow, exposing parameters for SDK branch and the sdk-test-data branch to use in testing.

  • The test workflow runs using the main branch of sdk-test-data on all main pushes and Pull Requests using the PR's branch

External to this Change
sdk-test-data get two testing workflows.

  1. ♻️ "Local Testing"- For all pull request changes, the "Local Testing" workflow calls the reusable SDK workflows, test results are recorded only in the sdk-test-data repository. This is run using the main SDK branch and the "current" branch of workflow, i.e. the pull request branch. (SDK repo does not see/is not notified of test failures during PR lifecycle, only on push to main)
  2. ♻️ πŸ§‘β€πŸ’» "Remote Testing" - On all pushes to the main branch of sdk-test-data, the SDK testing workflows are triggered to run within their respective repositories, alerting all subscribers of failed runs (repo is red/green).

Copy link
Member

@leoromanovsky leoromanovsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some test failures related to file paths. Can I help here?

@typotter
Copy link
Collaborator Author

There are some test failures related to file paths. Can I help here?

Thanks. I dropped cloning the test data for python and ruby since they didn't have universal tests but I missed that they're got a couple static tests using the data files.

Copy link
Member

@leoromanovsky leoromanovsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python will get fixed in #77

@typotter typotter enabled auto-merge (squash) December 2, 2024 21:38
@rasendubi rasendubi dismissed their stale review December 3, 2024 13:09

comments addressed

@typotter typotter merged commit 691f7ac into main Dec 3, 2024
26 checks passed
@typotter typotter deleted the tp/testDataBranch branch December 3, 2024 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants