Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ci): per provider integration tests #465

Merged
merged 4 commits into from
Jan 15, 2024
Merged

Conversation

geekbrother
Copy link
Contributor

@geekbrother geekbrother commented Jan 12, 2024

Description

The context of the issue (#464):

At the moment we are running providers integration tests on all changes. Sometimes some of the providers are unavailable (rate-limited etc.) that results in a blocking of the CI/CD process.
We should run the provider tests only when the provider implementation is changed and run the test only for that provider that was changed.

This PR introduces the following changes:

  • Making all provider tests optional by making them #[ignore]
  • Unifying providers test names to the {provider_name}_provider pattern
  • Adding the sub-providers.yml workflow that:
    • Determining what provider was changed
    • Run the specific provider test (or multiple providers that were changed)
    • This workflow can be called manually from actions by providing the list of providers to test
    • This workflow will be called as a dependency step from sub-validate.yml when merged and after testing it manually (to not block the CI until tested and possible fixes are merged) by the following feat(ci): run provider specific tests from sub-validate #466 PR.
  • Adding README file with the descriptions on how providers testing work

Resolves #464

How Has This Been Tested?

Tested only partly because of the GitHub CI nature.
The sub-providers.yml workflow was tested isolated and reduced in the forked repo by pushing different commits on it:

Due Diligence

  • Breaking change
  • Requires a documentation update
  • Requires a e2e/integration test update

@geekbrother geekbrother self-assigned this Jan 12, 2024
@geekbrother geekbrother changed the title feat: per provider integration tests run feat(ci): per provider integration tests Jan 12, 2024
@geekbrother geekbrother marked this pull request as ready for review January 12, 2024 15:41
@geekbrother geekbrother requested a review from xav as a code owner January 12, 2024 15:41
runs-on: ubuntu-latest
if: needs.providers-list.outputs.providers != '[]'
strategy:
fail-fast: false
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need to fail all providers in case of one for cases like: we are manually validating a list of providers and want to know if there is a global issue or just one provider fails.

fi
done

PROVIDERS_LIST="${PROVIDERS_LIST% }"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing whitespace at the end.

PROVIDERS_LIST="${PROVIDERS_LIST% }"
fi

JSON_FMT=$(printf '[%s]' "$(echo $PROVIDERS_LIST | awk '{for(i=1;i<=NF;i++) printf "\"%s\",", $i}' | sed 's/,$//')")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making JSON array from a space separated list.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered jq?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's not installed by default, so it's not worth to install it for one line.


JSON_FMT=$(printf '[%s]' "$(echo $PROVIDERS_LIST | awk '{for(i=1;i<=NF;i++) printf "\"%s\",", $i}' | sed 's/,$//')")
echo "providers=$JSON_FMT" >> $GITHUB_OUTPUT
- name: Print list of changed providers
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I leave this step for debugging purposes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat(ci): run providers tests only when the providers code is changed
3 participants