Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect CI environment and send associated data along with the test suite #21

Merged
merged 11 commits into from
Jul 12, 2024

Conversation

thebhumilsoni
Copy link
Contributor

@thebhumilsoni thebhumilsoni commented Jun 7, 2024

This PR adds support for passing environment variables from various CI systems to enrich test reports. It also introduces significant improvements to the collector architecture, enhancing flexibility, testability, and separation of concerns.

💬 Summary of Changes

  • Simplified environment variable handling for InstrumentedTestCollector. New setup uses Gradle configuration to pass environment variables directly through instrumentation arguments instead of using BuildConfig. This simplifies the instrumentation test collector setup as now environment variables are passed to InstrumentedTestCollector without a need to create a custom test collector implementation.
  • Introduced TestDataUploader and TestObserver interfaces to allow for custom implementations.
  • Refactored the code to extract TestListener implementation into a new class UnitTestListener, improving the separation of concerns and making the UnitTestCollectorPlugin cleaner.
  • Similarly, extracted RunListener implementation into a new class InstrumentedTestListener from InstrumentedTestCollector.
  • Resolves Detect CI environment and send associated data along with the test suite details #9 by adding support for passing CI environment values along with test results to enrich test analytics. The framework now supports Buildkite, CircleCI, and GitHub Actions. If no CI system is detected, it defaults to a local environment setup.
  • Introduced TestEnvironmentProvider and BaseTestEnvironmentProvider to handle environment variables effectively from various sources such as system properties or Android Bundle arguments.
  • Implemented a validation check for the presence of the tests suite API token before creating instances of UnitTestListener and InstrumentedTestListener. This ensures that no test data is collected or uploaded without a valid token, preventing unnecessary operations.
  • Updated the logger to be defined by the caller and passed to TestDataUploader.
  • Updated the README and created a new CI_CONFIGURATION documentation file with detailed setup instructions and examples for configuring environment variables for supported CI platforms.
  • Aligned the example project and workflows with the new configuration requirements.

These changes significantly improve the framework's architecture and maintainability.

🧾 Checklist

  • 🧐 Performed a self-review of the changes
  • ✅ Added tests to cover changes
  • 🏎 Ran Unit Tests and they all passed
  • 🏷 Labeled this PR appropriately

📝 Items of Note

- Updated setup to use Gradle configuration to pass environment variables directly to instrumentation arguments instead of using BuildConfig.
- Simplified the instrumentation test collector as InstrumentedTestCollector is no longer abstract and now handles environment variables internally, thus removing the need to create a custom test collector class.
Added TestDataUploader and TestObserver interfaces to allow custom implementations and also improve flexibility and testability.
Extracted TestListener implementation to a new class UnitTestListener from UnitTestCollectorPlugin for better separation of concerns.
Extracted RunListener implementation to a new class InstrumentedTestListener from InstrumentedTestCollector for better separation of concerns and testability.
@thebhumilsoni thebhumilsoni requested a review from a team as a code owner June 7, 2024 03:59
The commit adds support for uploading environment variables from various CI system - Buildkite, CircleCI and GitHubActions via accessing variables each system provides. If no CI system is detected, it sends default environment data.
Moved validation to check for the presence of the 'BUILDKITE_ANALYTICS_TOKEN' environment variable before creating instances of `UnitTestListener` and `InstrumentedTestListener`.

This change ensures that no test data is collected or uploaded without a valid API token unnecessarily.
@thebhumilsoni thebhumilsoni force-pushed the feature/send-ci-info-to-test-analytics branch from 2259a45 to 40f9300 Compare June 7, 2024 04:07
@thebhumilsoni thebhumilsoni force-pushed the feature/send-ci-info-to-test-analytics branch from 40f9300 to 5531f25 Compare June 7, 2024 04:31
@thebhumilsoni thebhumilsoni force-pushed the feature/send-ci-info-to-test-analytics branch from 5531f25 to ec0838d Compare June 7, 2024 04:34
@thebhumilsoni thebhumilsoni force-pushed the feature/send-ci-info-to-test-analytics branch from 57170be to ea62982 Compare June 7, 2024 06:05
@thebhumilsoni thebhumilsoni force-pushed the feature/send-ci-info-to-test-analytics branch from ea62982 to b581775 Compare June 7, 2024 06:11
@thebhumilsoni thebhumilsoni changed the title wip - Detect CI environment and send associated data along with the test suite Detect CI environment and send associated data along with the test suite Jun 7, 2024
Copy link

@niceking niceking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know kotlin, and I have no thoughts on the refactoring, but the added CI environment support for GitHub looks good and I checked the ENV vars against the values in the GitHub docs 👍

Seeing as this PR is quite a significant refactor, is it possible to integration test these changes before they are released? Again, I don't know how any of this works, but is it possible to build a collector with these changes and verify that everything works as intended and the test results are received by Buildkite on a local test run with an API key from a nominated production Suite?

@thebhumilsoni
Copy link
Contributor Author

thebhumilsoni commented Jun 24, 2024

I don't know kotlin, and I have no thoughts on the refactoring, but the added CI environment support for GitHub looks good and I checked the ENV vars against the values in the GitHub docs 👍

Seeing as this PR is quite a significant refactor, is it possible to integration test these changes before they are released? Again, I don't know how any of this works, but is it possible to build a collector with these changes and verify that everything works as intended and the test results are received by Buildkite on a local test run with an API key from a nominated production Suite?

Hi @niceking,

Thank you for your patience regarding the delayed response.

We have an example project that uses the test collectors which should be used to test any changes locally before pushing up a PR. This project provides a working representation of the SDK. Detailed steps on how to test changes locally are noted in the Contribution guide.

Additionally, we have a CI workflow that tests changes via CI runs. This workflow sends the example project's instrumented and unit test data to the analytics suite using the token provided in secrets. @KatieWright26 might be aware of the associated dev account with the token. It's important to note that this workflow is/should be disabled (and should only be enabled and when there is a need to test the changes) as it is set up to fail due to intentionally failing tests in the example project.

I have added some comprehensive unit tests to verify environment variables access for different CI systems. Although, there are some additional tests that can now be written due to this refactoring (as it sets up more testable architecture) that I couldn't cover due to the limited time available before I transitioned to another engagement. I will open an issue to ensure these tests are added and tracked for future completion.

Once this PR is merged, it will publish the changes to the SNAPSHOT version on Maven. This means that the changes will be available for testing in other projects that use the SDK, not just the example project before we actually release the changes. Following that and the setup instructions in the README, we can easily test the new features without needing to go through the Contribution guide.

I hope this answers your questions, but please feel free to ask if you need additional information.

@thebhumilsoni thebhumilsoni requested a review from niceking June 30, 2024 23:48
@niceking
Copy link

niceking commented Jul 2, 2024

Thanks for your due diligence around this @thebhumilsoni :)

@thebhumilsoni thebhumilsoni merged commit d878ce0 into main Jul 12, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Detect CI environment and send associated data along with the test suite details
2 participants