Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added DVC, teamcity-messages and Julia TC integration #1915

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

evetion
Copy link
Member

@evetion evetion commented Oct 21, 2024

Fixes #1879

This does three things

  • Adds teamcity-messages as a Python test dependency, enabling TC to keep track of pytest tests (number of tests, number of failures, linking to specific logs), making debugging much easier.
Screenshot 2024-10-21 at 22 35 54
  • Adds teamcity specific logging in Julia (much like teamcity-messages) to keep track of the integration test time and differences with a previous model. These parameters are tracked by TC (and can be failed on), and can be plotted over time.
Screenshot 2024-10-21 at 22 38 31
  • Finally, and biggest, this adds DVC to our project, with the minio repo tracked, and the integration test as stage. It tracks a new default toml for that model (models/integration.toml), and like TC in 2, the time and differences. Once someone changes either the hws model, or the core code, or the integration.toml, the results are outdated and dvc repro will rerun the test. We can save these experiments over time, and compare results with the VS Code plugin:
Screenshot 2024-10-21 at 22 09 27

This disables the threshold test in the integration test, and opts for monitoring results (could be done automatically) instead. I've also opted for keeping track of results and settings threshold outside of Julia.

This is a backwards compatible setup PR. Changes/additions for benchmark tuning can be made in another PR.

@evetion evetion requested a review from visr October 21, 2024 20:31
@visr
Copy link
Member

visr commented Oct 21, 2024

Wonderful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Save benchmark results over time in readable/plottable format
2 participants