Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Migration] Update dashboard to process results from openxla-benchmark #39

Open
pzread opened this issue Jul 1, 2023 · 5 comments
Open
Assignees

Comments

@pzread
Copy link

pzread commented Jul 1, 2023

Benchmark results uploaded from openxla-benchmark are a little different from iree-samples.

Here is the example from T5_LARGE_FP32_JAX benchmarks:

From iree-samples: https://gist.github.com/pzread/2be3b2db7c0ffa14518085f08e33b815
From openxla-benchmarks: https://gist.github.com/pzread/cf3e089d0a4e0d5c452aebc1df6dbc44

The major differences are:

  • Changes in benchmark id and benchmark name
  • Changes in some metadata values (e.g. data_type and device field)
  • Changes in the format of python_environment field

I think the pain point will be changing the benchmark id, which means we might need a backfill of the historical data. We can still show new data in the dashboard first then backfill the previous data

@pzread
Copy link
Author

pzread commented Jul 1, 2023

@beckerhe temporarily assigned this to you. Let me know if you have time to work on this : )

@pzread
Copy link
Author

pzread commented Jul 1, 2023

So the format of the benchmark id needs to be finalized first (so we won't change it later...). This is blocked by #41

@pzread
Copy link
Author

pzread commented Jul 5, 2023

As an offline discussion, I'll create a new GCS bucket to store new benchmark results #20 (currently the new benchmark results are uploaded to postsubmit GCS bucket with random job id, which is not well-organized). Then we can start testing and doing data migration with it

@pzread
Copy link
Author

pzread commented Jul 13, 2023

All benchmarks are running and uploading results to GCS, so I think we can start importing them into the dashboard.

Here is an example of the benchmark results gs://comparative-benchmark-artifacts/2023-07-12.1689195682

@beckerhe
Copy link
Collaborator

Awesome. I will prepare the switch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants