Add a README.md for benchmarking infra #5403
Conversation
guangy10
requested review from
shoumikhin,
huydhn,
mergennachin and
kirklandsign
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5403
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 6d293c3 with merge base 7c661d7 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
guangy10
force-pushed
the
doc_benchinfra
branch
3 times, most recently
from
b35c3e8
to
fde8fd7
Compare
|
@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
guangy10
force-pushed
the
doc_benchinfra
branch
from
fde8fd7
to
255acbc
Compare
|
@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
|
||
| | Model | Backend | Model Load Time (ms) | Avg Inference Latency (ms) | Device | | ||
| |-----------------------|--------------|----------------------|----------------------------|------------------------| | ||
| | DeepLabV3 (dl3) | XNN Q8 | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | Samsung S22 | |
There was a problem hiding this comment.
Just FYI, the GitHub link here expires after 90 days, so let's work on getting the dashboard ready before that date :)
|
|
||
| ## Scheduling On-Demand Benchmarking | ||
|
|
||
| Users can schedule a benchmarking workflow on a pull request through GitHub Actions using the workflow dispatch UI. Follow the steps below to trigger benchmarking: |
There was a problem hiding this comment.
Not sure if we want to be clear here, only people with write permission to the repo can run the workflow.
There was a problem hiding this comment.
Will add it to the readme. Having write permission will limit the usage of it, especially for backend vendors. Except for explicitly grant write permissions to individuals, is there other more convenient way for vendors to start dogfooding?
There was a problem hiding this comment.
There are similar questions/requests to GH from what I see https://github.com/orgs/community/discussions/49688. We could consider building something like @pytorchbot here, for example, posting a comment like @executorchbot benchmark to run perf benchmark. There are pros and cons, but that's a familiar approach that PyTorch is using
guangy10
force-pushed
the
doc_benchinfra
branch
from
255acbc
to
dceb33c
Compare
guangy10
force-pushed
the
doc_benchinfra
branch
from
dceb33c
to
6d293c3
Compare
|
@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@pytorchbot cherry-pick --onto release/0.4 -c docs |
Cherry picking #5403The cherry pick PR is at #5413 The following tracker issues are updated: Details for Dev Infra teamRaised by workflow job |
Add a README.md for benchmarking infra (#5403) Summary: Pull Request resolved: #5403 Reviewed By: mergennachin, huydhn Differential Revision: D62790095 Pulled By: guangy10 fbshipit-source-id: fb0ec5e7bfcc49db6de86ca3a52729026827d567 (cherry picked from commit c5d1661) Co-authored-by: Guang Yang <[email protected]>
No description provided.