Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Python script that runs LLM benchmark #406

Merged
merged 17 commits into from
Oct 5, 2023
Merged

Conversation

nv-hwoo
Copy link
Contributor

@nv-hwoo nv-hwoo commented Sep 28, 2023

Running benchmark 1: Prefill phase

python profile.py --prompt-size-range 100 1000 200 --max-tokens 1

# Sample output
# [ Benchmark Summary ]
#   Prompt size: 100, Average first-token latency: 0.0421 sec
#   Prompt size: 300, Average first-token latency: 0.0312 sec
#   Prompt size: 500, Average first-token latency: 0.0289 sec
#   Prompt size: 700, Average first-token latency: 0.0358 sec
#   Prompt size: 900, Average first-token latency: 0.0327 sec

Running benchmark 2: Generation phase

python profile.py --prompt-size-range 100 1000 200 --max-tokens 256 --ignore-eos

# Sample output
# [ Benchmark Summary ]
#   Prompt size: 100, Average first-token latency: 0.0419 sec, Average token-token latency: 0.0068 sec
#   Prompt size: 300, Average first-token latency: 0.0513 sec, Average token-token latency: 0.0070 sec
#   Prompt size: 500, Average first-token latency: 0.0325 sec, Average token-token latency: 0.0069 sec
#   Prompt size: 700, Average first-token latency: 0.0325 sec, Average token-token latency: 0.0071 sec
#   Prompt size: 900, Average first-token latency: 0.0368 sec, Average token-token latency: 0.0071 sec

src/c++/perf_analyzer/docs/examples/profile.py Outdated Show resolved Hide resolved
src/c++/perf_analyzer/docs/examples/profile.py Outdated Show resolved Hide resolved
src/c++/perf_analyzer/docs/examples/profile.py Outdated Show resolved Hide resolved
@nv-hwoo nv-hwoo requested a review from nv-braf October 3, 2023 21:15
src/c++/perf_analyzer/docs/llm.md Show resolved Hide resolved
src/c++/perf_analyzer/docs/examples/profile.py Outdated Show resolved Hide resolved
src/c++/perf_analyzer/docs/examples/profile.py Outdated Show resolved Hide resolved
src/c++/perf_analyzer/docs/examples/profile.py Outdated Show resolved Hide resolved
@nv-hwoo nv-hwoo changed the title [WIP] Add a Python script that runs LLM benchmark Add a Python script that runs LLM benchmark Oct 4, 2023
@nv-hwoo nv-hwoo marked this pull request as ready for review October 4, 2023 21:01
@nv-hwoo nv-hwoo merged commit eca94a8 into main Oct 5, 2023
3 checks passed
@nv-hwoo nv-hwoo deleted the hwoo-llm-py-guide branch October 5, 2023 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants