Skip to content

Files

Latest commit

divakar-amddivakar-amd
[Misc][MoE] add Deepseek-V3 moe tuning support (vllm-project#12558)
Jan 30, 2025
1c1bb0b · Jan 30, 2025

History

History
This branch is 520 commits behind vllm-project/vllm:main.

benchmarks

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
cutlass_benchmarks
cutlass_benchmarks
Dec 18, 2024
disagg_benchmarks
disagg_benchmarks
Dec 25, 2024
fused_kernels
fused_kernels
Dec 13, 2024
kernels
kernels
Jan 30, 2025
overheads
overheads
Oct 17, 2024
structured_schemas
structured_schemas
Dec 4, 2024
README.md
README.md
Nov 5, 2024
backend_request_func.py
backend_request_func.py
Jan 24, 2025
benchmark_guided.py
benchmark_guided.py
Dec 4, 2024
benchmark_latency.py
benchmark_latency.py
Jan 9, 2025
benchmark_long_document_qa_throughput.py
benchmark_long_document_qa_throughput.py
Jan 10, 2025
benchmark_prefix_caching.py
benchmark_prefix_caching.py
Jan 10, 2025
benchmark_prioritization.py
benchmark_prioritization.py
Oct 22, 2024
benchmark_serving.py
benchmark_serving.py
Jan 28, 2025
benchmark_serving_guided.py
benchmark_serving_guided.py
Dec 4, 2024
benchmark_throughput.py
benchmark_throughput.py
Dec 19, 2024
launch_tgi_server.sh
launch_tgi_server.sh
Nov 7, 2024
sonnet.txt
sonnet.txt
Mar 27, 2024

README.md

Benchmarking vLLM

Downloading the ShareGPT dataset

You can download the dataset by running:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

Downloading the ShareGPT4V dataset

The json file refers to several image datasets (coco, llava, etc.). The benchmark scripts will ignore a datapoint if the referred image is missing.

wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json
mkdir coco -p
wget http://images.cocodataset.org/zips/train2017.zip -O coco/train2017.zip
unzip coco/train2017.zip -d coco/