Skip to content

Files

Latest commit

huydhnhuydhn
Run v1 benchmark and integrate with PyTorch OSS benchmark database (v…
Feb 17, 2025
4518683 · Feb 17, 2025

History

History
This branch is 148 commits ahead of, 364 commits behind vllm-project/vllm:main.

benchmarks

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
cutlass_benchmarks
cutlass_benchmarks
 
 
disagg_benchmarks
disagg_benchmarks
 
 
fused_kernels
fused_kernels
 
 
kernels
kernels
 
 
overheads
overheads
 
 
structured_schemas
structured_schemas
 
 
README.md
README.md
 
 
backend_request_func.py
backend_request_func.py
 
 
benchmark_guided.py
benchmark_guided.py
 
 
benchmark_latency.py
benchmark_latency.py
 
 
benchmark_long_document_qa_throughput.py
benchmark_long_document_qa_throughput.py
 
 
benchmark_prefix_caching.py
benchmark_prefix_caching.py
 
 
benchmark_prioritization.py
benchmark_prioritization.py
 
 
benchmark_serving.py
benchmark_serving.py
 
 
benchmark_serving_guided.py
benchmark_serving_guided.py
 
 
benchmark_throughput.py
benchmark_throughput.py
 
 
benchmark_utils.py
benchmark_utils.py
 
 
launch_tgi_server.sh
launch_tgi_server.sh
 
 
sonnet.txt
sonnet.txt
 
 

README.md

Benchmarking vLLM

Downloading the ShareGPT dataset

You can download the dataset by running:

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

Downloading the ShareGPT4V dataset

The json file refers to several image datasets (coco, llava, etc.). The benchmark scripts will ignore a datapoint if the referred image is missing.

wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json
mkdir coco -p
wget http://images.cocodataset.org/zips/train2017.zip -O coco/train2017.zip
unzip coco/train2017.zip -d coco/

Downloading the BurstGPT dataset

You can download the BurstGPT v1.1 dataset by running:

wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv