Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[V1] E2E (prompt )logprobs tests (PR 3/N for v1 sample and prompt logprobs support) #11974

Draft
wants to merge 346 commits into
base: main
Choose a base branch
from
Draft

[V1] E2E (prompt )logprobs tests (PR 3/N for v1 sample and prompt logprobs support) #11974

wants to merge 346 commits into from
+2,182 −132

Conversation

afeldman-nm
Copy link
Contributor

@afeldman-nm afeldman-nm commented Jan 12, 2025 •
edited by github-actions bot
Loading

Analogous to v0 test_completions, added end-to-end sample logprobs and prompt logprobs unit tests.

DarkLight1337 and others added 30 commits December 2, 2024 11:36
@DarkLight1337 @afeldman-nm
[Bugfix] Ignore lm_head when loading embedding models (vllm-project…
8239c6f 
…#10719)

Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@tomeras91 @afeldman-nm
[Frontend] don't block event loop in tokenization (preprocess) in Ope…
5a3a0eb 
…nAI compatible server (vllm-project#10635)

Signed-off-by: Tomer Asida <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@youkaichao @afeldman-nm
[misc] upgrade filelock version (vllm-project#10731)
b22e27c 
Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@zixuanzhang226 @afeldman-nm
[Model] support bitsandbytes quantization with minicpm3 model (vllm-p…
b5864e2 
…roject#10682)

Signed-off-by: Ubuntu <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@spacewander @afeldman-nm
[Doc] Update model in arch_overview.rst to match comment (vllm-projec…
b9cabc9 
…t#10701)

Signed-off-by: spacewander <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@rickyyx @afeldman-nm
[Bug][CLI] Allow users to disable prefix caching explicitly (vllm-pro…
d61d661 
…ject#10724)

Signed-off-by: rickyx <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@WoosukKwon @afeldman-nm
[V1] Do not allocate beyond the max_model_len (vllm-project#10730)
39f4494 
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@WoosukKwon @afeldman-nm
[Kernel] Update vllm-flash-attn version (vllm-project#10736)
dcdf2f3 
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@richardsliu @afeldman-nm
[TPU] Update requirements-tpu (vllm-project#10726)
ea6ed6b 
Signed-off-by: Richard Liu <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@sixsixcoder @Isotr0py
@DarkLight1337 @afeldman-nm
[Model] Added GLM-4 series hf format model support vllm==0.6.4 (vllm-…
ac0b495 
…project#10561)

Signed-off-by: Isotr0py <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@WoosukKwon @afeldman-nm
[Kernel] Update vllm-flash-attn version to reduce CPU overheads (vllm…
1362dac 
…-project#10742)

Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@WoosukKwon @afeldman-nm
[V1] Optimize the CPU overheads in FlashAttention custom op (vllm-pro…
bc6637c 
…ject#10733)

Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@Isotr0py @afeldman-nm
[Model] Add Internlm2 LoRA support (vllm-project#5064)
3733796 
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@DarkLight1337 @afeldman-nm
[Model] Clean up MiniCPMV (vllm-project#10751)
170a30c 
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@noooop @afeldman-nm
[Misc] typo find in sampling_metadata.py (vllm-project#10740)
8d83244 
Signed-off-by: Andrew Feldman <[email protected]>
@jeejeelee @afeldman-nm
[Bugfix] Fix Idefics3 bug (vllm-project#10778)
d8499c0 
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@wangxiyuan @afeldman-nm
[platform] Add verify_quantization in platform. (vllm-project#10757)
3c8ced2 
Signed-off-by: wangxiyuan <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@NickLucche @DarkLight1337 @afeldman-nm
[Bugfix] Fix OpenVino/Neuron driver_worker init (vllm-project#10779)
5146352 
Signed-off-by: NickLucche <[email protected]>
Signed-off-by: Cyrus Leung <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@Isotr0py @afeldman-nm
[Model] Refactor Molmo weights loading to use AutoWeightsLoader (vllm…
d95da87 
…-project#10771)

Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@patrickvonplaten @youkaichao @afeldman-nm
[Interleaved ATTN] Support for Mistral-8B (vllm-project#10591)
7831672 
Signed-off-by: youkaichao <[email protected]>
Co-authored-by: youkaichao <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@wangxiyuan @afeldman-nm
[doc] format fix (vllm-project#10789)
a877540 
Signed-off-by: wangxiyuan <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@DarkLight1337 @afeldman-nm
[Model] Replace embedding models with pooling adapter (vllm-project#1…
cbf1489 
…0769)

Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@DarkLight1337 @afeldman-nm
[Misc] Improve type annotations for support_torch_compile (vllm-pro…
db1ca39 
…ject#10763)

Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@DarkLight1337 @afeldman-nm
[Misc] Rename embedding classes to pooling (vllm-project#10801)
d198e8f 
Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@youkaichao @afeldman-nm
[doc] add warning about comparing hf and vllm outputs (vllm-project#1…
cf04e11 
…0805)

Signed-off-by: youkaichao <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@ywang96 @heheda12345
@Isotr0py @afeldman-nm
[Misc] Adding MMMU-Pro vision dataset to serving benchmark (vllm-pr…
b58062b 
…oject#10804)

Signed-off-by: Roger Wang <[email protected]>
Co-authored-by: Chen Zhang <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
removed fast tests from pipeline
bcdb5b8 
Signed-off-by: Andrew Feldman <[email protected]>
@KuntaiDu @afeldman-nm
[Core] Implement disagg prefill by StatelessProcessGroup (vllm-projec…
88f7f57 
…t#10502)

This PR provides initial support for single-node disaggregated prefill in 1P1D scenario.
Signed-off-by: KuntaiDu <[email protected]>
Co-authored-by: ApostaC <[email protected]>
Co-authored-by: YaoJiayi <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@Isotr0py @afeldman-nm
[Model] Add BNB support to Llava and Pixtral-HF (vllm-project#10795)
02eb179 
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Andrew Feldman <[email protected]>
@cduk @afeldman-nm
[core] Avoid metrics log noise when idle - include speculative decodi… (
8d5035d 
vllm-project#10809)

Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
torch serialization with msgpack via enc_/ext_hooksgit status!
f707191 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
Merge branch 'main' into v1_logprobs_merge
637c45c
@afeldman-nm
Merge branch 'main' into v1_logprobs_merge
cd5e7c6
@afeldman-nm
Merge branch 'main' into v1_logprobs_merge
8b1b995
@afeldman-nm
Merge branch 'main' into v1_logprobs_merge
3d00348
@afeldman-nm
Merge branch 'main' into v1_logprobs_merge
ce4f081
@afeldman-nm
Merge branch 'main' into v1_logprobs_merge
62d648a
@afeldman-nm
wip
3546639 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
Merge branch 'main' into v1_logprobs_merge
a8c0167
@afeldman-nm
Merge branch 'v1_logprobs' into v1_logprobs_prompt
0bba8f9
@afeldman-nm
GPU returns num_prompt_logprobs + 1 prompt logprobs
69218ab 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
now prompt logprobs include prompt token
2505244 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
wip making prompt logprobs line up with tok ids
e1058ac 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
partial req peek token
5f33902 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
refactoring
199a834 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
refactoring; non-blocking cpu->gpu transfer
879fc44 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
wip detokenizer tests
0f425fe 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
detok test fix
1089127 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
passing detok tests
d2742d8 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
Merge branch 'main' into v1_logprobs
cf28c9b
@afeldman-nm
Merge branch 'main' into v1_logprobs_merge
749be5a
@afeldman-nm
LLMEngine test working, wip AsyncLLM test
a55e679 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
reverted unwanted changes
b2c0c95 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
success
9a40c5f 
Signed-off-by: Andrew Feldman <[email protected]>
@afeldman-nm
Merge branch 'main' into v1_logprobs_apc_merge
ca94fd4
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@mergify Mergify
Copy link

mergify bot commented Jan 12, 2025

⚠️ The sha of the head commit of this PR conflicts with #11910. Mergify cannot evaluate rules on this PR. ⚠️

@mergify Mergify
Copy link

mergify bot commented Jan 12, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @afeldman-nm.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jan 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon will be requested when the pull request is marked ready for review WoosukKwon is a code owner

@robertgshaw2-neuralmagic robertgshaw2-neuralmagic Awaiting requested review from robertgshaw2-neuralmagic robertgshaw2-neuralmagic will be requested when the pull request is marked ready for review robertgshaw2-neuralmagic is a code owner

@njhill njhill Awaiting requested review from njhill njhill will be requested when the pull request is marked ready for review njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 will be requested when the pull request is marked ready for review ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac will be requested when the pull request is marked ready for review comaniac is a code owner

@alexm-neuralmagic alexm-neuralmagic Awaiting requested review from alexm-neuralmagic alexm-neuralmagic will be requested when the pull request is marked ready for review alexm-neuralmagic is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
needs-rebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.