Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft : API Server #18

Open
wants to merge 27 commits into
base: v1.1
Choose a base branch
from
Open

Draft : API Server #18

wants to merge 27 commits into from

Conversation

fearnworks
Copy link
Contributor

  • Implements openai mock api server
  • Implements basic api server
  • Examples for consumption through jupyter notebook and gradio
  • Includes streaming response example

@nivibilla
Copy link

Hey, quick question. Since vLLM doesn't support LoRA. How are you planning to have different experts loading at the same time? I'm asking as I've been trying to figure out the same but I didn't get anywhere

@fearnworks
Copy link
Contributor Author

We are currently exploring adding a custom mistral_moe model to vllm to handle loading the lora weights and any changes we need to apply to the forward passes.

@nivibilla
Copy link

Ah I see. Makes sense. Would that work with tensor parallel too?

@fearnworks
Copy link
Contributor Author

Pivoting this to a general fastapi server

@fearnworks fearnworks changed the title Draft : vllm Integration Draft : API Server Oct 25, 2023
fearnworks and others added 23 commits October 25, 2023 11:51
@fearnworks
Succesful api model inference
e23751b
@fearnworks
Streaming fix
a859eb2
@fearnworks
Add api notebook
fe7986e
@fearnworks
Newline fixes
81c19bf
@fearnworks
Decouple model loading from chat
560d184
@fearnworks
Add installable hydramoe base
ce32263
@fearnworks
Fix memory hog on model load
b1c9d69
@fearnworks
Update to run api from compose
61cbf76
@fearnworks
ModelWorker & RabbitMQ Init, First Handshake
9c0cf52
@fearnworks
Streaming Inference over RabbitMQ
670acd8
@fearnworks
Integrate fixes for eos stream
9cc1c04
@fearnworks
Skeleton for openai routing and schemas
b0fb4db
@fearnworks
Manual merge from pharouks fork, openai api init
4993241
@fearnworks
Add validation for max_tokens api param
7f07de7
@fearnworks
Add read of model_info from inference yaml file
94a8beb
@fearnworks
Model Info Endpoint provided by ModelWorker
d644712
@fearnworks
Add cli option for choosing a new config file
bfa6a83
@fearnworks
Connect max_tokens api input to model gen
45f417e
@fearnworks
A params for temp, top_p, top_k, and rep penalty
77119eb
@pharaouk
Update chat_service.py
93224a0
@pharaouk
Update chat.py
ae39630
@fearnworks
Updates to infer script and new tokenizer logic
115fe56
@fearnworks
Add check for max_token limit being breached
7192317
pharaouk added 2 commits November 2, 2023 22:41
a
e0dd9dc
Revert "a"
5846241 
This reverts commit e0dd9dc.
@fearnworks fearnworks marked this pull request as ready for review November 7, 2023 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@fearnworks @nivibilla @pharaouk