How to get access to the vllm backend model

[lianyiyi]

Hi, I want to get access to the backend model, do you know how to make it happen? Thanks!

[tanmayv25]

The term model within Triton vLLM backend is quite overloaded now :P

The python model.py which serves vLLM engine within Triton's python backend is now called one of the python-based backend.
If you are searching for it, the file can be found here: https://github.com/triton-inference-server/vllm_backend/blob/main/src/model.py

See vLLM section within: https://github.com/triton-inference-server/backend?tab=readme-ov-file#where-can-i-find-all-the-backends-that-are-available-for-triton

This python-based backend loads model.json which is basically the EngineArgs that gets passed to the vLLM engine. See here to learn more: https://github.com/triton-inference-server/vllm_backend?tab=readme-ov-file#using-the-vllm-backend

The vLLM backend's model.json can point to a HF model via "model" field.