Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Clarify Logits Processor and TRT-LLM examples #116

Merged
merged 2 commits into from
Sep 24, 2024
Merged

docs: Clarify Logits Processor and TRT-LLM examples #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3b7a99e
Eliminated confusions, clarified flags
oandreeva-nv Sep 24, 2024
6d70032
Update Popular_Models_Guide/Hermes-2-Pro-Llama-3-8B/README.md
oandreeva-nv Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions AI_Agents_Guide/Constrained_Decoding/README.md
View file
Open in desktop
oandreeva-nv marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -469,7 +469,7 @@ class TritonPythonModel:

def get_executor_config(self, model_config):
+ tokenizer_dir = model_config['parameters']['tokenizer_dir']['string_value']
+ logits_lmfe_processor = LMFELogitsProcessor(tokenizer_dir, AnswerFormat.model_json_schema())
+ logits_processor = LMFELogitsProcessor(tokenizer_dir, AnswerFormat.model_json_schema())
kwargs = {
"max_beam_width":
get_parameter(model_config, "max_beam_width", int),
Expand All @@ -491,7 +491,7 @@ class TritonPythonModel:
"decoding_config":
self.get_decoding_config(model_config),
+ "logits_post_processor_map":{
+ LMFELogitsProcessor.PROCESSOR_NAME: logits_lmfe_processor
+ LMFELogitsProcessor.PROCESSOR_NAME: logits_processor
+ }
}
kwargs = {k: v for k, v in kwargs.items() if v is not None}
Expand Down Expand Up @@ -602,7 +602,7 @@ class TritonPythonModel:

def get_executor_config(self, model_config):
+ tokenizer_dir = model_config['parameters']['tokenizer_dir']['string_value']
+ logits_lmfe_processor = OutlinesLogitsProcessor(tokenizer_dir, AnswerFormat.model_json_schema())
+ logits_processor = OutlinesLogitsProcessor(tokenizer_dir, AnswerFormat.model_json_schema())
kwargs = {
"max_beam_width":
get_parameter(model_config, "max_beam_width", int),
Expand All @@ -624,7 +624,7 @@ class TritonPythonModel:
"decoding_config":
self.get_decoding_config(model_config),
+ "logits_post_processor_map":{
+ OutlinesLogitsProcessor.PROCESSOR_NAME: logits_lmfe_processor
+ OutlinesLogitsProcessor.PROCESSOR_NAME: logits_processor
+ }
}
kwargs = {k: v for k, v in kwargs.items() if v is not None}
Expand Down
7 changes: 7 additions & 0 deletions Popular_Models_Guide/Hermes-2-Pro-Llama-3-8B/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,13 @@ python3 ${FILL_TEMPLATE_SCRIPT} -i ${MODEL_FOLDER}/tensorrt_llm/config.pbtxt tri

3. Launch Tritonserver

> [!NOTE]
> This tutorial was prepared for serving a TensorRT-LLM model on a single GPU.
> Thus, in the following command use `--world_size=1` if the engine was built
> for a single GPU. Alternatively, if the engine requires multiple GPUs
> make sure to specify the exact number of GPUs required by the engine
> in `--world_size`.

Use the [launch_triton_server.py](https://github.com/triton-inference-server/tensorrtllm_backend/blob/release/0.5.0/scripts/launch_triton_server.py) script. This launches multiple instances of `tritonserver` with MPI.
```bash
python3 /tensorrtllm_backend/scripts/launch_triton_server.py --world_size=<world size of the engine> --model_repo=/opt/tritonserver/inflight_batcher_llm
Expand Down
Loading