triton-inference-server tensorrtllm_backend · Discussions · GitHub

What's the difference between ensemble and tensorrt_llm_bls?
Q&A Saigut

Sort by: Latest activity

Categories

Community guidelines

Discussions

You must be logged in to vote

Whisper model support

Saeedmatt3r started Oct 28, 2024 in General · Closed

0
You must be logged in to vote

What is the purpose of all_models/inflight_batcher_llm/tensorrt_llm/1/model.py?

owenonline asked Jul 21, 2024 in Q&A · Answered

3
You must be logged in to vote

Disabling dynamic batching and enabling TRTLLM's continuous batching

dhruvmullick asked Aug 14, 2024 in Q&A · Unanswered

0
You must be logged in to vote

Speculative decoding example in e2e_grpc_speculative_decoding_client.py

jxchenus asked May 31, 2024 in Q&A · Closed · Unanswered

1
You must be logged in to vote

Any Speculative Decoding Performance Numbers?

jxchenus asked May 30, 2024 in Q&A · Closed · Unanswered

0
You must be logged in to vote

Why do I get this error after executing the command python3 scripts/launch_triton_server.py --world_size 1 --model_repo=llama_ifb/?

SevenEmotion asked Apr 12, 2024 in Q&A · Unanswered

0
You must be logged in to vote

What's the difference between ensemble and tensorrt_llm_bls?

Saigut asked Apr 10, 2024 in Q&A · Closed · Answered

1