Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (and the Triton backend) this January 16th, 2024.

This update includes:

• Features
  • Medusa decoding support for batch size 1
    • The support is limited to Python runtime for Ampere or newer GPUs with fp16 and bf16 accuracy, and under the condition that linear or paged KV cache is used without removing input padding, and the temperature parameter of sampling configuration should be 0.
  • Add Weight-Only Support To Whisper Add Weight-Only Support To Whisper #794, thanks to the contribution from @Eddie-Wang1120
• Performance
  • Add bfloat16 and paged kv cache support for optimized generation MQA/GQA kernels
  • Custom allreduce performance optimization by introducing a ping-pong buffer to avoid an extra synchronization cost
  • Top-P sampling performance optimization
  • LoRA performance optimization
  • Disable enable_trt_overlap argument for GPT manager by default
• Documentation
  • Update to the docs/source/new_workflow.md documentation

Thanks,
The TensorRT-LLM Engineering Team