IPDPS'23 Best Paper "ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs"

[hwu36]

@yzhaiustc er al just won IPDPS'23 best paper. It describes an MHA algorithm on top of cutlass group gemm for variable sequence length.

The implementation is 2 year old, but the idea is still valid.

Paper link: https://arxiv.org/abs/2210.03052

[yzhaiustc]

Thanks a lot, Haicheng, for your help & support with this work.
CUTLASS enabled most of architectural-related designs in the paper.
Also many thanks and congrats to all the co-authors @Meteorix, @kongroo, @Laurawly, @shangz-ai :)