[Feature]: Flash Attention 3 Support for MI300X GPUs

[codinggosu]

Suggestion Description

Context:
We are currently using the latest MI300X GPUs for performance evaluation and have observed that Flash Attention 2 provides a 20% performance improvement over our existing setup. This enhancement has been significant for our projects.

Request:
We are eager to explore the benefits that Flash Attention 3 could offer. However, we were disappointed to find that it currently only supports Hopper architectures.

Inquiry:

Is there a timeline or plan for supporting Flash Attention 3 on MI300X GPUs?
If there is no official plan, are there any guides or resources available that could assist us in implementing support ourselves?
Thank you for your attention to this matter. We look forward to any information you can provide.

Operating System

Ubuntu

GPU

MI300X

ROCm Component

No response