Created by Tri Dao and collaborators at the Dao-AILab research group.

FlashAttention is an IO aware exact attention algorithm that computes attention faster and with far less memory than naive implementations.

Hosted on GitHub at Dao-AILab/flash-attention and distributed via PyPI.

First published in 2022, with FlashAttention-2 and FlashAttention-3 following in 2023 and 2024.

It removed the quadratic memory bottleneck of standard attention, enabling much longer context windows in transformer training and inference.

Built as a CUDA extension for PyTorch and used inside Transformers, vLLM and many other frameworks.

BSD 3-Clause licensed, open source, maintained by Tri Dao and Dao-AILab, roughly 25k GitHub stars.