FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Tiled exact self-attention computation between fast GPU SRAM and slow HBM using online softmax renormalization, eliminating quadratic O(N^2) memory reads/writes.

Integrated into PyTorch (scaled_dot_product_attention), vLLM, Hugging Face, TensorRT-LLM, and every modern LLM training run.

Assumptions

• GPU compute FLOPs are abundant, but Memory IO (HBM bandwidth) is the fundamental bottleneck in transformer self-attention

Limitations

• Requires specialized hand-tuned CUDA/Triton kernels targeting specific GPU tensor core microarchitectures (Ampere, Hopper)