Back to Gap Map Infrastructure / Compilers & Model Optimization

FlashAttention

open source / Overall score: 3.5

FlashAttention is the reference implementation of IO-aware exact attention, computing attention without materializing the full attention matrix and so cutting memory traffic rather than approximating the result. The repository carries FlashAttention and FlashAttention-2 alongside the papers describing them, with FlashAttention-3 Hopper kernels in a separate tree and FlashAttention-4, written in CuTeDSL, for Hopper and Blackwell.

Openness

5 high confidence

5.0

license: BSD-3-Clause(OSI)
source: public
core features withheld: no — the default flash-attn build needs only PyTorch and einops
optional-dependency: nvidia-cutlass-dsl(NVIDIA proprietary license, required by the optional flash-attn-4 package and not by the default build)

The LICENSE file is BSD-3-Clause and the repository is public. The default flash-attn package builds FlashAttention-2 from this source with only PyTorch and einops as dependencies. FlashAttention-4 ships separately as flash-attn-4, which requires NVIDIA's proprietary nvidia-cutlass-dsl. Because that package is optional rather than part of the default build, the closed dependency does not gate the core.

https://api.github.com/repos/Dao-AILab/flash-attention recorded 2026-08-18
Repo metadata - license spdx_id BSD-3-Clause, private false, archived false, default branch main - for Dao-AILab/flash-attention.
https://pypi.org/pypi/nvidia-cutlass-dsl/json recorded 2026-10-08
PyPI metadata for nvidia-cutlass-dsl carries the classifier "License :: Other/Proprietary License".
https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/flash_attn/cute/pyproject.toml recorded 2026-10-08
The separate flash-attn-4 package, "Flash Attention CUTE (CUDA Template Engine) implementation", lists "nvidia-cutlass-dsl>=4.6.2" among its dependencies.
https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/LICENSE recorded 2026-10-08
The LICENSE file is the BSD 3-Clause text, headed "BSD 3-Clause License" and continuing "Redistribution and use in source and binary forms".
https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/README.md recorded 2026-10-08
README installs the default package with "pip install flash-attn --no-build-isolation" and documents FlashAttention-4 as a separate package, "pip install flash-attn-4", imported with "from flash_attn.cute import flash_attn_func".
https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/setup.py recorded 2026-10-08
setup.py excludes "flash_attn.cute.*" from the default package and, under "# Build install_requires based on platform", requires only "torch" and "einops" on CUDA.

Adoption

3 high confidence

3.0

Adoption is measured from PyPI downloads of the `flash-attn` package.

https://pypistats.org/api/packages/flash-attn/recent recorded 2026-08-18
last_month downloads = 402,966 for flash-attn

Capability

4 medium confidence

4.0

FlashAttention supplies one narrow kernel family, exact IO-aware attention on NVIDIA GPUs with an AMD ROCm backend, but its usage page lists PyTorch, DeepSpeed and Megatron-LM among the projects that integrate it. Narrow coverage with broad reliance places it one step below Apache TVM.

https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/README.md recorded 2026-09-30
README documents "FlashAttention-3 is optimized for Hopper GPUs (e.g. H100)." and "FlashAttention-4 is written in CuTeDSL and optimized for Hopper and Blackwell GPUs (e.g. H100, B200)." and, for AMD, "ROCm version has two backends." with FlashAttention-2 support.
https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/usage.md recorded 2026-09-30
Usage page opens "This page contains a partial list of places where FlashAttention is being used." and lists integrations in "Pytorch", "Huggingface's [transformers]", "Microsoft's [DeepSpeed]", "Nvidia's [Megatron-LM]", "MosaicML [Composer]", "EleutherAI's [GPT-NeoX]" and "PaddlePaddle".

Verified 2026-09-24