Dolma toolkit
Allen Institute for AIHigh-performance toolkit for curating LM pretraining data: taggers, quality and content filtering, and Rust Bloom-filter deduplication. The software that built the Dolma corpus and the OLMo/OLMoE mixes.
Ai2 Dolma toolkit (allenai/dolma), distinct from the Dolma dataset artifact. Built the Dolma corpus and the OLMo/OLMoE pretraining mixes. Verified 2026-08-13 via the GitHub README and the LICENSE body.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public(allenai/dolma)
- governance
- Allen Institute for AI
- core-gated
- ungated
Fully OSI-licensed (Apache-2.0), full source public.
- https://github.com/allenai/dolma/blob/main/LICENSE recorded 2026-08-13
Apache License Version 2.0 text
- https://raw.githubusercontent.com/allenai/dolma/main/README.md recorded 2026-08-13
README documents the whole pipeline building and installing from the published repo; no paid tier, enterprise edition or license key appears anywhere in it, so nothing is withheld from the source
Adoption
1 high confidence4,390 downloads in the trailing 30 days for dolma, read from the pypistats API, which bands at <10K, level 1 on the software and model adoption scale. It is the curation toolkit behind the Dolma corpus and the OLMo family, so its importance runs ahead of its raw download volume.
- https://pypistats.org/api/packages/dolma/recent recorded 2026-08-13
4,390 downloads in the trailing 30 days for dolma
Capability
3 high confidenceBenchmark-anchored on the corpus the toolkit builds, as a proxy for the tool: capable and fully open, but the released Dolma corpus trails the datatrove and NeMo Curator frontier on DataComp-LM.
- https://arxiv.org/html/2406.11794v3 recorded 2026-08-13
DataComp-LM Table 2 in the full text - Dolma-V1 scores 35.0 Core and 18.4 Extended at the 7B-1x scale, below RedPajama 35.3 and RefinedWeb 36.9
Verified 2026-08-13