Model components / Dataset Processing Tools

Dolma toolkit

High-performance toolkit for curating LM pretraining data: taggers, quality and content filtering, and Rust Bloom-filter deduplication. The software that built the Dolma corpus and the OLMo/OLMoE mixes.

Ai2 Dolma toolkit (allenai/dolma), distinct from the Dolma dataset artifact; repo active (v1.2.1, 2025-07-07), Apache-2.0 confirmed live 2026-07-21. Built the Dolma corpus and OLMo/OLMoE pretraining mixes.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public(allenai/dolma)
governance: Allen Institute for AI
core-gated: ungated

Fully OSI-licensed (Apache-2.0), full source public.

https://github.com/allenai/dolma/blob/main/LICENSE recorded 2026-07-21
Apache License Version 2.0 text

Adoption

2 high confidence

2.0

~5.1k PyPI downloads/month; the curation toolkit behind the Dolma corpus and OLMo family.

https://pypistats.org/packages/dolma recorded 2026-07-21
Downloads last month: 5,097

Capability

3 high confidence

3.0

Benchmark-anchored on the output corpus (proxy): capable and fully open, but its released corpus trails the datatrove / NeMo Curator frontier on DCLM. Down from a feature-only read of 4.

https://arxiv.org/abs/2406.11794 recorded 2026-07-21
DataComp-LM Table 2: Dolma-V1 35.0 CORE vs RefinedWeb 36.9 at 7B-1x scale

Unchanged since 2026-07-30 (last edited, not re-checked)