DCLM pipeline
DataComp-LM (ML Foundations)The DataComp-LM toolkit and competition harness: a standardized Common Crawl pool, open_lm training recipes, a 50+ task eval suite, and the reference curation pipeline that produces DCLM-baseline.
Not on PyPI - installs from source and includes a Rust component. The DataComp-LM benchmark framework itself, which the capability scores of datatrove, NeMo Curator and Dolma are measured against. Verified 2026-08-13 via GitHub and the repository README.
Openness
5 high confidence- license
- MIT(OSI)
- source
- public
- core-gated
- ungated
Fully OSI-licensed (MIT), full source public.
- https://github.com/mlfoundations/dclm/blob/main/LICENSE recorded 2026-08-13
MIT License text
- https://raw.githubusercontent.com/mlfoundations/dclm/main/README.md recorded 2026-08-13
README documents the whole pipeline building and installing from the published repo; no paid tier, enterprise edition or license key appears anywhere in it, so nothing is withheld from the source
Adoption
2 medium confidence1,463 GitHub stars on mlfoundations/dclm, read from the repository API, which bands at 1K-10K stars, level 2 on the stars fallback scale. That scale caps at level 3, because a star is not a use. No PyPI package is published, so the toolkit installs from source and stars are the only signal available. As the reference DataComp-LM harness its importance runs well ahead of its adoption, so the level is directional.
- https://api.github.com/repos/mlfoundations/dclm recorded 2026-08-13
stargazers_count = 1463 for mlfoundations/dclm
Capability
5 high confidenceBenchmark-defining: DCLM-baseline is the reference-topping corpus of the DataComp-LM leaderboard.
- https://arxiv.org/abs/2406.11794 recorded 2026-08-13
DataComp-LM abstract - DCLM-baseline trains a 7B model to 64% 5-shot MMLU on 2.6T tokens, 6.6 points above MAP-Neo, the previous open-data state of the art
Verified 2026-08-13