Model components / Dataset Processing Tools

DCLM pipeline

The DataComp-LM toolkit and competition harness: a standardized Common Crawl pool, open_lm training recipes, a 50+ task eval suite, and the reference curation pipeline that produces DCLM-baseline.

mlfoundations/dclm; MIT, active (last push 2025-09). Not on PyPI (installs from source, includes a Rust component). This is the DataComp-LM benchmark framework itself, which the capability scores of datatrove/NeMo Curator/Dolma are measured against.

Openness

5 high confidence

5.0

license: MIT(OSI)
source: public
core-gated: ungated

Fully OSI-licensed (MIT), full source public.

https://github.com/mlfoundations/dclm/blob/main/LICENSE recorded 2026-07-21
MIT License text

Adoption

2 medium confidence

2.0

~1,454 GitHub stars; no PyPI package (source install). The reference DataComp-LM toolkit.

https://github.com/mlfoundations/dclm recorded 2026-07-21
1,454 stars; not published to PyPI

Capability

5 high confidence

5.0

Benchmark-defining: DCLM-baseline is the reference-topping corpus of the DataComp-LM leaderboard.

https://arxiv.org/abs/2406.11794 recorded 2026-07-21
DataComp-LM: DCLM-baseline tops existing corpora; 7B/2.6T ~64% MMLU

Unchanged since 2026-07-30 (last edited, not re-checked)