datatrove
Hugging FacePlatform-agnostic library of composable pipeline blocks for large-scale text data processing (reading, filtering, deduplication, tokenization) that runs unchanged across local, Slurm, and cloud backends. The reference pretraining-data pipeline behind the FineWeb family.
Built FineWeb, FineWeb-Edu, FineWeb-2 and FineMath; the de-facto open pretraining-data processing pipeline. Verified 2026-08-13 via the GitHub README and the LICENSE body.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public(huggingface/datatrove)
- governance
- Hugging Face
- managed-tier
- none
- core-gated
- ungated
Fully OSI-licensed (Apache-2.0), full source public, no proprietary core or managed gate.
- https://github.com/huggingface/datatrove/blob/main/LICENSE recorded 2026-08-13
Apache License Version 2.0 text
- https://raw.githubusercontent.com/huggingface/datatrove/main/README.md recorded 2026-08-13
README documents the whole pipeline building and installing from the published repo; no paid tier, enterprise edition or license key appears anywhere in it, so nothing is withheld from the source
Adoption
2 high confidence65,009 downloads in the trailing 30 days for the declared `datatrove` package, which bands at 10K-100K, level 2 on the software and model adoption scale.
- https://pypistats.org/api/packages/datatrove/recent recorded 2026-08-13
65,009 downloads in the trailing 30 days for datatrove
Capability
5 high confidenceBenchmark-anchored on the corpus datatrove builds, as a proxy for the tool itself: FineWeb-Edu leads the public pretraining corpora on the FineWeb ablation suite. The arXiv abstract supports that ranking claim; the per-task ablation figures recorded here are plotted in the paper rather than printed on the abstract page.
- https://arxiv.org/abs/2406.17557 recorded 2026-08-13
FineWeb abstract - FineWeb produces better-performing LLMs than other open pretraining datasets, and FineWeb-Edu lifts knowledge and reasoning benchmarks such as MMLU and ARC dramatically. The per-task ablation figures behind the aggregate are plots in the paper rather than text on this page.
Verified 2026-08-13