datatrove
Hugging FacePlatform-agnostic library of composable pipeline blocks for large-scale text data processing (reading, filtering, deduplication, tokenization) that runs unchanged across local, Slurm, and cloud backends. The reference pretraining-data pipeline behind the FineWeb family.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public(huggingface/datatrove)
- managed-tier
- none
- core features withheld
- no
Fully OSI-licensed (Apache-2.0), full source public, no proprietary core or managed gate.
- https://github.com/huggingface/datatrove/blob/main/LICENSE recorded 2026-08-13
Apache License Version 2.0 text
- https://raw.githubusercontent.com/huggingface/datatrove/main/LICENSE recorded 2026-09-29
LICENSE file is the "Apache License" "Version 2.0, January 2004" text.
- https://raw.githubusercontent.com/huggingface/datatrove/main/pyproject.toml recorded 2026-09-29
pyproject.toml declares "Apache-2.0" as the license and the classifier "License :: OSI Approved :: Apache Software License". Searched the body for enterprise, license key and paid tier: none; optional extras list only public packages such as "vllm>=0.10.0".
- https://raw.githubusercontent.com/huggingface/datatrove/main/README.md recorded 2026-09-29
README install section says "Requires Python 3.10+." and installs every extra from the repo with `uv sync --extra all`, and states "DataTrove processing pipelines are platform-agnostic, running out of the box locally or on a slurm cluster". Searched the body for enterprise, paid tier, license key and managed service: none.
Adoption
2 high confidencePyPI downloads for the datatrove package are the usage signal recorded for this tool.
- https://pypistats.org/api/packages/datatrove/recent recorded 2026-08-13
65,009 downloads in the trailing 30 days for datatrove
Capability
5 high confidenceDatatrove is judged through the corpus it builds, since the library has no benchmark of its own. The FineWeb paper reports that corpus outperforming other open pretraining datasets.
- https://arxiv.org/abs/2406.17557 recorded 2026-09-29
FineWeb abstract: "we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets" and "FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC"; it says "we publicly release our data curation codebase".
Verified 2026-09-24