CCNet
MetaThe CCNet pipeline: downloads and cleans Common Crawl into high-quality monolingual corpora via language identification, KenLM perplexity filtering, and deduplication. Built CC-100.
Meta facebookresearch/cc_net, archived read-only 2023-10-31 and no longer maintained. Foundational Common Crawl pipeline that built CC-100 and seeded the perplexity-filtering approach reused across later multilingual corpora. Verified 2026-08-13 via the GitHub README and the LICENSE body.
Openness
5 high confidence- license
- MIT(OSI)
- source
- public
- core-gated
- ungated
Fully OSI-licensed (MIT), full source public.
- https://github.com/facebookresearch/cc_net/blob/main/LICENSE recorded 2026-08-13
MIT License text
- https://raw.githubusercontent.com/facebookresearch/cc_net/main/README.md recorded 2026-08-13
README documents the whole pipeline building and installing from the published repo; no paid tier, enterprise edition or license key appears anywhere in it, so nothing is withheld from the source
Adoption
2 medium confidence1,045 GitHub stars on facebookresearch/cc_net, read from the repository API, which bands at 1K-10K stars, level 2 on the stars fallback scale. That scale caps at level 3, because a star is not a use. The repo has been archived read-only since October 2023, but it is foundational and widely reused, and there is no PyPI package, so stars are the only signal.
- https://api.github.com/repos/facebookresearch/cc_net recorded 2026-08-13
stargazers_count = 1045 for facebookresearch/cc_net
Capability
3 medium confidenceHistorically foundational (CC-100), but a single-method pipeline and frozen since 2023.
- https://raw.githubusercontent.com/facebookresearch/cc_net/main/README.md recorded 2026-08-13
README documents the language-ID plus KenLM perplexity plus dedup pipeline of the CCNet paper; the repo is archived read-only
Verified 2026-08-13