CCNet
MetaThe CCNet pipeline: downloads and cleans Common Crawl into high-quality monolingual corpora via language identification, KenLM perplexity filtering, and deduplication. Built CC-100.
Meta facebookresearch/cc_net; MIT, ARCHIVED read-only 2023-10-31 (no longer maintained). Foundational Common Crawl pipeline that built CC-100 and seeded the perplexity-filtering approach reused across later multilingual corpora.
Openness
5 high confidence- license
- MIT(OSI)
- source
- public
- core-gated
- ungated
Fully OSI-licensed (MIT), full source public.
- https://github.com/facebookresearch/cc_net/blob/main/LICENSE recorded 2026-07-21
MIT License text
Adoption
2 medium confidence~1,000 GitHub stars; research repo, no PyPI package. Archived since 2023 but a foundational, widely-reused pipeline.
- https://github.com/facebookresearch/cc_net recorded 2026-07-21
~1,000 stars; archived read-only 2023-10-31
Capability
3 medium confidenceHistorically foundational (CC-100), but a single-method pipeline and frozen since 2023.
- https://github.com/facebookresearch/cc_net recorded 2026-07-21
CCNet perplexity-filtering pipeline (built CC-100)
Unchanged since 2026-07-30 (last edited, not re-checked)