AI Potluck
Model components / Dataset Processing Tools

CCNet

Meta

The CCNet pipeline: downloads and cleans Common Crawl into high-quality monolingual corpora via language identification, KenLM perplexity filtering, and deduplication. Built CC-100.

Meta facebookresearch/cc_net, archived read-only 2023-10-31 and no longer maintained. Foundational Common Crawl pipeline that built CC-100 and seeded the perplexity-filtering approach reused across later multilingual corpora. Verified 2026-08-13 via the GitHub README and the LICENSE body.

Openness

5 high confidence
5.0
license
MIT(OSI)
source
public
core-gated
ungated

Fully OSI-licensed (MIT), full source public.

Adoption

2 medium confidence
2.0

1,045 GitHub stars on facebookresearch/cc_net, read from the repository API, which bands at 1K-10K stars, level 2 on the stars fallback scale. That scale caps at level 3, because a star is not a use. The repo has been archived read-only since October 2023, but it is foundational and widely reused, and there is no PyPI package, so stars are the only signal.

Capability

3 medium confidence
3.0

Historically foundational (CC-100), but a single-method pipeline and frozen since 2023.

Verified 2026-08-13