CCNet

license

MIT(OSI)

source

public

core features withheld

no

Fully OSI-licensed (MIT), full source public.

• https://github.com/facebookresearch/cc_net/blob/main/LICENSE recorded 2026-09-22

  MIT License text

• https://raw.githubusercontent.com/facebookresearch/cc_net/main/README.md recorded 2026-09-22

  README documents the whole pipeline building and installing from the published repo; no paid tier, enterprise edition or license key appears anywhere in it, so nothing is withheld from the source

This repository has no PyPI package, so GitHub stars are the only usage signal published for it, and a star is not a use. The repository is now archived and read-only.

• https://api.github.com/repos/facebookresearch/cc_net recorded 2026-08-13

  stargazers_count = 1045 for facebookresearch/cc_net

A single-method pipeline built around language ID, perplexity filtering and deduplication.

• https://raw.githubusercontent.com/facebookresearch/cc_net/main/README.md recorded 2026-09-29

  The README describes the pipeline steps: hashes "downloads one Common-Crawl snapshot, and compute hashes for each paragraph" and mine "removes duplicates, detects language, run the LM and split by lang/perplexity buckets", with a "dl_cc_100.py" script for CC-100 and KenLM installed for the language models.