AI Potluck
Model components / Dataset Processing Tools

Ungoliant

OSCAR Project

Rust pipeline that generates the OSCAR corpus from Common Crawl WET files: language identification, filtering, and metadata annotation. The official OSCAR / Colossal-OSCAR construction pipeline.

OSCAR project Ungoliant, still receiving commits (last push November 2025). The pipeline that produces the OSCAR / Colossal-OSCAR multilingual corpora. Verified 2026-08-13 via the GitHub README and the LICENSE body.

Openness

5 high confidence
5.0
license
Apache-2.0(OSI)
source
public(oscar-project/ungoliant)
governance
OSCAR project
core-gated
ungated

Fully OSI-licensed (Apache-2.0), full source public.

Adoption

1 medium confidence
1.0

178 GitHub stars on oscar-project/ungoliant, which bands at <1K stars, level 1 on the stars fallback scale. That scale caps at level 3, because a star is not a use. As the official pipeline behind the OSCAR corpora it matters more than its star count suggests, but importance is not adoption, and no download, install or customer figure is published for this product, so stars remain the only honest signal and the level is directional.

Capability

3 medium confidence
3.0

Focused single-corpus pipeline; mature for its scope.

Verified 2026-08-12