Model components / Dataset Processing Tools

Ungoliant

Rust pipeline that generates the OSCAR corpus from Common Crawl WET files: language identification, filtering, and metadata annotation. The official OSCAR / Colossal-OSCAR construction pipeline.

OSCAR project Ungoliant; Apache-2.0 confirmed live 2026-07-21 (v2.0.0, still receiving commits). The pipeline that produces the OSCAR / Colossal-OSCAR multilingual corpora.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public(oscar-project/ungoliant)
governance: OSCAR project
core-gated: ungated

Fully OSI-licensed (Apache-2.0), full source public.

https://github.com/oscar-project/ungoliant/blob/main/LICENSE recorded 2026-07-21
Apache License Version 2.0 text

Adoption

2 medium confidence

2.0

~178 GitHub stars; niche but the official pipeline behind the widely-used OSCAR multilingual corpora.

https://github.com/oscar-project/ungoliant recorded 2026-07-21
178 stars

Capability

3 medium confidence

3.0

Focused single-corpus pipeline; mature for its scope.

https://github.com/oscar-project/ungoliant recorded 2026-07-21
ungoliant repo feature set confirmed

Unchanged since 2026-07-30 (last edited, not re-checked)