Ungoliant

license

Apache-2.0(OSI)

source

public(oscar-project/ungoliant)

core features withheld

no

Fully OSI-licensed (Apache-2.0), full source public.

• https://github.com/oscar-project/ungoliant/blob/main/LICENSE recorded 2026-09-15

  Apache License Version 2.0 text

• https://raw.githubusercontent.com/oscar-project/ungoliant/main/README.md recorded 2026-09-15

  README documents the whole pipeline building and installing from the published repo; no paid tier, enterprise edition or license key appears anywhere in it, so nothing is withheld from the source

GitHub stars are the only usage signal published for this repository, and a star is not a use. It is the official pipeline behind the OSCAR and Colossal-OSCAR corpora, so its importance runs well ahead of that count.

• https://api.github.com/repos/oscar-project/ungoliant recorded 2026-08-12

  stargazers_count = 178 for oscar-project/ungoliant

Focused single-corpus pipeline.

• https://raw.githubusercontent.com/oscar-project/ungoliant/main/Cargo.toml recorded 2026-09-29

  Cargo.toml gives the package description "The pipeline for the OSCAR corpus." and lists the "fasttext" crate as a dependency.

• https://raw.githubusercontent.com/oscar-project/ungoliant/main/README.md recorded 2026-09-29

  README describes Ungoliant as "a high-performance pipeline that provides tools to build corpus generation pipelines from CommonCrawl", currently "the generation pipeline for" the OSCAR corpus, with a fastText language identification model ("lid.176.bin") and an optional KenLM feature.

• https://raw.githubusercontent.com/oscar-project/ungoliant/main/src/filtering/mod.rs recorded 2026-09-29

  Rust source of the filtering module: "Filters can operate on sentence or record level"

• https://raw.githubusercontent.com/oscar-project/ungoliant/main/src/pipelines/oscardoc/types/mod.rs recorded 2026-09-29

  The oscardoc types module re-exports the OSCAR document metadata type: "pub use oscar_io::v3::Metadata;"