duplodocus

license

Apache-2.0(OSI)

source

public(allenai/duplodocus)

governance

Allen Institute for AI

core-gated

ungated

Fully OSI-licensed (Apache-2.0), full source public.

• https://github.com/allenai/duplodocus/blob/main/LICENSE recorded 2026-08-13

  Apache License Version 2.0 text

• https://raw.githubusercontent.com/allenai/duplodocus/main/README.md recorded 2026-08-13

  README documents the whole pipeline building and installing from the published repo; no paid tier, enterprise edition or license key appears anywhere in it, so nothing is withheld from the source

93 GitHub stars on allenai/duplodocus, read from the repository API, which bands at <1K stars, level 1 on the stars fallback scale. That scale caps at level 3, because a star is not a use. A recent single-purpose Rust CLI used in the Dolma 3 pipeline, and a niche one.

• https://api.github.com/repos/allenai/duplodocus recorded 2026-08-13

  stargazers_count = 93 for allenai/duplodocus

Covers one stage of the Dolma 3 toolchain, deduplication, rather than a full pipeline.

• https://raw.githubusercontent.com/allenai/duplodocus/main/README.md recorded 2026-08-13

  README documents exact and MinHash-LSH fuzzy deduplication in memory- and disk-based variants, a single-purpose dedup tool