duplodocus

license

Apache-2.0(OSI)

source

public(allenai/duplodocus)

core features withheld

no

Fully OSI-licensed (Apache-2.0), full source public.

• https://github.com/allenai/duplodocus/blob/main/LICENSE recorded 2026-08-13

  Apache License Version 2.0 text

• https://raw.githubusercontent.com/allenai/duplodocus/main/LICENSE recorded 2026-09-29

  LICENSE file carries the header "Apache License" and "Version 2.0, January 2004" followed by "TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION".

• https://raw.githubusercontent.com/allenai/duplodocus/main/README.md recorded 2026-09-29

  README documents "Build from Source" with "cargo build --release" and four strategies for "exact and fuzzy (MinHash) document deduplication tool, natively implemented in Rust"; searched the body for paid, enterprise, license key and weights: none.

GitHub stars are the only usage signal published for this repository, and a star is not a use. It is a single-purpose Rust tool that covers one stage, deduplication, of the Dolma 3 pipeline behind OLMo 3, so its natural audience is small.

• https://api.github.com/repos/allenai/duplodocus recorded 2026-08-13

  stargazers_count = 93 for allenai/duplodocus

Covers a single task, deduplication, rather than a full pipeline.

• https://raw.githubusercontent.com/allenai/duplodocus/main/README.md recorded 2026-09-29

  README lists four strategies (Exact + Memory, Exact + Disk, MinHash + Memory, MinHash + Disk) and "Identifies near-duplicates using MinHash LSH based on" Lee et al. 2021, "Removes documents with identical content using fast hash-based matching".