datamap-rs

license

Apache-2.0(OSI)

source

public(allenai/datamap-rs)

core features withheld

no

Fully OSI-licensed (Apache-2.0), full source public.

• https://github.com/allenai/datamap-rs/blob/main/LICENSE recorded 2026-08-13

  Apache License Version 2.0 text

• https://raw.githubusercontent.com/allenai/datamap-rs/main/LICENSE recorded 2026-09-29

  LICENSE file carries the license text headed "Apache License" and "Version 2.0, January 2004" followed by "TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION".

• https://raw.githubusercontent.com/allenai/datamap-rs/main/README.md recorded 2026-09-29

  README describes a Rust toolkit that builds from the published repo: "Clone the repository" then "cargo build --release" producing "The binary will be available at `target/release/datamap`". Searched the body for enterprise, license key, paid, pricing and subscription: none; the only optional external download is the "lid176.bin" FastText model used by tests and some example configs.

GitHub stars are the only usage signal published for this repository, and a star is not a use. It is a single-purpose Rust tool that covers one stage, cleaning, of the Dolma 3 pipeline behind OLMo 3, so its natural audience is small.

• https://api.github.com/repos/allenai/datamap-rs recorded 2026-08-13

  stargazers_count = 56 for allenai/datamap-rs

Covers the data-processing steps around corpus cleaning and deduplication rather than model training or a full end-to-end data platform.

• https://raw.githubusercontent.com/allenai/datamap-rs/main/README.md recorded 2026-09-29

  README opens with "A high-performance data processing pipeline for large-scale datasets built in Rust" and says "DataMap is a Rust-based toolkit designed for efficient processing, filtering, and transformation of large text datasets, primarily in JSONL format", with commands Map, Reshard, Reservoir Sample, Discrete Partition, Range Partition, Group, GroupFilter, Shuffle and Count. Searched the body for Dolma: none.