Dolma toolkit

license

Apache-2.0(OSI)

source

public(allenai/dolma)

core features withheld

no

Fully OSI-licensed (Apache-2.0), full source public.

• https://github.com/allenai/dolma/blob/main/LICENSE recorded 2026-08-13

  Apache License Version 2.0 text

• https://raw.githubusercontent.com/allenai/dolma/main/LICENSE recorded 2026-09-29

  LICENSE file text reads "Apache License", "Version 2.0, January 2004" and "TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION".

• https://raw.githubusercontent.com/allenai/dolma/main/pyproject.toml recorded 2026-09-29

  pyproject.toml declares license = { text = "Apache-2.0" } and the classifier "License :: OSI Approved :: Apache Software License"; dependencies are ordinary PyPI packages. Searched the body for enterprise, license key and non-commercial: none.

• https://raw.githubusercontent.com/allenai/dolma/main/README.md recorded 2026-09-29

  README states "this repo contains the source code for the Dolma Toolkit" and "To install, simply type `pip install dolma` in your terminal." Searched the body for enterprise, license key, paid tier and non-commercial: none.

PyPI downloads of the dolma package are the usage signal for this tool, counted with mirror traffic as for every PyPI-routed product. Mirrors account for most of this package's downloads, so the figure without them is far lower.

• https://pypistats.org/api/packages/dolma/overall?mirrors=true recorded 2026-10-08

  Daily with_mirrors downloads for dolma summing to 11,619 over the 30 days 2026-09-08 to 2026-10-07.

• https://pypistats.org/api/packages/dolma/recent recorded 2026-10-08

  "last_month":1727 - the without-mirrors series, which this reading does not use.

The Dolma toolkit is judged through the corpus it builds, since the toolkit has no benchmark of its own. That corpus scores below RedPajama and RefinedWeb on DataComp-LM, which places the toolkit two tiers below datatrove.

• https://arxiv.org/html/2406.11794v3 recorded 2026-09-29

  Table 2 of the DataComp-LM paper compares Dolma-V1 with RedPajama and RefinedWeb at the 7B-1x scale, and the text notes that "all four datasets use various heuristic filters and data cleaning steps" before ranking RefinedWeb first; the table caption adds "Despite not mixing high-quality sources" RefinedWeb performs best, with Dolma-V1 at 35.0 Core and 18.4 Extended below RedPajama and RefinedWeb.