RedPajama-Data

license

Apache-2.0(OSI)

source

public

core features withheld

no

Fully OSI-licensed (Apache-2.0), full source public.

• https://github.com/togethercomputer/RedPajama-Data/blob/main/LICENSE recorded 2026-08-13

  Apache License Version 2.0 text

• https://raw.githubusercontent.com/togethercomputer/RedPajama-Data/main/LICENSE recorded 2026-09-29

  LICENSE file holds the text "Apache License" "Version 2.0, January 2004" with "TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION".

• https://raw.githubusercontent.com/togethercomputer/RedPajama-Data/main/README.md recorded 2026-09-29

  README states "Licensed under the Apache License, Version 2.0" and documents building and running the whole pipeline from the repo ("Here we provide code to run exact and fuzzy" deduplication); searched the body for enterprise, license key, paid tier and non-commercial: none.

This project has no PyPI package, so GitHub stars are the only usage signal published for it, and a star is not a use. Development has stopped, but the project is still widely referenced as the RedPajama-v2 quality-signal pipeline.

• https://api.github.com/repos/togethercomputer/RedPajama-Data recorded 2026-08-13

  stargazers_count = 4977 for togethercomputer/RedPajama-Data

Comprehensive quality-signal pipeline that produced a frontier-scale open corpus.

• https://raw.githubusercontent.com/togethercomputer/RedPajama-Data/main/README.md recorded 2026-09-29

  README documents quality signals and dedup: "Here we provide code to run exact and fuzzy" deduplication, "Exact Deduplication using a Bloomfilter", "Fuzzy Deduplication with Locality Sensitive Hashing", quality signal "ccnet_perplexity", "rps_doc_ml_wikiref_score" classifier, importance weights and "rps_doc_ldnoobw_words" blocklist; title "an Open Dataset with 30 Trillion Tokens" and languages "English, German, French, Italian, Spanish".