Model components / Dataset Processing Tools

RedPajama-Data

The RedPajama construction and quality-signal pipeline: quality classifiers, per-document quality/toxicity/importance annotations, and exact + fuzzy deduplication. Produced RedPajama-Data-v2 (30T tokens, 5 languages).

Together togethercomputer/RedPajama-Data, dormant since December 2024. The construction pipeline, not the dataset, behind RedPajama-v2 and the RedPajama-INCITE models. Verified 2026-08-13 via the GitHub README and the LICENSE body.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core-gated: ungated

Fully OSI-licensed (Apache-2.0), full source public.

https://github.com/togethercomputer/RedPajama-Data/blob/main/LICENSE recorded 2026-08-13
Apache License Version 2.0 text
https://raw.githubusercontent.com/togethercomputer/RedPajama-Data/main/README.md recorded 2026-08-13
README documents the whole pipeline building and installing from the published repo; no paid tier, enterprise edition or license key appears anywhere in it, so nothing is withheld from the source

Adoption

2 medium confidence

2.0

4,977 GitHub stars on togethercomputer/RedPajama-Data, read from the repository API, which bands at 1K-10K stars, level 2 on the stars fallback scale. That scale caps at level 3, because a star is not a use. There is no PyPI package and the last real commit was December 2024, though the project is still widely referenced as the RedPajama-v2 quality-signal pipeline.

https://api.github.com/repos/togethercomputer/RedPajama-Data recorded 2026-08-13
stargazers_count = 4977 for togethercomputer/RedPajama-Data

Capability

4 medium confidence

4.0

Comprehensive quality-signal pipeline; produced a frontier-scale open corpus, but dormant.

https://raw.githubusercontent.com/togethercomputer/RedPajama-Data/main/README.md recorded 2026-08-13
README documents the RedPajama-v2 quality-signal annotations and the exact plus fuzzy dedup stages over 30T tokens

Verified 2026-08-13