Model components / Dataset Processing Tools

RedPajama-Data

The RedPajama construction and quality-signal pipeline: quality classifiers, per-document quality/toxicity/importance annotations, and exact + fuzzy deduplication. Produced RedPajama-Data-v2 (30T tokens, 5 languages).

Together togethercomputer/RedPajama-Data; Apache-2.0, dormant since Dec 2024. The construction pipeline (not the dataset) behind RedPajama-v2 and the RedPajama-INCITE models.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public
core-gated: ungated

Fully OSI-licensed (Apache-2.0), full source public.

https://github.com/togethercomputer/RedPajama-Data/blob/main/LICENSE recorded 2026-07-21
Apache License Version 2.0 text

Adoption

2 medium confidence

2.0

~4,969 GitHub stars; pipeline repo, no PyPI package. Widely referenced quality-signal pipeline; dormant since Dec 2024.

https://github.com/togethercomputer/RedPajama-Data recorded 2026-07-21
4,969 stars; last real commit Dec 2024

Capability

4 medium confidence

4.0

Comprehensive quality-signal pipeline; produced a frontier-scale open corpus, but dormant.

https://github.com/togethercomputer/RedPajama-Data recorded 2026-07-21
RedPajama-v2 quality-signal + dedup pipeline (30T tokens)

Unchanged since 2026-07-30 (last edited, not re-checked)