Back to Gap Map Model components / Dataset Processing Tools

TxT360 (pipeline)

LLM360
source available / Overall score: 1.8

Construction pipeline for the TxT360 open pretraining corpus: global deduplication and source processing across 99 Common Crawl snapshots and 14 curated sources. Distinct from the released TxT360 dataset.

Openness

2 low confidence
2.0
license
none-declared(the README badge asserts Apache-2.0 but the repo ships no LICENSE file and the GitHub license endpoint 404s)
source
public(LLM360/TxT360)

The README badge claims Apache-2.0, but the repository ships no LICENSE file: the GitHub license endpoint returns 404, the repository API reports no license, and the badge links to a LICENSE in a different repository. That leaves no actual redistribution grant, so the code can be read but not freely reused, which the record treats as no license declared rather than as the badge's claim. No license is declared anywhere the publisher could declare one, so the license tier is unstated.

Adoption

1 medium confidence
1.0

GitHub stars are the only usage signal published for this repository, and a star is not a use. What matters is the 5.7-trillion-token corpus these scripts build, not the tool's own adoption.

Capability

3 medium confidence
3.0

Replication-oriented construction scripts rather than an evolving tool.

  • https://raw.githubusercontent.com/LLM360/TxT360/main/README.md recorded 2026-09-29

    README states "This repository contains the scripts for replicating TxT360" and structures the scripts as "common-crawl", "curated-sources" and "deduplication", with "Run the deduplication scripts to globally deduplicate data from both Common Crawl and curated sources."

Verified 2026-09-27, except some scores, still awaiting re-confirmation