Back to Gap Map Model components / Training & synthetic datasets

The Pile

open / Overall score: 3.0

EleutherAI's 825 GiB English pretraining corpus from 2020, drawn from 22 sources spanning web text, academic papers, code, books and forums. It set the template for documented open pretraining data and trained Pythia, GPT-NeoX-20B, GPT-J and Cerebras-GPT among others.

A family record covering the original and deduplicated releases.

Openness

3 medium confidence

3.0

access: ungated
license: mixed-per-subset
dataset_card: legacy-repo-only

The corpus downloads freely and ungated from the deduplicated Hugging Face repository. The original repository now holds only a dataset card and a loading script, and licensing defers to each subset's own terms.

https://huggingface.co/api/datasets/EleutherAI/the_pile_deduplicated recorded 2026-09-29
Hub API record for the deduplicated repository, which reports gated and disabled as false and lists parquet shards such as "data/train-00000-of-01650-f70471ee3deb09c0.parquet".
https://huggingface.co/datasets/EleutherAI/pile recorded 2026-09-29
Legacy repo page shows license other and a viewer notice that "The viewer is disabled because this dataset repo requires arbitrary Python code execution."
https://huggingface.co/datasets/EleutherAI/pile/raw/main/README.md recorded 2026-09-29
Raw legacy dataset card declares "license: other" and says "Please refer to the specific license depending on the subset you use:" listing only PubMed Central under MIT.
https://huggingface.co/datasets/EleutherAI/the_pile_deduplicated recorded 2026-09-29
Dataset page for "EleutherAI/the_pile_deduplicated" showing "No dataset card yet" beside the parquet files, with 134318121 rows (451 GB) served for download.

Adoption

3 medium confidence

3.0

Adoption is measured by Hugging Face downloads of the deduplicated repository, EleutherAI/the_pile_deduplicated, the corpus's current distribution point since the original loading-script mirrors went dark. The count sits near the bottom of its range, so confidence is medium rather than high.

https://huggingface.co/api/datasets/EleutherAI/the_pile_deduplicated recorded 2026-08-14
downloads field = 13,360 in the trailing 30 days; 116 likes; gated false

Capability

3 high confidence

3.0

The foundational 2020-era template, proven for a wide 2022-23 family (Pythia, GPT-NeoX-20B, GPT-J, Cerebras-GPT), but dated and beaten by modern web corpora in FineWeb ablations.

https://arxiv.org/abs/2204.06745 recorded 2026-09-29
GPT-NeoX-20B abstract says it is "a 20 billion parameter autoregressive language model trained on the Pile".
https://arxiv.org/abs/2304.03208 recorded 2026-09-29
Cerebras-GPT abstract says "We train Cerebras-GPT models on the Eleuther Pile dataset".
https://arxiv.org/html/2406.17557 recorded 2026-09-29
FineWeb paper says the FineWeb model "outperforms all other open datasets we compared on our aggregate group of tasks", and its comparison set lists "The Pile (340B tokens)".

Verified 2026-09-27