Pythia

weights

open(Apache-2.0)

data

open(the Pile, with dataloader reconstruction tools)

code

open(training+analysis code, GitHub)

checkpoints

open(154 per model)

license

Apache-2.0(OSI)

Pythia releases Apache-2.0 weights, the Pile as its training corpus together with dataloader tools that reconstruct the exact token order, training and analysis code, and 154 intermediate checkpoints per model size. Every part of the release is downloadable and the training run itself is reproducible, not just the model, which is what separates full openness from open weights.

• https://arxiv.org/abs/2304.01373 recorded 2026-09-29

  Abstract: "We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders" and "Trained models, analysis code, training code, and training data can be found at".

• https://github.com/EleutherAI/pythia recorded 2026-08-13

  Repository page, described as "The hub for EleutherAI's work on interpretability and learning dynamics" - training and analysis code, public.

• https://huggingface.co/api/datasets/EleutherAI/pile recorded 2026-09-29

  Hub API record for the Pile - "The Pile is a 825 GiB diverse, open source language modelling data set" - with gated and private both false; the repo itself holds only a loading script (pile.py) and a card, tagged "license:other".

• https://huggingface.co/api/datasets/EleutherAI/pile-deduped-pythia-preshuffled recorded 2026-09-29

  Hub API record for "EleutherAI/pile-deduped-pythia-preshuffled", the deduplicated Pile in Pythia training order, with gated and private both false and 21 shards plus "document.idx".

• https://huggingface.co/api/datasets/EleutherAI/pile-standard-pythia-preshuffled recorded 2026-09-29

  Hub API record for "EleutherAI/pile-standard-pythia-preshuffled", the tokenized Pile in Pythia training order, with gated and private both false and 21 shards plus "document.idx" (about 600 GB stored).

• https://huggingface.co/api/models/EleutherAI/pythia-12b recorded 2026-09-29

  Hub API record tagged "license:apache-2.0" with card data license apache-2.0 and "text-generation-inference"; gated, private and disabled are all false, and the tags include safetensors weights.

• https://huggingface.co/EleutherAI/pythia-12b/blob/main/README.md recorded 2026-09-29

  Rendered Pythia-12B model card lists "License: Apache 2.0" and "We also provide 154 intermediate checkpoints per model, hosted on Hugging Face as branches"; the embedded repo data records gated as false.

• https://huggingface.co/EleutherAI/pythia-12b/raw/main/README.md recorded 2026-09-29

  Raw model card front matter declares "license: apache-2.0" with dataset EleutherAI/pile, and the card says "All 8 model sizes are trained on the exact same data, in the exact same order" and points to the GitHub repository "for training procedure, config files, and details on how to use".

Adoption is measured as Hugging Face downloads for the single declared artifact, EleutherAI/pythia-12b.

• https://huggingface.co/api/models/EleutherAI/pythia-12b recorded 2026-08-13

  30,007 downloads in the trailing 30 days for EleutherAI/pythia-12b

Pythia was never built to compete on capability. Its sixteen models top out at 12B, were trained on the 2020 Pile corpus, and are research tools for studying how language models learn rather than production systems. No benchmark measures it against the 2026 frontier, and it trails every model built for competitive performance.

• https://arxiv.org/abs/2304.01373 recorded 2026-09-29

  Abstract: "a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters"; the case studies it presents are "memorization, term frequency effects on few-shot performance, and reducing gender bias", not capability benchmarks.

• https://huggingface.co/EleutherAI/pythia-12b/raw/main/README.md recorded 2026-09-29

  Model card: "The Pythia model suite was deliberately designed to promote scientific research on large language models, especially interpretability research" and "Despite not centering downstream performance as a design goal"; its evaluations compare against OPT and GPT-Neo.