Back to Gap Map Model components / Training & synthetic datasets

OpenWebText

Aaron Gokaslan (Skylion007)
open / Overall score: 3.0

OpenWebText is an open re-creation of WebText, the corpus OpenAI used to train GPT-2. Aaron Gokaslan and Vanya Cohen of Brown University gathered the web pages linked from Reddit submissions, kept the English ones, removed near-duplicates and very short documents, and ended with about 38GB of text from 8,013,769 documents. It went into the pretraining mix of RoBERTa and DeBERTa and is the corpus nanoGPT uses to reproduce the smallest GPT-2.

Openness

5 high confidence
5.0
license
cc0-1.0 (covers the packaging, while the creators claim no rights in the extracted text)
ungated
yes
dataset_card
present

OpenWebText downloads without a sign-in, with a card describing how it was collected, and its creators release it under CC0. The text itself comes from third-party web pages, which the creators do not claim to own and will remove on request.

Adoption

3 high confidence
3.0

Hugging Face downloads of the dataset over the trailing month, the only usage channel on record for OpenWebText.

Capability

3 medium confidence
3.0

OpenWebText helped train RoBERTa and DeBERTa and remains the usual corpus for reproducing GPT-2 at small scale. Like C4, it is a 2019-era English web corpus that has proven its worth but is no longer a first choice for pretraining.

Verified 2026-11-12