Back to Gap Map Model components / Training & synthetic datasets

RefinedWeb

open / Overall score: 3.0

English web dataset of roughly 968 million rows and about 2.8TB, built by the Technology Innovation Institute through stringent filtering and exact-plus-fuzzy deduplication of Common Crawl. The public release is an extract of the larger corpus used to train the Falcon 7B and 40B models.

Openness

5 high confidence

5.0

license: ODC-BY-1.0
gated: false
dataset_card: present
note: public_extract

Permissive ODC-BY 1.0 license, ungated download, with a dataset card (a public extract of the full corpus). The public extract is 600B tokens.

https://huggingface.co/api/datasets/tiiuae/falcon-refinedweb recorded 2026-09-29
The Hub API record for "tiiuae/falcon-refinedweb" lists the tag "license:odc-by". The same record reports gated as false (the JSON quote characters prevent quoting that stretch), so the download is ungated.
https://huggingface.co/datasets/tiiuae/falcon-refinedweb recorded 2026-09-29
The dataset page renders the Falcon RefinedWeb card, which says "This public extract of RefinedWeb contains about 1B instances".
https://huggingface.co/datasets/tiiuae/falcon-refinedweb/raw/main/README.md recorded 2026-09-29
The raw dataset card is present and states "Falcon RefinedWeb is a massive English web dataset built by [TII](https://www.tii.ae) and released under an ODC-By 1.0 license." and "This public extract should contain 500-650GT depending on the tokenizer you use".

Adoption

3 high confidence

3.0

Adoption is measured by Hugging Face downloads of the dataset over the trailing month, the only usage channel on record for RefinedWeb.

https://huggingface.co/api/datasets/tiiuae/falcon-refinedweb recorded 2026-08-13
93,929 downloads in the trailing 30 days for tiiuae/falcon-refinedweb

Capability

3 high confidence

3.0

Filtered web alone beat The Pile and was the best legacy corpus in DCLM's tests; powered Falcon (2023-24) but is now a component inside newer pipelines rather than a standalone default.

https://arxiv.org/abs/2306.01116 recorded 2026-08-13
RefinedWeb: filtered/deduplicated web alone outperforms models trained on The Pile
https://arxiv.org/pdf/2406.11794 recorded 2026-09-11
DCLM paper full text. The legacy-corpus ranking is in the body: Table 2 evaluates C4, RefinedWeb, RedPajama and Dolma-V1, "we find that RefinedWeb performs the best on our CORE and EXTENDED metrics at the 7B-1x scale", and the takeaway is "For DCLM-BASELINE and other experiments, we adopt RefinedWeb's heuristic filters." The arXiv abstract page carries the DCLM-Baseline headline and no corpus ranking.

Verified 2026-09-27, except some scores, still awaiting re-confirmation