Back to Gap Map Model components / Base / pretrained models

RoBERTa

Meta
open weights / Overall score: 2.2

RoBERTa is Facebook AI's retraining of BERT: the same encoder-only architecture, trained longer, on larger batches and on about 160GB of English text, without BERT's next-sentence task and with masks that change on every pass over the data. Released in 2019 through the fairseq toolkit in Base (125M parameters) and Large (355M) sizes, it is built to be fine-tuned for classification, tagging and question answering rather than to generate text.

XLM-RoBERTa, the multilingual model trained with the same recipe, was released separately and is not counted here.

Openness

4 high confidence
4.0
weights
open (Base and Large checkpoints, ungated on Hugging Face and on fairseq's download server)
data
documented-not-released (BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories, about 160GB, are named, but the assembled corpus was never released)
code
open (fairseq ships the model and a recipe for pretraining RoBERTa from scratch)
license
MIT (OSI)

RoBERTa's checkpoints and the fairseq code that trained them are published under the MIT license, with a written recipe for pretraining from scratch. The five corpora it trained on are named, but the 160GB mix was never released, so the original run cannot be rebuilt exactly.

Adoption

5 high confidence
5.0

Hugging Face downloads summed across the Base and Large checkpoints, split fairly evenly between them.

Capability

1 high confidence
1.0

RoBERTa showed that BERT had been undertrained and, retrained carefully, matched or beat every model published after it in 2019, but it is an encoder of at most 355M parameters that cannot generate text. It sits a step below Comma, a 7B model that only matches 2023-era peers.

Verified 2026-11-12