RoBERTa
MetaRoBERTa is Facebook AI's retraining of BERT: the same encoder-only architecture, trained longer, on larger batches and on about 160GB of English text, without BERT's next-sentence task and with masks that change on every pass over the data. Released in 2019 through the fairseq toolkit in Base (125M parameters) and Large (355M) sizes, it is built to be fine-tuned for classification, tagging and question answering rather than to generate text.
XLM-RoBERTa, the multilingual model trained with the same recipe, was released separately and is not counted here.
Openness
4 high confidence- weights
- open (Base and Large checkpoints, ungated on Hugging Face and on fairseq's download server)
- data
- documented-not-released (BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories, about 160GB, are named, but the assembled corpus was never released)
- code
- open (fairseq ships the model and a recipe for pretraining RoBERTa from scratch)
- license
- MIT (OSI)
RoBERTa's checkpoints and the fairseq code that trained them are published under the MIT license, with a written recipe for pretraining from scratch. The five corpora it trained on are named, but the 160GB mix was never released, so the original run cannot be rebuilt exactly.
- https://huggingface.co/api/models/FacebookAI/roberta-base recorded 2026-11-12
Model API for FacebookAI/roberta-base: `gated: false`, cardData.license `mit`, with model.safetensors among the files.
- https://huggingface.co/api/models/FacebookAI/roberta-large recorded 2026-11-12
Model API for FacebookAI/roberta-large: `gated: false`, cardData.license `mit`, with model.safetensors among the files.
- https://huggingface.co/FacebookAI/roberta-base/raw/main/README.md recorded 2026-11-12
Model card: RoBERTa "was pretrained on the reunion of five datasets" (BookCorpus, English Wikipedia, CC-News, OpenWebText and Stories) that "together ... weigh 160GB of text"; no download of the assembled corpus is offered.
- https://raw.githubusercontent.com/facebookresearch/fairseq/main/examples/roberta/README.md recorded 2026-11-12
fairseq's RoBERTa README lists roberta.base (125M) and roberta.large (355M) as tar.gz downloads and links a "tutorial for pretraining RoBERTa using your own data".
- https://raw.githubusercontent.com/facebookresearch/fairseq/main/examples/roberta/README.pretraining.md recorded 2026-11-12
"Pretraining RoBERTa using your own data": preprocessing with the GPT-2 BPE, binarizing with fairseq-preprocess and training RoBERTa base with fairseq-train.
- https://raw.githubusercontent.com/facebookresearch/fairseq/main/LICENSE recorded 2026-11-12
LICENSE body is the MIT License, Copyright (c) Facebook, Inc. and its affiliates.
Adoption
5 high confidenceHugging Face downloads summed across the Base and Large checkpoints, split fairly evenly between them.
- https://huggingface.co/api/models/FacebookAI/roberta-base recorded 2026-11-12
"downloads":5879284 for FacebookAI/roberta-base, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/FacebookAI/roberta-large recorded 2026-11-12
"downloads":6402732 for FacebookAI/roberta-large, the trailing 30-day count the Hub API reports.
Capability
1 high confidenceRoBERTa showed that BERT had been undertrained and, retrained carefully, matched or beat every model published after it in 2019, but it is an encoder of at most 355M parameters that cannot generate text. It sits a step below Comma, a 7B model that only matches 2023-era peers.
- https://arxiv.org/abs/1907.11692 recorded 2026-11-12
Abstract: "We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD."
- https://raw.githubusercontent.com/facebookresearch/fairseq/main/examples/roberta/README.md recorded 2026-11-12
README results for roberta.large: GLUE dev MNLI 90.2, SST-2 96.4, CoLA 68.0; SQuAD 2.0 EM/F1 86.5/89.4; RACE accuracy 83.2; parameter counts 125M and 355M.
Verified 2026-11-12