Back to Gap Map Model components / Base / pretrained models

DeBERTa

Microsoft
open weights / Overall score: 1.9

DeBERTa is Microsoft's encoder-only language model, which keeps each word's content and position apart inside attention. Its third version, DeBERTaV3, replaces masked-word prediction with ELECTRA-style replaced-token detection and comes in XSmall to Large sizes, plus mDeBERTaV3 Base, a multilingual model covering 102 languages. Earlier DeBERTa and DeBERTaV2 checkpoints remain available. It is built to be fine-tuned for classification and question answering rather than to generate text.

Openness

4 high confidence
4.0
weights
open (DeBERTaV3 XSmall, Small, Base and Large and mDeBERTaV3 Base, ungated on Hugging Face)
data
documented-not-released (the English models use the 160GB mix of Wikipedia, BookCorpus, OpenWebText, Stories and CC-News shared with DeBERTa and RoBERTa, named but never released, while mDeBERTaV3 uses the public CC100)
code
open (rtd.sh in microsoft/DeBERTa pretrains DeBERTaV3 models with replaced-token detection)
license
MIT (OSI)

DeBERTaV3's checkpoints and its pretraining scripts are published under the MIT license. The English models were trained on the same 160GB mix of encyclopedia, book, news and web text as RoBERTa, which is described in the papers but was never released as a corpus.

Adoption

4 high confidence
4.0

Hugging Face downloads summed across Microsoft's DeBERTa checkpoints, most of them for mDeBERTaV3 Base and the English DeBERTaV3 Base.

Capability

1 high confidence
1.0

DeBERTaV3 Large posted the best GLUE result of any encoder its size when it was released, and the multilingual Base model led XNLI, but these are encoders that cannot generate text. It sits a step below Comma, a 7B model that only matches 2023-era peers.

Verified 2026-11-12