DeBERTa
MicrosoftDeBERTa is Microsoft's encoder-only language model, which keeps each word's content and position apart inside attention. Its third version, DeBERTaV3, replaces masked-word prediction with ELECTRA-style replaced-token detection and comes in XSmall to Large sizes, plus mDeBERTaV3 Base, a multilingual model covering 102 languages. Earlier DeBERTa and DeBERTaV2 checkpoints remain available. It is built to be fine-tuned for classification and question answering rather than to generate text.
Openness
4 high confidence- weights
- open (DeBERTaV3 XSmall, Small, Base and Large and mDeBERTaV3 Base, ungated on Hugging Face)
- data
- documented-not-released (the English models use the 160GB mix of Wikipedia, BookCorpus, OpenWebText, Stories and CC-News shared with DeBERTa and RoBERTa, named but never released, while mDeBERTaV3 uses the public CC100)
- code
- open (rtd.sh in microsoft/DeBERTa pretrains DeBERTaV3 models with replaced-token detection)
- license
- MIT (OSI)
DeBERTaV3's checkpoints and its pretraining scripts are published under the MIT license. The English models were trained on the same 160GB mix of encyclopedia, book, news and web text as RoBERTa, which is described in the papers but was never released as a corpus.
- https://arxiv.org/html/2006.03654 recorded 2026-11-12
DeBERTa paper, Table 7 and Appendix A.2: pretraining data named as Wikipedia and BookCorpus (16GB), OpenWebText (38GB), Stories (31GB) and CC-News (76GB), with no corpus download.
- https://arxiv.org/html/2111.09543 recorded 2026-11-12
DeBERTaV3 paper: "We train those models with 160GB data, which is the same as DeBERTaV2 and RoBERTa"; mDeBERTaV3 Base is trained "with the 2.5T CC100 multi-lingual dataset which is the same as XLM-R".
- https://huggingface.co/api/models/microsoft/deberta-v3-base recorded 2026-11-12
Model API for microsoft/deberta-v3-base: `gated: false`, cardData.license `mit`, with pytorch_model.bin among the files.
- https://huggingface.co/api/models/microsoft/deberta-v3-large recorded 2026-11-12
Model API for microsoft/deberta-v3-large: `gated: false`, cardData.license `mit`.
- https://huggingface.co/api/models/microsoft/mdeberta-v3-base recorded 2026-11-12
Model API for microsoft/mdeberta-v3-base: `gated: false`, cardData.license `mit`.
- https://huggingface.co/microsoft/deberta-v3-base/raw/main/README.md recorded 2026-11-12
Model card: "This model was trained using the 160GB data as DeBERTa V2", license `mit`.
- https://raw.githubusercontent.com/microsoft/DeBERTa/master/experiments/language_model/README.md recorded 2026-11-12
"Run `rtd.sh` to train a ELECTRA like model using RTD as the pre-training task", with `rtd.sh deberta-v3-base` and `rtd.sh deberta-v3-large` among the documented targets.
- https://raw.githubusercontent.com/microsoft/DeBERTa/master/LICENSE recorded 2026-11-12
LICENSE body is the MIT License, Copyright (c) Microsoft Corporation.
- https://raw.githubusercontent.com/microsoft/DeBERTa/master/README.md recorded 2026-11-12
Repository README: "The code for DeBERTaV3 pre-training and continous training is added", and a model table listing DeBERTa-V3 XSmall to Large and mDeBERTa-V3-Base (102 languages).
Adoption
4 high confidenceHugging Face downloads summed across Microsoft's DeBERTa checkpoints, most of them for mDeBERTaV3 Base and the English DeBERTaV3 Base.
- https://huggingface.co/api/models/microsoft/deberta-base recorded 2026-11-12
"downloads":37791 for microsoft/deberta-base, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/deberta-large recorded 2026-11-12
"downloads":16134 for microsoft/deberta-large, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/deberta-v2-xlarge recorded 2026-11-12
"downloads":25345 for microsoft/deberta-v2-xlarge, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/deberta-v2-xxlarge recorded 2026-11-12
"downloads":4824 for microsoft/deberta-v2-xxlarge, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/deberta-v3-base recorded 2026-11-12
"downloads":2440007 for microsoft/deberta-v3-base, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/deberta-v3-large recorded 2026-11-12
"downloads":1092468 for microsoft/deberta-v3-large, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/deberta-v3-small recorded 2026-11-12
"downloads":617911 for microsoft/deberta-v3-small, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/deberta-v3-xsmall recorded 2026-11-12
"downloads":39058 for microsoft/deberta-v3-xsmall, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/deberta-xlarge recorded 2026-11-12
"downloads":359 for microsoft/deberta-xlarge, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/microsoft/mdeberta-v3-base recorded 2026-11-12
"downloads":5327502 for microsoft/mdeberta-v3-base, the trailing 30-day count the Hub API reports.
Capability
1 high confidenceDeBERTaV3 Large posted the best GLUE result of any encoder its size when it was released, and the multilingual Base model led XNLI, but these are encoders that cannot generate text. It sits a step below Comma, a 7B model that only matches 2023-era peers.
- https://arxiv.org/abs/2111.09543 recorded 2026-11-12
Abstract: "the DeBERTaV3 Large model achieves a 91.37% average score" on GLUE, "setting a new state-of-the-art (SOTA) among the models with a similar structure", and "the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI".
- https://raw.githubusercontent.com/microsoft/DeBERTa/master/README.md recorded 2026-11-12
README model table gives DeBERTa-V3-Large 304M backbone parameters, Base 86M, Small 44M and XSmall 22M.
Verified 2026-11-12