Back to Gap Map Model components / Base / pretrained models

ModernBERT

Answer.AI
open weights / Overall score: 1.9

ModernBERT is an encoder-only language model from Answer.AI and LightOn that brings BERT's design up to date, with rotary position embeddings, alternating local and global attention and a native context of 8,192 tokens. It was pretrained on 2 trillion tokens of mostly English text and code and comes in Base (149M parameters) and Large (395M) sizes. Like BERT, it is meant to be fine-tuned for classification, retrieval and similar tasks rather than to generate text.

Openness

3 high confidence
3.0
weights
open (Base and Large checkpoints, ungated on Hugging Face)
data
closed (2 trillion tokens of web documents, code and scientific literature, described only in those terms, with no sources or mixture named)
code
open (pretraining and evaluation code in AnswerDotAI/ModernBERT)
checkpoints
open (intermediate training checkpoints for both sizes on Hugging Face)
license
Apache-2.0 (OSI)

ModernBERT's weights, its training code and its intermediate checkpoints are all published under Apache-2.0. The training data is described only as web documents, code and scientific literature, without the sources or the mixture, so the corpus cannot be rebuilt.

Adoption

4 high confidence
4.0

Hugging Face downloads summed across the Base and Large checkpoints, most of them for Base.

Capability

1 high confidence
1.0

ModernBERT leads the older encoders it was tested against on most retrieval and code tasks, and its Base model is the first to pass DeBERTaV3 Base on GLUE, but it is an encoder of at most 395M parameters that cannot generate text. It sits a step below Comma, a 7B model that only matches 2023-era peers.

Verified 2026-11-12