ModernBERT
Answer.AIModernBERT is an encoder-only language model from Answer.AI and LightOn that brings BERT's design up to date, with rotary position embeddings, alternating local and global attention and a native context of 8,192 tokens. It was pretrained on 2 trillion tokens of mostly English text and code and comes in Base (149M parameters) and Large (395M) sizes. Like BERT, it is meant to be fine-tuned for classification, retrieval and similar tasks rather than to generate text.
Openness
3 high confidence- weights
- open (Base and Large checkpoints, ungated on Hugging Face)
- data
- closed (2 trillion tokens of web documents, code and scientific literature, described only in those terms, with no sources or mixture named)
- code
- open (pretraining and evaluation code in AnswerDotAI/ModernBERT)
- checkpoints
- open (intermediate training checkpoints for both sizes on Hugging Face)
- license
- Apache-2.0 (OSI)
ModernBERT's weights, its training code and its intermediate checkpoints are all published under Apache-2.0. The training data is described only as web documents, code and scientific literature, without the sources or the mixture, so the corpus cannot be rebuilt.
- https://arxiv.org/html/2412.13663v2 recorded 2026-11-12
Paper section 2.2.1: "trained on 2 trillion tokens of primarily English data from a variety of data sources, including web documents, code, and scientific literature", with no source list or proportions; it says the release includes "all intermediate training checkpoints".
- https://huggingface.co/answerdotai/ModernBERT-base/raw/main/README.md recorded 2026-11-12
Model card: "We release the ModernBERT model architectures, model weights, training codebase under the Apache 2.0 license"; training data given only as "2 trillion tokens of English text and code".
- https://huggingface.co/api/models/answerdotai/ModernBERT-base recorded 2026-11-12
Model API for answerdotai/ModernBERT-base: `gated: false`, cardData.license `apache-2.0`, with model.safetensors among the files.
- https://huggingface.co/api/models/answerdotai/ModernBERT-base-training-checkpoints recorded 2026-11-12
Model API for answerdotai/ModernBERT-base-training-checkpoints: ungated, 120 files of intermediate .pt checkpoints such as context-extension/ep0-ba12000-rank0.pt.
- https://huggingface.co/api/models/answerdotai/ModernBERT-large recorded 2026-11-12
Model API for answerdotai/ModernBERT-large: `gated: false`, cardData.license `apache-2.0`, with model.safetensors among the files.
- https://huggingface.co/api/models/answerdotai/ModernBERT-large-training-checkpoints recorded 2026-11-12
Model API for answerdotai/ModernBERT-large-training-checkpoints: ungated, 99 files of intermediate .pt checkpoints.
- https://raw.githubusercontent.com/AnswerDotAI/ModernBERT/main/LICENSE recorded 2026-11-12
LICENSE body is the Apache License, Version 2.0.
- https://raw.githubusercontent.com/AnswerDotAI/ModernBERT/main/README.md recorded 2026-11-12
Repository README: "All code used in this repository is the code used as part of our experiments for both pre-training and GLUE evaluations", with training launched by `composer main.py yamls/main/modernbert-base.yaml`.
Adoption
4 high confidenceHugging Face downloads summed across the Base and Large checkpoints, most of them for Base.
- https://huggingface.co/api/models/answerdotai/ModernBERT-base recorded 2026-11-12
"downloads":3011393 for answerdotai/ModernBERT-base, the trailing 30-day count the Hub API reports.
- https://huggingface.co/api/models/answerdotai/ModernBERT-large recorded 2026-11-12
"downloads":583958 for answerdotai/ModernBERT-large, the trailing 30-day count the Hub API reports.
Capability
1 high confidenceModernBERT leads the older encoders it was tested against on most retrieval and code tasks, and its Base model is the first to pass DeBERTaV3 Base on GLUE, but it is an encoder of at most 395M parameters that cannot generate text. It sits a step below Comma, a 7B model that only matches 2023-era peers.
- https://arxiv.org/abs/2412.13663 recorded 2026-11-12
Abstract: ModernBERT models "exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval".
- https://arxiv.org/html/2412.13663v2 recorded 2026-11-12
Paper: "On GLUE, ModernBERT-base is the first encoder to beat DeBERTaV3-base since its release in 2021".
- https://huggingface.co/answerdotai/ModernBERT-base/raw/main/README.md recorded 2026-11-12
Model card Table 1: ModernBERT Base GLUE 88.4, BEIR 41.6 (DPR) and 51.3 (ColBERT), CSN 56.4; Large GLUE 90.4, BEIR 44.0 and 52.4, CSN 59.5; sizes listed as 149 million and 395 million parameters.
Verified 2026-11-12