Back to Gap Map Model components / Base / pretrained models

BERT

Google Research
open weights / Overall score: 2.2

BERT (Bidirectional Encoder Representations from Transformers) is Google's encoder-only language model, pretrained by masking words and predicting the next sentence on English Wikipedia and BookCorpus. Released from 2018 in Base (110M parameters) and Large (340M) sizes, cased and uncased, with later multilingual and Chinese checkpoints, it is built to be fine-tuned for classification, tagging and question answering rather than to generate text.

Openness

4 high confidence
4.0
weights
open (Base, Large, multilingual and Chinese checkpoints, ungated on Hugging Face)
data
documented-not-released (English Wikipedia and BookCorpus are named, but the preprocessed corpus was never released and BookCorpus is no longer distributed)
code
open (create_pretraining_data.py and run_pretraining.py in google-research/bert)
license
Apache-2.0 (OSI)

BERT's checkpoints and its pretraining code, from data generation to the training loop, are published under Apache-2.0. The pretraining corpus is described but was never released, and BookCorpus is no longer offered by its collectors, so the original training run cannot be rebuilt exactly.

Adoption

5 high confidence
5.0

Hugging Face downloads summed across Google's original BERT checkpoints, most of them for bert-base-uncased.

Capability

1 high confidence
1.0

BERT set the state of the art on eleven language-understanding tasks in 2018, but it is an encoder of at most 340M parameters that cannot generate text, and the benchmarks used for today's language models do not apply to it. It sits a step below Comma, a 7B model that only matches 2023-era peers.

  • https://arxiv.org/abs/1810.04805 recorded 2026-11-12

    Abstract: BERT "obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% ..., MultiNLI accuracy to 86.7% ..., SQuAD v1.1 question answering Test F1 to 93.2 ... and SQuAD v2.0 Test F1 to 83.1".

  • https://raw.githubusercontent.com/google-research/bert/master/README.md recorded 2026-11-12

    README lists BERT-Large as "24-layer, 1024-hidden, 16-heads, 340M parameters" and BERT-Base as "12-layer, 768-hidden, 12-heads, 110M parameters".

Verified 2026-11-12