AI Potluck
Contribute
Back to Gap Map Model components / Embeddings & retrieval models

ColPali

ILLUIN Technology
open weights / Overall score: 4.0(strong)

ColPali applies the ColBERT late-interaction recipe to page screenshots: a vision-language model embeds each image patch of a rendered document page, a query is embedded token by token, and MaxSim scores the two multi-vector sets. That removes the OCR-and-chunk pipeline from document retrieval entirely. The line has grown past the original PaliGemma-based ColPali into ColQwen2, ColQwen2.5, the sub-1B ColSmol checkpoints and ColQwen-Omni, all released from the vidore Hub org with the shared colpali-engine training and inference code.

Published by ILLUIN Technology - code at github.com/illuin-tech/colpali, weights and the ViDoRe benchmark under the `vidore` Hugging Face org - not by a company called ViDoRe. Verified 2026-09-11 via the colpali-v1.3 and colqwen2-v1.0 model cards, the colpali repository README and the Hugging Face API.

Openness

3 high confidence
3.0
weights
open(45 ungated checkpoints on the vidore org, covering ColPali, ColQwen2/2.5, ColSmol and ColQwen-Omni)
data
open(vidore/colpali_train_set, the 118,195-pair training set, published ungated on the Hub)
code
open(colpali-engine[train] ships scripts/train/train_colbert.py and the per-model training configs used for the released checkpoints)
license
Gemma-License(non-OSI, resolved most-restrictive across the distributed SKUs - the PaliGemma-backed colpali-v1.2/v1.3-hf and -merged weights carry the Gemma terms)+MIT(OSI, the ColPali adapters and, with Apache-2.0 Qwen2-VL backbones, the ColQwen SKUs)

Data and code are as open as the ladder's top rung asks - a published training set and a real training pipeline - but the licence rung fires first. Some distributed SKUs merge PaliGemma weights and inherit the Gemma terms, and multi_sku_rule resolves the family on the most restrictive licence actually shipped, which caps this at 3. The ColQwen line alone would reach higher.

Adoption

4 high confidence
4.0

1,255,462 downloads in the trailing 30 days summed across the 45 repos in the vidore org (colqwen2-v1.0 471,319; colqwen2.5-v0.2 328,936; colSmol-500M 314,435; the ColPali checkpoints themselves a long tail behind their ColQwen successors).

Capability

4 high confidence
4.0

Rung 4, competitive frontier: the reference family for the whole task, current-generation, with published numbers within a few points of the leader rather than at it. Placed one rung above ColBERT on the rung definitions, the late-interaction retriever it descends from - ColPali carries the same MaxSim mechanism into a modality where it is still the reference family, while ColBERT's text checkpoint has been overtaken. ColModernVBERT is placed one below this in turn, off the same ViDoRe table. Placed against the rung definition rather than a peer: no single measurement reports this product and a category peer together - ViDoRe against ColBERT's BEIR and LoTTE - and rule (d) forbids an edge without one.

Verified 2026-09-11