VLMEvalKit
OpenCompass CommunityEvaluation toolkit for large vision-language models that runs 220+ models over 80+ multimodal benchmarks with one command and generation-based scoring. It produces the numbers behind the OpenVLM Leaderboard, the reference leaderboard for open multimodal models, and is installed from source rather than a package registry.
Kept as a separate product from OpenCompass, with its own repository, scope and OpenVLM Leaderboard.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public(GitHub)
- core features withheld
- no
VLMEvalKit carries the stock Apache-2.0 text, with no appended terms, under a VLMEvalKit Authors copyright. It installs from a git clone with `pip install -e .`, and the README names no pricing, paid tier, enterprise edition or license key - the only commercial references are the commercial model APIs it can evaluate.
- https://api.github.com/repos/open-compass/VLMEvalKit/license recorded 2026-10-06
GitHub's license endpoint reports spdx_id Apache-2.0 and returns the stock Apache-2.0 text (Copyright 2023 VLMEvalKit Authors) with no appended terms
- https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/README.md recorded 2026-10-06
open-source evaluation toolkit, one-command evaluation of LVLMs; all evaluation code, model configs and benchmark adapters ship in the repo; free HF leaderboards beside it; no pricing, paid tier or license key
Adoption
3 medium confidenceVLMEvalKit is the toolkit behind the OpenVLM Leaderboard and OpenVLM Video Leaderboard, producing results for hundreds of vision-language models and serving as a backend for OpenCompass and EvalScope. It has no download channel to measure - the package was never published to PyPI and installs from a git clone - so this reading follows its leaderboard role rather than a star count.
- https://api.github.com/repos/open-compass/VLMEvalKit recorded 2026-09-01
stargazers_count = 4,370, pushed_at 2026-08-31, description "support 220+ LMMs, 80+ benchmarks"
- https://pypi.org/simple/vlmeval/ recorded 2026-09-01
404 — no package named vlmeval exists on PyPI, so no download channel is available to count
- https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/README.md recorded 2026-09-01
header links to the OpenVLM Leaderboard and OpenVLM Video Leaderboard HF spaces and the OpenVLMRecords dataset of official evaluation records
Capability
4 medium confidenceThe multimodal peer of its own org's flagship: comparable breadth (220+ models, 80+ benchmarks against opencompass's 70+ datasets), the same leaderboard-producing role (OpenVLM against CompassRank), scoped to vision-language models rather than the cross-domain standardization reference — so at opencompass's 4, one below lm-evaluation-harness.
- https://api.github.com/repos/open-compass/VLMEvalKit recorded 2026-09-01
repo description "support 220+ LMMs, 80+ benchmarks"
- https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/README.md recorded 2026-09-01
one-command generation-based evaluation of LVLMs; exact matching and LLM-based answer extraction; supported-benchmark and supported-model lists; official leaderboards downloadable
Verified 2026-10-06, except some scores, still awaiting re-confirmation