VLMEvalKit
OpenCompass CommunityEvaluation toolkit for large vision-language models that runs 220+ models over 80+ multimodal benchmarks with one command and generation-based scoring. It produces the numbers behind the OpenVLM Leaderboard, the reference leaderboard for open multimodal models, and is installed from source rather than a package registry.
Kept as a separate product from opencompass: separate repo, separate scope and its own OpenVLM Leaderboard, integrated by OpenCompass as a backend. No PyPI package exists (vlmeval is the import name only; 404 on the JSON API and the simple index). Verified 2026-09-01 via the open-compass/VLMEvalKit repository, its license endpoint, its README and Quickstart.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public(GitHub)
- core-gated
- ungated
Apache-2.0 and fully public. The LICENSE body is the stock Apache-2.0 text under "Copyright 2023 VLMEvalKit Authors" with no appended terms. Install is git clone + `pip install -e .`; the README documents no pricing, paid tier, enterprise edition or license key — the only commercial references are the commercial model APIs it can evaluate.
- https://api.github.com/repos/open-compass/VLMEvalKit/license recorded 2026-09-01
GitHub's license endpoint reports spdx_id Apache-2.0 and returns the stock Apache-2.0 text (Copyright 2023 VLMEvalKit Authors) with no appended terms
- https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/README.md recorded 2026-09-01
open-source evaluation toolkit, one-command evaluation of LVLMs; all evaluation code, model configs and benchmark adapters ship in the repo; free HF leaderboards beside it; no pricing, paid tier or license key
Adoption
3 medium confidenceThe toolkit behind the OpenVLM Leaderboard and OpenVLM Video Leaderboard (Hugging Face spaces under the opencompass org), with 4,370 stars and results published for 220+ models — the reference evaluation surface for open multimodal models, and integrated as a backend by both OpenCompass and EvalScope. No download channel exists to count: there is no package (vlmeval is the import name only, PyPI 404 this session) and the install path is a git clone, so the level rests on reported traction rather than stars alone, the same shape as opencompass's own record. Conservative alternative if the writer prefers the mechanical instrument: stars_fallback at level 2 (1K-10K stars).
- https://api.github.com/repos/open-compass/VLMEvalKit recorded 2026-09-01
stargazers_count = 4,370, pushed_at 2026-08-31, description "support 220+ LMMs, 80+ benchmarks"
- https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/README.md recorded 2026-09-01
header links to the OpenVLM Leaderboard and OpenVLM Video Leaderboard HF spaces and the OpenVLMRecords dataset of official evaluation records
- https://pypi.org/simple/vlmeval/ recorded 2026-09-01
404 — no package named vlmeval exists on PyPI, so no download channel is available to count
Capability
4 medium confidenceThe multimodal peer of its own org's flagship: comparable breadth (220+ models, 80+ benchmarks against opencompass's 70+ datasets), the same leaderboard-producing role (OpenVLM against CompassRank), scoped to vision-language models rather than the cross-domain standardization reference — so at opencompass's 4, one below lm-evaluation-harness.
- https://raw.githubusercontent.com/open-compass/VLMEvalKit/main/README.md recorded 2026-09-01
one-command generation-based evaluation of LVLMs; exact matching and LLM-based answer extraction; supported-benchmark and supported-model lists; official leaderboards downloadable
- https://api.github.com/repos/open-compass/VLMEvalKit recorded 2026-09-01
repo description "support 220+ LMMs, 80+ benchmarks"
Verified 2026-09-01