lmms-eval

license

Apache-2.0/MIT(OSI

split license

MIT for the pipeline code inherited from lm-evaluation-harness, Apache-2.0 for the added code in lmms_eval/tasks and lmms_eval/models. Both parts OSI, so the most restrictive part is still osi.)

source

public(GitHub)

core-gated

ungated

GitHub's classifier reports NOASSERTION because the LICENSE file is a composite, but the body read this session is two stock OSI texts with a scope note each: "For the main pipeline structure-related code, we maintain the original license provided with lm-evaluation-harness, which is the MIT License", then "For the multimodal models and datasets that we have added (defined as code in the lmms_eval/tasks and lmms_eval/models folders), we apply the Apache License." Both parts resolve to the osi tier — the rubric already names Apache-2.0/MIT verbatim — so the compound governs at osi. Fully public, and the 23KB README documents no pricing, enterprise edition, paid tier or license key.

• https://api.github.com/repos/EvolvingLMMs-Lab/lmms-eval/license recorded 2026-09-01

  LICENSE body is a split license: the MIT text (Copyright (c) 2025 LMMs-Lab) for the harness-derived pipeline code, then the Apache-2.0 notice for code in lmms_eval/tasks and lmms_eval/models. GitHub reports spdx_id NOASSERTION for the composite; both component licenses are OSI.

• https://raw.githubusercontent.com/EvolvingLMMs-Lab/lmms-eval/main/README.md recorded 2026-09-01

  README documents installing editable from a clone or from git and running the harness locally against local weights or APIs; all tasks ship in lmms_eval/tasks and models in lmms_eval/models. Nothing in the 23KB mentions pricing, a paid tier, an enterprise edition or a license key.

The de-facto standard multimodal eval harness: lm-evaluation-harness's own README directs multimodal evaluation to it by name, model teams cite it for reported LMM numbers, and the repo carries 4,386 stars with releases through v0.7 (Feb 2026). The PyPI package (lmms-eval, backlink-verified) is genuinely this project but is not the measure used: it sees 7,115 downloads a month against a project whose documented install path is a git clone, so banding on it would read a minority channel rather than the project. Same shape and same remedy as opencompass in this category.

• https://api.github.com/repos/EvolvingLMMs-Lab/lmms-eval recorded 2026-09-01

  stargazers_count = 4,386, forks 648, pushed_at 2026-09-01, active issue flow

• https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md recorded 2026-09-01

  EleutherAI's README suggests users "check out lmms-eval, a wonderful project originally forking off of the lm-evaluation-harness, for a broader range of multimodal tasks, models, and features" — third-party standing from the category anchor itself.

• https://pypistats.org/api/packages/lmms-eval/recent recorded 2026-09-01

  last_month = 7,115 downloads of the lmms-eval package — recorded to document why the registry channel was NOT banded on, not as the level's basis

The widest multimodal task and model coverage in the category, and the harness the category anchor's own README recommends for multimodal work — but scoped to LMMs rather than being the cross-domain standardization reference lm-evaluation-harness is, and without the leaderboard-backend role. Sits beside opencompass at 4.

• https://raw.githubusercontent.com/EvolvingLMMs-Lab/lmms-eval/main/README.md recorded 2026-09-01

  100+ tasks and 30+ models linked from the header; v0.7 release notes list agentic task evaluation, HTTP eval server, audio expansion, CI and paired t-test support