lmms-eval

license

Apache-2.0/MIT(OSI

split license

MIT for the pipeline code inherited from lm-evaluation-harness, Apache-2.0 for the added code in lmms_eval/tasks and lmms_eval/models. Both parts OSI, so the most restrictive part is still osi.)

source

public(GitHub)

core features withheld

no

GitHub's license classifier reports NOASSERTION because the LICENSE file mixes two licenses: MIT for the pipeline code inherited from lm-evaluation-harness, and Apache-2.0 for the multimodal code added under lmms_eval/tasks and lmms_eval/models. Both are OSI licenses, the repository and README are fully public, and neither offers a paid tier, enterprise edition or license key.

• https://api.github.com/repos/EvolvingLMMs-Lab/lmms-eval/license recorded 2026-09-01

  LICENSE body is a split license: the MIT text (Copyright (c) 2025 LMMs-Lab) for the harness-derived pipeline code, then the Apache-2.0 notice for code in lmms_eval/tasks and lmms_eval/models. GitHub reports spdx_id NOASSERTION for the composite; both component licenses are OSI.

• https://raw.githubusercontent.com/EvolvingLMMs-Lab/lmms-eval/main/LICENSE recorded 2026-10-07

  The LICENSE body, read rather than the classifier's label. It opens "# For the main pipeline structure-related code, we maintain the original license provided with lm-evaluation-harness, which is the MIT License." followed by the MIT text under "Copyright (c) 2025 LMMs-Lab", and continues with Apache-2.0 terms for the rest. That two-licence split in one file is what GitHub's classifier cannot match, and it is why the API reports NOASSERTION.

• https://raw.githubusercontent.com/EvolvingLMMs-Lab/lmms-eval/main/README.md recorded 2026-10-07

  README documents installing from a clone ("git clone https://github.com/EvolvingLMMs-Lab/lmms-eval") or from git and running the harness locally against local weights or APIs. Nothing in it mentions pricing, a paid tier, an enterprise edition or a license key.

lm-evaluation-harness's README recommends lmms-eval by name for multimodal evaluation, and model teams cite it when reporting LMM benchmark numbers, making it the de-facto standard. Its PyPI package is genuine but not this reading's basis: the documented install path is a git clone, so banding on downloads would read a minority channel rather than the project, the same shape as OpenCompass.

• https://api.github.com/repos/EvolvingLMMs-Lab/lmms-eval recorded 2026-09-01

  stargazers_count = 4,386, forks 648, pushed_at 2026-09-01, active issue flow

• https://pypistats.org/api/packages/lmms-eval/recent recorded 2026-09-01

  last_month = 7,115 downloads of the lmms-eval package.

• https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/README.md recorded 2026-09-01

  EleutherAI's README for lm-evaluation-harness suggests users "check out lmms-eval, a wonderful project originally forking off of the lm-evaluation-harness, for a broader range of multimodal tasks, models, and features."

lmms-eval offers the widest multimodal task and model coverage in the category, and lm-evaluation-harness's own README recommends it by name for multimodal work. It is scoped to multimodal models rather than serving as the cross-domain standardization reference lm-evaluation-harness provides, and it lacks a leaderboard-backend role, which places it alongside OpenCompass.

• https://raw.githubusercontent.com/EvolvingLMMs-Lab/lmms-eval/main/README.md recorded 2026-10-07

  Header links "100+ Tasks" and "30+ Models"; release notes list "agentic task evaluation" for v0.7 and, for v0.6, a "Standalone HTTP eval server, ~7.5x throughput over v0.5, statistically grounded results (CI, paired t-test)", with async serving and video I/O optimizations named among its principles.