Model components / Evaluation code

EvalScope

One-stop evaluation framework from the ModelScope community that runs LLM, VLM, embedding and AIGC benchmarks and inference stress tests from a single CLI. It drives OpenCompass and VLMEvalKit as pluggable backends and adds the performance benchmarking side that plain accuracy harnesses leave out.

Distinct from the corpus's swift (ms-swift), the same community's training framework. Verified 2026-09-01 via the modelscope/evalscope repository, its license endpoint, its README, the PyPI JSON backlink and pypistats.

Openness

5 high confidence

5.0

license: Apache-2.0(OSI)
source: public(GitHub)
core-gated: ungated

Apache-2.0 and fully public. The LICENSE body is the stock Apache-2.0 text under "Copyright 2022-2023 Alibaba ModelScope" with no appended terms. `pip install 'evalscope[service]'` in the README is an ordinary optional-dependency extra, not a paid gate; no pricing, enterprise edition or license key appears in the 28KB README.

https://api.github.com/repos/modelscope/evalscope/license recorded 2026-09-01
GitHub's license endpoint reports spdx_id Apache-2.0 and returns the stock Apache-2.0 text (Copyright 2022-2023 Alibaba ModelScope) with no appended terms
https://raw.githubusercontent.com/modelscope/evalscope/main/README.md recorded 2026-09-01
README installs with `pip install evalscope` and runs evaluation and perf stress tests locally; backends (OpenCompass, VLMEvalKit, RAGEval) ship as extras. No pricing, paid tier, enterprise edition or license key in the README.

Adoption

2 high confidence

2.0

The evalscope package draws 75,444 PyPI downloads in the trailing 30 days, which bands at level 2 (10K-100K) on the software scale. Unlike opencompass and lmms-eval, `pip install evalscope` is the README's front-and-center install path, so the registry is the product's primary channel and the count is the instrument. Package identity backlink-verified (Homepage -> modelscope/evalscope, ModelScope team publisher).

https://pypistats.org/api/packages/evalscope/recent recorded 2026-09-01
last_month = 75,444 downloads of the evalscope package (last_week 5,981)
https://pypi.org/pypi/evalscope/json recorded 2026-09-01
project_urls.Homepage = github.com/modelscope/evalscope, author ModelScope team <contact@modelscope.cn>, version 1.11.1 uploaded 2026-08-31 — the package is the product, not a name collision

Capability

3 medium confidence

3.0

Broad model-type coverage and a performance-testing side the accuracy harnesses lack, but much of its benchmark depth is delegated — OpenCompass and VLMEvalKit run as its backends — and it carries no leaderboard or standardization role of its own, so it sits one below opencompass alongside bigcodebench.

https://raw.githubusercontent.com/modelscope/evalscope/main/README.md recorded 2026-09-01
key-features list — built-in benchmarks, multi-modal and multi-domain support, multi-backend integration (OpenCompass, VLMEvalKit, RAGEval), agent evaluation mode, inference stress testing

Verified 2026-09-01