EvalScope
ModelScope (Alibaba)One-stop evaluation framework from the ModelScope community that runs LLM, VLM, embedding and AIGC benchmarks and inference stress tests from a single CLI. It drives OpenCompass and VLMEvalKit as pluggable backends and adds the performance benchmarking side that plain accuracy harnesses leave out.
Distinct from the corpus's swift (ms-swift), the same community's training framework.
Openness
5 high confidence- license
- Apache-2.0(OSI)
- source
- public(GitHub)
- core features withheld
- no
EvalScope carries the stock Apache-2.0 text, with no appended terms, under a ModelScope copyright. The README's optional `[service]` install extra is an ordinary dependency rather than a paid gate, and it names no pricing, enterprise edition or license key.
- https://api.github.com/repos/modelscope/evalscope/license recorded 2026-09-01
GitHub's license endpoint reports spdx_id Apache-2.0 and returns the stock Apache-2.0 text (Copyright 2022-2023 Alibaba ModelScope) with no appended terms
- https://raw.githubusercontent.com/modelscope/evalscope/main/LICENSE recorded 2026-10-07
LICENSE body, read in full: "Copyright 2022-2023 Alibaba ModelScope. All rights reserved." above the stock Apache License 2.0 text, with the appendix notice filled in and no appended terms.
- https://raw.githubusercontent.com/modelscope/evalscope/main/README.md recorded 2026-10-07
README installs with "pip install evalscope" and runs evaluation and stress tests locally; backends ship as integrations ("Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval"). No pricing, paid tier, enterprise edition or license key in the README.
Adoption
2 high confidenceEvalScope's adoption is measured from its PyPI downloads in the trailing 30 days. Unlike opencompass and lmms-eval, pip install evalscope is the README's front-and-center install path, so PyPI is the closest usage signal for the package rather than a stand-in for a different one.
- https://pypi.org/pypi/evalscope/json recorded 2026-09-01
project_urls.Homepage = github.com/modelscope/evalscope, author ModelScope team <contact@modelscope.cn>, version 1.11.1 uploaded 2026-08-31 — the package is the product, not a name collision
- https://pypistats.org/api/packages/evalscope/recent recorded 2026-09-01
last_month = 75,444 downloads of the evalscope package (last_week 5,981)
Capability
3 medium confidenceBroad model-type coverage and a performance-testing side the accuracy harnesses lack, but much of its benchmark depth is delegated — OpenCompass and VLMEvalKit run as its backends — and it carries no leaderboard or standardization role of its own, so it sits one below opencompass alongside bigcodebench.
- https://raw.githubusercontent.com/modelscope/evalscope/main/README.md recorded 2026-10-07
Key features: "Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval", an "Agent Evaluation Mode" that drives benchmarks inside a multi-turn AgentLoop, and "Provides powerful model service stress testing tools", alongside built-in benchmarks across model types.
Verified 2026-09-24