languagebench

license

MIT(OSI)

source

public(GitHub)

core-gated

ungated

MIT and fully public. The LICENSE body is the stock MIT text (copyright 2025 Datenlabor BMZ & GIZ & DFKI) with no appended terms. The evaluation pipeline, the FastAPI backend and the React frontend all ship in the one repository and the Hugging Face Space builds from it; the result tables publish as open datasets. No pricing, paid tier or license key appears anywhere in the repo.

• https://api.github.com/repos/datenlabor-bmz/evals-for-every-language/license recorded 2026-09-10

  GitHub's license endpoint reports spdx_id MIT and returns the stock MIT text (Copyright (c) 2025 Datenlabor BMZ & GIZ & DFKI) with no appended terms

• https://raw.githubusercontent.com/datenlabor-bmz/evals-for-every-language/main/README.md recorded 2026-09-10

  README documents the whole product from the one repo — `uv run evals/main.py` to evaluate, `uv run evals/backend.py` plus the npm frontend to explore, and results/ committed in the tree — with no pricing, paid tier or license key

There is no installable package — the product is a hosted leaderboard plus the pipeline behind it — so the measurable artifacts are the two Hugging Face result datasets the README names as the programmatic access path. They draw 102 downloads between them in the trailing 30 days, which bands at level 1. The other visible counts corroborate the band rather than move it: 21 GitHub stars and 2 forks on the repository, 26 likes on the Space.

• https://huggingface.co/api/datasets/fair-forward/evals-for-every-language-results recorded 2026-09-10

  downloads = 57 in the trailing 30 days on the aggregated results dataset

• https://huggingface.co/api/datasets/fair-forward/evals-for-every-language-results-detailed recorded 2026-09-10

  downloads = 45 in the trailing 30 days on the per-sample results dataset

• https://api.github.com/repos/datenlabor-bmz/evals-for-every-language recorded 2026-09-10

  21 stars, 2 forks, last push 2026-07-28

• https://huggingface.co/api/spaces/fair-forward/languagebench recorded 2026-09-10

  likes = 26, runtime stage RUNNING, and four resolved domains including datenlabor-bmz-ai-language-monitor.hf.space and fair-forward-languagebench.hf.space

The widest language coverage of any harness in the category — up to 200 languages against EuroEval's 30+ — bought by a narrow task surface. Seven borrowed benchmarks at ten samples each, hosted API models only through a single router, no locally-run weights and no installable package, against EuroEval's encoders, decoders and encoder-decoders and its own reproducible dataset-creation scripts. Deep on one axis and thin on the rest in each direction, so level with EuroEval rather than a rung apart, and below the general-purpose harnesses at 4.

• https://arxiv.org/abs/2507.08538 recorded 2026-09-10

  "a comprehensive multilingual benchmark that systematically assesses LLM performance across up to 200 languages, with a particular focus on low-resource languages"; aggregates translation, question answering, math and reasoning over FLORES+, MMLU, GSM8K, TruthfulQA and ARC; "an open-source, auto-updating leaderboard and dashboard"; a global proficiency map and trends over time

• https://raw.githubusercontent.com/datenlabor-bmz/evals-for-every-language/main/notes/system-architecture-diagram.md recorded 2026-09-10

  seven task functions (translation_from/to with BLEU+ChrF, classification, mmlu, arc, truthfulqa, mgsm), a static curated list of ~42 models filtered to a default top 40 through OpenRouter, languages sorted by speaker count with a 1000-language default ceiling, 10 samples per model x language x task, and bootstrap confidence intervals on the aggregate