AI Potluck
Contribute
Back to Gap Map Model components / Benchmark / eval datasets

TruthfulQA

Unknown
open / Overall score: 4.0(strong)

817 questions across 38 categories, written to elicit imitative falsehoods - the misconceptions a model picks up from its training text rather than errors of reasoning. It ships generation and multiple-choice configurations and dates from 2021.

Relatively dated and largely saturated by frontier models, though still widely reported.

Openness

5 high confidence
5.0
license
Apache-2.0(OSI/open,redistributable)
access
public(not gated)
datasheet
present(card+creation methodology+citation)
splits
public(817 Qs, two configs, no hidden held-out)

Fully open: Apache-2.0, ungated, redistributable, complete card with creation methodology.

  • https://arxiv.org/abs/2109.07958 recorded 2026-09-29

    The abstract reads "The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics".

  • https://huggingface.co/api/datasets/truthfulqa/truthful_qa recorded 2026-09-29

    The API record lists the tag "license:apache-2.0" and the repo files "generation/validation-00000-of-00001.parquet" and "multiple_choice/validation-00000-of-00001.parquet", with gated false.

  • https://huggingface.co/datasets/truthfulqa/truthful_qa recorded 2026-09-29

    The dataset page carries the tag "license:apache-2.0", a card section titled "Curation Rationale" and one titled "Citation Information", and the card opens "TruthfulQA is a benchmark to measure whether a language model is truthful in generating answers to questions"; the embedded repo state has gated false.

Adoption

4 medium confidence
4.0

Adoption is measured by Hugging Face downloads of truthfulqa/truthful_qa. TruthfulQA is a long-standing truthfulness benchmark, though one of the older ones in the category and increasingly saturated.

Capability

not assessed

A dataset is not 'capable', so capability is left unscored; openness and adoption carry this category.

  • https://huggingface.co/datasets/truthfulqa/truthful_qa recorded 2026-09-29

    Card summary: "The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics"; the card makes no performance, throughput or feature claim for the corpus itself beyond the section heading "Supported Tasks and Leaderboards", and the Hub download counter on the page is usage data, not a capability claim.

Verified 2026-09-27