Amazon Bedrock Evaluations

license

proprietary(AWS managed service)

source

closed

deployment

cloud-only(AWS-hosted eval jobs, S3 I/O)

billing

pay-as-you-go(judge-model inference + AWS-managed human workforce)

Amazon Bedrock Evaluations is a proprietary AWS managed service with no published source code. It runs only as evaluation jobs billed through Bedrock, with no self-hosted option.

• https://aws.amazon.com/bedrock/evaluations/ recorded 2026-09-29

  The page describes "Amazon Bedrock provides evaluation tools for you to accelerate adoption of generative AI applications" and lists no repository, download or license terms.

• https://aws.amazon.com/bedrock/evaluations/ recorded 2026-09-29

  The AWS product page says "you can engage a team managed by AWS to perform the human evaluation" and offers LLM-as-a-Judge, programmatic and human evaluation. Searched the body for self-host, open source, license and github: none.

• https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html recorded 2026-09-29

  The AWS user guide describes a managed Bedrock service that runs evaluation jobs, saying "Required Cross Origin Resource Sharing (CORS) permissions on S3 buckets" and "CloudTrail management events in model evaluation jobs". Searched the body for self-host, on-prem, open source and license: none.

Amazon Bedrock Evaluations has been generally available since 2025-03-20, but AWS publishes no separate usage or job-count figure for the feature itself. As a closed service with no repository or download count either, this reading reflects the broader Bedrock platform's reach rather than the evaluation feature specifically.

• https://aws.amazon.com/bedrock/evaluations/ recorded 2026-08-13

  No published usage or customer count for the Evaluations feature.

• https://aws.amazon.com/blogs/machine-learning/evaluate-models-or-rag-systems-using-amazon-bedrock-evaluations-now-generally-available/ recorded 2026-06-04

  GA announcement (2025-03-20) of model + RAG evaluations as a broadly available Bedrock feature

• https://aws.amazon.com/blogs/machine-learning/evaluate-models-or-rag-systems-using-amazon-bedrock-evaluations-now-generally-available/ recorded 2026-08-13

  The GA announcement for model and RAG evaluations carries no usage, customer or job-count figure.

Bedrock Evaluations covers models and RAG systems with judge-model, programmatic and human grading, four families of metrics, and bring-your-own inference responses for models hosted elsewhere. It evaluates custom datasets rather than running a standardized public benchmark leaderboard.

• https://aws.amazon.com/bedrock/evaluations/ recorded 2026-06-04

  model eval (LLM-as-judge/programmatic/human) + RAG eval (retrieval, retrieve-and-generate) with safety/quality metrics

• https://aws.amazon.com/bedrock/evaluations/ recorded 2026-09-29

  Model evaluation uses "Use an LLM as a Judge to evaluate model outputs using your custom prompt datasets" plus programmatic BERT Score and F1 metrics, human evaluation with "your own workforce or have AWS manage your evaluations", and both RAG retrieval and retrieve-and-generate evaluation.

• https://aws.amazon.com/blogs/machine-learning/evaluate-models-or-rag-systems-using-amazon-bedrock-evaluations-now-generally-available/ recorded 2026-09-29

  The post documents citation metrics: "Citation precision measures how accurately a RAG system cites its sources" and lists built-in metrics such as Builtin.FollowingInstructions, Builtin.Harmfulness, Builtin.Stereotyping and Builtin.Refusal.

• https://aws.amazon.com/blogs/machine-learning/evaluate-models-or-rag-systems-using-amazon-bedrock-evaluations-now-generally-available/ recorded 2026-06-04

  four metric areas (quality/UX/instruction-compliance/safety) and citation precision/coverage metrics