Amazon Bedrock AgentCore Evaluation Playbook: Built-In, DeepEval, and AutoEval

On November 10, 2026 AWS published an agent evaluation playbook for Amazon Bedrock AgentCore Evaluations — a practical map from business outcomes to built-in, DeepEval, and AutoEval judges. The post complements the decision-model screening guide AWS shipped six days earlier and the seven-layer agent framework from November 9.

If you run agents on AgentCore — like the Trovata treasury pattern AWS documented the same morning — this playbook answers a question decision models do not: which full evaluators to run after you know what "working" means.

TL;DR

Why surface metrics hide agent failure

AWS opens with a warning teams already feel in production: high completion rate can coexist with PII leakage, skipped policy steps, or incoherent multi-turn drift. Latency and spot checks measure whether the agent ran, not whether it met the business bar.

Agent evaluation playbook diagram showing surface latency signals versus hidden agent behaviors on Amazon Bedrock AgentCore Image: AWS

That framing matches what explainx.ai saw in Raindrop's tau2-bench agent memory work and in WorkflowArena orchestration benchmarks: orchestration scores can look fine while per-turn policy fails. AgentCore Evaluations is AWS's hosted answer for measuring below the waterline behaviors at scale.

Start with outcomes, not metric labels

The post's core discipline is simple and easy to skip:

1. Business outcome — e.g. first-contact resolution for support, fully sourced answers for research, zero false negatives for moderation.
2. Observable behaviors — goal completion, tool choice, compliance boundaries, skill steps.
3. Evaluator mapping — trace-level for per-turn signals; session-level for whole-conversation results.

Three-step flow from business outcomes to evaluators in the Bedrock AgentCore evaluation playbook Image: AWS

AWS argues that picking "helpfulness" because the name sounds good produces numbers without insight. The same lesson applies outside Bedrock: Andy Hall's audit-agent study found high recall with costly false alarms when judges are not tied to explicit behaviors.

Three evaluator tiers

AgentCore exposes three sources AWS diagrams as a control-versus-QA tradeoff:

AgentCore evaluator catalog spanning built-in AWS, DeepEval, AutoEval, and custom judges Image: AWS

Built-in catalog (selected IDs)

AWS groups built-ins into:

• Response quality: Builtin.Correctness, Builtin.Faithfulness, Builtin.Helpfulness, Builtin.ResponseRelevance, Builtin.Conciseness, Builtin.Coherence, Builtin.InstructionFollowing, Builtin.Refusal.
• Task completion: Builtin.GoalSuccessRate (session-level anchor for many stacks).
• Tool use: Builtin.ToolSelectionAccuracy, Builtin.ToolParameterAccuracy.
• Skills: Builtin.SkillSelectionAccuracy, Builtin.SkillInstructionFollowing.
• Safety: Builtin.Harmfulness, Builtin.Stereotyping.
• Trajectories: Builtin.TrajectoryExactOrderMatch, Builtin.TrajectoryInOrderMatch, Builtin.TrajectoryAnyOrderMatch — require expected paths (batch / golden sets).

Exact rubrics live in the AgentCore prompt template library per AWS. Call ListEvaluators to enumerate built-ins plus account customs.

DeepEval managed metrics

DeepEval's managed set targets multi-turn failure modes AWS highlights:

• Task and goal: ThirdParty.DeepEval.TaskCompletion, ThirdParty.DeepEval.GoalAccuracy, ThirdParty.DeepEval.ConversationCompleteness.
• Conversation quality: ThirdParty.DeepEval.TurnRelevancy, ThirdParty.DeepEval.KnowledgeRetention, ThirdParty.DeepEval.Summarization.
• Tool use: ThirdParty.DeepEval.ToolUse.
• Safety: ThirdParty.DeepEval.Bias, ThirdParty.DeepEval.Toxicity, ThirdParty.DeepEval.PIILeakage.

ConversationCompleteness catches agents that answer the latest message but drop earlier user requests — a pattern we also track in ChatGPT Business Agents hub launches where long-running tasks need session memory.

AutoEval specialized signals

AutoEval adds narrower judges:

• ThirdParty.AutoEval.Security — instruction injection or system-prompt reveal attempts.
• ThirdParty.AutoEval.Possible — valid tasks wrongly declared impossible (over-refusal).
• ThirdParty.AutoEval.Humor — tone for consumer-facing agents.

Pair AutoEval.Security with governance patterns from Okta cross-app access on AgentCore Gateway when agents touch multiple SaaS tools.

Multi-dimension evaluators

When accuracy, policy compliance, and tone can fail independently, AWS recommends a multi-dimension custom LLM judge: one trace read, separate scores per dimension, no blended composite that hides which criterion broke.

Keep criteria in separate catalog evaluators when:

• A built-in already covers them (Builtin.Correctness, ThirdParty.DeepEval.PIILeakage).
• They need different levels (trace vs session) or different judge models.
• Some need ground truth fields like {expected_response} — those are batch-only on live traffic.

This is the hosted version of a pattern Hugging Face Decision Models pursues for cheap per-turn gates — full judges stay expensive, so dimension design matters.

Example stacks AWS publishes

AWS gives three starter tables. All mark trajectory and some correctness evaluators as needing ground truth (batch / golden data).

Customer support

Multi-step research

Regulated domain content

AWS notes derived evaluators for trajectory and skill metrics when inference must stay in your account — relevant for Bedrock tokenomics and FinOps teams metering judge spend.

Three run modes (same IDs)

The Python sample in the post uses ThirdParty.DeepEval.TaskCompletion with sessionSpans — the same ID works when you move from notebook to production online eval.

For physical-world agents, compare this software stack to Crossnokaye Aura on AgentCore: industrial agents need trajectory and safety dimensions, not helpfulness alone.

How to adopt this without boiling the ocean

AWS's closing advice is deliberately minimal:

1. Answer what success looks like for this agent.
2. Name the trace evidence that proves it.
3. Pick two or three evaluators that target those behaviors.
4. Add evaluators when production or batch runs surface a missed failure mode.

If you already run AgentCorruption-style hijack tests, treat ThirdParty.AutoEval.Security and PII judges as continuous checks, not pre-release one-offs.

Pair full judges with the Jev / Strands Decider screening layer when per-turn LLM judge cost dominates.

What AWS does not publish here

The playbook is a catalog and mental model, not benchmark leaderboard numbers. AWS does not claim accuracy rates for each built-in judge in this post. Validation is on you: labeled edge cases, human majority vote, compare decision-model screeners to full judges — the same workflow AWS described in the November 4 decision-model article.

For agent security incidents rather than quality drift, see november cyber tracker patterns and evaluate whether your runtime needs action-level gates, not just transcript judges.

Practitioner checklist before you turn on online eval

AWS assumes you already run agents on AgentCore with tracing enabled. Before you sample production traffic into evaluators, walk through this sequence — it mirrors the post's three-step model without requiring every catalog metric on day one.

Define the failure you fear most. For a coding agent, that might be merging unsafe shell commands. For a finance agent, leaking account numbers in summaries. Write that failure as a user-visible outcome, not as a metric name. If your team cannot articulate the outcome in one sentence, pause before buying judge capacity.

Pick one session anchor and two trace checks. Customer support stacks in the post anchor on Builtin.GoalSuccessRate and add conversation completeness plus PII leakage. Research stacks anchor on trajectory match when you have golden paths. Regulated stacks anchor on correctness with references. Your first week should not include every row in AWS's tables.

Validate on labeled exports before online mode. Export fifty to two hundred historical sessions with human labels on the behaviors you care about. Run on-demand evaluate() calls in a staging account. Compare built-in and DeepEval scores to human majority vote. Calibrate thresholds for online sampling only after agreement is acceptable on held-out sessions.

Document judge model residency. Managed third-party evaluators run on AWS capacity per the post. Custom derived evaluators exist so judge inference can stay inside your account when data governance requires it — the same reason Amazon Quick Skills teams segregate prompts and tools per tenant.

Plan spend. Each online eval multiplies inference cost on every sampled turn or session. Pair expensive judges with the decision-model screeners AWS documented November 4 so only low-confidence spans hit a frontier judge.

Re-run batch eval on every prompt or tool change. AWS positions batch mode as your pre-release gate. Tie it to the same golden set you use for regression tests on semantic caching or routing changes so eval debt does not drift from product debt.

How this differs from generic LLM-as-judge tutorials

Community guides often show one helpfulness rubric and a Python script. The AgentCore playbook's value is naming IDs you can move from notebook to production without rewriting glue code, and grouping metrics so you do not double-pay for near-duplicate judges.

DeepEval's conversation metrics matter because enterprise agents are multi-turn by default now — ChatGPT Business Agents, Claude office suite, and Meta Muse wearables all assume sessions that stretch beyond a single prompt-response pair.

AutoEval's Possible metric addresses a failure mode safety teams see in red-team exercises: models that refuse valid work because caution prompts were overtuned. That is different from Builtin.Refusal, which measures policy declines on actually harmful requests.

If you are standardizing across clouds, compare this catalog to Google's multimodal knowledge agent benchmark work and to open harnesses like Beneath the Surface agent evolution audit — AgentCore is the managed AWS slice, not the whole eval universe.

Related reading

• AgentCore Evaluations + decision models screening
• AWS Pluggable Agentic AI seven-layer framework
• Trovata treasury agents on AgentCore
• A2A vs MCP for agent tooling
• Heelo agent-native primitives and Terminal Bench

Primary sources: AWS evaluation playbook (Nov 10, 2026) and Amazon Bedrock AgentCore Evaluations documentation.