On November 11, 2026, AWS published the Universal Agent Evaluation Framework (UAEF) — an open evaluation layer that answers a question every agent team hits after the demo: did the prompt, model, or tool change actually improve behavior? Instead of rebuilding metrics per SDK, UAEF adapters normalize LangGraph, LangChain, Strands, Bedrock agents, AgentCore, Langfuse, or custom JSON into one AgentTrace, then score the same 36 metrics whether the agent ran locally, behind HTTP, or on a managed runtime.
The code lives on GitHub as awslabs/uaef under Apache 2.0. AWS says 12 customer organizations in telecom, healthcare, and financial services used UAEF before the public post — a signal this is meant for release gating, not slideware.
TL;DR
| Piece | AWS (Nov 11, 2026) |
|---|---|
| Project | UAEF — Universal Agent Evaluation Framework |
| License | Apache 2.0 (awslabs/uaef) |
| Adapters | LangGraph, LangChain, Bedrock, Langfuse, Strands, AgentCore, Generic JSON |
| Metrics | 36 built-in across 7 dimensions |
| Storage | DynamoDB metadata + S3 full JSON per experiment |
| Judges | Deterministic metrics + Bedrock LLM-as-judge with prompt-injection guards |
| Surfaces | Python library, notebooks, deployable web UI, optional EKS API |
The problem UAEF targets
Teams on AWS already mix Strands decider patterns, Bedrock AgentCore, and LangGraph graphs in different products. Each framework logs messages, tool calls, and handoffs differently, so evaluation code rots per stack — exactly the fragmentation WorkflowArena-style orchestration benchmarks try to measure from the outside.
UAEF separates normalization from scoring: adapters emit AgentTrace; every metric consumes that shape. AWS claims you can compare an agent built on one framework against another on one scale — useful when a platform team standardizes gates but not SDK choices.
Closed-loop flow
AWS diagrams a repeatable loop:
- Adapter ingest — pass an existing trace or let UAEF invoke the agent via ARN/HTTP and capture fresh output.
- PII screen — detect emails, phone numbers, payment data before storage.
- Evaluate — run selected metrics (subset or full 36).
- Persist — optional experiment row in DynamoDB plus full results in S3.
- Compare & report — highlight regressions vs baseline; optional LLM-written insights on what to change next.
That maps directly to CI patterns explainx.ai recommends in Raindrop reasoning memory benchmarks and Infrathon infrastructure agent evals — except UAEF is AWS-native storage with a shipped UI.
Metrics and judges
Seven dimensions span behaviors a single accuracy number hides:
| Dimension | What it catches |
|---|---|
| Tool calling | Wrong tool, bad argument shape, sequence errors |
| Response quality | Relevance, completeness, hallucination vs context |
| Responsible AI | Safety, bias, toxicity, prompt injection resistance |
| Performance | Latency, token and cost efficiency |
| Multi-turn | Retention, coherence, goal completion |
| Multi-agent | Delegation, utilization, workflow completion |
| Reasoning | Chain consistency, fallacies |
Deterministic metrics replay identically; judge metrics call Bedrock with untrusted-output handling — rubric separated from content, escaped delimiters, evidence spans required or scores reject. AWS explicitly says judge scores are not a safety gate; use Bedrock Guardrails on the runtime path for enforcement — same split we document in our agent sandbox security post.
Integrations include RAGAS and DeepEval metrics for retrieval-heavy agents.
Experiment tracking builders actually need
When persist=True, UAEF records framework, model, tools, prompts, optional git SHA, and per-metric averages. Mark a baseline run and the comparison engine flags regressions — the workflow MemCurrent-style memory papers chase with token counts, but for full agent traces.
Multi-tenant note: AWS's SECURITY.md says UAEF expects host-level tenant isolation — separate stacks or accounts per customer — and scopes reads to Cognito sub inside a deployment. Do not bolt multi-customer SaaS onto one shared UAEF stack without that separation.
How to start this week
git clone https://github.com/awslabs/uaefanduv syncper README.- Run notebook 00_langgraph.ipynb or 06_agentcore.ipynb against a toy agent.
- Pick metrics from AWS's scenario table (RAG Q&A vs tool-using vs multi-turn columns in the blog).
- Call
batch_evaluate(..., persist=True, experiment_name='v2 prompt')before merging prompt changes. - Optional: deploy uaef-service with CDK when you want a reviewer UI on top of the same engine.
AWS also documents dataset columns — Question, optional Answer, expected_tool_calls JSON, and session_id / turn_id for conversations — so exports from Quick Skills agent catalogs or internal ticket bots can reuse CSVs with minimal reformatting.
Playbook highlights from the AWS post
AWS's metric chooser table is the fastest onboarding doc: start from scenario, not the full metric list. Retrieval agents begin with faithfulness and context precision; tool agents need selection accuracy and parameter quality; compliance-heavy flows prioritize safety and injection detection.
Pipeline choices are explicit: passive trace evaluation vs active invoke_* helpers, single trace vs batch, offline CI vs OnlineEvaluator on live traffic, ephemeral scores vs persisted experiments (only persisted runs compare).
Deployment surfaces in more detail
AWS documents three runnable surfaces that share one evaluation core:
Python library and notebooks. The fastest path for a platform team is evaluate() on a trace captured from staging, then batch_evaluate() on a CSV exported from support tickets. Notebooks under notebooks/ map frameworks to scenarios — LangGraph batch runs, AgentCore multi-agent sessions, Langfuse imports for production traces you already store.
Serverless web UI. The uaef-service CDK stack provisions Cognito auth, API Gateway, worker Lambdas, Step Functions, DynamoDB, S3, and CloudFront for a React reviewer UI. AWS stresses that cors_allowed_origins is mandatory: blank means the deploy script derives the CloudFront origin on first UI deploy, while EKS mode requires an explicit origin because no distribution exists.
EKS API. Teams that already run Bedrock inference on Kubernetes may prefer the containerized FastAPI deployment in deploy_eks/ — same persistence schema, different ingress. Bring your own SSO and UI; keep experiment comparison on the Python API.
Across modes, only persisted experiments compare. Ephemeral evaluate() calls help debug one bad trace; they will not show up in regression dashboards.
Example metric bundles AWS recommends
Rather than enabling all 36 metrics on day one, AWS's scenario table implies starter packs:
| Agent shape | Start with | Why |
|---|---|---|
| RAG helpdesk | answer_relevance, hallucination_score, ragas_faithfulness | Catches wrong retrieval and invention without tool metrics |
| Ticket router | tool_selection_accuracy, parameter_quality, latency_score | Measures wrong API and slow escalations |
| Compliance bot | safety_score, prompt_injection_detection, toxicity_score | Separates policy incidents from helpfulness |
| Pair programmer | accuracy, completeness, reasoning chain metrics | Aligns with IDE agent regressions |
| Swarm workflow | delegation_quality, workflow_completion, coordination_efficiency | Mirrors WorkflowArena failure modes |
Add multi-turn metrics only when datasets include session_id and turn_id columns; otherwise UAEF treats rows as independent single-turn cases.
CI/CD pattern
A minimal release gate looks like this:
- Export 50–200 production queries with redacted PII into a CSV using
PIIDetector. - Mark the current production agent run as
baselinein UAEF experiment metadata. - On each pull request that changes prompts, tools, or model ARNs, run
batch_evaluate(..., persist=True, experiment_name='pr-1234'). - Fail the build if any dimension drops more than an agreed tolerance versus baseline averages.
- Attach the HTML report link to the PR for human review of judge-reasoning fields stored in S3.
This is the AWS-native version of patterns Amazon's Bedrock evaluation playbook describes for DeepEval inside AgentCore — complementary, not redundant.
Judge security details worth copying
AWS's judge path treats agent output as untrusted input: rubric and content are separated, content is escaped, and judge answers must cite verbatim evidence spans or the score is rejected. That design responds to the same class of attacks Claude memory exfiltration research documents — models that see their own evaluation prompt can game it.
UAEF explicitly says do not use judge scores as runtime policy. Pair UAEF offline gates with Bedrock Guardrails or application-level tool allowlists on the live agent.
Online evaluation versus offline batches
AWS distinguishes OfflineEvaluator batch runs on frozen datasets from OnlineEvaluator hooks that score live conversations turn by turn. Offline runs belong in CI and release trains; online hooks belong in canary deployments where you sample a percentage of production traffic (with PII redaction) to catch drift when user phrasing shifts.
The blog's passive versus active pipeline choice matters here: passive mode ingests traces your observability stack already captured (Langfuse, CloudWatch logs, custom JSON); active mode lets UAEF invoke agents so every test uses the same temperature and tool manifest. Mixed environments often passive-import staging traces but actively invoke production builds before promotion.
Human-in-the-loop review
UAEF documentation references HITL flows for disputed judge scores — important when LLM judges disagree with deterministic metrics on borderline safety cases. Treat HITL queues as part of the product, not an afterthought: without them, teams mute judge metrics and ship anyway.
When not to adopt UAEF yet
UAEF is heavy if you have one LangGraph demo and no baseline traffic. It pays off when:
- Multiple teams ship agents on different frameworks under one platform org.
- You already pay Bedrock judge costs and need shared storage.
- Regulated customers ask for experiment audit trails.
If you only need a single RAG faithfulness number, starting with RAGAS inside the app may be faster — then graduate to UAEF when regressions become cross-team.
How UAEF fits next to other AWS agent posts
- Bedrock tokenomics data lake — UAEF adds quality time series beside cost telemetry.
- Trovata treasury agents on AgentCore — UAEF is how a platform team regression-tests those graphs across releases.
- American Express workforce agent rollout — enterprise scale is where Dunbar-number reporting problems appear; UAEF is the technical mirror of "what broke after the model swap."