Your agent skill passed the linter. It might still make the agent worse.
NVIDIA's ACES framework — Agentic Continuous Evaluation of Skills — and the open SkillEvaluator tool measure what document scans cannot: live agents with and without a skill installed on the same task.
The August 2026 headline from digest coverage: ~27% of paired runs show zero or negative lift — skills that do not beat baseline once an agent actually executes them.
TL;DR
| Finding | Number |
|---|---|
| Paired production cases | 947 (58 of 64 skills) |
| Mean composite Skill Lift | +0.2134 (95% CI ~0.20–0.23) |
| Positive / zero / negative | 689 / 171 / 87 (~27% ≤ baseline) |
| Scan vs live correlation | Tier 1 ρ ≈ -0.02, Tier 2 ρ ≈ -0.03 |
| Verified catalog snapshot | 300+ skills; +31 pt avg lift (+39 excl. Security) |
| Tool | Open SkillEvaluator + nvidia/skills benchmarks.json |
Why scans are not enough
NVIDIA's ACES paper (arXiv:2608.20614) argues skills are the application layer for agents — SKILL.md packages loaded via progressive disclosure across Claude Code, Codex, Cursor, and registries like /skills.
Scan-only eval (frontmatter lint, LLM rubric on prose) is like compiling with -Wall — no warnings ≠ correct behavior.
Runtime failure modes scans miss:
| Failure | What happens |
|---|---|
| Never discovered | Agent never loads skill on relevant queries |
| Wrong script | Reads docs but invokes bad tool/args |
| Misinterpretation | Output correct, report wrong |
| Collision | Two skills conflict in workspace |
| Model drift | New model reasons differently about same SKILL.md |
On 145 skills, structural vs LLM-judge scan scores correlated at Spearman ρ = 0.14 — scans do not even agree with each other, let alone predict runtime.
What live paired eval shows
ACES runs with-skill and baseline agents on identical tasks, grades trajectories (discovery, workflow, tool efficiency, accuracy), and reports Skill Lift = with − without.
Production-skill subset highlights:
- Largest gains: skill execution (+0.33), behavior checking (+0.30), efficiency (+0.28) — not just final-answer accuracy (+0.14)
- Negative lift (87 cases): skill found but misused, truncated answers, extra tool calls, goal accuracy fell despite "activation"
- Efficiency metric: positive lift in only 41.7% of pairs — high-variance tradeoff
Separately, NVIDIA's verified catalog snapshot (Aug 12, 2026) shows when skills work, they work strongly:
| Dimension | Without skill | With skill | Lift |
|---|---|---|---|
| Correctness | 46 | 87 | +41 |
| Discoverability | 42 | 82 | +40 |
| Effectiveness | 39 | 78 | +39 |
| Efficiency | 43 | 78 | +35 |
| Security | 97 | 98 | +1 |
Skill Lift varies more by product than harness — Claude Code vs Codex ~5 pt apart on average; per-product spread ~+2 to +46.
Some skills save tokens (example cited: 76.9% reduction); others increase tokens and time — flagged for rework, not celebrated.
How ACES compares to other evaluation approaches
Most teams evaluating agent skills today fall into one of three buckets — and ACES is the first to treat runtime paired comparison as the default gate:
| Approach | What it measures | Predicts live lift? | Cost |
|---|---|---|---|
| Structural scan (Tier 1) | Frontmatter, file layout, script presence | No (ρ ≈ -0.02) | Low |
| LLM rubric on prose (Tier 2) | Readability, completeness of SKILL.md | No (ρ ≈ -0.03) | Medium |
| Single-run task eval | Did the agent succeed once with the skill? | Partial — no baseline | Medium |
| ACES paired eval | With-skill vs without on identical task | Yes — direct lift metric | Higher |
| Human spot-check | Reviewer reads trajectories | Yes, but not scalable | High |
The gap ACES fills is the same one eval-driven development addresses for models: you need a control group. A skill that helps on one lucky run but hurts on average is invisible without paired comparison.
NVIDIA's data also shows scan tiers barely agree with each other (Spearman ρ = 0.14 between Tier 1 and Tier 2). That means a skill can pass your linter and your LLM-judge rubric while still ranking in the negative-lift bucket once an agent loads it.
For explainx.ai readers who maintain skills on /skills, the practical read is: structure checks are hygiene, not proof of value. Hygiene catches broken scripts and missing frontmatter; ACES catches skills that agents discover but misuse.
What people are asking about NVIDIA ACES
These questions showed up across NVIDIA's blog comments, Hacker News threads on agent skills, and GitHub issues on nvidia/skills:
"Does this mean my skill is bad if lift is zero?" Not necessarily. Zero lift often means the task was solvable without the skill — the skill added no marginal value on that query class. That is useful signal for trimming scope: if a skill never beats baseline on its intended trigger set, rewrite the description or delete it.
"Why does efficiency lift only hit 41.7% of pairs?" Because skills frequently add tool calls, longer reasoning chains, or extra context reads. Correctness can rise while token count doubles. ACES treats that as a first-class tradeoff — not every "helpful" skill is economical.
"Can I run ACES locally?" SkillEvaluator ships open on the nvidia/skills repository with a benchmarks.json that updates continuously. You point it at your skill, define task fixtures, and run paired trajectories against your harness (Claude Code, Codex, or others). Exact CLI flags evolve — check the repo README for the August 2026 interface.
"Does ACES replace security review?" No. Security dimension lift was already near ceiling (97→98) because baselines were strong. Agent skills threat model coverage — prompt injection via SKILL.md, malicious scripts, supply-chain trust — stays a separate review pass. ACES adds a behavioral dimension security scans cannot see.
"Which harness benefits most?" Per-product spread was ~+2 to +46 points — wider than the ~5 pt average gap between Claude Code and Codex. Skill quality and discoverability matter more than which host you pick.
Running paired evals in practice
You do not need NVIDIA's full 947-case production suite to start. A minimal ACES-style loop for a skill you ship:
- Define 10–20 representative tasks — the queries your skill's description claims to handle.
- Run each task twice — identical prompt, same model, same harness config; once with the skill installed, once without.
- Grade trajectories, not just final answers — did the agent load the skill? Did it follow the workflow? Did tool calls match the spec?
- Compute per-dimension lift — Correctness, Discoverability, Effectiveness, Efficiency, Security.
- Flag negative-lift cases — read the trace; categorize as never-discovered, misused, collision, or harness regression.
- Gate merges on lift — same discipline as unit tests: no merge if composite lift ≤ 0 on your fixture set.
For teams already using loop engineering or the Gauntlet loop, ACES slots in as the critic pass for skills — the builder writes SKILL.md, the evaluator runs paired trajectories, the loop repeats until lift is positive on fixtures.
Token economics deserve explicit thresholds. If your skill adds +15 Correctness but +80% token use, set a budget rule: reject skills where efficiency lift ≤ 0 unless correctness gain exceeds your threshold. NVIDIA flagged several catalog skills in exactly that bucket.
ACES and the explainx.ai skills ecosystem
The /skills registry and companion posts like What are agent skills? and Agency agents assume skills are loadable, discoverable packages — not guaranteed improvements. ACES gives that assumption a quantitative test.
If you publish skills publicly, add a benchmarks.json entry or document your paired-eval fixtures so consumers can reproduce lift numbers. Transparency beats star counts: a skill with +40 Correctness lift on 20 fixtures is more trustworthy than one with polished prose and no runtime data.
LangChain deep agents 0.7 trimmed harness overhead; ACES trims skill overhead — complementary optimizations on opposite sides of the agent stack.
When auditing third-party skills from registries, ask for paired-eval results the same way you ask for SBOMs: proof beats prose. A skill that fails baseline on your fixtures should not ship because its SKILL.md reads well.
Negative-lift patterns worth fixing
The 87 negative-lift cases in ACES production data cluster into repeatable failure modes:
| Pattern | Symptom | Fix |
|---|---|---|
| Over-broad description | Skill loads on irrelevant queries, adds noise | Tighten trigger text; add negative examples |
| Script mismatch | Agent reads docs, runs wrong command | Pin script paths; add validation steps |
| Conflicting skills | Two skills fight for routing | Namespace triggers; document exclusions |
| Verbose overhead | Long SKILL.md burns context before task starts | Thin prompts, thick artifacts |
| Model-specific assumptions | Works on Opus, fails on Haiku | Test across models in paired eval |
The 171 zero-lift cases are a separate bucket — not harmful, but not worth the maintenance cost if your skill registry is growing. Periodic pruning against ACES fixtures keeps /skills-class catalogs honest.
What builders should do
- Do not ship skills on prose review alone — add paired evals to CI like unit tests.
- Investigate negative-lift traces — ACES separates "never discovered" from "discovered but harmful."
- Optimize discoverability — description/trigger text is a routing problem; isolated content quality is insufficient.
- Watch token economics — a skill that adds 2× tokens for +1 pt correctness may fail your budget.
- Cross-read security — agent skills threat model + runtime security metrics stay separate from behavior pass rates.
For explainx.ai readers maintaining **agent skills` or MCP-adjacent workflows, ACES is the closest thing to "does this SKILL.md actually help?" as an automated gate.
Related on explainx.ai
- What are agent skills? Complete guide
- Agent skills security threat model
- Agency agents — AI specialists guide
- Loop engineering for coding agents
- Gauntlet loop — builder/critic technique
- Claude Code commands reference
- Codex slash commands reference
- LangChain deep agents 0.7
Sources
- NVIDIA — Evaluating agent skills with SkillEvaluator — August 2026
- arXiv — Evaluating Skills, Not Just Agents (ACES)
- nvidia/skills benchmarks.json — commit 738d79e snapshot cited in NVIDIA blog
Lift statistics from NVIDIA ACES paper and SkillEvaluator catalog snapshot August 2026. Follow @explainx_ai for agent evaluation coverage.