claude plugin eval: How to Score Your Claude Code Plugins

Most Claude Code plugin authors have never actually measured whether their plugin helps. They ship a SKILL.md, watch a few sessions go well, and assume it's working. On September 12, 2026, Anthropic's Claude Code team (via @ClaudeDevs) shipped a command that replaces that assumption with a number: claude plugin eval.

The pitch, in Anthropic's own words: "See what value your plugin is adding, or if it needs more work. You can create test cases, run your plugin or skill against those test cases, score those runs, then run each case again without the plugin to see the differences." It's a built-in A/B test for Agent Skills and Claude Code plugins, run entirely from your terminal.

TL;DR

The command reference

Two commands cover the whole workflow. Run both from inside your plugin's own folder — the eval suite lives alongside the plugin it tests, not in a separate repo.

# 1. Set up a suite from scratch (only needed once)
cd my-plugin/
claude plugin eval init

# 2. Run the suite (repeat this every time you change the plugin)
claude plugin eval

# 3. Pilot cheaply before spending on a full run
claude plugin eval --runs 1

# 4. Get the latest CLI build if the subcommand isn't there yet
claude update

init is the interactive setup step. You tell Claude what good output looks like for your plugin, what bad output looks like, and bring a handful of real prompts you've actually used. Claude drafts the test cases and the scoring checks, pilots the draft suite once to sanity-check it, and tells you what a full run across every case will cost — before you spend anything on the real thing.

claude plugin eval (no arguments) then runs every case twice: once with your plugin active, once with it disabled, across the number of runs each case is configured for. Because it calls the model for every run of every case, "evals call the model, so they use tokens and results vary" is Anthropic's own framing — this is not a static test suite, it's a live, LLM-graded comparison, and the same suite can return a slightly different score on a different day.

What the report actually looks like

Anthropic published a sample report from its own internal eval suite. Reproduced as a table:

Two things stand out in that sample, and they're exactly what makes this more useful than a single manual test:

• Case 05 scores low even with the plugin (0.11). A naive "did it help" glance would say yes — it beat the 0.00 baseline — but 0.11 is still a failing score in absolute terms. The plugin author now knows this case needs actual work, not just a relative win.
• Cases 06 and 07 are negative controls — both score 1.00 with or without the plugin, meaning the plugin correctly does nothing on prompts it shouldn't touch. A delta of 0.00 there is the correct outcome, not a wasted case. Anthropic's own suite deliberately includes cases designed to show no lift, which is the kind of test discipline this tool encourages practitioners to copy.

That distinction — absolute score vs. delta — is the main thing to read past the headline number for. A plugin can post a great average delta while still failing a specific case outright, and this report format is the first place that shows up in one glance.

What people are asking

"How is this different from just testing my plugin manually?" Manual testing is what most plugin authors already do — run a prompt, glance at the output, decide it "feels" better. claude plugin eval formalizes that into a repeatable suite: the same cases run every time, scored by a consistent check rather than a vibe, with a with/without baseline built in so you're not just checking "is the output good" but "is the output better because of the plugin." It's the same shift testing frameworks made for code — from "I ran it once and it looked fine" to a suite you rerun on every change.

"Do I need to write the test cases myself?" No — that's the point of init. You supply real prompts you've actually sent the plugin plus a description of what good and bad output looks like, and Claude drafts the cases and scoring checks for you, then pilots them once before asking you to commit to a full run.

"What if my plugin's score doesn't move much?" A delta near zero on a case like 06-neg-feedback-only above is a good sign, not a bad one, if that case is a negative control — a prompt your plugin shouldn't affect. A near-zero delta on a case the plugin is supposed to help with is the real warning sign, and it means the skill's instructions likely need to be more specific or better triggered, the same diagnosis this blog covered in AGENTS.md for code quality — a prose instruction can influence behavior, but only a check tells you whether it actually did.

"Should I trust an eval run against someone else's plugin?" Be careful here. Anthropic's own guidance is blunt: "Your plugin's hooks and MCP servers run as you, so only evaluate plugins you trust." Running claude plugin eval on a plugin is not a sandboxed, read-only check — the plugin's hooks and any MCP servers it wires up execute under your own account permissions during every eval run, identical to using the plugin live. Evaluating an unfamiliar third-party plugin from a registry is the same trust decision as installing it and running it in production, not a safer preview step.

"What does a full run cost me?" It scales with case count × runs per case × model choice, since every run is a real model call, scored by another model call. Anthropic's own 7-case, 6-runs-per-case suite came to $9.59 — a little under $1.37 per case. init estimates the cost of your specific suite before you run it, and --runs 1 lets you pilot at roughly a sixth of the per-case cost to sanity-check the setup before committing to the full run count.

The same week, a converging idea from OpenAI

claude plugin eval landed the same week OpenAI's Codex team published its own guidance on tightening skill descriptions and AGENTS.md files for GPT-6 Astra — covered in explainx.ai's GPT-6 Astra skills and prompting guide. The two announcements aren't the same feature, but they're pointed at the same underlying problem from opposite ends: OpenAI's guidance is about writing leaner skills before you ship them, while Anthropic's tool is about proving what a skill actually did after you ship it. Put together, the pattern across both labs this month is the same: skills and plugins are being treated less like static documentation and more like code that needs a lint pass and a test suite before you trust it in a real session.

That mirrors a shift explainx.ai has tracked for months. Thin prompts, thick artifacts, thin skills argued skills should carry the smallest instruction that reliably triggers correct behavior — hard to know without measuring. Microsoft's SkillOpt tried to automate that tightening loop entirely; claude plugin eval instead hands the measurement step to a human who still writes and owns the skill. And Terminal-Bench 2.0 already showed the industry that a scored, repeatable comparison beats a demo — this is the same idea applied one level down, to a single plugin instead of an entire model.

Honest limitations

• Scores are model-graded and noisy. A single run's score is a sample, not a ground truth — that's exactly why the tool defaults to multiple runs per case (6 in Anthropic's sample) rather than one.
• Cost is real and recurring. Every case, every run, with and without the plugin, is a separate model call plus a separate grading call. A suite you rerun on every commit adds up fast; pilot with --runs 1 before scaling up.
• Trust is not sandboxed. As covered above, hooks and MCP servers execute with your real permissions during an eval — this is not a dry-run environment.
• A good average can hide a bad case. The 0.11 score on 05-rewrite-request in Anthropic's own sample report, despite a positive delta, is the clearest illustration: read every row, not just the mean delta at the bottom.
• It measures your test cases, not universal quality. A suite built from a handful of real prompts reflects the scenarios you thought to include. A plugin can pass its own eval suite and still fail on a prompt nobody wrote a case for — the same blind spot Terminal-Bench 2.0 and other benchmarks share.

Getting started today

1. Run claude update to make sure your CLI build includes plugin eval.
2. cd into an existing plugin or skill folder — see explainx.ai's complete guide to Agent Skills if you haven't built one yet, or browse top Claude plugins for examples to eval against.
3. Run claude plugin eval init and bring 3-5 real prompts plus a plain-language description of good vs. bad output.
4. Pilot with claude plugin eval --runs 1 to confirm the suite behaves and check the estimated cost.
5. Run the full claude plugin eval and read every case row, not just the mean delta — and only ever eval plugins whose hooks and MCP servers you actually trust, per the permission-mode guidance that applies to any Claude Code extension running with your credentials.

Related reading

• What are Agent Skills? A complete guide
• GPT-6 Astra: rethinking skills, AGENTS.md, and prompting
• AGENTS.md for code quality: file vs. CI
• Terminal-Bench 2.0: the AI agent benchmark that actually matters
• Microsoft SkillOpt: self-evolving agent skills optimization
• Thin prompts, thick artifacts, thin skills
• Skills vs hooks vs prompts: when to use each
• Top 25 Claude plugins in 2026
• Claude Code permission modes explained

Primary source: @ClaudeDevs on X, September 12, 2026

This post reflects the claude plugin eval announcement from September 12, 2026. Command syntax, scoring format, and artifact publishing behavior may change as Anthropic iterates on the feature — run claude plugin eval --help for the current flags on your installed CLI version.