If you have ever shipped an MCP server and then had no real way to answer "does this actually work well with an agent," a new paper from Apple researchers gives you a path that does not start with writing test cases by hand.
Agent Seer: Synthesizing Scenarios from Specification Understanding (arXiv 2608.26133, submitted June 24, 2026, by Harish Karumuri, Mahesh Vemula, and David Lopes Pegna) generates realistic multi-turn agent evaluation scenarios from a single MCP server specification — the function names, natural-language descriptions, and typed parameter schemas that already ship with any spec-compliant server. No example transcripts. No live access to the tools being tested. No domain-specific tuning. The paper tested the pipeline across seven MCP specifications spanning different domains and tool-suite sizes, with complete tool coverage on small and medium specs.
The paper surfaced widely on X via Elvis Saravia (DAIR.AI founder), who framed the core problem well: "Hand-built agent benchmarks demand deep domain expertise, do not scale across tool ecosystems, and go stale as soon as an API changes. Generating them from the live spec keeps pace with the ecosystem instead."
TL;DR
| Question | Answer |
|---|---|
| What is it | A method that synthesizes multi-turn agent eval scenarios from an MCP server's spec alone |
| Who built it | Apple researchers Harish Karumuri, Mahesh Vemula, David Lopes Pegna |
| Where | arXiv 2608.26133, submitted June 24, 2026 |
| Do I need examples or live tools? | No — spec only, no examples, no live tool access, no domain tuning |
| How many specs tested | Seven, spanning different domains and tool-suite sizes |
| Coverage | Complete tool coverage on small and medium specs |
| Finding #1 | Parameter schema complexity predicts eval quality far better than tool-suite size |
| Finding #2 | Argument-value accuracy is the dominant failure mode, invisible to name-matching metrics |
| Can I run it today? | Not as a released tool — it's a research pipeline, not a public CLI or package yet |
| The catch | Synthesis quality likely inherits documentation quality; thinly-documented servers may get thin evals |
Why hand-built agent benchmarks break down
Anyone who has tried to properly evaluate an MCP server runs into the same wall: writing good multi-turn test scenarios by hand requires you to already understand your own tool suite well enough to predict how a real agent would misuse it. That's expensive, it doesn't transfer to the next server you build, and it stops being accurate the moment you rename a parameter or add a new tool.
This is the same failure mode covered in our guide to reading AI benchmarks: a benchmark is only as good as the process that generated it, and a benchmark that goes stale the day after an API changes was never really measuring the live system in the first place. Agent Seer's pitch is to replace the human-authoring step with a pipeline that regenerates the eval suite from the spec itself, so the benchmark tracks the API instead of lagging it.
How the Agent Seer pipeline actually works
Per the paper's introduction, Agent Seer runs in three stages:
- Enrich the raw schema. The pipeline takes an MCP spec — function names, natural-language descriptions, and typed parameter schemas — and infers richer semantics than what's explicitly written: what a parameter is actually for, what values are plausible, and how tools relate to each other.
- Generate graded scenarios with synthetic tool outputs. Instead of calling real APIs, the pipeline fabricates plausible tool outputs (mock data) and builds scenarios of varying difficulty around them, so evaluation doesn't require live credentials or a running backend.
- Expand into multi-turn dialogues. Those graded scenarios get turned into full mock-data-grounded conversations that exercise both tool-calling correctness (did the agent call the right tool, with the right arguments) and conversational coherence (does the dialogue read like a real multi-turn interaction, not a scripted Q&A).
The result is an evaluation suite generated entirely from documentation the server author already wrote — nothing you would not already ship in a compliant MCP spec.
Do I need example scenarios or live tool access?
No. That's the headline constraint the paper is built around: no examples, no live tool access, no domain-specific tuning. This matters practically because most teams evaluating a new connector or MCP tool don't have a library of prior test transcripts to draw from, and giving an eval pipeline live write access to production APIs is its own security and cost problem. Agent Seer sidesteps both by working purely from the static contract the client already sees.
Finding #1: parameter schema complexity beats tool-suite size
The paper's first major result inverts a common assumption. It's tempting to think a server with 40 tools is harder to evaluate well than one with 5 — more surface area, more combinations, more room to fail. Agent Seer's results say otherwise: parameter schema complexity is the strongest predictor of evaluation quality, while tool-suite size plays a smaller, largely orthogonal role.
In practice, that means a server with few tools but deeply nested, loosely-typed, or ambiguous parameter schemas is a harder evaluation target than a server with many tools but simple, well-typed parameters. If you're deciding where to invest documentation effort on your own MCP server, this is a concrete signal: tighten your parameter schemas before you worry about trimming your tool count.
Finding #2: argument-value accuracy is the blind spot that matters most
The second finding is arguably more important for anyone currently grading agents on tool use. Most coarse evaluation setups check tool-name matching — did the agent call create_invoice when it should have called create_invoice? That check passes even when the agent calls the right function with the wrong argument values: a malformed date, a truncated ID, a unit mismatch, a hallucinated enum value.
Agent Seer's analysis found argument-value accuracy is the dominant failure mode — and it's a failure category that coarse name-matching metrics simply cannot see, because they stop checking once the function name lines up. This lines up with what we've written about writing MCP tool descriptions for reliable selection: getting an agent to pick the right tool is only half the job. Getting it to fill that tool's arguments correctly is the harder, less-measured half — and apparently the one that breaks most often.
What's the catch?
The most honest caveat came up in the replies to the original thread, not the abstract: coverage quality inherits documentation quality. Agent Seer enriches semantics from function names, descriptions, and schemas — it has nothing else to work with. A well-documented MCP server gives the pipeline rich signal to synthesize good scenarios from. A thinly-documented one gives it thin material, and thin material likely produces a thin, lower-quality eval suite.
That's a real problem because the servers that are thinly documented are often exactly the ones a team most needs a rigorous, independent eval for — the ones nobody has had time to write good tests for by hand either. It's not clear from the publicly available abstract and introduction whether the paper measures how synthesis quality degrades as documentation quality drops, or reports that spread across its seven test specifications. Treat this as an open question rather than a resolved one until the full paper or a follow-up addresses it directly.
What this means if you're shipping an MCP server this week
Agent Seer is a research result, not a released tool — there's no public CLI or package to install as of this writing, so don't expect an npx agent-seer command. But the workflow it describes is something you can approximate conceptually against your own MCP server right now:
- Audit your spec as if it were the only input an evaluator gets. Read your own function names, descriptions, and parameter schemas cold, the way the pipeline would. If a human reader can't infer what a parameter means or what values are valid, an automated evaluator can't either.
- Tighten parameter schemas before adding more tools. Given finding #1, ambiguous or loosely-typed parameters are a bigger risk to eval quality — and real-world agent reliability — than a large tool count.
- Test argument values, not just tool names. If you're building your own eval harness in the meantime, don't stop at "did it call the right function." Check whether the arguments it passed are actually correct, well-typed, and in range — that's where Agent Seer says most of the real failures live.
- Treat thin documentation as an evaluation risk, not just a UX nuisance. If your server's docs are sparse, both human integrators and automated eval pipelines will struggle with it in the same way.
None of this requires waiting for a public release. It's the same discipline the paper's pipeline formalizes — just applied by hand until (or if) Apple or someone else ships an implementation.
Related reading
- Introducing MCP servers on explainx.ai — browse the MCP servers directory this post's guidance applies to
- What is MCP (Model Context Protocol)? — protocol fundamentals
- Build your first MCP server: step-by-step guide — where to apply Agent Seer's schema-quality lessons
- MCP tool descriptions: how to write them for reliable agent selection — the companion problem of tool-name selection vs. argument accuracy
- How to read an AI benchmark and not get fooled — why benchmarks that don't track the live system go stale
- AI benchmarks in 2026: the complete guide — broader evaluation landscape
- Top 10 MCP server directories in 2026 — where to find and compare MCP servers
- RAG vs MCP: complete comparison — MCP fundamentals in context
Official source: Agent Seer on arXiv (2608.26133)
Paper details, dates, and findings above reflect the publicly available abstract and introduction as of August 30, 2026 — the full paper may contain additional methodology and results not summarized here.