FDE-Gym: A Benchmark for the Last Mile of AI Delivery, Where the Best Agent Resolves 14 of 95 Tasks

Most agent benchmarks hand the model a clear task and check whether it was done. Real work rarely arrives that way. FDE-Gym tests the messier version: a customer asks for something that may be the wrong thing, key rules are never stated, and the answer has to work in a live system. The result is sobering. Among 13 model configurations from seven providers, the best one, Claude Opus 5.5, averaged 0.56 and resolved just 14 of 95 tasks.

The paper on arXiv comes with a GitHub repository under Apache-2.0 and a dataset on Hugging Face. The authors call it a test of the "last mile of AI delivery". For anyone selling or buying agent-built software, it is one of the more useful reality checks to appear recently.

TL;DR

What is a forward-deployed engineer, and why test for it?

A forward-deployed engineer (FDE) sits with a customer and turns a vague business need into working software inside the customer's own systems. The job is less about writing code than about asking the right questions: is the sponsor's request aimed at the real problem, which rules are out of date, who holds the critical information, what must not be touched?

That is exactly where agents are being sold into the enterprise, and where typical benchmarks say little. The authors point out that well-defined coding tasks are now solved at high rates, so they build a test where the difficulty is in the situation, not the syntax.

How is the benchmark built?

Scenarios and variants. Industry experts adapted 20 scenarios from real projects. Each has an original task plus two to seven variants that change a key fact, such as which rule is outdated or who holds critical information, for 95 tasks in total.

Environment. The agent works in a copy of the customer's system with documents, data and a staff directory. It can interact with more than 130 scripted stakeholders who have limited patience, conditions under which they disclose information, and contact restrictions. The delivered system is deployed unchanged and tested on held-out data.

Four instruction levels. The same environment and grader run at four levels:

• L0: the problem, rules and scoring are stated.
• L1: the problem is stated.
• L2: a vague but correct request.
• L3: the sponsor's original request, which may be misdirected. This is the main setting.

Scoring. Business uplift is measured on the customer's own metric, normalized so the do-nothing baseline is 0 and the expert reference is 1. Incidents, such as unauthorized actions, set positive uplift to zero. Stakeholder contact cost charges for messages, trial runs and waiting. The final score is adjusted uplift minus contact cost, and a task is resolved at 0.8 or higher with no incident.

Guardrails against gaming. A task is admitted only if the reference scores at least 0.95, the do-nothing baseline scores 0, the literal solution scores at most 0.2 and shortcut strategies score at most 0.8. The authors say they tested the graders against more than 270 shortcut and bypass strategies.

What did the results show?

• The best model resolves few tasks. Claude Opus 5.5 scored 0.56 at L3 (95 percent confidence interval 0.50 to 0.62) and resolved 14 of 95. Five configurations resolved none.
• Ensembles do not fix it. Picking the best configuration per task reaches only 0.70 and 30 resolved tasks.
• Constraints are the bottleneck. The three leading models score 0.73 to 0.82 at L0 but 0.39 to 0.62 at L1, so the loss comes from having to discover rules, not from the later move to a misdirected request.
• Knowing facts is not enough. Leading models recover 72 to 85 percent of key facts but reach only 0.60 to 0.70 of the reference uplift, and contact costs remove another 0.14 to 0.22.
• Weak models can do harm. Claude Haiku 4.5 scores below the do-nothing baseline on 79 of 95 tasks after contact costs.
• Scenario matters a lot. In five of 20 scenarios, no model averages 0.5 or more.
• Spend does not track score. Cost and turn counts explain little of the ranking.
• Other benchmarks flatter agents. Models that solve 61 to 90 percent of tasks on Terminal-Bench 2.1, Vibe Code Bench and Finance Agent score 0.19 to 0.56 here, though the authors caution the metrics are not directly comparable.

For context on how the top model fares elsewhere, see our posts on Claude Opus 5.5's launch benchmarks and the FrontierCode maintainer-merge benchmark.

What does a misdirected request look like?

The L3 setting is the one that matters, and the idea is easy to miss. Suppose a customer's operations lead asks for a dashboard that flags late shipments. The real driver of lost revenue turns out to be an outdated rule in a pricing system that nobody mentioned, and the only person who knows it is a colleague who will answer only if asked the right question. An agent that builds the dashboard exactly as requested produces a literal solution and, by the benchmark's admission checks, would score at most 0.2. An agent that asks the right stakeholder, finds the rule, and fixes the pricing logic can reach the reference score.

That scenario is our illustration, not one of the benchmark's tasks, but it captures the pattern the authors describe: variants change a key fact, such as which rule is outdated or who holds critical information, so memorizing one scenario does not transfer. A model has to ask, listen and verify in each case.

The scoring then pushes against two bad habits. Agents that never ask miss constraints and deliver the wrong thing. Agents that ask everything burn contact cost, because messages, trial runs and waiting all carry charges. The best behavior is targeted, early questions, which is exactly what human forward-deployed engineers are valued for.

How does it compare with other agent benchmarks?

Terminal-Bench, Vibe Code Bench and Finance Agent measure whether an agent can execute well-specified work in a controlled environment, and strong models now solve 61 to 90 percent of those tasks according to the paper. FDE-Gym differs in three ways: the request can be wrong, success is measured on the customer's own business metric after deployment, and asking questions is priced. The paper situates it near work on ambiguity (such as Ambig-SWE and LHAW), discovery (TheAgentCompany) and agent construction (tau-bench), and says its distinguishing features are misdirected requests, a business-metric outcome and a cost for asking.

Because the authors warn that the numbers are not directly comparable across benchmarks, the safest takeaway is directional: when ambiguity and hidden rules are added, the same models lose most of their headline performance. That echoes our reading of Pine's SaaS-Bench claims, where a high headline number depended on the exact setup.

What does it mean for people deploying agents?

Expect the gap between demo and delivery. An agent that writes correct code on a clear ticket may still fail a customer whose requirements are partly unspoken. If you resell agent-built work, price in a human who owns discovery.

Budget for questions. The benchmark charges for contact, because real stakeholders have limited patience. Agents that ask nothing miss constraints; agents that ask too much run up costs. Evaluate both.

Watch incidents, not just output. Unauthorized actions zero the score. This matches concerns we raise in agent security coverage, such as the APEX-Harm study showing correct deliverables can hide unsafe actions.

Use it on your own stack. Because the harness is open, you can plug in your own agent framework and compare, rather than relying on a vendor's number.

What are the limitations?

The authors list them plainly:

• Stakeholders are scripted, so real human unpredictability is not captured.
• Only models were tested; there is no human baseline yet, although an online platform for humans to attempt the same tasks is being used to collect one.
• All models ran in one minimal agent framework, which may under- or over-state some vendors.
• Sample sizes at L0 to L2 are small, with only 20 tasks each.
• Contact costs are expert estimates, not measurements.
• Training-data contamination cannot be ruled out.

Also note the public availability: the 20 original tasks and graders are available on request with three public examples, while the 75 variants and reference solutions are held out. That protects the benchmark but limits independent replication.

How to try it

The README gives these steps, which require Docker, Python 3.9 or later and an OpenRouter key stored outside the repository:

git clone https://github.com/deeplethe/fde-gym.git
cd fde-gym
mkdir -p ~/.fdegym
python3 scripts/fetch_cases.py
docker compose up --build

Store your OpenRouter key in ~/.fdegym/openrouter_key before starting. The site runs at http://127.0.0.1:8787 with a runner, PostgreSQL and an S3-compatible object store. The first account to sign up becomes admin. Because the site executes submitted code, the README recommends reading its self-hosting guide before opening it to others. A cheap model plays the customer's people, and choosing the coding agent's model is listed as a roadmap item. Scoring runs from 0 for changing nothing to 100 for the reference solution, and 80 or more counts as solved. The README asks users not to train on the cases.

What we still do not know

• How humans score on the same tasks.
• How results change with a stronger agent framework or with tools tuned for discovery.
• Whether newer models close the constraint-discovery gap, given the benchmark's results are for the models tested at the time of the paper.
• How well scripted stakeholders predict real customer behavior.

Related reading

• Claude Opus 5.5 launch, benchmarks and pricing
• FrontierCode: maintainer-merge benchmark and Opus 5.5
• APEX-Harm: agent safety in shared workspaces
• WipeBench: coding agent safety benchmark
• Pine SaaS-Bench claimed vs verified
• Claude Opus 5.5 vs Sonnet 5.5

Based on the FDE-Gym paper and repository as of October 16, 2026; results are the authors' and have not been independently reproduced.