OR-License: Can LLM Agents Do Real Operations Research Jobs?

Benchmarks for coding agents mostly hand the model a tidy problem. Real operations research (OR) work does not: a client sends spreadsheets, a vague requirement and a deadline, and someone has to work out the model, ask the right questions, write the solver code and deliver something that actually runs. OR-License is a new benchmark that tries to test exactly that. Its headline finding, from the authors' abstract: across 11 frontier LLMs using coding-agent harnesses, several beat the expert's solution quality on tasks where they succeed, but none matches the expert once failed deliveries are counted.

The paper (arXiv), project site, code and dataset are public. If you track agent evaluations, it sits alongside work like the FDE-Gym forward-deployed engineer benchmark and Real-SWE on private codebases, which also move away from clean, self-contained puzzles.

TL;DR: the questions most readers ask

Why is operations research a good stress test for agents?

OR is the discipline of optimizing real systems: routing, scheduling, inventory, portfolio construction, bike-sharing logistics. The hard part is rarely the solver. It is formulation, meaning deciding what the variables, constraints and objective should be from messy business descriptions, then implementing it faithfully and validating the output.

That makes it a natural test for the long-horizon agent behavior everyone is talking about. An agent has to acquire information, make modeling choices, write code, run solvers, check results and deliver in a required format. Mistakes compound, which is the same dynamic our loop engineering guide discusses for coding agents in general.

A benchmark scoreboard illustrating the gap between clean benchmarks and real client work

How is OR-License built?

According to the paper abstract and repository, the design has four pieces.

1. Real engagements. The 31 instances come from paid OR work. After de-identification they preserve the original requirements, input data, deliverable formats and information boundaries, which is what separates them from textbook problems.
2. ORSim, a client simulator. Rather than giving the agent everything up front, the benchmark includes a controlled simulator that discloses business facts when asked, without supplying OR expertise. The agent must work out what to ask. Per the README, a DeepSeek model powers the simulator and the source reviewer.
3. Hidden evaluation. Task-specific evaluators check feasibility and compute objectives, and selected tasks also require a review of the submitted optimization source. Quality is compared against the original expert solutions the clients accepted before the benchmark was built.
4. DAQ. Deployment-Adjusted Quality averages expert-relative quality across instances and assigns zero to infeasible deliveries.

The DAQ choice is the interesting one. Plenty of benchmarks report quality only on successes, which flatters unreliable systems. By scoring infeasible output as zero, OR-License measures what a client would experience: a plan that breaks a constraint is not partly useful, it is unusable.

What did the results show?

The abstract's findings, in the authors' words condensed:

Two readings follow. First, the gap is about reliability, not peak ability: models can occasionally out-optimize a human expert, but they fail often enough that the average, once failures are zeroed, stays below the expert. Second, the authors find that information acquisition goes beyond clarification. Agents also seek client information during solution development and validation, and some later stages show higher confirmed issue-resolution proportions. In plain terms, asking the client only at the start is not enough.

Trajectory analysis points to two dominant failure types, formulation errors (modeling the wrong problem) and implementation errors (modeling the right problem incorrectly). The authors say these findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for long-horizon agents.

We did not see a per-model leaderboard in the sources we could read, so we are not going to rank the other ten models. Check the paper tables for that.

How does this compare with other agent benchmarks?

OR-License belongs to a growing set of evaluations built from real professional work. Our AI benchmarks complete guide and how to read AI benchmarks explain the general pitfalls. What is distinctive here:

• Expert-accepted references. The comparison point is a solution a client actually signed off on, not a synthetic gold answer.
• Deployability scoring. Zero for infeasible output ties the metric to a real-world cost.
• Information boundaries. The simulator forces the agent to ask, rather than reading a perfect spec.
• Small and heterogeneous. 31 instances across 10 scenarios is rich per task but statistically thin, so single-point differences between models are fragile.

For the cost of unreliable agent output more broadly, see our coverage of the verification tax, the extra checking and redoing that AI work needs.

How do you run it?

The README lists the setup. You need Python 3.12, Docker Engine with the Compose plugin, and API access to the models you test. Harbor 0.18.0 is pinned in the requirements.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python scripts/download_data.py
docker build -t or-license:open harbor_integration/coding_agent/image

You then copy .env.example to .env, set TESTED_API_BASE, TESTED_API_KEY and DEEPSEEK_API_KEY, edit the model identifier in llm_config.yaml, and run a preflight and a task:

python run_eval.py --task 3_food3 --model tested --validate-only
python run_eval.py --task 3_food3 --model tested --tag example-run

Practical notes:

• Data size. The complete download is about 3.4 GB, of which 0.6 GB is shared-bike observations. Tasks 1 to 19 and 23 need no download.
• Solver licenses. Financial tasks 20 to 22 need a separate environment and licenses: task 20 requires MOSEK, tasks 21 and 22 require COPT. You obtain those yourself.
• Harness. Claude Code is the default; --agent codex switches to Codex.
• Cost. The README does not give a per-run API cost, and running the full suite across models will not be cheap, so start with a single task.

A code-block frame with a row of passing green dots, illustrating hidden feasibility checks on an agent's submitted solution

What can a team building an optimization agent borrow today?

You do not need to run the full suite to learn from its design. Three habits transfer directly to your own agent workflows.

First, score deployability, not just quality. If your agent produces schedules, routes or allocations, write an independent checker that validates every hard constraint and count a failure as zero, exactly as DAQ does. A model that is brilliant half the time and invalid the other half looks fine on average-of-successes charts and terrible in production.

Second, separate fact-finding from solving. Give the agent a way to ask for business facts, log what it asked, and flag assumptions it made without asking. The paper's observation that information acquisition continues into development and validation suggests prompting for a question round at each stage, for example:

Before you finalize the model, list every business fact you assumed
rather than confirmed, and ask for each one. After the first solution,
list what in the data or constraints would invalidate it.

Third, review the formulation before the solver runs. The two failure families the authors identify, formulation and implementation, are different bugs needing different checks: a human or second model reading the problem statement against the model, and tests that run the code against known small instances.

Limits and open questions

• Authors' evaluation. We have not seen independent reproductions.
• Small N. 31 instances is small; model rankings could shift with a few tasks.
• Simulator dependence. ORSim is itself an LLM-backed system, so results depend on how faithfully it reveals facts.
• Harness confounds. Results mix model and harness effects, since agents ran inside coding-agent harnesses, so a ranking of models is also partly a ranking of harness fit.
• Source-page lag. The project page still lists the paper link as coming soon, while the repository and Hugging Face already link the arXiv paper.

Why it matters

If you are deciding whether to trust an agent with a professional-grade optimization task, the useful number here is not the best-case quality but the deployability-adjusted one. OR-License suggests that frontier agents are already competitive with experts on what they can finish, and that the remaining distance is reliability and information-gathering discipline. For teams, that points to concrete mitigations: independent feasibility checks on every output, structured client or stakeholder question rounds during the work, not just at the start, and human review of the problem formulation before any solver time is spent.

Related reading

• FDE-Gym: forward-deployed engineer benchmark
• Real-SWE: coding agents on private codebases
• AI benchmarks: the complete guide
• How to read AI benchmarks
• Loop engineering for coding agents with Claude Code
• The AI verification tax
• Primary sources: paper page, arXiv, project site, GitHub, dataset

Details are accurate as of October 20, 2026, and come from the authors' abstract, README and project page.