OpenRouter Web Search Benchmarks: How to Pick a Search Tool for Agents

If you're wiring web search into an agent, the engine you pick matters less than how many search turns you let it take. That's the headline finding from OpenRouter's Web Search Benchmarks, published August 14, 2026 — a public leaderboard ranking combinations of models and search backends across four different evaluation sets. Perplexity co-founder and CEO Aravind Srinivas quote-tweeted the launch with a single crown emoji; the underlying data is more useful than the flex.

OpenRouter tested four models across four search depths on Exa, Parallel, Perplexity, and each model's own native search, ran all of those combinations against four benchmarks, and ranked every combination by quality, cost, and speed. If you're building anything that needs to ground an agent in live web results — a research assistant, a fact-checker, a pricing bot — this is the first apples-to-apples comparison of the tradeoffs, not just a single provider's marketing benchmark.

TL;DR — what the data says

The four benchmarks, and what each one actually measures

OpenRouter didn't pick one benchmark and call it done — a single eval tends to reward one narrow skill. Each of the four tests a different failure mode agents hit when they search the live web:

BrowseComp and HLE reward an agent's ability to chase down one hard-to-find answer through several dependent search steps. DeepSearchQA punishes agents that pad list answers with irrelevant filler — completeness matters, but so does precision. WideSearch is closer to a real analyst task: build a table, not a paragraph, and get credit for partial matches rather than an all-or-nothing grade.

That spread is why the leaderboard doesn't have one winner. Perplexity paired with Claude Opus 5 at high reasoning effort took top quality on three of the four; GPT-5.6 Sol paired with Perplexity took the fourth. No engine or model swept everything, which is the actual lesson — a leaderboard-topper on one workload is not automatically the right default for yours.

The finding that should change how you build: search budget beats everything else

OpenRouter's own framing is direct: "Of all the factors we tested, increasing the search budget had the biggest impact on results." Going from a budget of 1 turn to 25 turns roughly doubles scores on BrowseComp — a bigger effect than switching engines or switching models produced independently.

That reorders the default engineering instinct. Most agent-search integrations spend their first design pass picking a provider — Exa vs. Perplexity vs. a lab's built-in web_search tool — and treat the turn limit as an afterthought capped low to control cost. OpenRouter's data says that's backwards for quality-sensitive workloads: the turn budget is the lever with the largest single effect, and it's the one most agent builders under-tune.

Model choice edges out engine choice — but neither dominates

Two more comparisons from the same dataset, in order of effect size:

1. Swapping models moved scores more than swapping engines. The average difference between a frontier model and a cost-efficient model was ~15 points. Holding the model constant and swapping the search engine shifted scores by ~10 points on average.
2. A lab's own native search is not automatically your best option. OpenRouter's example: native search on GPT-5.6 Sol was roughly 50-50 against third-party engines across the benchmarks — a coin flip, not a clear win. Which engine wins depends on the workload, not on which lab shipped the model.

The practical read: if you're optimizing an agent's search stack, spend your first tuning pass on model + turn budget, and treat the engine choice as workload-specific rather than a fixed default. "Just use the built-in search tool because it's one line of code" is a real cost-saver, but OpenRouter's data says it's not a quality guarantee.

The cost trap: failed searches are expensive, and budget doesn't fix bad queries

The most actionable finding for anyone paying per-search or per-token: failed searches burn far more turns than successful ones. On a 25-turn BrowseComp budget, models averaged 10.3 searches when they got the answer right and 19.7 searches when they got it wrong.

That's not a small gap — an agent heading toward a wrong answer uses almost double the search calls of one heading toward a right one, and every one of those extra calls is billed. Two consequences follow directly:

• Raising the turn budget helps quality on the margin, but it also compounds the cost of the failures you were already going to have. A generous budget doesn't rescue a bad query strategy; it just lets the agent search longer before giving up on the wrong path.
• If you already expect a high failure rate on a given task type — ambiguous queries, sparse-coverage domains, adversarial or trick questions — cutting the turn budget is a legitimate cost control, not just a quality tradeoff. You're mostly cutting the expensive failure tail, not the cheap successes.

For teams tracking monthly AI agent spend, this is the kind of lever that's easy to miss in a token-cost spreadsheet: the marginal cost of a search-grounded agent isn't dominated by tokens per call, it's dominated by how many calls a wrong turn generates before the budget runs out.

Cost and speed: no engine wins everything

Quality leaderboards get the attention, but the value and speed columns tell a different story — and matter more once you're running at volume:

• Cheapest strong result: Parallel + DeepSeek V4 Flash 0731 (high) hit $0.091 on DeepSearchQA — nowhere near the quality leader's cost, and not the quality leader either (Perplexity + Claude Opus 5 led that benchmark's quality at $0.16 on HLE, roughly 89% more expensive per query for the top result).
• Fastest strong result: Perplexity + GPT-5.6 Luna (xhigh) posted 1.6 minutes on DeepSearchQA, faster than the Claude Opus 5 pairing that led quality on the same benchmark.
• HLE was the outlier on speed: Perplexity + Claude Opus 5 (high) completed HLE in 48 seconds — far faster than the multi-minute times on the other three benchmarks, likely reflecting HLE's narrower, single-answer question format versus BrowseComp's multi-step chase or WideSearch's table-fill task.

None of the four benchmarks had one combination winning quality, cost, and speed simultaneously. That's the actual takeaway for anyone building a production system rather than chasing a leaderboard number: pick the axis that matters for your product, then check which combination wins on that axis for a benchmark that resembles your actual task — not the aggregate leaderboard topper.

What this means for how you build search-grounded agents

1. Tune search budget before you tune the provider. It's the single largest lever OpenRouter measured — bigger than model or engine swaps individually.
2. Don't default to a lab's native search tool on quality-sensitive tasks without testing. It's a reasonable default for cost and simplicity, but OpenRouter's ~50-50 result against third-party engines means it's not a safe quality assumption.
3. Match the benchmark to your task shape, not the aggregate leaderboard. A table-filling research agent should weight WideSearch results; a fact-verification agent should weight BrowseComp or HLE.
4. Cap turns aggressively on tasks with a known high failure rate. The 10.3-vs-19.7 turn gap between correct and incorrect runs means uncapped budgets mostly pay for the failures, not the wins.
5. Re-check this periodically. OpenRouter's benchmark runs are dated (August 11, 2026, at time of writing) and search-tool quality shifts as engines and models update — treat the leaderboard as a snapshot, not a permanent ranking.

Bottom line

OpenRouter's Web Search Benchmarks are the first public, cross-provider comparison of search grounding for agents, and the standout finding isn't which engine wins — it's that how many search turns you allow matters more than which engine or model you pick. Perplexity paired with Claude Opus 5 at high reasoning effort leads quality on three of four benchmarks, but no single combination wins on cost, speed, and quality together, and a model lab's own native search is a coin flip against third-party alternatives rather than a safe default.

If you're building or tuning a search-grounded agent, the fastest win available from this data isn't a provider swap — it's checking whether your turn budget is actually calibrated to the task, and whether you're capping runaway searches on the queries most likely to fail.

Related on explainx.ai

• RAG vs MCP: complete comparison — how live search grounding differs from retrieval over your own indexed corpus
• Perplexity's Search as Code: rethinking search for the agentic era — Perplexity's own architecture for programmable, agent-driven search
• Perplexity's Wandr benchmark for open-source research agents — a related open benchmark effort from one of the engines tested here
• AI agent monthly cost: a real workflow breakdown — where search-call volume shows up in an actual agent spend sheet
• AI benchmarks: the complete guide — how to read benchmark methodology and avoid leaderboard traps
• Claude Sonnet 5 vs GPT-5.6: Luna and Max compared — deeper comparison of two of the models tested in this benchmark
• What are agent skills? A complete guide — packaging retrieval and tool-use patterns like search grounding into reusable skills
• What is MCP? — the protocol most agents use to wire in a search tool like the ones benchmarked here

Primary source: OpenRouter, "Web Search Benchmarks" (openrouter.ai/benchmarks, launched August 14, 2026) · OpenRouter announcement thread on X, August 14, 2026

Accurate as of August 15, 2026, based on OpenRouter's benchmark runs dated August 11, 2026. Benchmark rankings on openrouter.ai/benchmarks update as models and search engines change — check the live leaderboard for current standings. Follow @explainx_ai for updates.