On September 1, 2026, Parallel Web Systems published How to eval web search for AI — a methodology post from @everythingmeta (MTS at Parallel) that answers a question explainx.ai readers keep hitting after OpenRouter's search benchmarks: how do you compare Exa, Parallel, Perplexity, and native model search fairly inside a real agent?
The short answer: do not score search snippets in a vacuum. Score answers from a fixed agent harness.
TL;DR — eval equation
Agent Harness + LLM + Search (+ Extract) = Answer
| Rule | Why |
|---|---|
| Hold constant | Same model, prompts, judge, turn budget |
| Swap only | Search (and Extract if applicable) provider |
| Allow multi-turn | Agents search → read → search again |
| Cap total budget | Reflect production cost, not unlimited turns |
| Grade | Final answer vs gold labels — not intermediate SERPs |
Search vs Extract vs Task API
| API | Agent use | Eval mistake |
|---|---|---|
| Search | Ranked, token-dense excerpts | Judging snippet BLEU instead of answer |
| Extract | URL → compressed content | Forgetting to wire fetch after search |
| Task | Long-horizon research jobs | Comparing to single-hop Search latency |
Parallel's public benchmarks page runs GPT-5.4-class agents with up to 20 tool calls (search_web plus web_fetch where Extract exists). Match your harness to what you ship — search-only providers (Brave, SerpAPI in their table) behave differently from Search+Extract stacks.
Building a gold set you trust
Parallel's first principle: no eval without labels you believe.
- Start from production logs — anonymized user tasks beat synthetic trivia
- Mix task shapes — factual, list, structured JSON, open research
- Timestamp freshness — include breaking-news items (WISER-Fresh style) so stale indexes fail loudly
- Verify ground truth — human sign-off or dual-label agreement
- Version the set — git-track prompts + labels; rerun when providers change
OpenRouter found search budget (allowed turns) moves scores as much as engine choice — your gold set should specify max tool calls and max USD per task.
Harness setup — isolate the search tool
Step-by-step from Parallel's guide:
- Expose one search provider as an MCP tool or function (see MCP search patterns)
- Keep Extract configured identically when the provider supports it
- Fix the LLM and system prompt — no provider-specific prompt hacks in v1
- Log every tool call, latency, token use, and final answer
- Re-run with the next provider — same harness binary, different API key
This mirrors Terminal-Bench-style harness engineering: the middleware is the experiment.
Parallel best practices (API + agent prompts)
From Search API best practices:
| Lever | Guidance |
|---|---|
objective | Natural-language research goal — Parallel cites +10–20% vs keywords-only |
search_queries | 1–3 keyword queries, 3–6 words, entity in each |
| Modes | turbo (speed), fast (default agent balance), advanced (max quality) |
| Avoid | site: and after: operators in agent-generated queries — brittle for agents |
| Both fields | Send objective and queries when the model can fill both |
Example request shape (fast mode):
curl https://api.parallel.ai/v1/search \
-H "x-api-key: $PARALLEL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"mode": "fast",
"objective": "Find latest NVIDIA stock price and today'\''s move percent",
"search_queries": ["NVIDIA stock price today", "NVDA daily change"]
}'
Wire the Search Tool Definition into your agent so the model fills objective — do not substitute hand-written queries in eval unless that is your production path.
Grading — LLM judge + human audit
| Task shape | Judge tip |
|---|---|
| Factual | Exact match or normalized number/date |
| List | Set F1 on items; order may not matter |
| Structured | JSON schema validation + field-level match |
| Open research | Rubric + cite-required sources |
Use a strong judge model (Parallel's public benchmarks reference GPT-5.4 graders). Always hand-check ≥10% of successes and failures — judges miss subtle hallucinations and outdated facts.
Failure taxonomy — four buckets
Track separately in your spreadsheet:
| Bucket | Symptom | Fix lever |
|---|---|---|
| No search call | Agent answered from parametric memory | Prompt / tool-choice harness |
| Provider error | 429, 524, timeout | Retries, mode downgrade |
| Retrieval miss | Search ran; right URL never appeared | Provider, mode, objective wording |
| Synthesis failure | Good excerpts; wrong conclusion | Model, prompt, Extract step |
OpenRouter's BrowseComp data fits here: failed runs averaged more searches than successes — burning budget without fixing retrieval.
Cost, Pareto frontier, confidence intervals
Report cost per resolved task (search API + LLM tokens), not API list price alone. Plot accuracy vs cost per provider — Parallel's Search MCP benchmark claims higher accuracy at up to 50% lower total cost vs native web search on WISER blends, but your curve depends on turn budget and model.
Publish methodology:
- Gold set size and date range
- Harness version hash
- Judge model + rubric
- 95% CI on pass rate (Wilson interval works for binary tasks)
- Raw failure taxonomy counts
End-to-end eval checklist
- Gold labels verified by a human you trust
- Harness + LLM + judge fixed across providers
- Only search/extract tool swapped per run
- Multi-turn allowed within production budget cap
- Task shapes cover factual, list, structured, open research
- Log tool calls, latency, tokens, USD per task
- LLM judge + ≥10% manual audit
- Failure taxonomy tagged on every miss
- Pareto chart: accuracy vs cost per resolved task
- Confidence intervals published alongside point estimates
- Re-run when provider ships new modes (turbo/fast/advanced)
Related on explainx.ai
- OpenRouter Web Search Benchmarks — engine vs budget
- Agent harness engineering — Terminal-Bench gains
- AI benchmarks complete guide
- RAG vs MCP — when to use live search
- Firecrawl keyless agent web search
- Perplexity Search API index debut
- Max Stoiber joins OpenAI Plugin Platform — connectors layer
Primary sources: Parallel — How to eval web search for AI · Search API best practices · Parallel benchmarks · @everythingmeta (Sep 1, 2026)
Benchmark figures and mode names reflect Parallel's public docs as of September 1, 2026. Re-run evals before production provider lock-in. Follow @explainx_ai for updates.