GPT-5.6Production MigrationAgent EvalsPrompt CachingTool SchemasClaude Opus
Ploy did not "switch models." They switched a stack — and discovered the stack was tuned to Opus.
On July 9, 2026, Lorenzo Gentile published how Ploy moved its website-building agent from Claude Opus 4.8 to GPT-5.6 Sol after four months where nothing beat Opus on their bar. Hacker News gave it 132 points — with a parallel argument about whether the prose sounded LLM-generated. Fair complaint. The engineering underneath is what enterprises should steal.
This pairs with Systima's harness overhead study: tokens are not the model; they are model + harness + cache config + tool contracts.
Catch: Raw first eval run lied until they fixed the harness, schemas, cache keys, and reasoning replay.
Step 0 — Your harness is tuned to your incumbent
Ploy runs hundreds of cases against real fixture workspaces — homepage builds, clone-safety checks. Scoring includes:
Visual judge: 10 binary design checks
Content assertions
Tool-trajectory checks
File assertions
Full trace triage on every failure
First cross-model surprise:
"Your harness is tuned to your incumbent model, and you don't know it."
Harness assumption
Opus behavior
GPT-5.6 behavior
Tool-call budgets
Sequential
Parallel fan-out → budget blow
File reads
Rarely batched
Constant batch reads → executor unsupported
Pass threshold
Missing minScore → default 1.0
0.98 hero "failed" while passing checks
~⅓ of first-run failures were harness — not evenly distributed between models.
explainx.ai rule: Before any model procurement decision, run private evals with trace triage as a gate. Same lesson as 37% lab-to-production gap on public benchmarks — worse in private, because your harness is bespoke.
Step 1 — Tool schemas are provider behavior, not prompts
Silent corruption: GPT-5.6 sent all 25 code-tool properties on every call.
// Optional → required + nullable at provider boundary// anyOf: [T, null] — model expresses honesty// Strip nulls at single invocation seam before validation
Lesson: "The model" in production is model + tool contract. Structured output guides help; provider-specific emission patterns need boundary transforms.
Step 2 — Prompt caching is not one feature
Surface: both vendors offer "prompt caching." Under the hood: different designs.
Claude (Anthropic)
cache_control breakpoints
~29K static prefix (tools + system)
Org-scoped shared cache — 92–96% hit rates, fades to background
Key is cache identity — same prompt, different key → zero hits
~15 requests/minute per cache node — global key → fan-out to cold nodes
Ploy's workspace-scoped key
snippet
request ──► hash(prompt head + prompt_cache_key) ──► cache node (~15 rpm/key)
[ tools + static prefix ]············ A every session
[ + workspace context ]·············· B same context
[ + turn history + latest ]·········· C this session
Key strategy
First-call hit
Problem
Per-conversation
0%
Never shares prefix
One global
Spill at scale
15 rpm budget blown
Per-workspace
83.7% after fix
Sweet spot
Entry B self-heals: workspace memory change misses B, still hits A — one context write vs full 29K re-bill.
No workaround: cross-workspace static prefix sharing is structurally impossible on OpenAI vs Anthropic org cache. Every workspace pays ~$0.18 per idle-window cold write — bounded, predictable.
Post-fix: uncached input −28%; GPT-5.6 per-suite cost below Opus. The 50% "GPT is expensive" gap was config, not list price.
Debugging corollary: server-side reasoning state means effective prompt can change upstream even when your client bytes are append-only. Log at the boundary.
Quality caveats — design convergence
Ploy's honest read: GPT-5.6 ships clean, modern, gridded layouts — can ignore existing design systems without steering. Opus reproduced Clay brand system; GPT-5.6 shipped generic-but-sharp.
Visual score still won on their suite — but brand adherence needed harness/workflow fixes (separate post promised).
HN thread split: some preferred Opus aesthetics for marketing; others cared about time × dollars × score.
Enterprise framing: define "good" in charter — visual judge + brand rubric + human slice, not one number.
Migration checklist (explainx.ai)
Phase
Action
0. Harness audit
Re-run incumbent vs challenger; triage traces; fix budgets, batching, thresholds
1. Tool boundary
Provider-specific schema transforms; null-strip seam; no tool impl churn
2. Cache design
Map breakpoints; choose key scope; measure first-call hit % cold
3. Reasoning
store: false or equivalent; verify mid-session continuity
4. Regression suite
Freeze golden workspaces; version harness with evals
5. Cost dashboard
$/build, p95 latency, human edit ratio — not pass@1 alone
Top comments debated LLM writing style ("llmish"). Substance comments that aged well:
Consistency > one benchmark run for production agents
Luna routing for tool-touching subtasks (5× samples per Sol dollar)
Subagent isolation — research duplication eats uncached tokens (Systima 4.2×)
Fable extension to July 19 kept competitive pressure on OpenAI subs
We cite Ploy for engineering, not copy tone. If you publish migration numbers, human-edit the prose — credibility is part of the metric.
explainx.ai read
Ploy's post is the production complement to Nadella's essay and Systima's microscope:
Eval harness bias is invisible until you switch models — ~⅓ failure rate inflation
Tool verbosity is a silent correctness bug — not a prompt problem
Cache key design is architecture — 0% vs 84% first-call hits
Cross-vendor "caching" is homonym — procurement must compare $/successful-task at steady-state cache
Migration is a loop — eval fail → trace → harness fix → re-run (compound learning)
GPT-5.6 Sol winning on website agents does not generalize to your compliance workflow — run your private benchmark. Ploy proved the method; you bring the tasks.
Migration details reflect Ploy's July 2026 publication and public HN discussion. Pricing, cache TTLs, and model IDs change — verify against current OpenAI and Anthropic docs before production cutover.
