Armature Study: What Claude Code, Codex, and Cursor Actually Pick

On September 3, 2026, a YC-backed startup called Armature published a study titled "Which tools do Claude Code, Codex and Cursor choose? We measured 16,893 sessions to find out." It hit Hacker News the same day as "Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out" and pulled 92 points. The headline number worth sitting with first: only 42% of comparable sessions saw all three agents pick the same tool for the same task in the same repo.

Before any of the data, one sentence from Armature's own post needs to stay visible through the rest of this one: "Disclaimer: Armature sells growth services to dev tools. This study is part of our broader work on how to influence coding agents choices and get products picked." Armature is not a neutral research lab. It is a company whose revenue depends on getting vendors chosen by exactly the systems this study measures. That's not a reason to throw the data out — the methodology is unusually rigorous for a vendor blog post, and the numbers are checkable — but it is a reason to read every finding with a raised eyebrow, especially anywhere the write-up edges toward "here's how you'd move an agent's decision." We covered a related, less rigorous version of this dynamic recently: a study found 215,128 machine-generated "best software" pages built specifically to be cited by Perplexity. Armature's study is the more sophisticated, more commercial cousin of that same story — a company that has turned "get picked by an AI" into a paid service line, publishing research about how that picking happens.

TL;DR

How the study was actually built

Armature's methodology is worth walking through in full, because it's the part of the post doing the most work to earn credibility — and the part worth checking hardest given who wrote it.

The team built a panel of 75 real GitHub repositories across 10 languages, deliberately correcting for the fact that public GitHub over-represents startups relative to enterprises. They scrubbed each repo with fake company names, fake git histories, and fake API keys, while keeping real lockfiles verified against npm and equivalent registries — the goal being repos that read as genuine codebases to an agent, not synthetic test fixtures.

On top of that panel, they ran four simulated personas: a vibe-coder who describes only symptoms ("emails aren't sending"), a junior engineer who names the category ("I need an email service"), a senior engineer who's precise about requirements and anti-requirements, and an enterprise engineer who raises compliance and procurement details. That's a reasonable proxy for how differently real developers prompt the same underlying task, and it produced 1,163 prompt variations, 20-25% of which deliberately varied cost or usage-volume mentions to test whether budget framing moved the outcome.

Sessions ran in ephemeral sandboxes rotated across three providers — E2B, Blaxel, and Daytona — specifically to rule out any one sandbox vendor's environment biasing the result. A "simulated human" (Gemini 3.7 Flash acting as an orchestrator) sat in the loop, either accepting the agent's top recommendation or telling it to just pick and implement — mirroring how a real developer interacts with an agent instead of letting it run fully autonomous. That detail mattered more than it sounds: letting the agent ask permission first, rather than being told to just build in-house, measurably reduced how often platform-native defaults won. Cloudflare R2 started beating Amazon S3 in object-storage sessions specifically when there was no human-in-the-loop step forcing a pause before the agent defaulted to the biggest name it already knew.

A second Gemini 3.7 Flash instance acted as judge — rejecting sessions where a repo was pre-biased toward one vendor or where the agent never actually made a choice, then identifying every tool mentioned in the conversation and the actual winner from the real code diffs, not just from what the agent said out loud. That diff-level check is the single most important design choice in the whole study, and it's the one that produces the "mentioned ≠ picked" finding below. Out of 16,893 raw runs, 5,292 valid sessions across 51 codebases and 18 sectors made the published cut, with Armature saying more waves may follow. All traces — prompts, model thinking, diffs — are published publicly, which is a genuinely higher bar than most vendor research clears.

Finding 1: the three agents barely agree with each other

Only 42% of "cells" — comparable prompt/repo combinations — saw Claude Code, Codex, and Cursor converge on the same tool. The rest split. Armature's example: for a voice-agent feature, Claude Code picked Twilio, Codex picked the OpenAI Realtime API, and Cursor picked Vapi — three different vendors for the same ask in the same repo. If you're comparing agent output across Claude Code, Codex, and Cursor for your own stack, this is the number to keep in mind: don't assume any one agent's suggestion represents consensus best practice. It represents that one model's training priors plus whatever it found searching (or didn't).

Finding 2: web search behavior is wildly different across agents

The three tools research the web at completely different rates and in completely different ways:

• Codex searches the web in 94% of sessions, and roughly 9 out of 10 of those queries use search operators like site: to target specific trusted domains — e.g. site:auth0.com password reset MFA social connections.
• Cursor searches in about two-thirds of sessions.
• Claude Code searches only about 30% of the time, leaning on training priors instead — but when it does search, it browses roughly 3x more pages per session than Codex, and in newer or less-established categories (sandboxes, per Armature's example) it searched around 80% of the time.

That's a real, practical difference if you're choosing between agents for a task in an unfamiliar or fast-moving category: Claude Code's default behavior is closer to "trust what I already know unless the category is genuinely new," while Codex's is closer to "always verify against a small set of trusted sources, narrowly." Neither is strictly better — they trade off recency against consistency differently.

Finding 3: Claude Code builds in-house nearly twice as often

Claude Code chose to build a feature in-house rather than adopt a third-party tool 19% of the time, versus 10% for Codex and Cursor. Whether that's a virtue (less vendor lock-in, no surprise pricing tier) or a cost (reinventing wheels that a mature SaaS already solved well) depends entirely on the task — but it's a measurable, consistent behavioral difference between the agents worth knowing before you pick one as your default.

Finding 4: the repo's programming language decides the winner

This is the most actionable finding for a working developer, and it's stark. For the exact same email-sending feature request, across four different languages, four completely different vendors won:

Deployment showed the same pattern even more sharply: Vercel won virtually every TypeScript repo (100% when Next.js was detected) and was never once recommended on a Python repo, where Render dominated instead. That's not a subtle statistical lean — it's a near-total split driven by ecosystem convention, not by any evaluation of which platform is actually better for the task at hand. If you're picking infrastructure based on what an agent suggests, the honest read is that it's often reporting "what's idiomatic in this language's ecosystem," not "what's objectively best" — those two things frequently coincide, but not always, and the study gives no way to tell which case you're in from inside a single session.

Finding 5: mentioned is not the same as picked

This is where the diff-level judging pays off, and it's the finding most worth internalizing if you're evaluating your own product's AI visibility (a theme we've covered in the AEO-measurement-tools space before):

A tool showing up constantly in an agent's reasoning trace, or even being weighed out loud, tells you almost nothing about whether it wins. Any vendor — or any GEO consultant selling "get mentioned by AI agents" as a service — measuring success by mention count alone is measuring the wrong thing by this data.

Finding 6: presentation details on a vendor's own pages can flip the decision

Armature found that some of what looks like a product-quality objection is actually a documentation or pricing-page framing problem. Mailgun regularly lost to Postmark specifically when agents read "1-day retention" language on Mailgun's free tier. Supabase often lost "database only" use cases because its bundled pricing page surfaces auth, storage, and realtime features the agent wasn't asked about, muddying a simple database decision. Only a minority of cost or platform-overhead objections in the dataset (388 and 195 mentions respectively, out of 5,292 sessions) were actually about disqualifying facts — many were about how information was presented on a docs or pricing page, not the underlying product tradeoff. That's the exact mechanism a growth-services vendor like Armature would want to know about and could plausibly sell as a lever — worth remembering when reading the rest of the post as "just data."

Finding 7: category leaders, when there is consensus

Where agents did converge, dominance was often lopsided: Stripe won roughly 9 of 10 payment sessions (losing mainly to Paddle or Mollie in EU-regulated cases), Neon won 66% of database sessions over cloud-native AWS/Azure options, Amazon S3 dominated file storage at 45% (Azure and GCP tied at 20% each), and Resend (35.6%) and Postmark (27.4%) led email closely. If you're weighing Stripe against alternatives for agent-driven commerce, this is independent-ish confirmation that agents converge hard on it as a default — though "independent-ish" is doing real work in that sentence given who ran the study.

Reading the room: the HN thread didn't let this slide

The Hacker News discussion pushed back on Armature's framing harder than the study's own disclaimer did. One commenter joked the field now needs a name — "SEA for AI agents (AEA?)" — a nod to "Agent Engine Optimization" becoming a real, salaried discipline, not a hypothetical one. Another was much sharper about the business model itself: "if your business is basically 'we do growth hacking and SEO tricks on models and get them to use products that aren't actually best for the job,' you are scum... Part of the reason people use AI is because of how useless search is due to the previous generation doing the same kind of thing you propose." A founder responded diplomatically rather than disputing the substance. And in a small, almost too-perfect irony, at least one commenter reported that Armature's own website has an onboarding-tour popup that gets in the way on mobile — a minor UX complaint, but a fitting one for a company whose product is, in part, about making other people's presentation choices work harder on an AI audience.

Why this is a genuine "SEO for AI agents" moment

The blog's own recent coverage of manufactured GEO farms citing Perplexity documented low-effort, high-volume content designed purely to be scraped and cited. Armature's study is the more professionalized, well-funded version of the same underlying dynamic — a company with actual research infrastructure (real repos, sandbox rotation, LLM-as-judge, published traces) studying exactly which levers move a coding agent's tool choice, then selling the answer as a service to vendors who want to be the tool that gets picked. The methodology here is legitimately better than a content farm's; the incentive structure is the same. Both stories point at the same underlying fact: AI recommendation systems are now a market, and someone is always optimizing to be recommended, whether that's 215,128 templated pages or a rigorously designed 16,893-session experiment. Neither kind of optimization pressure makes an agent's suggestion more likely to be objectively correct for your specific case — it just makes certain answers more likely to surface regardless of fit.

What this means if you're actually choosing a tool

• Don't treat an agent's first suggestion as a procurement decision. Ask it to justify the pick against your specific constraints (compliance, existing vendor relationships, EU data residency, cost at your actual volume) rather than accepting a default.
• Know that your repo's language is quietly voting. If you're in Python and get pointed at Render or SendGrid, that's partly "what's idiomatic here" — check whether a TypeScript-favored tool like Vercel or Resend would still serve you better if you're polyglot or migrating.
• Discount high mention-counts as a signal of quality. A tool an agent talks about a lot (LangChain, Supabase, Adyen in this dataset) isn't necessarily the one it will actually wire into your code.
• Read a vendor's pricing/docs page the way an agent would, i.e., literally — vague retention windows or bundled feature framing cost real vendors real recommendations in this study, and the fix on the vendor side is usually just clearer copy, not a better product.
• If you're building or teaching agent workflows, this is a useful case study for why "ask the agent what to use" is a starting point, not an audit — something we get into more generally in our agent harness guide.

Honest limitations

• Armature has a direct, disclosed financial interest in the outcome that agents can be influenced toward particular vendors — they sell exactly that service. Every finding here should be read as "data from a company that profits from this dynamic," not neutral academic research, even where the methodology is unusually transparent (published traces, disclosed judge model, disclosed sandbox rotation).
• The judge and orchestrator are both Gemini 3.7 Flash — a single model family making both the simulated-human decisions and the pick/mention classifications. That's not independent triangulation; it's one model's judgment applied twice, similar to the "one retrieval layer sampled twice" caveat that applied to the Perplexity GEO-farms study.
• This is the first published wave (5,292 of 16,893 sessions) — Armature says more may follow, so category leaders and win rates could shift as the dataset grows.
• We have not independently reproduced this study. The traces are public, which is commendable, but we have not re-run the methodology or spot-checked the underlying diffs ourselves.
• HN's skepticism about the business model is worth taking seriously, not as a dismissal of the data, but as a reason to expect follow-up "how to win agent recommendations" content from Armature that should get the same scrutiny as the study itself.

Closing

The useful version of this story is: repository language, prompt framing, and a vendor's own documentation wording are measurably swaying which tool a coding agent picks — sometimes more than the actual merits of the product. That's real, checkable, and worth knowing whether you're choosing infrastructure or building on top of an agent's suggestion. The uncomfortable version is that the company that discovered and quantified this now sells the service of exploiting it, and says so in its own disclaimer. Both things are true of the same 5,292 sessions. Read the data, use the practical lessons, and treat the next "how to get your product picked by AI agents" pitch — from Armature or anyone else — as a sales conversation, not a research finding.

Related on explainx.ai

• AI recommendation sources: manufactured "GEO farms" feeding Perplexity citations
• Codex vs Claude Code: the developer verdict
• Claude Code vs Cursor vs GitHub Copilot
• Top 10 open and closed source agent harnesses
• AI visibility tools and AEO measurement, explained
• Stripe directory: AI agent commerce and machine payments
• HarnessDev: can LLMs build and evolve their own agent harness? — a related benchmark on harness quality itself, not just tool-choice bias inside one
• What is an agent harness? Complete guide
• What is AI slop? SEO/GEO content quality

Sources

• Armature — Which tools do Claude Code, Codex and Cursor choose? We measured 16,893 sessions to find out. (September 3, 2026)
• Hacker News discussion — "Which tools do Claude, Codex and Cursor choose?" (92 points)

This post summarizes a third-party study published September 3, 2026, by Armature, a company that discloses a direct financial interest in influencing coding-agent tool recommendations. We have not independently reproduced the methodology or verified the underlying session data. Tool win rates, agent behaviors, and vendor names are accurate to Armature's publication as of this writing and may shift as later waves of the dataset are published.