Mercor Benchmark Map: 90% of US Working Hours Are Barely Measured by AI Benchmarks

Most claims about what AI can do at work rest on benchmarks, and a new study from Mercor argues those benchmarks measure only a sliver of the economy. In its US Benchmark Map, published in October 2026, the AI hiring and evaluation company mapped 29 benchmarks onto the US Department of Labor's task taxonomy and estimates that about 90% of American working hours go to tasks with little or no benchmark coverage.

The number is easy to misread. It does not say AI fails at 90% of work. It says we have little standardized evidence either way. For anyone who relies on leaderboards to decide what to build, buy or worry about, that is a useful correction, and we walk through the method, the findings and the limits below. Mercor makes its own benchmarks, so read its framing with that in mind.

TL;DR: what the benchmark map says

How was the benchmark map built?

The team started from O*NET, the Department of Labor's database of occupations and the tasks that make them up. It selected 29 benchmarks using five criteria: public availability, agentic tasks, economic realism, diversity across job families, and unsaturated difficulty.

An LLM judge then read each benchmark prompt and tagged it with the O*NET tasks it resembles. A task is called covered only when it has at least five mapped prompts, a threshold meant to avoid declaring victory on a single lucky question.

This is a coverage audit, not a performance test. It asks whether anyone has written enough questions about a task to evaluate a model on it. That distinction explains most of the headline.

What does it find about who gets measured?

The pattern is skewed toward work that is already attractive to AI labs.

• Exposure predicts coverage. AI-exposed tasks are about six times more likely to be covered than unexposed ones. Software engineering and financial analysis are heavily covered; truck driving and gas plant operation are sparse.
• Money predicts coverage. Even after controlling for AI exposure, tasks in occupations with larger wage bills are overrepresented.
• Importance predicts coverage. Tasks rated most important within an occupation are roughly 6 to 10 times more likely to be covered than the least important.
• Within a profession, the edges are missing. Lawyers are well covered, while legal secretaries and court clerks are barely covered.

This matches what builders already sense. Coding agents get a stream of new benchmarks, such as the FDE-Gym forward-deployed engineer test where Opus 5.5 completed 14 of 95 tasks, while the administrative and physical work that fills most of the economy gets little attention.

Do simple averages mislead? The APEX check

Mercor tested one practical question: if you weight model scores by how often tasks occur in an occupation, do the rankings change? For investment banking, law, accounting and software engineering, it compared simple averages of APEX scores with O*NET-weighted averages, where each task counts equally.

The two methods gave generally similar results. In one example, a top model scored 73.3% on financial analysis with a simple average and 70.2% weighted. That is mildly reassuring for the benchmarks that exist, but it comes with a catch: fewer than a third of relevant O*NET tasks had enough APEX prompts to be included in the weighted scores at all.

What obstacles does the study name?

Mercor lists several reasons the gap is hard to close.

1. Low statistical power. With few prompts per task, a handful of weak items can swing a score.
2. Uneven quality. Some benchmarks are saturated, and some items are flawed or easy to game. There is no explicit quality weighting yet.
3. Automation versus augmentation. Benchmarks mostly test whether AI can do a task alone, while people usually use AI as an aid.
4. Physical work. Roughly 63% of jobs involve physical components that current benchmarks do not capture.

How far should you trust the 90% figure?

Treat it as an estimate with wide error bars. The authors say the 29-benchmark list is incomplete, which likely understates coverage. Capping benchmarks per job family understates coverage in heavily benchmarked fields such as software engineering. Excluding saturated benchmarks may understate coverage of easier tasks, while including them could overstate it. Selection criteria such as economic realism require subjective judgment, and an LLM judge decided which prompt matches which task.

There is also an incentive to note. Mercor sells evaluation data and publishes APEX, so a finding that the world needs more and better benchmarks fits its business. That does not make the analysis wrong, and the method and caveats are published, but it is a reason to wait for outside replication. We could not find independent confirmation of the numbers yet.

What does this mean for what you build or pay?

• Be skeptical of "AI can do X% of jobs" claims. They often extrapolate from covered tasks. Our look at the World Bank jobs tsunami claim shows how quickly such numbers spread.
• Build your own evals for uncovered work. If your task is not on a leaderboard, a leaderboard score tells you little. Collect real examples and score them yourself.
• Budget for checking. Where benchmarks are thin, outputs need more review, which links to the AI verification tax that workplace surveys describe.
• Read job-impact headlines carefully. Reports about entry-level hiring in India's IT sector face the same measurement problem.

For a related view of where model scores can mislead, see Google's report on the jagged frontier in AI for science and the Artificial Analysis Intelligence Index v4.2.

Why do benchmarks cluster around a few jobs?

The skew has understandable causes. Benchmarks are cheapest to build where answers can be checked automatically. Code either passes tests or it does not, and a financial model either reconciles or it does not. Work whose quality is judged by a supervisor's taste, a customer's mood or a physical result is expensive to score, so nobody writes thousands of prompts for it.

Labs also have their own incentives. A model release needs a chart that moves, so teams pick tests where recent models make visible progress. Once a benchmark saturates, attention shifts to the next hard test in the same domain rather than to a neglected one. Over several years that compounds: the covered areas get deeper while the uncovered ones stay dark. Mercor's finding that important, high-wage, exposed tasks are over-covered is what this dynamic would predict.

The result is a feedback loop for hiring and policy as well. Employers deciding where to deploy agents, and governments estimating exposure, lean on the same few leaderboards. If the leaderboards describe a narrow slice of work, estimates built on them describe that slice too.

How can you check whether your own job is covered?

You can approximate the study's logic in an afternoon.

1. List ten tasks you do weekly, using O*NET's task list for your occupation as a starting point.
2. For each, search for a public benchmark whose prompts resemble it. If you cannot find five realistic ones, treat the task as uncovered under Mercor's rule.
3. For uncovered tasks that matter to you, write ten real examples with a short grading note each, and run two or three models on them.
4. Record not only accuracy but also how long checking took, since that review time is the real cost.

This turns a vague worry into your own small benchmark, which will tell you more about your workflow than any aggregate score.

What are people asking about this study?

Is 90% a measurement or a guess? It is an estimate built on a model-judged mapping and a coverage threshold, with the authors' own caveats. It is best read as an order of magnitude: the large majority of hours, not a precise share.

Does it mean benchmarks are useless? No. The APEX check suggests simple averages and task-weighted averages often agree in the fields that are covered. The problem is breadth, not that existing scores are wrong.

Will more benchmarks fix it? Partly. Mercor points to low statistical power, uneven quality and the automation-versus-augmentation gap as separate problems. Even a perfect map of prompts would not capture the 63% of jobs with physical components.

What to watch next

Mercor says the map is a starting point. Useful follow-ups would be an open list of the 29 benchmarks and prompt-to-task tags so others can audit the judge, versions that weight coverage by wages or hours, and new benchmarks aimed at the least covered job families. If outside groups reproduce the 924-of-18,797 figure with different benchmark sets, the 90% claim would gain weight; if coverage jumps with a broader list, it would shrink. We will update this post when independent analyses appear.

Related reading

• FDE-Gym: forward-deployed engineer benchmark
• The AI verification tax: rework surveys
• World Bank: AI jobs tsunami claim
• India IT entry-level hiring and AI: claimed vs verified
• Google AI in science: the jagged frontier
• Artificial Analysis Intelligence Index v4.2
• Official: Mercor US Benchmark Map and O*NET

Figures are accurate as of October 16, 2026 and reflect Mercor's own provisional analysis.