METR''s famous time horizon curve measures verifiable software tasks well, but the November update probes fuzzier work that looks more like research judgment. MIT Technology Review reported on November 20, 2026 that METR added judgment-heavy and open-ended tasks to its suite, finding models still trail humans on designing useful evaluations even as coding horizons rise.
This article pairs outlet reporting with METR''s own time horizon FAQ and limitations note so readers separate headline plots from error bars.
TL;DR
Benchmark checklist grid, echoing mixed verifiable and judgment tasks
| Slice | MIT TR summary of METR findings | Caveat |
|---|---|---|
| Verifiable coding-style tasks | Exponential improvement continues | Classic plot easily misread as autonomy hours |
| Evaluation design | Zero staff-usable evals from models | Opus 5.5 borderline on some bars |
| Philosophy argument scoring | Below human performance | Staff disagree on relevance to AI R&D |
| Measurement | Many human times now estimates | Hard to forecast judgment progress |
What the time horizon plot actually means
METR''s public page defines 50% time horizon as the human-calibrated task length where an agent is predicted to succeed half the time. It is not how many hours an agent runs unattended. Agents often finish faster than humans when they succeed.
Thomas Kwa''s limitations note warns error bars can span factor-of-two swings and that domain mixes change results by orders of magnitude. Readers who treat one point estimate as gospel misread the graph MIT TR calls the most misunderstood in AI.
Why judgment tasks entered the suite
Stacked rulers suggesting uneven progress across task types
Reinforcement learning with verifiable rewards excels at coding and math where success is checkable. Research direction, evaluation design, and taste-heavy choices lack cheap graders.
METR''s reported new tasks include:
- Design a new AI evaluation METR could adopt, later scored by METR employees
- Score philosophical arguments as a proxy for nuanced judgment
MIT TR quotes METR staff disagreement about how predictive the philosophy task is for automating AI R&D. That honesty matters: benchmarks encode values.
Connection to OpenAI''s jagged frontier
Days earlier OpenAI published Research acceleration: The jagged frontier, showing 79% success on some monitoring buckets versus 25% on inference-code tasks internally. METR''s external judgment slice rhymes with that uneven automation story.
Neither source proves stagnation; both argue against single-number AGI timelines.
Anthropic RSI politics
MIT TR mentions community reaction to a redacted METR acceleration report about Anthropic earlier in November. explainx.ai covered METR–Anthropic acceleration claims separately. The judgment update is not a replacement for that document; it widens task types.
Opus 5.5 and borderline performance
MIT TR says Opus 5.5 lands borderline above the bar on evaluation-design tasks where every model a year ago sat clearly below. Borderline is not shippable: METR employees still judged proposals not useful.
Product marketers should not convert borderline bars into autonomous evaluator launch copy.
Methodological limits you should cite
From METR''s prior methodological writing plus MIT TR''s November reporting:
- Long tasks may lack fresh human baselines; durations are estimated
- Philosophy scoring may reflect one primary rater
- Few judgment tasks mean wide confidence on trends
- Time horizons page notes measurements above 16 hours are unreliable with the current suite
Builders exporting METR charts into board decks should paste caveats beside the PNG.
What Sayash Kapoor and Najoung Kim said
MIT TR quotes Sayash Kapoor welcoming broader eval creativity while noting models still fail open-ended research reproduction in his prior work. Najoung Kim cautions that short-term judgment gains may not extrapolate.
Those quotes are expert opinion, not METR official conclusions.
Practical guidance for engineering managers
| If your roadmap assumes… | November METR reporting suggests… |
|---|---|
| Agents pick next quarter''s eval suite | Still need senior researchers |
| Coding horizon doubles === full lab automation | Category errors remain |
| One vendor leaderboard | Ask for component splits like MedChemBench |
Invest in human review loops for fuzzier milestones; keep CI-style gates for verifiable ones.
What would change our mind
Signals that judgment curves are catching up would include:
- Public METR task definitions with multi-rater agreement stats
- Independent labs reproducing eval-design usability scores
- Corporate deployments where model-written evals ship without human rewrite
Until then, treat judgment metrics as early warning, not capacity planning.
Verifiable versus fuzzy: reinforcement learning angle
MIT Technology Review''s November 20 piece explains why RLVR lifted coding: successes and failures are cheap to score. Judgment tasks lack that loop, so gradients do not sculpt taste the same way.
Product teams should expect plateaus on fuzzy milestones even while SWE-bench charts keep climbing. Your roadmap needs separate tracks for lintable outputs and editorial choices.
How to read exponential doubling claims
The classic METR plot interpreted four-month doubling of completable task duration. Kwa''s note reminds readers that slope mattered more than any single model dot.
When vendors repost METR charts, ask whether they cherry-picked 50% horizons while ignoring 80% reliability needs for production automation.
Enterprise procurement language
RFP lines like matches METR time horizon are meaningless without specifying:
- Task suite version (1.0 versus 1.1 versus judgment add-on)
- Scaffold (ReAct, Claude Code, Codex)
- Success threshold (50% versus 80%)
See METR Claude Code versus Triframe note for how much scaffold choice moves numbers.
Policy and IPO-week context
MIT Technology Review situates the METR update beside Anthropic IPO-week anxiety about recursive self-improvement. Market narratives conflate faster coding with self-upgrading labs. Judgment-task gaps are a factual counterweight: no model shipped a usable eval to METR staff.
Investors should not treat time horizon slopes as revenue guidance; builders should not treat them as headcount replacement ratios.
Comparison table: verifiable vs judgment slices
| Property | Verifiable software tasks | Judgment-heavy METR additions |
|---|---|---|
| Grader | Automated tests | Human staff review |
| RL-friendly | Yes | Mostly no |
| Sample size in report | Large suite | Few tasks per MIT TR |
| Production analogy | CI merge gates | Staff meeting decisions |
| Error bars | Still wide per METR | Wider plus rater disagreement |
Monitoring product implications
If you sell AI SOC or agent observability, METR''s eval-design failure mode suggests a product opening: humans still author detection semantics. Do not promise autonomous eval generation until usability studies replicate METR''s staff bar.
Academic citations for papers
When citing METR in LaTeX, use the limitations note for methodology caveats and MIT Technology Review only for the November judgment-task reporting until METR posts a dedicated PDF URL on metr.org.
Scenario planning for labs
| Scenario | If judgment gap persists 12 months | If judgment gap closes quickly |
|---|---|---|
| Hiring | Keep senior eval designers | Shift some eval work to agents with audit |
| Release gates | Human sign-off on eval suites | Faster iteration with spot checks |
| CapEx | More human review FTE | More GPU for agent loops |
Neither scenario is forecast here; both are planning stubs for risk officers.
Connection to UK and US eval policy
Government eval labs and Safe AI Act outlines assume benchmarks keep pace with models. METR''s struggle to accept model-written evals shows how hard that procurement promise is in practice.
Builder exercises you can run Monday
- Ask your best coding agent to propose an eval for your product''s riskiest tool call.
- Have staff score usefulness blind.
- Compare results to your SWE-bench-style automated score.
You will likely reproduce METR''s gap locally without waiting for their PDF.
Glossary reminders
Time horizon is not wall-clock agent runtime. Borderline above the bar is METR''s internal label, not a public leaderboard tier. Judgment-heavy means human graders disagreed or qualitative scoring dominated.
Timeline for watchers
| Date | Event |
|---|---|
| Mar 2025 | Original METR time horizon paper wave |
| Jan 2026 | Time Horizon 1.1 release with bigger suite |
| Sep 2026 | metr.org time-horizons page stops active updates |
| Nov 2026 | MIT TR reports judgment-heavy add-on findings |
If METR publishes a dated PDF on metr.org, replace the middle reporting layer with that primary URL in your internal wiki.
Why PHD hiring still makes sense
Startups interpreting exponential coding curves as near-term headcount zero should read the evaluation-design null result. Someone must still decide what to measure, what ship criteria mean, and when a capability is too dangerous to demo. Those are judgment tasks the November reporting says models failed.
Red-team versus eval-design confusion
Security red teams often write narrow exploits with clear pass/fail. METR''s eval-design task asks for durable measurement instruments — closer to science than penetration testing. Conflating the two in vendor RFPs produces false confidence when a model can break a sandbox yet cannot propose a regression suite your staff would adopt.
Leaders scheduling 2027 autonomy roadmaps should separate milestone A (merge bots with CI truth) from milestone B (agents author your eval catalog). November reporting suggests B still lags A by a wide margin on frontier models.
Share this split with boards so RSI headlines do not compress two unlike automation curves into one hockey stick chart. Revisit quarterly when METR publishes the next suite revision or any new judgment-heavy tasks on frontier models.
