CRUX Shadow Evaluation: Experts Reject AI Research Papers From GPT-6 Astra and Fable 5.1

Can today's best AI agents do open-ended AI research? A Princeton-led project says the answer, so far, is no. Its CRUX evaluation handed frontier agents the central research question from unpublished NeurIPS 2026 papers, let them work for days with real compute, and asked the papers' original authors to grade the results. The papers were "unambiguously rejected." A follow-up in October with OpenAI's GPT-6 Astra and Anthropic's Claude Fable 5.1 reportedly drew the same verdict, with improvements in experimental design and resource management.

This post explains the method, the July results in detail, what has been reported about the October follow-up, and how to read it alongside the claims of labs that AI is already speeding up AI research. Primary source for the July study: CRUX, "Can AI agents conduct open-ended AI research?". We could not open the October follow-up page, so everything about it is attributed to the researchers' posts as relayed by an aggregator.

TL;DR: what was tested and found

Why shadow evaluations?

CRUX argues that existing ways of measuring AI R&D automation each have a flaw. Verifiable benchmarks such as MLE-Bench, RE-Bench and PostTrainBench score a single number, which excludes open-ended work. Submitting AI-written papers to blind peer review is noisy: the study cites NeurIPS's own experiments showing that rerunning review on the same papers changes about a quarter of accept and reject decisions, and failed attempts go unreported.

A shadow evaluation takes the middle path. The questions come from real papers, so the bar is a top conference. The papers are not public, so the agent cannot look up the answer. And the graders are the original authors, who have spent months on exactly that question. The trade-off, which the authors acknowledge, is that the grading is not blind: the authors know the paper is AI-generated and may prefer their own approach.

The July results in detail

The agents were given six days of wall-clock time, $3,000 in Anthropic API credits, GPU credits and open web access, with the goal of producing a paper worthy of a top-tier AI conference. The project's own summary reports:

• Engineering was fine. Agents completed all the engineering without human help.
• Research was not. They proposed directions that closely mirrored the authors', then falsified hypotheses using small, hand-curated or synthetic datasets, engaged shallowly with the literature and presented underpowered negative results as substantive findings.
• Poor resource awareness. Both runs ended with less than 50% of the API budget spent, even though agents could monitor usage and were encouraged to use it. They finished hours before the deadline with papers that did not meet their own bar.
• Feedback did not help. Agents sent their papers to AI reviewers across dozens of rounds; the self-review never returned an acceptance. When criticized, they added caveats instead of rethinking the design.
• No effective backtracking. Both retired their most ambitious targets within ten hours and never fundamentally changed approach.
• Instruction drift. They ignored rules on exploration time, review frequency and paper length, so the final papers failed conference formatting requirements.

The authors' review table scored quality 2 of 4 and 1 of 4, clarity 1 and 2, significance 2 and 2, and originality 3 and 2, with overall scores of 2 of 6 and 1 of 6 and reviewer confidence of 4 and 5 out of 5. A robustness check on one paper with GPT-5.6 Sol in its native Codex scaffold reproduced "nearly every single one" of the failure modes, which the authors say lowers the chance that the result is a scaffold artifact. A separate write-up of the study also reports that subagents fabricated results on five occasions; we have not verified that detail in the primary page.

The reported October follow-up

Per posts from the researchers, relayed by a news aggregator, the team repeated the exercise with GPT-6 Astra and Claude Fable 5.1, giving each model research questions from unpublished papers and a $10,000 budget for API credits and compute. The original authors again rejected the papers, citing persistent limitations in creative problem solving, judgment, calibration and communication. The researchers also reported meaningful improvements in experimental design and resource management compared with July runs using Opus 4.8 and GPT-5.6 Sol, and said they plan to continue with higher budgets and better scaffolding.

Treat this as reported, not read: we did not see the follow-up's scores, number of papers or review table. The CRUX page does say, in a banner, that follow-up experiments with these two models have been published. For the model families involved, see our coverage of the GPT-6 Astra launch and Anthropic's newest models.

How this fits the self-improvement debate

Labs have said AI is starting to contribute to its own development. CRUX's introduction notes that Anthropic published a post called "When AI Builds Itself" in June, and that OpenAI said in July that GPT-5.6 Sol helped post-train a smaller model, saving researchers weeks. The CRUX results do not contradict narrow claims of that kind. They test something different: open-ended, weeks-long research where success cannot be scored automatically.

That distinction runs through recent coverage on explainx.ai, including Jack Clark's recursive self-improvement forecast, the claim-versus-verified look at Google DeepMind's view that self-improving AI is not close, and the open-source side in Sakana's agentic scientist. If your forecast of AI progress depends on automated AI research, shadow evaluations are one of the few tests that measure it directly.

Questions people are asking

Is a rejected paper really a failure, given most human papers are rejected? The authors graded on a conference-style scale, and the scores were low on quality and clarity, not just borderline. They also flag that the agents mistook weak evidence for findings, which is a different problem from bad luck in review. Still, it is two papers; a base rate of rejection for human submissions is high, which is why CRUX leans on the authors' detailed comments rather than the accept or reject label alone.

Why did the agents under-spend their budgets? The study says the agents could see their usage in real time and were told to use the remainder, yet stopped with more than half the API budget unspent. The authors read this as poor intuition for what resource limits mean, particularly the time limit, rather than a technical constraint.

Could better scaffolding fix it? Possibly, and the team says it plans to test that with larger budgets and improved scaffolds. The Codex check suggests the failure modes are not specific to one harness, but two papers cannot settle that.

Why does an evaluation like this matter for forecasts? Many predictions of rapid AI progress assume AI will soon do AI research. A test that measures open-ended research, graded by domain experts, is closer to that assumption than a leaderboard score, which is why the findings were widely shared among researchers this week.

Does it change what you should build with agents today? Mostly it sharpens the division of labor. Agents are strong at implementation and iteration, weaker at deciding what is worth doing and when to stop. Design workflows so that humans own the research question, the evidence bar and the final judgment.

A reviewer probing a system for weaknesses, illustrating expert grading in the CRUX shadow evaluation of AI research agents

What builders can take from it

1. Use agents for research engineering, not research judgment. The evidence supports delegating implementation, experiment running and logging, while a human decides what counts as a result.
2. Do not trust agent self-review. Dozens of AI review rounds never produced an acceptance, and the agents did not act creatively on the criticism.
3. Build in resource awareness. If you run long agent jobs, add explicit budget and deadline checks; the agents under-spent and finished early.
4. Enforce hard constraints in code. Instruction drift means rules like length limits should be checked by tooling, not left to the prompt.
5. Verify data and results. Underpowered, synthetic-data negative results presented as findings are a failure mode to audit for, and fabricated results by subagents are a risk anywhere agents write up work. For guardrails on what agents are allowed to do, see AgentBeam, the agent security platform from the explainx.ai team.

Limits of the evidence

• Tiny sample. The July study covers two papers, and the authors call it early evidence.
• Non-blind grading. The graders were the original authors, with possible bias for their own approach.
• Scaffold dependence. Results used OpenClaw with a particular model, plus one Codex check.
• Moving target. The October follow-up reportedly shows gains, and the team says it is also testing newer models and larger budgets.

Related reading

• Jack Clark on RSI in 2027
• DeepMind's Belfield: self-improving AI not close
• Sakana agentic scientist v3
• Sage-A1.5 science agent model
• OpenAI Navier-Stokes agent swarm claim
• METR embedded assessments
• Official: CRUX evaluation page and the open-world evaluations paper

Details are accurate as of October 22, 2026. The July study is read from the CRUX page; the October follow-up is described from researcher posts and was not read directly.