Caltech-trained physicist Casey Handmer on November 17, 2026 published an AI-assisted audit of theoretical astrophysics that reads like a product demo for science, not a press release. Starting from 158,059 papers and 554,670 citations around Caltech's TAPIR group, the site screens 50,269 open full texts, recomputes 659 highly cited works, and traces 872 upheld errors through papers that copied them — often years after someone already objected in print.
For AI builders, the story is workflow economics: cheap screening at field scale, expensive verification only where citations concentrate, and citation-graph follow-through humans rarely fund. For policymakers, it is evidence that peer review catches only a slice of mistakes that stay in PDFs anyway.
TL;DR: what the map claims
| Question | Answer from Handmer's site |
|---|---|
| What corpus? | 158,059 papers seeded from TAPIR (4,093 papers by 326 members since 2004) plus citation closure |
| What was screened? | 50,269 papers with open full text (76% of the drawn map; pre-1991 mostly database-only) |
| What was recomputed? | 659 papers with line-by-line model review and second-pass confirmation |
| Uphold rate | 872 errors in 433 papers after dual reviewers |
| Propagation | 367 papers copied an error; only 68 criticized one in print (~5:1 copy-to-criticism) |
| Cost (author estimate) | ~$2,720 list API price; ~$3 per verified error; ~44 hours human time over 41 days |
| Fraud scan | No paper-mill language; no hidden reviewer prompts; 13 retractions in DOI set |
Why Handmer built this
Handmer opens with a numerical relativity example: a 1997 gravitational-wave paper carried known sign errors for decades while fixes lived in email threads, not the PDF students cite. His thesis work, he writes, could only patch the record in his own papers — the standard human limit when nobody is paid to re-derive every cited equation.
The AI pitch is symmetrical: read the whole field, recompute flagged lines, and follow each mistake forward through citing papers. Handmer says the pipeline also flagged errors in his own work.
That framing differs from consumer AI literature review tools that summarize abstracts. The method page (linked from the hero site) describes staged model roles, tool access for recomputation, and explicit human-in-the-loop limits — though, as HN commenter sudo_cowsay noted, the landing page does not prominently name which model family ran each stage.
What the screening pass actually checks
Before expensive recomputation, the project runs cheap fingerprints on every open full text:
- Paper-mill phrase patterns
- Chatbot residue and hidden instructions to AI reviewers
- Suspicious text overlap between unrelated author lists
- Retraction status against DOI records
- Unfinished-draft markers (such as FIXME notes) shipped to arXiv
Handmer reports astrophysics looks relatively clean on fraud: 43 suspicious overlap pairs queued for human review, mostly successive survey catalogues, not identity theft. The needle that does move is vocabulary: words like "delve" and "crucial" appear 2.5× more often in 2025 papers than in 2016 on his chart — a soft signal readers already argue about in generative-AI authorship debates.
Deep read: 659 papers, dual reviewers
The expensive stage targets citation inequality: 200 top TAPIR-cited papers (2006–2019) plus 459 additional high-signal works. Each deep read gets:
- LaTeX or text source
- Tool use to re-derive equations and rebuild numbers from tables
- A second independent reviewer — flags alone are not findings
Handmer reports 2,623 problems flagged initially, 84 major, with 56 standing as full or partial errors after confirmation. Thirteen errors were fixed between arXiv preprint and journal version — peer review removing only about one in five errors in that subset, by his accounting.
Two shapes inspecting one paper beside a green stamp, illustrating dual-model confirmation in Casey Handmer's physics literature audit
Field-scale literature maps depend on cheap screening, then expensive line-by-line recomputation on high-citation nodes.
The Smith–Kamionkowski–Cooray factor-of-four story
The site spotlights a 2005–2014 propagation case: a factor-of-four mistake in an inflationary gravitational-wave energy-density conversion. A 2008 paper criticized it, yet six copy papers kept the wrong factor until 2014. Handmer animates the timeline on the live map — a concrete example of slow correction even when objection exists.
At the field level, he estimates ~34,000 likely errors among papers cited 100+ times if the 4% deep-read sample scales — explicitly a floor, not a census.
What developers are saying on Hacker News
The HN discussion on November 18, 2026 mixes praise and skepticism:
- spongebobstoes argued the visualization could be replicated field-by-field and that grad students with modest AI subscriptions might clear low-hanging correction fruit.
- jdlshore called the landing copy AI-toned and bounced before the method depth.
- joe_the_user questioned whether "the 4% that mattered" was defined clearly enough to trust extrapolation.
- lexandstuff joked about Claude-style "nobody" rhetorical ticks in the prose — a reminder that voice detection is now part of reader trust.
- swampdeer reported heavy Firefox lag on the interactive map — a deployment lesson for huge client-side graphs.
None of these comments verify or falsify the error list; they surface UX, disclosure, and rhetoric risks for the next team that ships a field audit.
How this compares to other AI-for-science stacks
Handmer's work sits beside — not against — institutional efforts:
| Thread | How Handmer differs |
|---|---|
| Nvidia Nemotron for Science | Vendor agents optimizing new experiments; Handmer optimizes retroactive literature hygiene |
| Sage S2 science agent | Forward hypothesis generation; this is backward graph tracing |
| Epoch AI cyber eval agents | Safety red-team on live sites; Handmer red-teams PDFs |
| OpenAI CoT monitorability | Watches model reasoning traces; Handmer watches human paper trails |
If you run RAG over papers in production, the Handmer map is a warning: retrieval amplifies stale mistakes unless your pipeline includes version-aware citation checks or primary-source recompute on high-impact claims.
Can builders copy the pipeline?
Handmer links a Method page describing stages, cost, and known failure modes. Treat it as a reference architecture, not a hosted product:
- Seed graph from a real collaboration you understand (not arbitrary keywords).
- Screen everything with deterministic cheap checks before any opus-class spend.
- Spend deep tokens only on high in-degree nodes — citations are the pagerank of truth maintenance.
- Trace forward through citing DOIs; correction value lives in downstream copies, not the original typo.
- Publish uncertainty: Handmer notes 40 random errors sent to a third reviewer found 33/40 — some listed errors will fall on human challenge.
Research papers stacked for review, echoing citation tracing in Casey Handmer's AI astrophysics audit
Tracing citations matters because most copied errors never received an in-print criticism.
Practical limits Handmer admits
- 24% of mapped papers lack open full text — mostly pre-arXiv or paywalled journal PDFs.
- No human has checked each listed error yet; the site invites corrections.
- Interactive map performance may exclude low-power clients.
- Model disclosure on the landing hero is thinner than safety reviewers want — read the method appendix before citing dollar figures in procurement docs.
Thirty-two subfields and the AI-opportunity flags
The public site groups the corpus into 32 columns — from fast radio bursts to the Galactic halo — each with a subpage listing open, contested, or settled questions, central authors, and institutions. Handmer adds an AI opportunities layer: of 155 flagged open questions, the project's classifiers marked 8 as answerable now from existing data and codes, 119 as tasks AI would accelerate, and 28 as blocked on telescopes or detectors not online yet.
That split is useful for grant writers and startup roadmaps. It shows where literature automation is a substitute for new instruments versus where it only prepares experiments. It also mirrors how Shanghai AI Lab's Sage S2 markets science agents — but Handmer's map is backward-looking integrity, not forward hypothesis generation.
The explore view exposes 158,059 nodes with year filters and error highlighting. Performance complaints on HN suggest teams cloning the UI should cap initial render or paginate aggressively for low-end laptops.
What not to overclaim
Handmer is not saying AI replaces referees or that astrophysics is broken. He is saying:
- Errors with known fixes can persist and spread faster than criticism.
- arXiv versions often keep mistakes journal versions fixed.
- Field-scale reading is now affordable enough to treat as infrastructure, not a one-off stunt.
Regulators and universities debating AI exam cheating should notice the mirror image here: models that ace tests also scale literature review, which cuts both ways for integrity tools and integrity risks.
What you can do this week
- Open the map at science.caseyhandmer.com/physics and click Numerical relativity if that is your subfield.
- Read the Method page before budgeting a clone — token math is the product.
- If you maintain citation RAG, add a "cited error registry" check for papers in Handmer's public error list when they appear in your index.
- Challenge errors through the site's contact path if you hold primary sources — Handmer promises updates when challenged.
Related reading
- OpenAI CoT monitorability and latent reasoning risks
- Epoch AI and live-site agent eval incidents
- Sage S2 Flash science agent from Shanghai AI Lab
- Nvidia Nemotron for Science and LitmusBench
- Terminal-Bench audit flaws and claimed-vs-verified scoring
- AI homework versus exam scores research
- HybridAgent latent reasoning with tools
Handmer's figures are accurate as of the November 17, 2026 site publication; error counts may change as third-party reviewers respond.
