CodeRabbit Verify entered public beta on November 18, 2026 as an agentic QA layer on top of AI code review. Instead of only reading diffs, Verify deploys your app in CodeRabbit's sandbox, walks user journeys in a browser, and returns functional bugs with screenshots, recordings, and logs. Official sources: the launch blog, product page, and documentation.
Code review at agent speed breaks when nobody runs the app. Verify is CodeRabbit's bet that runtime behavior must join static review—the same week higher SWE-bench scores failed to raise merge rates in community experiments.
TL;DR: November 18 launch facts
| Item | CodeRabbit says |
|---|---|
| Start command | PR comment @coderabbitai verify |
| Hosts | GitHub Cloud, GitLab.com, Bitbucket Cloud |
| Sandbox | Agent prepares dependencies and services |
| Outputs | Bugs, blockers, screenshots, session recordings |
| Fixes | CodeRabbit Agent or exported Markdown for other agents |
| Beta pricing | Verify runs free, including Agent fix handoffs per billing docs |
| Mobile | Not supported |
What problem Verify targets
Models and agents can ship PRs faster than humans can click through UI. Teams using CodeRabbit with Codex already saw review automation; Verify adds execution.
The launch post describes three gaps:
- Behavior versus diff. A coupon change might pass unit tests yet charge the wrong amount in checkout.
- Regression across the repo. Journeys can span multiple modules beyond the PR footprint when you run full-repo schedules.
- Human QA tax. Verify aims so QA does not manually replay every agent-generated change.
That aligns with Harvard's code-review bottleneck research theme: review capacity, not generation capacity, is the constraint.
How Verify builds and runs scenarios
CodeRabbit Verify pull request results showing three passing scenarios and one reported bug
Image: CodeRabbit
Per the blog, Verify:
- Reads commits, relevant source, user docs, and optional user-provided tests.
- Uses prior CodeRabbit review context when present so scenarios reflect known risks.
- Derives expected results from docs and existing test expectations, not only from the model's guess of correctness.
- Lets engineers edit scenarios before execution.
- Re-plans when new commits land while keeping historical results per commit.
When a scenario fails, you can Fix with CodeRabbit Agent—carrying reproduction evidence—or copy Markdown for Claude Code, Codex, or another harness. That handoff mirrors OpenAI's Codex harness philosophy: separate find from fix, but do not lose repro.
Sandbox setup: what the agent does for you
CodeRabbit Verify sandbox environment settings and repository startup scripts
Image: CodeRabbit
Verify's agent inspects requirements, provisions a sandbox, installs dependencies, configures emulators or side services, starts the app, and health-checks readiness. Testing stays off production.
Compare to bank-agent incidents regulators now probe (Arizona's OpenAI investigation): Verify's isolation is vendor-hosted, so secrets you inject for staging still need rotation discipline. CodeRabbit explicitly splits application bugs from blockers (missing creds, cannot boot).
Access, permissions, and billing
Access and billing docs state:
- Verify inherits Coding Agent permissions; Billing Admins cannot run it by default.
- Automatic PR runs skip public GitHub repos; use
@coderabbitai verifymanually there. - Self-hosted CodeRabbit and enterprise git flavors are unsupported.
- During beta, Verify runs are not charged and no Agent minutes are billed for them, including Fix with CodeRabbit Agent handoffs, per billing docs.
Teams should still watch whether Verify exits beta as a paid SKU—similar uncertainty to Vercel Board's experimental label.
How Verify fits next to unit tests and MCP
Unit tests catch functions; Verify catches wiring—broken redirects, stale feature flags, CSS that hides pay buttons. It does not replace contract tests or microsoft-agent-framework-grade supply-chain checks.
If you expose MCP tools to agents, Verify is one downstream consumer that treats the built UI as ground truth. Pair it with iFixAi-style harness audits on the agent configuration itself.
Example shapes from the launch materials
Demo checkout screenshot where order total and simulated charge disagree
Image: CodeRabbit
CodeRabbit's demo coupon flow is illustrative even though it uses a fake payment backend. Verify scripted three scenarios: valid coupon application, invalid coupon handling, and coupon removal. Two scenarios surfaced charge bugs; the invalid-coupon path passed. That split matters operationally: passing negative tests does not excuse failing happy paths.
In your repo, expect Verify to propose journeys like sign-in, create resource, edit settings, and delete account when those flows touch changed files. You should delete or rewrite proposals that reference feature flags you have not shipped, or you will burn sandbox minutes on blockers that are really spec drift.
Repository journeys versus PR-scoped runs
The product supports two speeds:
- PR-scoped runs answer "did this change break a user-visible workflow?"
- Repository journeys answer "did any recent change break a critical path we saved?"
The blog promises schedules for full-repo journeys—nightly or on-demand—so teams can treat Verify like a synthetic monitoring layer that follows agentic velocity. That is distinct from GAO-style attack planning probes; Verify is product QA, not red-team science.
When schedules fail, docs say blockers (cannot log in, missing API key) report separately from bugs. Do not mark a release green because blockers were "expected"; fix staging secrets or document required env vars in the scenario preamble.
Jira, Linear, and human triage
Verify can turn findings into Jira or Linear tickets per the documentation trail in the launch post. That helps QA leads who still want a human gate before Fix with Agent fires on production branches.
A practical workflow for regulated teams:
- Verify posts recording plus steps to Linear.
- A human assigns severity.
- Only sev-1/sev-2 auto-hand to CodeRabbit Agent; others wait for sprint planning.
This mirrors how Credit Nirvana's agentic collections CVE forced explicit human oversight when automation touched money movement—even though Verify demos use fake payments, the governance pattern transfers.
Security notes for sandboxed production-like data
Verify runs outside your VPC. If your staging stack needs PII, production database dumps, or HSM-backed keys, expect blockers unless CodeRabbit adds bring-your-own-secret integrations you trust.
Minimum hygiene:
- Use synthetic users and masked data in verify environments.
- Rotate any credential you paste into CodeRabbit setup during beta.
- Treat session recordings as internal-only; they may capture tokens in URLs if your app is sloppy.
For finance flows, keep human approval on Fix with Agent until you have audit logs that satisfy your risk team—the same week Arizona opened a criminal probe into rogue bank agents.
Operational metrics to track in beta
Track internally, because CodeRabbit has not published industry benchmarks yet:
- Sandbox ready time per stack (Next.js versus Django monolith).
- Scenario acceptance rate (how often engineers delete proposed steps).
- False positive rate on blockers versus bugs.
- Agent fix merge rate after Verify findings.
Compare merge rate to your pre-Verify baseline—the same metric higher-scoring models failed to improve in a natural experiment post.
Getting started on a real PR
- Enable CodeRabbit on a web app repo (not mobile).
- Merge or open a PR with a user-visible change.
- Comment
@coderabbitai verifyafter the latest CodeRabbit review finishes. - Open the scenario plan; delete fantasy steps that do not match your product.
- Run; download recordings for failing flows.
- Either Fix with Agent or paste Markdown into your preferred coding agent.
For regression libraries, generate user journeys from the codebase and schedule nightly runs—the blog promises on-demand or scheduled full-repo journeys.
Limits and honest caveats
- Web only in beta.
- Cloud git only—no GitHub Enterprise Server.
- Vendor sandbox means custom hardware, on-prem auth, or air-gapped builds may block as blockers, not bugs.
- Demo apps in marketing may show payment flows that are not live charges—read captions carefully.
Verify also does not prove security; pair runtime QA with OWASP-style agent pentest surveys themes when agents can modify prod configs.
How this compares to rolling your own Playwright agent
| Approach | You operate | Tradeoff |
|---|---|---|
| DIY Playwright in CI | Browsers, secrets, flake triage | Full control, high upkeep |
| Verify | Sandbox prep + scenario agent | Faster start, vendor lock-in |
| Preview deploy + manual QA | Humans | Slow, does not scale with Codex |
If you already run Prime Intellect train-deploy loops, Verify is closer to deployment verification than training eval.
What to watch after beta
CodeRabbit will need to publish flake rates, mean time to sandbox ready, and which stacks (monorepos, Docker Compose, serverless) are first-class. Until then, treat Verify as best for teams already on CodeRabbit Cloud who ship web UI weekly.
Where Verify does not help yet
Verify will not load-test your API at Black Friday scale, fuzz auth cookies, or prove compliance with PCI or HIPAA controls. It also cannot validate native mobile binaries. Teams shipping Nvidia Cosmos-style physical-AI datasets need simulation QA, not browser walks.
If your agent workflow is CLI-only with no server UI, Verify adds little; invest in integration tests and Rust Anything parity MCP instead.
FAQ deep dive: who should enable Verify first?
Small product teams with weekly web releases benefit most: they feel agent PR volume but lack a dedicated QA bench. Platform teams should pilot on one service with clean docker-compose before monorepo-wide schedules. Open-source maintainers on public GitHub must trigger runs manually because automatic PR verification skips public repos per docs—plan for @coderabbitai verify discipline in contributor docs.
