OpenAI published an engineering post on November 19, 2026 that explains how Codex agents help keep ChatGPT fast. The system is called the Performance Factory. It runs in the cloud, opens a task when a performance alert fires, and has agents find the cause, propose a fix and check recovery after deployment. Per OpenAI's post by Okay Zed and Cyrus Forbes, the first workflow tracked more than 1,700 investigations and root caused over 600 cases in just over three months.
That is 17 automated investigations every day, in OpenAI's words. A human engineer still reviews every proposed fix. The post is useful for builders because it shows a working agent pipeline with its failures included.
TL;DR: the Performance Factory at a glance
| Question | Answer (from OpenAI's post) |
|---|---|
| What is it? | A Codex-powered system that finds and fixes performance regressions |
| What did it do so far? | 1,700+ investigations tracked, 600+ root caused, in just over three months |
| How accurate? | 70% of 100 reviewed root cause suggestions graded accurate |
| Who merges fixes? | An engineer reviews the PR and decides |
| What are its parts? | Task service, dispatch loop, Codex in the cloud, subscriptions |
| What is the first workflow? | The Regression Workflow, from alert to verified recovery |
| What is next? | Proactive monitoring, more workflows, possibly low-risk fixes without a human |
What is the Performance Factory?
OpenAI says agentic development moved so fast that traditional performance engineering became unsustainable. The Performance Factory is its answer: engineers define workflows for recurring jobs and write instructions for the Codex agents that carry them out.
The first workflow is the Regression Workflow. It opens a task when a performance alert fires. An agent investigates, records findings in the task and proposes a fix. After deployment, another agent picks up the same task to check whether performance returned to baseline.
How did OpenAI get from manual work to a factory?
OpenAI describes six steps. The progression is a good template for any team adopting agents.
- Pre-Codex. Engineers read metrics and logs, compared traces, debugged live machines and read code. Over time they built more than 30 performance tools, such as CPU and allocation profilers.
- Codex as a partner. Engineers asked "why did this get slower?" and worked with Codex through the investigation, even in unfamiliar systems. They still had to steer.
- Reusable skills. The team captured its process in a shared skill called perf-forensics. It lists available tools and how to work a slowdown: find when it began, identify the affected requests, and check deployments and configuration changes. OpenAI tested it with evals based on past regressions. For the regression types it focused on, such as ChatGPT latency and availability drops, an alert could now yield a diagnosis in one run.
- The human proxy. Engineers noticed they were copying alerts into Codex and pasting findings back into Slack.
- Automated investigations. Alerts started Codex sessions in the cloud, which posted findings to the Slack thread. The automation stopped there. An engineer still had to build the fix and check it.
- The factory. A system that tracks what each task needs next and assigns an agent when a step is ready. Each task keeps its state and findings, so work passes between agents and engineers and resumes after a code review or deployment.
If you want the background on skills, our agent skills guide explains the idea, and the loop engineering guide covers repeatable agent loops.
What are the four parts of the system?
OpenAI lists four components.
| Component | What it does |
|---|---|
| Task service | Engineers define workflows with configuration files and Markdown prompts. The config lists task states, allowed transitions and which steps need an engineer |
| Dispatch loop | Periodically checks which tasks are ready and starts a Codex turn with instructions for the task's current state |
| Codex in the cloud | An internal API runs a Codex turn inside OpenAI's network, with access to code repositories and observability tools (metrics, logs, traces, CPU profiles, memory snapshots) |
| Subscriptions | Wake an agent when something changes, such as a GitHub review comment or a failed CI check |
An agent testing inside a sealed dome, showing how Codex runs performance investigations in a controlled cloud environment
Two details stand out. First, the agent runs under a service identity, so OpenAI can scope its permissions and see which actions came from the performance agents. Second, subscriptions mean an agent does not have to keep running while it waits for a person or a deployment. For how OpenAI limits what Codex can do on a machine, see our post on the Codex sandbox and permission profiles.
How does the Regression Workflow work, step by step?
A regression moves through four stages: needs investigation, root caused, fix proposed and resolved.
Finding the cause. The agent works out which metrics got worse, when they started to change and which requests were affected. An alert fires when a metric crosses a threshold, but the regression may have started hours earlier, so the agent looks back. It compares groups of requests: did similar requests get slower, or were more already-slow requests served? Those point to different causes. It then examines deployments and feature flag changes, reads the relevant code, and uses logs or traces to test whether the code could explain what happened.
Adversarial review. Early agents sometimes stopped at the first plausible explanation, and some of those were wrong. OpenAI added a separate Codex session whose job is to disprove the diagnosis. It rechecks assumptions and tests counterfactuals: if a rollout caused the slowdown, did similar unexposed requests stay healthy? If the suspected change was rolled back, did the metric recover?
OpenAI gives one example. An agent traced failing ChatGPT conversations to errors in image downloads and marked the cause as found. The reviewer noticed that the internal service that locates files had already recovered while image requests were still failing. The diagnosis did not explain the continuing failures, so the reviewer reopened the investigation.
Accuracy. In a manual review of 100 root cause suggestions, OpenAI graded 70% as accurate. It graded the other 30% as needing refinement or not fully establishing causality. That is an honest number, and it is the reason the human review stays.
The fix. After the diagnosis passes review, Codex implements a fix and opens a pull request. An engineer reviews and decides whether to merge. Each task has an auto-generated report with the regression, its impact, the root cause and the proposed fix, including metrics time series, CPU flame graphs and traces. Comments on the PR wake an agent to address them.
Verification. After deployment, an agent checks whether performance returned to baseline. Full recovery resolves the task. Partial recovery resolves the original regression and opens a new task for the rest. No improvement sends it back for investigation.
Why is fixing harder than finding the cause?
OpenAI says this directly: getting fixes right has been harder than finding the cause. Agent-proposed changes were sometimes more complicated than needed or introduced new issues. So OpenAI added an adversarial review of each proposed fix too.
Some fixes also need architectural judgment. OpenAI's example is a cache that improves speed but risks stale data. Those need an engineer who knows the service and can weigh the trade-offs.
OpenAI is building an evaluation loop that compares the fixes agents propose with the changes engineers merge. It is also working on synthetic performance tests so agents can measure a fix before asking for review, and on identifying low-risk changes that could ship without a human.
What lessons does OpenAI share for production agent workflows?
OpenAI lists three lessons.
Protect engineers' attention. Raw Codex output can be a wall of text. A first version that showed engineers a UI full of findings still left them to decide what mattered. One regression can also show up as several incidents across alerts, monitors and channels. The workflow now links related incidents, groups them by the metrics teams care about and ranks them by estimated impact.
Evaluate changes to your agents. Updating a skill or upgrading the model can improve some cases and worsen others. OpenAI tests two things separately: how well the agent investigates, and how useful its summaries are to engineers.
- For investigations, it backtests against past production incidents. Each case starts with a fresh Codex investigator and the original signal, stripped of details that give away the answer. The agent cannot use saved memories, the incident writeup or earlier runs. OpenAI scores when the problem began, how large it was and which requests were affected, and it scores the causal explanation separately, because an agent can get the timing right and still blame the wrong deployment.
- For handoffs, it grades task summaries from saved snapshots. A separate grader checks whether an engineer unfamiliar with the system could understand what happened.
Turn recurring failures into self-repair. Agents need monitoring too. A broken profiler or a missing permission can stop an agent from verifying a cause. A daily agent reads recent run logs for tool failures and access errors. When the same error blocks several investigations, the factory opens a task to fix it, and an engineer reviews the change.
What is next for the Performance Factory?
OpenAI calls the Regression Workflow one instance of an agent factory pattern: tasks move through explicit states, agents advance them, separate reviewers challenge conclusions and engineers step in when judgment is needed. Planned directions:
- Proactive discovery. Today the workflow starts from an alert. OpenAI wants agents to compare CPU and memory profiles, heap dumps and request traces across releases and notice small increases that alerts miss, then open a task in the Regression Workflow.
- More workflows. A service rightsizing skill already helps agents spot autoscaling settings that keep more replicas than needed. OpenAI wants to run such patterns regularly across all services.
- More autonomy. The post says the bottleneck today may be waiting for an engineer to review and merge. OpenAI is exploring whether agents can do more of that themselves.
Does this change how ChatGPT performs for users?
OpenAI says that when you notice a ChatGPT slowdown, "there's a good chance the Performance Factory is already figuring out why." The post does not publish before-and-after latency numbers, error budgets or uptime figures, so a user-visible speed gain is not measured in the text. The numbers it gives measure the process: investigations, root causes and review accuracy.
That fits a pattern in OpenAI's engineering writing. Its Pando Git backend post described infrastructure built to cope with agent-scale commit traffic, and its scaling Git at OpenAI article covers the same pressure. Codex is now both the tool engineers use and a cause of the load they must manage. For a customer view, see the CodeRabbit Codex case study, and for the human cost of heavy agent use, read our guide on AI brain fry and Git queues.
What can builders copy from this?
Here is a short checklist drawn from the post:
- Write the investigation playbook as a skill, then test it against past incidents.
- Store task state outside any one agent's context so agents and people can hand work back and forth.
- Add a second agent whose only job is to disprove the first one's conclusion.
- Give agents a service identity with scoped permissions.
- Wake agents on events instead of keeping them running.
- Keep a human on the merge decision until your evals show the fixes are reliable.
- Monitor the agents themselves for broken tools and missing access.
A starter prompt in the spirit of the perf-forensics skill:
Alert: p95 latency rose on endpoint X.
Step 1: find when the metric first got worse, not when the alert fired.
Step 2: compare affected and unaffected request groups.
Step 3: list deployments and flag changes in that window.
Step 4: state your confidence and what would disprove your diagnosis.
What is still unknown?
- OpenAI does not say how many of the 600+ root caused cases led to merged fixes.
- It gives no latency or availability improvements tied to the system.
- The 70% accuracy figure comes from a manual review of 100 suggestions, graded by OpenAI itself.
- The post does not say how much compute the investigations use.
Those gaps matter if you want to judge the system's return. The post is an engineering description, not an audited result.
Related reading
- OpenAI Pando: the Git backend built for Codex traffic
- Inside the Codex sandbox and permission profiles
- CodeRabbit and Codex: the OpenAI case study
- AI brain fry, Git queues and Codex
- What are agent skills? A complete guide
- Loop engineering for coding agents
- OpenAI's earlier engineering post: Inside the Codex sandbox
Figures reflect OpenAI's post of November 19, 2026.
