NVIDIA says the slow part of reinforcement learning is often not the learning. It is the inference that feeds it. In a technical post published November 19, 2026, NVIDIA authors Amr Elmeleegy, Harry Kim and Kyle Kranen describe how NVIDIA Dynamo can act as the serving layer for RL training. It targets three costs: waiting on the slowest rollout, recomputing the same prompt, and moving full model weights around.
This is an explainer on what the post claims, how the design works, and which numbers come from NVIDIA and its customers rather than from independent tests.
TL;DR: Dynamo for RL at a glance
| Question | Answer |
|---|---|
| What problem does it address? | Wasted GPU time in RL rollout generation |
| Three bottlenecks named | Rollout completion time, repeated context, weight freshness |
| Three fixes | KV-aware routing, ModelExpress weight distribution, worker recovery |
| Who still controls learning? | The RL framework (rewards, checkpoints, sample acceptance) |
| Open-source integrations | verl, NeMo RL, Slime, prime-rl |
| Reported results | CoreWeave: up to 15x faster weight updates. H Company: up to 20% lower step time, up to 5% fewer GPU-hours |
| Independently verified? | Not in the sources we checked |
Why does RL need its own serving infrastructure?
RL training is a loop. Workers generate rollouts. The framework scores them and updates the model. New weights go back to the workers for the next round. NVIDIA's Figure 1 shows that loop. Inference drives all of it, in NVIDIA's words.
The load is uneven. A reasoning rollout may hold one response. A coding-agent rollout can involve exploring a repository, editing files, calling tools and running tests, so rollouts finish at very different times. Our post on the DaisyRose-27B agent RL model shows how heavy these agent loops are in practice.
The three constraints NVIDIA names
- Rollout completion time. When training needs a group of samples, the slowest required rollout can hold up the next step. Giving each GPU the same number of requests does not give it the same amount of work.
- Repeated context. Several attempts often share the same instructions, task description or repository context. Recomputing that prefix burns GPU capacity before any new output exists.
- Weight freshness. Training keeps producing new weights. Workers must receive them within the staleness limits of the algorithm.
NVIDIA's point is that the right fix depends on which constraint dominates your workload. That is a useful caution: not every run is limited by the same thing.
How does Dynamo fit into the RL loop?
Dynamo sits between the rollout orchestration and the inference workers. The RL framework submits generation requests. Dynamo routes them to eligible capacity. The inference engine, such as SGLang in NVIDIA's diagram, runs the model.
NVIDIA lays out the split of duties:
| Component | Role |
|---|---|
| RL framework | Optimization, rewards, environments, checkpoints, sample acceptance |
| Dynamo | Routing, worker management, serving controls, telemetry |
| Inference engine | Generation and applying updated weights |
| ModelExpress | Model-weight and artifact distribution through supported integrations |
Generation and weight administration use different paths. Requests go through the shared serving frontend. Refresh operations target chosen workers under framework control. The framework therefore decides when workers may generate samples and which results count for training. NVIDIA describes this as improving serving without handing over ownership of the learning process.
For readers who follow Dynamo, this is a new use. Our earlier coverage looked at it for inference: Dynamo recipes for Korean open models and the Agent Substrate project on Kubernetes.
KV-aware routing: reuse the prompt without creating a queue
RL often samples several solutions to the same task from one shared prompt. If a worker has already cached that prompt in its KV cache, sending the next attempt there avoids repeated processing. If you send every attempt to that worker, you create a queue and delay the rollouts instead.
Dynamo's router balances cache reuse against worker load. NVIDIA says the goal is earlier completion of the samples that training needs. If you want the background on the cache itself, our KV cache explainer covers why prefix reuse saves compute.
The post gives no benchmark for routing alone, so the size of this gain is not stated.
The trade-off is practical. A router that only chases cache hits can overload one worker while others sit idle. A router that only balances load throws away the cached prefix and recomputes it. Because RL groups often share a long prompt, such as a repository description in a coding task, the right balance can change from one workload to the next. That is another reason to measure on your own run.
ModelExpress: move weights faster, send only what changed
Every RL update produces a new model. Sending a full checkpoint to each worker takes time and can pause generation. NVIDIA describes two ways ModelExpress cuts that cost.
Reuse weights already in the cluster. Compatible workers can copy weights straight from another GPU over NIXL instead of loading them from remote storage.
Send deltas. Rather than moving the full model, ModelExpress sends a compressed representation of what changed. Workers rebuild the new weights from the matching base version. That reduces transfer time and network traffic.
The framework still decides when workers apply updates, handles cached state and checks readiness before accepting new samples. That matters for correctness: a fast weight update is useless if samples are accepted from workers on the wrong version.
Recovery: what happens when a worker fails?
Long runs lose workers. NVIDIA says Dynamo detects the loss, sends new requests to eligible workers, and adds replacements to routing when they are ready. In supported configurations it can migrate eligible in-flight requests and drain workers gracefully.
Delta loading helps here too. A replacement worker loads a checkpoint and applies later deltas to catch up. Healthy workers keep generating in the meantime. The RL framework handles interrupted samples and makes sure replacements use the required weights.
NVIDIA's Figure 7 contrasts a failed rollout that restarts from zero with one that resumes from saved conversation history. For an agent rollout with many tool calls, that difference could be large, but NVIDIA gives no timing for it.
NVIDIA has published related work on training failures. See our posts on automated training workload recovery and failure root-cause attribution.
Who uses it, and what numbers are claimed?
NVIDIA lists these adopters:
| Organization | Reported use | Figure |
|---|---|---|
| Microsoft AI | RL rollout inference during post-training | None given |
| CoreWeave | Dynamo and ModelExpress for weight updates | Up to 15x faster |
| Cognition | Managing inference workers and routing rollouts for SWE-1.7 post-training | None given |
| Prime Intellect | Distributed inference orchestration | None given |
| H Company | Dynamo and ModelExpress | Up to 20% lower average training-step time, up to 5% fewer GPU-hours |
Open-source integrations cover verl, NeMo RL, Slime and prime-rl. Prime Intellect's own platform is covered in our post on the Prime Intellect Model Factory.
How to read these numbers
These are results published by NVIDIA about its own customers. "Up to" means a best case, not a typical run. The post does not state model size, cluster size, baseline or what the 15x is measured against. The 20% and 5% figures are also different measures: step time versus GPU-hours. That a step is 20% shorter yet GPU-hours fall only 5% suggests the gains do not all convert to saved hardware, though NVIDIA does not explain the gap. Treat these as a reason to benchmark on your own workload, not as expected results.
Should you try Dynamo for your RL run?
NVIDIA's own framing helps: pick the fix for the constraint you actually have.
- If rollouts share long prompts (coding agents with repository context), KV-aware routing is the first thing to test.
- If weight sync time shows up in your step profile, look at ModelExpress delta transfer.
- If long runs lose workers, test the recovery path, and check that your framework discards interrupted samples correctly.
- If you use verl, NeMo RL, Slime or prime-rl, start from the integration NVIDIA lists. The post points to the Dynamo RL documentation.
A practical way to evaluate it: measure step time, GPU-hours per accepted sample and weight-sync time on a fixed task, with and without Dynamo, over the same number of steps. Compare accepted samples, not raw tokens generated.
What this means for builders and teams
RL post-training is now a standard step for coding and agent models. That makes the infrastructure around it a cost line. If a serving layer can lift the share of GPU time spent on accepted samples, the run gets cheaper or finishes sooner. That is the pitch.
The wider pattern is that inference software is moving into training. Dynamo began as a serving system for large-scale inference. Putting it inside the training loop gives NVIDIA a role in post-training stacks that open-source frameworks also compete for. The claims here are specific and testable, which is good. They are also unverified outside NVIDIA's own account.
Related reading
- DaisyRose-27B: agent RL on Qwen3.8
- Prime Intellect Model Factory
- NVIDIA Agent Substrate and Dynamo
- Dynamo recipes for Korean open models
- What is the KV cache?
- NVIDIA automated training recovery
Sources: NVIDIA Technical Blog, Nov 19, 2026; NVIDIA Dynamo on GitHub.
Details are accurate as of November 19, 2026. Vendor-reported figures may change.
