NVIDIA Cosmos3-H-Dreams: A Generative Surgery Simulator You Can Steer in Real Time

NVIDIA has released Cosmos3-H-Dreams, a generative simulator for gallbladder surgery. Give it a starting frame and a stream of instrument commands, and it generates video of what the scene would look like next, in short blocks as each new action arrives. NVIDIA's own Hugging Face write-up calls it, "to our knowledge," the first real-time generative surgical simulator trained on clinical cholecystectomy recordings that supports live kinematic control, and in the same breath says it is a research simulator, not a validated system for patient care.

That honesty is the right frame for this release. It is a notable research artifact for robotics and world-model builders, with open code and data pointers, and no evidence yet that it is accurate enough for surgeons to trust.

TL;DR: Cosmos3-H-Dreams at a glance

What is a generative surgical simulator?

Classic surgical simulators render anatomy with hand-built 3D models and physics. They are precise but expensive to author and rigid across procedures. A generative simulator learns from recordings instead: given the current frame and an action, a video model predicts the next frames. The appeal is that the model can pick up lighting, tissue appearance and tool interaction directly from real footage, and can be steered like a game by a human or by a policy.

This is the same idea behind the wave of interactive world models we have tracked, such as Dream Labs' Dream-1 robot world-action model and the broader engineering-world-model push in Skyfall's Sky-001. Cosmos3-H-Dreams applies it to a clinical domain, where the question of whether the output is physically right matters far more than in a game.

How it is built: teacher, student, and streaming

The video in NVIDIA's post is the clearest demonstration of what the system does, so here it is, followed by the design.

Video: NVIDIA

According to the post, training happens in stages:

1. A bidirectional teacher starts from the 16B Cosmos3-H-Embodied-Nano base and is specialized on curated cholecystectomy video plus kinematics. It was first adapted on 24 future frames (2.4 seconds) and then extended to 72 frames (7.2 seconds).
2. A causal student is distilled from the teacher so it can generate in blocks as actions arrive, instead of denoising a whole clip at once. Distillation uses causal teacher forcing and Self-Forcing, which trains the student on its own generated context, guided by the teacher and a learned critic.
3. Two checkpoints ship: a four-step student (four denoising steps per block) with lower visual quality but faster inference, and a two-step student that NVIDIA says was trained separately rather than just run with fewer steps.
4. A streaming runtime, FlashDreams, serves it interactively, with an adapted FadeMem module that consolidates the growing key/value cache during long rollouts.

The action vector has 28 dimensions covering pose and gripper data for both arms, energy controls, and thumbstick axes and buttons. Camera commands are deliberately limited so that the model is driven by instrument interaction.

What is released and how to try it

NVIDIA points to these resources in its post:

• Code and examples: the isaac-for-healthcare/cosmos3-h-dreams repository on GitHub.
• The model: nvidia/Cosmos3-H-Dreams on Hugging Face, listed as image-to-video.
• The base: nvidia/Cosmos3-H-Embodied-Nano on Hugging Face and GitHub, with a step-by-step adaptation guide for teacher fine-tuning and Self-Forcing distillation.
• The dataset: Open-H-Embodiment on Hugging Face.
• The runtime: FlashDreams on GitHub, and the broader NVIDIA Cosmos 3 framework.

If you want to adapt it to your own data, the post cautions that you must match the action conventions and normalization used in training. That detail is where most world-model adaptations quietly fail: a mismatched action scale produces fluent video that ignores the controls.

We have not run the model, so we cannot report quality or speed from first-hand use. The post does not state a license or link a paper; read the model card and repository terms before any reuse, especially anything beyond research.

What the evidence actually shows

This is the part to read carefully. NVIDIA's post reports no quantitative benchmarks: no latency, frame rates, or image-quality metrics. The evidence is qualitative: two rollouts begin from the same frame and follow different VR-recorded action sequences, producing different continuations. The authors also claim interactive operation on one RTX PRO 6000 and "physically accurate tissue deformations." Those are the authors' claims, not independently measured in the post.

The authors list their own limitations plainly:

• Visual realism is not proof of biomechanical accuracy. Output that looks plausible but ignores controls is not useful as an interactive environment.
• Small errors accumulate over long rollouts, and temporal consistency remains a challenge.
• Scope is one procedure and two instruments, not every phase of cholecystectomy; occlusion and multi-tool coordination are not fully handled.
• Surgeon evaluation of anatomical accuracy and tissue response is future work.

So the right reading is "promising demonstration," not "validated simulator."

Why this matters beyond surgery

Three reasons builders should care.

Simulation as a training ground for robot policies. NVIDIA lists coupling the simulator with autonomous surgical policies in a closed loop as a future direction. If a learned simulator is faithful enough, policies could be trained and stress-tested against it without a patient or a physical rig. That is a big if, and the gap between plausible video and correct mechanics is exactly what has not been measured.

A recipe for domain-specific world models. The teacher-to-causal-student pipeline, with Self-Forcing and a critic, is a template: specialize a big bidirectional model on domain data, then distill it into something that streams. The same shape applies to warehouse robotics, driving, or lab automation. NVIDIA's wider push into this space shows up in its physical-AI partnership coverage in Germany, and its open data efforts, such as the Indic pretraining corpus, follow the same "ship the pipeline and data" approach.

A caution on medical AI claims. Surgery is where hype is most dangerous. Compare it with the first AI-assisted brain surgery, where the useful question was what the AI actually did, and the evidence problem in AI drug discovery, where benchmark wins did not equal clinical proof. Cosmos3-H-Dreams is honest about being at the start of that road.

Why the teacher-student split exists

A bidirectional video model sees a whole clip at once, which gives good temporal consistency but cannot react to an action that has not happened yet. A causal model generates block by block, so each new block can depend on the latest command, which is what interactivity needs. The cost is drift: the student conditions on its own earlier output, so small errors compound. Self-Forcing addresses this by training the student on its own generated context rather than on clean ground-truth frames, so it learns to recover from its mistakes. The learned critic and the teacher then pull the student toward realistic output.

The two-step versus four-step choice is a classic quality-versus-latency dial. Fewer denoising steps per block means the next block arrives sooner, which matters for a surgeon-in-the-loop or a policy that needs fast feedback, at the price of visual detail. NVIDIA published both so users can pick per use case, and the fact the two-step model was trained separately suggests it is not simply a truncated version of the four-step one.

The public code lives in the Cosmos3-H-Dreams GitHub repository, and the weights are on the Hugging Face model page. Start there for the adaptation recipe, and confirm the license on those pages, since the blog post itself does not state one.

Questions people are asking

Can it replace a physical surgical trainer? Not on current evidence. Without surgeon evaluation and quantitative fidelity metrics, there is no basis to say so, and NVIDIA does not claim it.

Why only one procedure? Cholecystectomy is common and well recorded, and the clinical data came from one platform. Generalizing needs more instruments, phases and data, which the authors list as next steps.

Why 10 Hz actions? The post says actions are aligned at 10 Hz for training and notes this differs from runtime generation throughput. Do not read it as the display frame rate.

Is the dataset clinical patient data? The post describes video and kinematics recorded during clinical procedures. It does not spell out consent, de-identification or licensing in the text we read, so check the dataset card before using it.

What to do with it

• Researchers in world models: read the adaptation guide and try Self-Forcing distillation on your own domain, matching action normalization first.
• Robotics teams: treat it as a reference architecture for causal, streaming world models, and run your own closed-loop tests rather than trusting visuals.
• Healthcare and policy readers: note that the maker's own limitations section is the most reliable summary of readiness.

Related reading

• Dream Labs Dream-1: robot world-action model API
• Skyfall Sky-001 engineering world model: claimed vs verified
• World-first AI-assisted brain surgery: what the AI did
• AI drug discovery has an evidence problem
• NVIDIA and Germany's physical-AI partnership
• NVIDIA Indic pretraining dataset

Details are from NVIDIA's October 20, 2026 post and may be updated as the repositories change.