NVIDIA Cosmos Interactive Simulation Guide: Sim-Auto, Transfer-Auto, Surgical Dreams and NIM

NVIDIA has published a hands-on guide to building interactive simulators with its Cosmos world models. The November 2, 2026 NVIDIA Technical Blog post tells developers which released Cosmos 3 Nano checkpoint to use for driving, robot and surgical simulation, what GPU memory each model needs, and how to launch the NIM container for serving.

This is a follow-up to the product announcements we covered earlier, not a new launch. If you want the big picture first, read our write-up of the Cosmos Data Factory blueprints and Framework skills and the Cosmos 3 launch guide. This post turns the new material into a decision guide. All performance figures below are NVIDIA's own.

What a world model does in a simulator

A classic simulator models geometry, physics and rendering by hand. A world foundation model learns visual dynamics from video and sensor data, then generates the next observation. NVIDIA's pitch is that generative simulation complements classic simulators, especially where physics is hard, such as tissue in surgery. If the idea is new to you, our world models explainer covers the basics.

NVIDIA frames the problem as two challenges for physical AI developers: adapting models to their own sensors, tasks and data, and getting those models into applications reliably. Cosmos Framework addresses the first, and Cosmos NIM microservices address the second.

Cosmos Framework: what it covers

NVIDIA describes Cosmos Framework as open and fully modular for world model training, optimization and inference. It is expanding beyond Cosmos 3 to support NVIDIA Alpamayo, GR00T, Qwen, pi0.5 and other compatible Hugging Face models. Per the post:

• Training. Text, images, video, audio and actions, with LoRA for parameter-efficient adaptation or full supervised fine-tuning. Data loads through JSONL, WebDataset, LeRobot or custom adapters.
• Reinforcement learning. Newly supported pipelines are DPO (offline preference optimization), GRPO (online RL with a rule or model reward and vLLM rollouts), PPO (actor-critic, including RL for embodied policies in simulation), OPD (on-policy distillation from a frozen teacher) and DiffusionNFT (online RL for diffusion world models).
• Optimization and inference. Distillation to cut model size or generation steps, quantization to cut memory, and compatibility with Hugging Face Diffusers, PyTorch, SGLang, vLLM-Omni, vLLM, TRT-LLM, TRT-Edge-LLM and NIM.
• Agent skills. Setup, inference, post-training and troubleshooting skills live under .agents/skills. NVIDIA's example prompt asks an agent to post-train a checkpoint, pin all revisions, validate the dataset, keep training and evaluation episodes separate, stop at a GPU-hour budget, and compare against the base model on held-out data using three seeds.

That last point is worth copying even if you never use Cosmos. A bounded, auditable task prompt with a budget cap and a held-out comparison is a sound way to hand any training job to a coding agent. For another example of agents running training loops, see our piece on NeMo Gym and Siemens EDA chip-design agents.

Which checkpoint for which job

NVIDIA says the new Cosmos 3 Nano simulation recipes and example checkpoints support PyTorch and vLLM-Omni, tested on Hopper and Blackwell GPUs. Here is the decision table distilled from the post.

Autonomous vehicles

Both AV checkpoints take a prompt and World Scenario Map controls that represent lanes, objects, road markings, traffic signs and road boundaries. The joint RGB and LiDAR generation also uses HD-map rangemap controls and optionally an initial LiDAR observation. Prompts describe weather, lighting, time of day and road conditions.

The practical trick NVIDIA highlights: change the prompt to turn a daytime drive into a rainy night while keeping the same road layout and vehicle trajectories. That gives perception teams controlled variation for training and evaluation.

The reference Transfer-Auto run uses four GPUs and the published sample with seven synchronized videos, per-camera captions, first-frame RGB conditioning and HD-map LiDAR controls. It produces seven 720p videos of 297 frames at 30 FPS plus 99 LiDAR sweeps. NVIDIA tells you to check sample_outputs.json for success, check lidar.safetensors for finite range and intensity values, and then verify alignment and consistency across camera views and LiDAR. Model weights come from the Hugging Face model page for Cosmos3-Nano-Transfer-Auto, which lists the model at 15B parameters.

Sim-Auto closes the loop. According to NVIDIA's figure, a policy model such as Alpamayo, or a user, sends an action to the AlpaSim simulation runtime. AlpaSim updates the simulation state and passes context to Cosmos3-Nano-Sim-Auto, which synthesizes the next camera frames and returns them to the policy. The reference run needs the published Sim-Auto checkpoint plus the sample data from the Transfer-Auto repository, and the same four-GPU launch pattern.

Robots

Sim-Transfer and Sim-Bimanual do different jobs, and NVIDIA is careful to separate them. Sim-Transfer generates visuals from a physics simulator's updated state, so the simulator still owns physical state. Sim-Bimanual generates the next observation directly from the scene and robot actions, with no physics engine in the loop. The second is more flexible and the first is easier to trust. For robot demonstrations, Sim-Transfer can also change the third-person view while keeping actions, states and wrist-camera recordings.

Surgery

NVIDIA argues that surgical scenes are among the hardest for classical simulators because of complex multi-physics. H-Embodied-Nano is built on Cosmos 3 and post-trained on Open-H-Embodiment. It takes an initial RGB frame, a task prompt, the last 16 executed actions and later robot actions. H-Dreams is the real-time version for closed-loop tests. We covered the clinical side in our H-Dreams report. The key caution applies here too: a generated surgical video is a learned approximation, so it can evaluate a policy offline but does not replace testing on physical hardware, which is exactly how NVIDIA words it.

GPU memory: what fits on your hardware

Wireframe world model cube over a grid floor representing GPU memory planning for Cosmos world models

NVIDIA's Table 1 gives the VRAM needed per model and task, in GiB.

Read the table by task. Reasoning with Nano needs 23.1 GiB, which is about 24.8 GB. That is more than a 24 GB card holds, so plan for a 32 GB or larger GPU. Generation with Nano needs 58 GiB in BF16, so it targets an 80 GB data-center GPU unless you quantize to FP8 or offload. With offloading the Nano generator drops to 31 GiB, trading memory for speed. Super generation at 150 GiB in BF16 needs more than one card. The AV reference examples above use four GPUs, and NVIDIA's recipes help you choose GPU count, memory configuration and inference settings.

Cosmos NIM: serving the models

NVIDIA's new Cosmos NIM packages supported models as optimized containers. The post lists these features:

• Quantization, speculative decoding, caching, token pruning and KV-cache-aware routing. NVIDIA reports "measured mean speedups of roughly 2–3x on H200" from speculative decoding.
• Support for fine-tuned Cosmos checkpoints and streaming VLM inference for live video, which NVIDIA says enables sub-second alerts with 10x lower latency.
• Automatic configuration selection, validated performance across supported GPUs, licensing support and quarterly feature, optimization and security updates.

The speed and latency numbers are NVIDIA's, measured on its own setup, and the post gives no baseline details. Treat them as an upper-end guide, and benchmark on your own workload before you commit.

Launching it

The steps, as NVIDIA gives them:

1. Pick a GPU and model configuration from the VRAM table. Install the NVIDIA driver, Docker and the NVIDIA Container Toolkit on Linux.
2. Create a personal key in NGC API Keys with NGC Catalog access, and export it as NGC_API_KEY.
3. Log in to nvcr.io and run the nvcr.io/nim/nvidia/cosmos3:2.0.0 image with NIM_MODEL_TYPE=generator and NIM_MODEL_VARIANT=nano, a 16 GB shared memory setting and port 8000 published.
4. Wait a few minutes, then check /v1/health/ready and /v1/metadata.

Choose a performance profile at launch. The default, latency, finishes one request sooner and suits experimentation, and multi-GPU setups combine GPUs on a request, with 1, 2, 4 or 8 GPUs supported. The throughput profile completes more requests over time using independent replicas and suits dataset generation or many users. It needs 2, 4 or 8 compatible GPUs, and NVIDIA warns that changing only the profile to throughput in the single-GPU example will fail profile selection. NVIDIA also notes that the cache permissions shown in its example, which open the cache directory to all users, are meant for local development only. Tighten them before you run anything shared.

What to take from this

First, the release is practical. Checkpoints, sample data and reference commands exist, so a team with four Hopper or Blackwell GPUs can reproduce the AV example this week. Second, generative simulation is moving from demo to workflow: closed-loop checkpoints (Sim-Auto, H-Dreams) are the important step, because they let a policy act and see consequences. Third, the numbers are still vendor numbers. NVIDIA gives speedups and latency claims without independent tests, and the post makes no claim about how faithfully a generated world matches a real one. Validate against real logs before you trust a policy score from a generated environment.

If you are choosing a first project, start with Transfer-Auto for weather and lighting variation on existing drive logs. It is offline, it fails safely and the output is easy to inspect. Move to Sim-Auto when you need a policy in the loop. For related reading on NVIDIA's physical AI stack, see real-time video faster than playback with Sol H3. NVIDIA also plans a livestream on using Cosmos Framework on Thursday, November 5, and the post links a full technical report for methodology and results.