NVIDIA on November 18, 2026 published Build Training-Ready Physical AI Datasets with Two Prompts in NVIDIA Cosmos, framing Cosmos Data Factory (CDF) as five reference blueprints plus six composable skills that a coding agent can run with a setup prompt and a task prompt. The goal is prompt-to-labeled-data for autonomy, robotics, retail analytics, and visual inspection without stitching together disconnected curation tools by hand.
Physical AI teams usually lose months on rare events—night glare, line defects, warehouse falls—that real cameras rarely capture. CDF turns those gaps into repeatable generation jobs with evaluators and auto-labeling in the loop.
What is Cosmos Data Factory?
NVIDIA Cosmos platform stack for physical AI dataset tooling
Image: NVIDIA
CDF is the dataset layer of the broader NVIDIA Cosmos open platform (models, datasets, frameworks, and skills for world models). The November 18 post emphasizes:
- Five cloud-native reference blueprints for common physical-AI dataset tasks
- Six core agent skills you can mix outside the canned blueprints
- CDF Orchestration, which deploys services, sequences steps, and runs jobs on your cluster
NVIDIA positions frontier coding agents as the driver: you paste prompts into an agent harness; the agent clones repos, runs preflight, and invokes blueprint APIs against your object storage buckets.
This is distinct from a single new checkpoint release—it is infrastructure for synthetic and augmented data at the scale robotics and AV teams already chase in NVIDIA GR00T and embodied-AI coverage and Cosmos world-model simulation posts.
Prompt 1: Orchestration preflight and setup
NVIDIA's first prompt is deliberately boring—in a good way:
Clone the CDF Orchestration repository, then run orchestration preflight and setup.
Preflight checks the environment without mutating it; setup prepares services for blueprint runs. That split matters for SRE-minded builders: you can fail fast in CI before touching production buckets.
If you already operate agent workspaces with multi-harness flows, treat CDF like any other MCP-or-shell tool chain: pin repos, log preflight output, and store setup artifacts per cluster.
Prompt 2: Pick a blueprint
A green geometric mold produces three matching cream shapes, illustrating controlled synthetic data variants
The second prompt selects one of five training-data patterns. NVIDIA documents example prompts with placeholder buckets such as <input-bucket> and <output-bucket>.
| Blueprint | What it produces | Example second prompt (abbreviated) |
|---|---|---|
| AV Dataset Generation | Weather and lighting variants of driving video with plausibility checks | Search scenes like video.mp4, add rain/snow/fog, check physics |
| Image Attribute Augmentation | Person-crop clothing variants with attribute labels | Three outfit variants per person in a bucket |
| Video Data Augmentation | Lighting and tabletop changes for manipulation video | Yellow/dim lighting and red table variants |
| Event Video Generation | Rare events (falls, collisions, spills) from a still | Traffic stall and intersection collision clips |
| Defect Image Generation | PCB/glass/metal defects from golden images or USD assets | Structural rotation defects with full board coverage |
Each path chains Cosmos Curator, Cosmos Dataset Search, Cosmos3 video or image models, Cosmos Evaluator plausibility checks, and auto-labeling where the post specifies it.
Evaluator: why generation is not "done" when pixels exist
Several blueprints call Cosmos Evaluator to reject clips with motion artifacts, missing requested conditions, or implausible physics. That is the difference between "synthetic data for demos" and training-ready data.
Builders should budget retry loops in orchestration—the Event Video blueprint explicitly retries failed clips before labeling. If you skip Evaluator to save GPU hours, you import silent label noise into downstream VLA or detection training—a problem inclusion benchmarks in healthcare AI and other vertical evals keep highlighting.
Six skills beyond the blueprints
NVIDIA lists standalone skills:
- Auto Labeling — detections, tracks, captions, scene understanding
- AnomalyGen — defect images from few examples
- Augmentation — Cosmos-driven visual variants
- Curation and Retrieval — Curator plus Dataset Search
- Simulation — Isaac Sim and Omniverse Replicator renders
- Orchestration — blueprint deployment and scaling
You can clone a single skill repo—for example auto labeling at github.com/NVIDIA/cdf-auto-labeling per the post—and point VLMs at a local endpoint if you cannot send video to a hosted API.
Cosmos Curator adds multicamera video, IMU, GPS, and early lidar support; Dataset Search adds agentic reranking in early access. Both tie into NVIDIA's sparse-attention and long-context inference work when your curation agents read long sensor logs.
Throughput and cost claims (compute only)
A machine producing green tiles beside a few gray ones, illustrating synthetic dataset generation at scale
NVIDIA publishes daily throughput on eight RTX PRO 6000 Blackwell GPUs at about $40/server-hour (~$960/day). Highlights from the post:
| Workflow | NVIDIA-stated daily output | Stated unit economics (GPU only) |
|---|---|---|
| AV edge-case clips | ~130 clips/day | ~$2.66 GPU per clip vs ~$125 road collection |
| Image attribute pairs | ~42,649 pairs/day | ~$0.023 per pair vs ~$0.25 third-party purchase |
| Video augmentation | ~305 five-minute clips/day | ~$3.15 per clip vs ~$22 purchased data |
| Defect assembly | two months to one day | 60x assembly speedup claim |
| Event videos | ~384 clips/day | ~$2.50 per ~4-minute clip |
NVIDIA warns these figures are compute-only—storage, human review, agent tokens, and source ingestion add cost. Treat them as order-of-magnitude planning numbers, the same way we handled vLLM Rubin throughput claims.
Customer outcomes NVIDIA cites
The post lists field-style results (always attribute them to NVIDIA's blog, not independent verification here):
- Up to 24% recall improvement on real-world anomalies with synthetic data at an offshore oil and gas site, with a cited 1,400x ROI on generation cost
- Up to 85% less manual engineering in AOI program generation with 40% fewer false calls (targeted)
- Up to 90% fewer repeated physical test drives and 85% shorter AI regression cycles (expected)
- Under two weeks to market with Cosmos Curator vs an estimated six months to build in-house
Pair those claims with your own holdout set before you bet a production model on generated rare events.
How this fits NVIDIA's November hardware narrative
The same week NVIDIA pitched RTX Spark compact desktops and Rubin-era CUDA core counts, CDF answers the boring question: what fills the GPUs once you buy them? Synthetic data pipelines are the consumption layer for world models and VLA backbones.
If you run on-prem, map blueprint GPU needs to your queue scheduler; if you burst to cloud, tie orchestration to TSMC upfront cash and credit limits debates—long curation jobs are capex-shaped even when labeled "opex."
November 19 livestream and ecosystem partners
NVIDIA schedules a November 19 livestream, Build Training-Ready Datasets in Two Prompts, featuring Voxel51 (Porsche via Voxel51) and Encord with ANYbotics. That is the right place to watch for bucket policies, agent harness choice, and Evaluator failure rates demos the blog does not show.
Labeling partners matter because CDF's value is not MP4 files—it is annotated manifests tracing each synthetic row to source assets and generation settings, especially for defect generation where regulators ask provenance questions.
Cosmos Curator and Dataset Search in production pipelines
The November 18 post separates blueprint orchestration from ongoing curation: Cosmos Curator ingests multicamera rigs, IMU traces, GPS tracks, and nascent lidar feeds, while Dataset Search retrieves clips by semantics and motion—not only filenames. For teams already versioning terabytes on object storage, that means agents can answer questions like "find sunset highway merges with hard braking" without hand-labeling entire drives first.
Dataset Search’s Apache 2.0 release matters for hybrid clouds: you can run retrieval on-prem while bursting augmentation to cloud GPUs, as long as Evaluator policies match between environments. Document embedding model versions in manifests so retraining does not mix incompatible vector spaces—a failure mode we have seen in RAG agent memory designs when retrieval indices drift.
Safety and consent for synthetic rare events
Event Video Generation can synthesize fires, fights, and falls without staging dangerous scenes. That helps model robustness but creates duty-of-care questions: synthetic violence in retail clips can still harm annotators; defect generation near worker imagery needs consent review. NVIDIA’s Evaluator reduces physics glitches; it does not replace legal review of surveillance-like outputs.
Treat CDF like any synthetic media pipeline: watermark training-only rows, keep human review queues on edge cases Evaluator passes but humans flag, and never ship generated defect images as real incident evidence in customer communications.
Getting started checklist
- Pick the blueprint closest to your failure mode (rare event vs defect vs AV weather).
- Run Orchestration preflight in a staging cluster; fix missing services before production buckets.
- Wire Evaluator thresholds to your downstream metric—do not accept default pass rates blindly.
- Store outputs with manifests linking to source USD or video IDs (Defect blueprint ships this explicitly).
- Compare a small real validation set before merging synthetic rows into pretraining mixes.
For robotics teams comparing open VLA stacks, cross-read SAGE S2.1 Flash science agents and ForceHarness frozen-frontier robot adapters—dataset factories and harness choice are twin bottlenecks.
Bottom line
Cosmos Data Factory is NVIDIA's bet that agent-orchestrated dataset assembly becomes as standardized as fine-tuning scripts. The November 18 post gives builders a concrete menu—five blueprints, six skills, two prompts—and enough throughput math to scope a pilot. Verify economics on your own cluster before you cancel camera rigs.
If you are standardizing on one coding agent for infra tasks, pin the CDF Orchestration git revision in your runbook so Prompt 1 does not silently pull breaking skill API changes.
Related reading
- ScaleVLA 552B vision-language-action robot backbone
- MBZUAI PAN-2 open world model for simulative reasoning
- NVIDIA Nemotron for Science and LitmusBench
- NVIDIA sparse attention for long-context inference
- RTX Spark compact AI desktops shipping
- SODUWORK multi-harness agent workspaces
- InclusionMed healthcare AI benchmark index
- vLLM Rubin throughput claimed vs verified
