NVIDIA’s Cosmos 3 gives robotics, autonomous‑vehicle and warehouse‑monitoring teams a single open model that handles perception, prediction and action in one pipeline. Earlier approaches split these tasks across separate vision‑language models, video generators and policy networks, creating integration overhead, inconsistent data formats and extra engineering effort. Cosmos 3 removes that friction by coupling an autoregressive reasoner tower with a diffusion‑based generator tower inside a Mixture‑of‑Transformers architecture. The reasoner understands images, video and text, while the generator creates physics‑aware future frames and action sequences conditioned on that understanding. Information flows one way, from reasoner to generator, so teams can run the reasoner alone for fast perception or activate both towers for guided generation.
Two ready‑to‑use checkpoints cover the main deployment scales. Cosmos3‑Nano (16 B total, dense 8 B backbone) fits workstation GPUs such as the RTX PRO 6000 and enables real‑time inference on‑device or at the edge. Cosmos3‑Super (64 B total, dense 32 B backbone) targets datacenter GPUs like Hopper and Blackwell for large‑scale synthetic data generation and advanced reasoning tasks. Both models share the same Qwen3‑VL‑based backbone, simplifying version control and reducing the need to maintain multiple codebases.
The release includes six synthetic‑data‑generation datasets covering robotics, physics, spatial reasoning, human motion, driving and warehouses, plus training recipes for supervised fine‑tuning and action post‑training. Teams can quantize checkpoints to BF16, FP8 or NVFP4, with NVFP4 delivering up to 2× speedup on supported hardware. Serving options range from research‑friendly Diffusers and Transformers to production‑ready vLLM‑Omni and vLLM stacks, and the Reasoner NIM microservice is available now, with the Generator NIM coming later.
To validate outputs, NVIDIA provides the Human Evaluation (HUE) framework, which breaks generated videos into yes/no fact questions across semantic alignment, physical laws, geometric reasoning and visual integrity for seven physical‑AI domains. This gives teams a practical, repeatable way to check temporal consistency, motion stability and sound‑video alignment before deploying in safety‑critical settings.
All code, weights, datasets and benchmarks are released under the OpenMDW‑1.1 license, removing licensing barriers and encouraging community contributions.
#AI #Robotics #AutonomousVehicles #WarehouseAutomation #OpenSource #PhysicalAI