Nvidia Cosmos 3: four open world models simulate traffic and robot arms

What's going on

On November 5, 2026, Nvidia released four new models for its Cosmos platform. Cosmos is Nvidia's collection of so-called world models. These are AI systems that predict how a scene in the real world will unfold, say when a car turns or a robot arm reaches for an object. The four new checkpoints (fully trained model snapshots) belong to the Cosmos 3 family and are post-trained offshoots of the Cosmos 3 Nano base model. Each has 16 billion trainable parameters.

The weights are available for download on Hugging Face. Nvidia says the OpenMDW 1.1 license permits commercial and non-commercial use within its terms. There's also a technical report, demos, notebooks and so-called agent skills: instructions an AI coding assistant can work through to get the models running.

The four models at a glance:

Why you'd need this

Robots and self-driving cars learn from data. But nobody wants to re-stage dangerous or rare situations, like a nighttime drive in heavy rain, hundreds of times on real roads. World models are meant to generate such situations convincingly. A robot's or car's control software, known in the jargon as a policy, can then be trained and tested in simulation before it goes onto real hardware.

In autonomous driving, appearance and geometry have to match. If the camera shows a car, the LiDAR sensor (a laser scanner that measures distances and builds a 3D point cloud from them) has to see that car in the same spot. That's why the automotive models generate both in sync. Using controls for the scene and traffic lights, researchers can change road layouts and signal states and watch how the driving software reacts.

The automotive models: from offline data to live simulation

Transfer-Auto is a so-called bidirectional model. It processes an entire clip at once and is allowed to look "into the future" while doing so. That makes it precise, but not interactive. It generates synchronized video from up to 11 camera views, including fisheye cameras, plus LiDAR data on request. It's controlled via World Scenario Maps (WSM), per-camera maps that define the road layout and traffic movement. A text prompt sets the look: weather, lighting, time of day. Optionally, real initial frames anchor the appearance of the scene. For LiDAR, control data from high-definition maps (HD maps) is added.

According to Nvidia's developer blog from November 2, as reported by blockchain.news, Transfer-Auto processes video at 30 frames per second, up to 400 frames in 480p or 720p. Longer sequences come from the model continuing to compute piece by piece. A reference run produces seven 720p videos with 297 frames each, plus 99 LiDAR scans.

Sim-Auto is the streaming version. Nvidia slimmed down Transfer-Auto via distillation, letting a faster "student" model learn from a slower "teacher". Sim-Auto generates small, consecutive chunks of camera and LiDAR data in just a few compute steps. It uses its own previous output as memory and takes updated driving commands into account. A true closed loop, where the driving software steers and the model delivers the next frames, also requires a simulation environment and a WSM renderer, though. The model doesn't ship with those.

Nvidia recommends a fixed path: if you have your own camera rigs or LiDAR sensors, first post-train Transfer-Auto on your own data, then distill it into your own Sim-Auto variant.

The robotics models: two opposite approaches

Sim-Transfer doesn't simulate physics itself. A classic physics simulator computes motion, contacts and states. Sim-Transfer takes its spatial auxiliary data and turns it into photorealistic frames that robot software can process. The control inputs are depth maps (distance to the camera), segmentation (which image regions belong to which object), edges (outlines) and a blurred rough template for layout and appearance. The text sets the look; the physical ground truth stays with the simulator.

Sim-Bimanual takes the opposite route and generates the next frame directly from the scene and robot motion, with no simulator in between. It supports two-armed robots of the YAM and AgiBot types. Control comes from a policy or from humans with joysticks. You start with a frame, move the robot or camera, and get the following frames predicted. A control sequence can switch between camera pans and grasping motions. The example uses 59 control values, padded to 64 during training. A published schema defines coordinate systems, rotations, gripper calibration and value ranges. Timing matters: for T motion steps, you need T+1 frames.

How the streaming works

Cosmos 3 is built on a Mixture-of-Transformers architecture: separate networks for text processing and image generation are connected via attention, the mechanism a model uses to decide which information matters right now. The model works with latents, compressed video representations. After the initial frame, each latent time step represents four actual video frames.

The flow: control data and history go in, the model generates a short chunk, it gets decoded into frames and appended to the history. During denoising, noisy data is gradually turned into a clean chunk. Temporally causal attention ensures the model only sees the past and can't peek into the future. A key/value cache stores intermediate results so they don't have to be recomputed every time. The smaller the chunks, the more often new commands can feed in.

Training runs in four stages:

• Stage 0: The model learns the new domain from complete clips.
• Stage 1 (teacher forcing): It predicts each chunk from real earlier footage.
• Stage 2 (consistency distillation): A frozen teacher model trains a student that gets by with four denoising steps.
• Stage 3 (self-forcing): The student trains on its own self-generated history.

The last stage is crucial. A slightly mispredicted object becomes the basis for every following frame, so errors compound. Self-forcing confronts the model with this problem during training. Nvidia also stresses that a successful short training run only proves the code works, not that the model has learned anything good.

What you need to try it

This isn't a model for your gaming PC. Nvidia lists Linux, an Nvidia GPU, Docker and the NVIDIA Container Toolkit as prerequisites. The published vLLM-Omni setup was tested on a single GB200, a data center chip. Nvidia stresses that this isn't a minimum requirement. According to the memory table reported by blockchain.news, Cosmos3-Nano needs 58 GiB of VRAM for video generation in BF16, 44 GiB in FP8 (lower-precision numbers) and 31 GiB with FP8 plus offloading to system RAM. Realistically, that means pro cards with 48 GB or more, or a rented cloud server. An RTX 4090 with 24 GB isn't enough by these numbers. It won't run on a Mac or CPU-only, and it's too big for free Google Colab. If you just want something small running locally, Pulse 2.0 says you're better served by the also-announced Cosmos 3 Edge (4 billion parameters, for Jetson Thor).

The bigger package

According to DotAI Times, the models are part of a larger announcement Nvidia made at GTC Berlin. It includes the Cosmos Framework for training, reinforcement learning and optimization, which also supports third-party models such as GR00T, Qwen and pi0.5, plus the Cosmos Data Factory with five blueprints for synthetic data. Nvidia says the revamped Cosmos NIM microservices run up to twice as fast. As real-world examples, Nvidia cites Porsche, which aims to cut real test drives by up to 90 percent, and Siemens, with 40 percent less calibration time in circuit board inspection. DotAI Times notes that these are individual cases that don't necessarily carry over elsewhere.

The big picture

The four checkpoints aren't flashy showcase models. They're tools for robotics and automotive professionals. What stands out is the openness: alongside the weights, Nvidia is publishing the complete post-training recipes. That lowers the barrier for companies and research groups that can't build their own simulations from scratch. It also fits Nvidia's strategy: open models that run best on Nvidia hardware sell GPUs.

The core question, which Nvidia itself openly acknowledges, remains: how well does a generated world actually hold up when you use it to train and evaluate robots or cars? Solid independent measurements are still missing. For individual users, the whole thing is barely usable given its memory requirement of more than 30 GB.