Kyutai CADRE: video autoencoder for world models with under 230 ms lag
Paris-based AI lab Kyutai has released CADRE, a family of seven video autoencoders for interactive world models. They aim to push the delay between input and response below the roughly 230 milliseconds humans can perceive. Weights are open, but only for non-commercial use.
What this is about
Kyutai, an AI lab focused on open research, has released CADRE: a family of video autoencoders built specifically for interactive world models. A world model is an AI system that predicts or generates video frames while reacting to inputs such as key presses, like a game that the AI draws live. The problem CADRE tackles is action lag: the time between an input and the visible response in the video. According to the accompanying paper, world models built on CADRE run at 20 frames per second on a single H100 GPU and respond within 230 milliseconds. All figures are Kyutai's own.
Why the lag arises
A world model doesn't generate video pixel by pixel, but in a compressed "latent space." An autoencoder translates videos into this compact representation and back again. Many world models borrow autoencoders from text-to-video generators. These pack several frames into a single latent block, so every input has to wait for the end of a block.
Compression is written as τ×s×s: τ is the number of frames per latent, s is the spatial downscaling factor. Latency is made up of τ/f (f = frame rate) plus generation and decoding time. At τ = 8 and 20 fps, the block alone costs 400 ms. According to the authors, people notice delays in their own movement from about 230 ms.
How CADRE is built
The encoder follows the recipe of the earlier MIRA: a frozen DINOv3 (a pretrained image model) extracts features per image patch, and a trained 3D convolution condenses them into a latent with 64 channels. Only this bottleneck differs between the seven family members. That lets you study compression as the only variable.
The decoder is a causal diffusion transformer with 24 blocks that streams chunk by chunk. "Causal" means it uses only the past, not future frames. Through progressive self-distillation (the model learns to reproduce its own slower outputs in fewer steps), it decodes in 1 or 2 steps. This happens in a single training phase and without an adversarial loss, meaning no GAN-style opponent. Training used 23 public video sources (about 76,000 hours) from the web, robotics, driving and games.
Evaluation and numbers
Kyutai trained the same 1.2B-parameter world model on each autoencoder, in four environments: robot manipulation (ABC), human manipulation (EgoDex), driving (PhysicalAI AV) and indoor navigation (RealEstate10K). The model sees 2 s of real video and then generates 4.8 s. The metrics are image quality (FID), motion quality (FVD, lower is better) and action fidelity (ARR, where 1 equals real video; measured with a frozen inverse-dynamics probe).
| CADRE-2×32×32 | CADRE-2×16×16 | Wan 2.2 VAE | Cosmos | LTX-2 | |
|---|---|---|---|---|---|
| Action latency (1 H100) | 185 ms | 216 ms | 317 ms | 605 ms | 492 ms |
| Max. fps | 23.4 | 17.3 | 34.2 | 39.0 | 87.4 |
| FVD driving | 171.3 | 162.6 | 599.7 | 452.5 | 913.3 |
| FVD robot | 105.0 | 47.6 | 52.9 | 39.5 | 69.6 |
| ARR robot | 0.45 | 0.73 | 0.85 | 0.78 | 0.25 |
| PSNR (reconstruction) | 22.86 | 25.12 | 35.02 | 31.39 | 28.24 |
Overall, Kyutai says CADRE variants cover most of the latency-quality frontier; in 42 of 48 comparisons with MIRA, Wan 2.2, Cosmos and LTX-2, they are equally fast and better. Compared with the retrained MIRA decoder, the 1-step diffusion decoder cuts FVD by a factor of 1.4 to 2.5. DC-AE is faster at 118 ms, but works frame by frame.
Four takeaways
- Decode with diffusion: At high compression, perceptual regression blurs images, while diffusion pulls out sharp video.
- Compress time in the autoencoder: If time is only merged inside the world model ("patching"), detail is lost; at 8 frames, ARR drops from 0.72 to 0.56.
- Camera motion decides: Static cameras tolerate temporal compression, while moving cameras benefit from fewer frames per latent and coarser spatial tokens.
- Coarse tokens hurt motion first: In robot manipulation, 32× spatial compression multiplies FVD by 2.4 compared with 16×; FID barely changes.
Weaknesses
CADRE doesn't win everywhere. In robot manipulation, Cosmos reaches an FVD of 39.5 versus 47.6 for the best 1-step CADRE, and on action following, Wan 2.2 with an ARR of 0.85 is clearly ahead of CADRE-2×32×32 (0.45). Wan 2.2 reconstructs unseen video much better (35.02 dB PSNR). Kyutai argues that reconstruction quality doesn't predict rollout quality; Wan 2.2 ranks last on indoor FVD. With long latents, you can see seams between chunks (boundary excess of 1.6 to 1.8 at τ = 8). CADRE-4×24×24 is an outlier with an FVD of 403 on driving. Each world model covers only one environment at 288×512; whether the results carry over to generalist models is an open question.
What you need to try it
The reference setup is an H100 (80 GB, bf16, batch size 1) for real time at 20 fps. On consumer cards, it will likely be much slower or not run smoothly at all; Kyutai doesn't mention quantized variants. There is a live demo where you control the models with your keyboard, and all you need is a browser. Released are 7 autoencoders, 4 longer-trained multi-resolution versions and 4 world models built on CADRE-2×16×16.
Our take
CADRE is above all a research contribution: it isolates clean design rules for world models (compression, decoder, camera) and provides open building blocks. For robotics, driving simulation or playable worlds, low latency is central. The caveats: all numbers are self-reported, testing covered only four narrow environments, and the weights are non-commercial under CC BY-NC 4.0 (code: MIT; the DINOv3 license applies on top). If you prioritize reconstruction quality or action following in robotics, Wan 2.2 or Cosmos sometimes score better.
Sources
- MarkTechPost marktechpost.com
Try it yourself: CADRE
Explained step by step, no prior knowledge needed – on your own computer or free in your browser.
What can I do with it?
CADRE provides video autoencoders and small world models that generate video live and react to keyboard input, such as driving or controlling a robot arm. You can use them to explore interactive, AI-generated scenes with low lag.
Who is it worth it for?
Interesting for researchers and developers working on world models, robotics and simulation projects. Not a great fit for everyday users or commercial projects (non-commercial license, hardware requirements).
What your computer needs
For low latency, pick CADRE-2×32×32 (185 ms); for better quality and action following, CADRE-2×16×16 (216 ms). The four released world models are based on CADRE-2×16×16. Quantized variants are not listed.
- Reference: one H100 in bf16, batch size 1, 20 fps at 288×512.
- The autoencoder has about 476M trained parameters plus 86M for DINOv3; the world model has 1.2B.
- Real time is unlikely on weaker GPUs; the source gives no details on this.
- A live demo in the browser lets you try it without your own hardware.
How do I find out what my computer has?
- Windows: Press Ctrl + Shift + Esc → Task Manager → Performance tab → click GPU on the left. At the bottom you'll see Dedicated GPU memory (that's your VRAM). You'll find system memory under Memory.
- Mac: Apple menu at the top left → About This Mac. You'll see your chip (e.g. M2) and memory (e.g. 16 GB). On Apple chips, graphics and processor share this memory.
- Linux: Type
nvidia-smiin the terminal (shows your graphics card and its memory) andfree -hfor system memory.
Guide
This model runs with its developers' own program. The commands below come from the Model Card and the developers' instructions.
Windows 10 / 11
Windows 10 / 11: not officially tested. The source only mentions CUDA (NVIDIA GPU) and doesn't test Windows. It should work with an NVIDIA card. Alternative: WSL2 (Linux on Windows) with the Linux steps.
- Check the prerequisites. You need Python (3.10 or later), Git and an NVIDIA GPU with a current driver. The source doesn't give the exact Python version, so check the README in the code folder. In PowerShell, check that everything is there.
python --version git --version nvidia-smi - Get the code package. The code (CADRE code release) is a separate package. The address is on the model's Hugging Face page; the source doesn't list it. Replace <CADRE-CODE-URL> with that address.
git clone <CADRE-CODE-URL> cadre cd cadre - Install uv. uv is a tool that sets up Python packages. The instructions require it for installation.
python -m pip install uv - Set up the environment. This command creates the environment and downloads all required packages. It may take a while. Stay in the cadre folder.
uv sync - Create the test file. Save this code as demo.py in the cadre folder. It loads the model (about 2.2 GB, downloaded automatically on first run) and encodes a random video.
import torch from model_autoencoder import CADRE with torch.device("cuda"): autoencoder = CADRE.from_pretrained("kyutai/cadre-2x16x16") video = torch.randint(0, 256, (1, 16, 3, 288, 512), dtype=torch.uint8, device="cuda") with torch.no_grad(): z = autoencoder.encode(autoencoder.preprocess(video)) video_hat = autoencoder.decode(z, decode_steps=1) print(z.shape, video_hat.shape) - Run it. If you get a download error, log in to Hugging Face with "hf auth login."
uv run python demo.py - Result. The script prints the shapes of the latents (1, 8, 64, 18, 32) and the reconstructed video (1, 16, 3, 288, 512).
macOS
macOS: not officially tested. The source uses only CUDA (NVIDIA) and doesn't test on Mac. Without an NVIDIA card, the example won't run as written. Switch to Windows/Linux with an NVIDIA card, or to Google Colab with a GPU.
- Check the prerequisites. You need Python 3, Git and the Xcode command-line tools. The source doesn't give the exact Python version, so check the README in the code folder. The example uses "cuda," which doesn't exist on Mac.
xcode-select --install python3 --version git --version - Get the code package. The address of the CADRE code is on the model's Hugging Face page, not in the source. Replace <CADRE-CODE-URL> with it.
git clone <CADRE-CODE-URL> cadre cd cadre - Install uv. uv is a tool that sets up Python packages.
python3 -m pip install uv - Set up the environment. Creates the environment and downloads the packages.
uv sync - Create the test file. Save the code as demo.py in the cadre folder. The example uses "cuda" and will probably crash on Mac. Whether "mps" or "cpu" works is untested.
import torch from model_autoencoder import CADRE with torch.device("cuda"): autoencoder = CADRE.from_pretrained("kyutai/cadre-2x16x16") video = torch.randint(0, 256, (1, 16, 3, 288, 512), dtype=torch.uint8, device="cuda") with torch.no_grad(): z = autoencoder.encode(autoencoder.preprocess(video)) video_hat = autoencoder.decode(z, decode_steps=1) print(z.shape, video_hat.shape) - Run it. If you hit download errors, log in to Hugging Face with "hf auth login."
uv run python demo.py - Result. The script prints the shapes of the latents (1, 8, 64, 18, 32) and the reconstructed video (1, 16, 3, 288, 512).
Linux
Linux: supported. The source shows an example using CUDA, meaning an NVIDIA GPU. Other cards (AMD) are not mentioned.
- Check the prerequisites. You need Python 3 (the exact version is in the README of the code folder), Git and an NVIDIA GPU with the driver installed. Check in the terminal.
python3 --version git --version nvidia-smi - Get the code package. The CADRE code is a separate package. The address is on the model's Hugging Face page; the source doesn't list it. Replace <CADRE-CODE-URL> with it.
git clone <CADRE-CODE-URL> cadre cd cadre - Install uv. uv is a tool that sets up Python packages. The instructions require it.
python3 -m pip install uv - Set up the environment. This command creates the environment and downloads all packages. It may take a while. Stay in the cadre folder.
uv sync - Create the test file. Save the code as demo.py in the cadre folder. It loads the model (about 2.2 GB, downloaded automatically on first run) and encodes a random video of 16 frames.
import torch from model_autoencoder import CADRE with torch.device("cuda"): autoencoder = CADRE.from_pretrained("kyutai/cadre-2x16x16") video = torch.randint(0, 256, (1, 16, 3, 288, 512), dtype=torch.uint8, device="cuda") with torch.no_grad(): z = autoencoder.encode(autoencoder.preprocess(video)) video_hat = autoencoder.decode(z, decode_steps=1) print(z.shape, video_hat.shape) - Run it. If you hit download errors, log in to Hugging Face with "hf auth login."
uv run python demo.py - Result. The script prints the shapes of the latents (1, 8, 64, 18, 32) and the reconstructed video (1, 16, 3, 288, 512).
Try it
In the live demo, steer the vehicle with your keyboard and watch the response time.Have the robot world model move a gripper arm and check whether it follows the actions.Compare CADRE-2×32×32 and CADRE-2×16×16 with a moving camera indoors.If something goes wrong
The weights are CC BY-NC 4.0: no commercial use.
The DINOv3 module has its own license, which you may need to accept on Hugging Face.
Each world model covers only one environment at 288×512; it is not a generalist.
“out of memory,” a crash, or your computer freezes
The model is too big for your VRAM (video memory). Pick a smaller variant from the table above and close other programs, especially games and browsers with lots of tabs.
It runs, but extremely slowly
Your CPU is probably doing the work instead of your graphics card. In Ollama, the PROCESSOR column of ollama ps shows how much runs on the GPU. If it says "100% CPU", update your graphics driver or pick a smaller variant that fits entirely into VRAM (video memory).
“ollama” or “python” is not found
Close the terminal window and open it again so it recognizes the newly installed program. For Python on Windows: reinstall it and check Add python.exe to PATH.
Windows: "Running scripts is disabled on this system"
Run the script with exactly the command above: powershell -ExecutionPolicy Bypass -File try_windows.ps1. This allows execution for this one time only.
Error message when loading the model (version, "trust_remote_code", unknown architecture)
Brand-new models often need the latest software version. With Ollama: update Ollama. With Python: pip install -U transformers in the activated environment. The Model Card usually lists the exact version.
Tokenstrom generates the scripts and notebook automatically. Please take a quick look before running them – they're plain, readable text. Mind the model's license: cc-by-nc-4.0.
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.