Nemotron 3.5 Super VL: NVIDIA's 120B model in NVFP4 for text, image and video

What this is about

NVIDIA has released the quantized version of Nemotron 3.5 Super VL on Hugging Face, specifically NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4. It is a vision-language model, meaning it can understand text, images and video and outputs text. It's intended for reasoning, coding, tool use, document understanding and video analysis. This variant uses NVFP4, a 4-bit number format, to cut memory requirements. The quantization was retrained from the BF16 model (the 16-bit variant) with "Quantization-Aware Distillation."

Architecture

The model has 120 billion parameters (roughly, the knobs adjusted during training), of which only 12 billion are active per token. That comes from the Mixture-of-Experts (MoE) principle: there are many specialized sub-networks ("experts"), here 512 per MoE layer, and only a small share is selected for each token. This saves compute.

The layers alternate between Mamba-2, MoE and attention. Mamba-2 is an alternative to classic attention that needs less memory on long texts. For images, a C-RADIO vision encoder with a connector is used, which translates image information into a form the language model can process. Context extends to 1 million tokens. The prompt, images, history, tool results and response all share this window. Individual deployment recipes often use less for memory reasons.

A Multi-Token Prediction (MTP) head predicts several future tokens, which enables speculative decoding: the main model checks a fast draft, and several tokens can be accepted per step. For DGX Spark there's also the "LiLiCorr" drafter, which NVIDIA recommends there in all scenarios.

Thinking can be controlled via the chat template: off, on (medium), or on. "Medium" generates fewer tokens but can reduce accuracy.

Training

According to NVIDIA, training ran in seven stages:

1. The language backbone starts from Nemotron 3 Super Base; the MTP head is trained on 100B tokens, followed by 1 trillion tokens of distillation from Nemotron 3 Ultra Base.
2. Continued pre-training (100B Indic tokens), extension to 1M context (35B tokens) and 67B tokens of software-engineering mid-training, using things like repositories, commit diffs, CI logs and agent trajectories.
3. Vision-language training with images, document pages and video frames.
4. Supervised fine-tuning with sequences of up to 524,288 tokens.
5. Reinforcement learning (verifiable rewards, human feedback) and multi-teacher distillation.
6. MTP boosting for better acceptance rates.
7. Quantization with NVIDIA Model Optimizer. The vision encoder, connector, Mamba convolutions and embeddings stay in BF16, and the MoE router in FP32. Inference uses W4A4 on Blackwell and W4A16 on other platforms.

Benchmarks

NVIDIA compares BF16 and NVFP4 with identical settings. A selection (vendor figures):

Quantization costs little on most tests, and NVFP4 even edges ahead in some. MMLU-ProX, WebVoyager and DeepSWE show clear drops. If you plan to run agentic tasks like these, keep that in mind. According to NVIDIA, the results come from NeMo Gym; some benchmarks used official implementations or internal, unreleased harnesses. The model card, at least in the text we have, includes no comparisons with competing models.

What you need to try it

The model is built for NVIDIA hardware (Blackwell and Hopper, Linux). NVIDIA lists 1× DGX Spark, 1× H200, 2× H100 or 1× GB200 as target systems. A normal gaming graphics card generally isn't enough for a model this size; the card also mentions local runs on RTX Spark and RTX Pro 6000 via llama.cpp (ggml-org collection) and Unsloth. According to the source, it isn't intended for Mac, CPU-only setups or Google Colab. Expect to need more than 60 GB of memory for the weights, so be generous with disk space too. For video, these limits apply: up to 2 minutes of MP4, at 1080p 1 FPS and max. 128 frames, at 720p 2 FPS and 256 frames. One image and one video per request are possible. Recommended sampling values: temperature 1.0 for text and 0.6 for vision, each with top_p 0.95.

For production, NVIDIA offers NIM containers and the Dynamo framework.

Our take

The model shows how large MoE models with 4-bit quantization can fit on a few GPUs or even a compact desktop supercomputer. The benchmarks show that quality mostly holds up, with a few exceptions. NVIDIA calls the Nemotron family open, with weights, training data and recipes. According to the card, the OpenMDW-1.1 license allows commercial use. What remains open is how the model stacks up directly against other open models, and the numbers are vendor claims. The source is also truncated (the training data section is missing), and the model card gives the release date as 10/15/2026.