NVIDIA PiD: This image decoder renders AI images directly at 4x resolution
NVIDIA's PiD replaces the VAE decoder in image generators like FLUX and SDXL. It draws the finished image at four times the width and height in just four steps. A community trick turns it into an upscaler for any image, and the weights are free on Hugging Face.
What it's about
NVIDIA has released PiD (Pixel Diffusion Decoder), a new building block for AI image generators. The weights are free to download from Hugging Face at nvidia/PiD. The accompanying research paper, "PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion", was submitted on May 22, 2026 (arXiv 2605.23902). In early November, a post on the r/StableDiffusion subreddit put PiD in the spotlight. Its author shows a trick that lets you use PiD not just with specific image models, but as an upscaler for any image at all.
The core idea: instead of just translating an image model's intermediate result back into an image of the same size, PiD draws the image directly at four times the width and height, in a few steps and with fine detail.
The problem with the classic decoder
Modern image generators like FLUX, Stable Diffusion and Qwen-Image don't work directly with pixels. They compute in a latent space, a heavily compressed internal representation of the image, a bit like a very compact sketch. At the end, that sketch has to be turned back into real pixels. That job usually falls to the VAE decoder (VAE stands for variational autoencoder, an encoder-decoder pair that compresses and restores images).
According to the paper, this decoder is tuned for reconstruction. It's meant to reverse what the encoder compressed as precisely as possible, without inventing extra detail. It also gets expensive at megapixel resolutions. So if you want large images, you've typically had to add a second step: decode the image at low resolution, then enlarge it with a separate super-resolution model (an AI upscaler). That costs time, and any errors or missing details from the first step get passed on to the second.
How PiD works
PiD merges both steps into one. Decoding is treated as conditional pixel diffusion: a diffusion model working directly in high-resolution pixel space generates the target image from noise, using the latent as a template. The latent provides the rough layout and content of the image. A pretrained pixel model, which alphaXiv's overview describes as a PixelDiT-style text-to-image prior, supplies the fine appearance: skin, hair, leaves and textures.
Two technical tricks make this fast and robust:
- Sigma-aware adapter: A small add-on module feeds the latent into the pixel model. During training, PiD sees latents with varying amounts of noise and learns how far it can trust them. A "gate" (a kind of dial) turns down the latent's influence the noisier it is. This lets the actual image generator stop earlier: it hands over a not-quite-finished latent, and PiD turns it into the final image.
- DMD2 distillation: This technique transfers a large, slow model into a faster "student" version. As a result, the decoder needs just four steps, and classifier-free guidance (a method that ties the image more closely to the prompt and normally doubles the compute) is baked right in.
The paper says there's an optimal point for stopping early. With FLUX.1 [dev] at 28 denoising steps, stopping three to five steps before the end gave the best image quality. Stop too early and the image content is still unclear. Run all the steps and you waste time while leaving PiD less room to add detail.
Numbers from the paper
NVIDIA reports the following measurements:
| Measurement | Value |
|---|---|
| 512×512 latent → 2048×2048 pixels on RTX 5090 | under 1 second, 13 GB peak memory |
| Same task on a GB200 GPU | approx. 210 ms |
| PiD (FLUX.1 stopped at step 24/28), 2048×2048, GB200 | 211.2 ms |
| VAE decoder + SeedVR2-3B upscaler, same task | 1237.5 ms |
| NIQE quality score on FLUX.1 (lower = better) | PiD 3.50 vs. 4.04 best competitor |
That works out to roughly 5.9x lower decoder latency. The alphaXiv overview points out that this covers decoding time only, not the entire image generation. On quality, PiD leads on several metrics, but not all of them. The tests used generated latents, not reconstructions of arbitrary real photos.
An ablation study (where parts of the system are removed for testing) showed that without the pretrained pixel prior, quality drops sharply: NIQE rises from 5.43 to 7.79. Removing the gate has a smaller effect (NIQE 5.84), and on some individual metrics the gate-free variant is even marginally better. So the gate helps, but according to alphaXiv it doesn't explain the overall gain on its own.
The community trick: PiD for any image
Officially, PiD decoders only exist for specific latent spaces: FLUX.1, FLUX.2, Qwen-Image (first version), SD3 and SDXL, plus the image encoders DINOv2 and SigLIP. The latter are used as semantic latents in newer so-called RAE models. A model with a different VAE therefore has no matching PiD decoder.
The Reddit author gets around this: any image, whether it's a render from another model, a photo or a video frame, is first encoded with the FLUX.2 VAE. The resulting latent goes to PiD's FLUX.2 decoder, which draws an image with four times the edge length in four steps, optionally guided by an image description. That turns PiD into a universal upscaler. The author provides ready-made workflows for ComfyUI (a popular node-based interface for image AI), both locally and in ComfyUI Cloud.
One important point: PiD is not a classic sharpener. It doesn't polish the existing pixels. It draws a new, larger image that matches the original. That looks far more detailed than simple bicubic upscaling, but it also means details are being made up.
What you need to try it
The weights are on Hugging Face (nvidia/PiD). The easiest way in is through ComfyUI with the community workflow. As a reference point, the paper cites 13 GB of peak VRAM (video memory) for a 2048×2048 output on an RTX 5090. An NVIDIA card with 16 GB of VRAM should therefore be enough for that size. With 12 GB or less, things get tight, but there are no reliable figures for that. For the upscaling trick, you'll also need the FLUX.2 VAE. Make sure you have enough disk space for the decoder files and the VAE; the sources don't give exact sizes. Whether PiD runs usefully on Apple Silicon Macs or on CPU alone isn't documented. If you don't have a suitable graphics card, you can use the workflow in ComfyUI Cloud.
The bigger picture
PiD tackles a real bottleneck in today's image generators: the final step from latent to pixel image. NVIDIA combining decoding and upscaling into a single fast model, and releasing the weights, is good news for anyone running FLUX, SDXL or Qwen-Image locally. Large images become faster to produce, with no second upscaler needed. The community trick via the FLUX.2 VAE extends that to any image.
You should know the limits, though. PiD generates details. The results look sharper, but they're not a true recovery of lost information, and text in images is explicitly not guaranteed to come out correctly. That makes it unsuitable for photos where accuracy matters, such as documents or evidence images. The speed and quality figures come from NVIDIA itself and refer to generated images. Independent comparisons with established upscalers are still pending, as are tests of how well the FLUX.2 VAE detour works on real photos.
Sources
- Reddit r/StableDiffusion reddit.com
- alphaxiv.org alphaxiv.org
Try it yourself: NVIDIA PiD (Pixel Diffusion Decoder)
Explained step by step, no prior knowledge needed – on your own computer or free in your browser.
What can I do with it?
PiD isn't a standalone image generator. It's a decoder. In models like FLUX.1, FLUX.2, SD3, SDXL or Qwen-Image (first version), it replaces the final step where the internal latent becomes the finished image. It draws that image directly at four times the width and height (up to eight times, according to the paper) in just four steps. With the community trick of first encoding any image with the FLUX.2 VAE, PiD becomes an upscaler for every photo, render and video frame. The results show far more fine detail in skin, hair, foliage and textures than simple enlargement. NVIDIA says PiD is about six times faster than combining a VAE with the SeedVR2-3B upscaler, with better image quality. The catch: PiD invents matching details rather than exactly recovering lost information. Text, faces of real people or documents can end up altered as a result.
Who is it worth it for?
Worth it if you generate images locally with ComfyUI and models like FLUX, SDXL or Qwen-Image and want large, detailed output. Also good for hobby photographers and designers who want to enlarge small images attractively. Less suitable when the enlarged image has to match the original exactly, such as documents, screenshots with text or evidence photos. Without an NVIDIA graphics card with enough memory, your only option is the cloud version.
Ideas: What you can do with it
Generate an image with FLUX or SDXL and let PiD decode it instead of the regular VAE. You get an image with four times the edge length and fine detail straight away, ready for printing or wallpapers, with no separate upscaler.
Encode small digital photos from the early days of phone cameras with the FLUX.2 VAE and enlarge them 4x with PiD. The result looks much more detailed. Keep in mind that details are invented, so this is better for decorative use than for documentation.
Run low-resolution textures like wood, stone or fabric through PiD and upscale them with a matching description. That gets you high-resolution variants for 3D scenes or mods in no time.
Export individual frames from your own videos as stills and enlarge them, for a thumbnail or a print, for example. PiD isn't built for whole videos, since every frame is redrawn individually.
In existing ComfyUI workflows, replace the VAE-plus-separate-upscaler step with PiD. According to NVIDIA, that cuts decoding time by about a factor of six for 2048×2048 output.
What your computer needs
NVIDIA ships a separate PiD decoder per latent space: FLUX.1, FLUX.2, Qwen-Image (first version), SD3, SDXL, plus DINOv2 and SigLIP. Pick the decoder that matches your image model's VAE. For upscaling arbitrary images, the FLUX.2 decoder together with the FLUX.2 VAE is the right choice. The sources don't mention quantized variants. According to the paper, about 13 GB of VRAM is enough for 2048×2048, so a 16 GB card is a sensible minimum.
- According to NVIDIA's paper, peak memory is 13 GB of VRAM to decode a 512×512 latent to 2048×2048 pixels (measured on an RTX 5090), taking under a second.
- An NVIDIA graphics card with at least 16 GB of VRAM is a realistic minimum. There are no official figures for 12 GB or less.
- Larger outputs, such as 8x enlargement, will likely need considerably more memory. The sources don't give concrete numbers.
- On data center GPUs like the GB200, decoding takes only about 210 milliseconds, according to the paper.
- Whether PiD runs usefully on Apple Silicon or on CPU alone isn't documented. As an alternative, there's the community workflow in ComfyUI Cloud.
How do I find out what my computer has?
- Windows: Press Ctrl + Shift + Esc → Task Manager → Performance tab → click GPU on the left. At the bottom you'll see Dedicated GPU memory (that's your VRAM). You'll find system memory under Memory.
- Mac: Apple menu at the top left → About This Mac. You'll see your chip (e.g. M2) and memory (e.g. 16 GB). On Apple chips, graphics and processor share this memory.
- Linux: Type
nvidia-smiin the terminal (shows your graphics card and its memory) andfree -hfor system memory.
Guide
This model runs with its developers' own program (bytetriper/RAE on GitHub). The commands below come from the Model Card and the developers' instructions.
Windows 10 / 11
Windows 10 / 11: not officially tested. The source doesn't mention Windows. With an NVIDIA card, it should generally work. WSL2 (Linux running inside Windows) is more reliable, though: there, just follow the Linux instructions.
- Check the prerequisites. Install Python (3.10 is a safe choice), Git and the latest NVIDIA driver. Open PowerShell and use these commands to check that everything is in place. Plan for about 80 GB of free disk space for the PiD checkpoints and FLUX.1-dev.
python --version git --version nvidia-smi - Download the code. Grab the PiD code from GitHub and change into the folder.
git clone https://github.com/nv-tlabs/PiD.git cd PiD - Create a virtual environment. The virtual environment keeps the packages separate from everything else. If PowerShell says scripts are blocked, run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned first. Afterwards, (.venv) appears at the start of the line.
python -m venv .venv .venv\Scripts\Activate.ps1 - Install PyTorch and dependencies. This installs PyTorch with CUDA support. We didn't have PiD's exact package list. If there's a requirements.txt in the folder, install it with the second line.
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu129 pip install -r requirements.txt - Log in to Hugging Face. While logged in, accept the license on the black-forest-labs/FLUX.1-dev Hugging Face page. Then log in with your token.
pip install huggingface_hub hf auth login - Download the PiD weights. This downloads the checkpoints folder (about 50 GB) into the PiD folder.
hf download nvidia/PiD --local-dir . --include "checkpoints/*" - Generate your first image. In PowerShell, you set PYTHONPATH on its own line. The ` character at the end of a line means the command continues on the next line. FLUX.1-dev is downloaded on the first run.
$env:PYTHONPATH="." python -m pid._src.inference.from_ldm --backbone flux ` --prompt "A photorealistic half-body portrait of a brown tabby cat with bold stripes sitting attentively on a rustic wooden kitchen table, soft morning light streaming sideways through a large window, fine fur detail and stripe patterns sharply visible, intense amber-green eyes in razor-sharp focus, warm farmhouse kitchen softly out of focus, cinematic shallow depth of field, ultra-detailed fur texture, photorealistic" ` --ldm_inference_steps 28 --save_xt_steps 24 ` --output_dir ./results/official_demo/flux ` --pid_inference_steps 4 - View the result. Open the results\official_demo\flux folder in File Explorer. That's where the images are.
dir results\official_demo\flux - Result. The folder results/official_demo/flux contains a high-resolution image of a tabby cat on a kitchen table, decoded by PiD.
macOS
macOS: not officially tested. The source doesn't mention macOS. NVIDIA's code is built for NVIDIA graphics cards (CUDA), which Macs don't have. So there's a good chance it won't run on a Mac at all. A rented cloud machine with an NVIDIA card is a good alternative.
- Check the prerequisites. Install the Xcode command line tools (they include Git) and Python 3.10 or newer. Plan for about 80 GB of free disk space. Be prepared for this to fail without an NVIDIA card.
xcode-select --install python3 --version - Download the code. Grab the PiD code and change into the folder.
git clone https://github.com/nv-tlabs/PiD.git cd PiD - Create a virtual environment. Create an isolated Python environment and activate it. Afterwards, (.venv) appears at the start of the line.
python3 -m venv .venv source .venv/bin/activate - Install PyTorch and dependencies. There's no CUDA on the Mac, so you install regular PyTorch. If there's a requirements.txt in the folder, install it with the second line.
pip install torch torchvision torchaudio pip install -r requirements.txt - Log in to Hugging Face. While logged in, accept the license on the black-forest-labs/FLUX.1-dev Hugging Face page. Then log in with your token.
pip install huggingface_hub hf auth login - Download the PiD weights. This downloads the checkpoints folder (about 50 GB) into the PiD folder.
hf download nvidia/PiD --local-dir . --include "checkpoints/*" - Generate your first image. This is the example command from the Model Card. If it aborts with a CUDA error, the code doesn't run on your Mac.
PYTHONPATH=. python -m pid._src.inference.from_ldm --backbone flux \ --prompt "A photorealistic half-body portrait of a brown tabby cat with bold stripes sitting attentively on a rustic wooden kitchen table, soft morning light streaming sideways through a large window, fine fur detail and stripe patterns sharply visible, intense amber-green eyes in razor-sharp focus, warm farmhouse kitchen softly out of focus, cinematic shallow depth of field, ultra-detailed fur texture, photorealistic" \ --ldm_inference_steps 28 --save_xt_steps 24 \ --output_dir ./results/official_demo/flux \ --pid_inference_steps 4 - Result. The folder results/official_demo/flux contains a high-resolution image of a tabby cat on a kitchen table, decoded by PiD.
Linux
Linux: supported. Linux with an NVIDIA graphics card is the standard environment for NVIDIA's PiD code. Note: the NSCLv1 license only allows non-commercial use, meaning research or experimentation.
- Check the prerequisites. You need Python (3.10 is a safe choice), Git and an NVIDIA graphics card with a current driver. Use nvidia-smi to check that the card is detected. Also plan for plenty of free disk space: all PiD checkpoints together come to about 50 GB, and FLUX.1-dev needs another 30 GB or more. Checkpoints are saved model weights.
python3 --version git --version nvidia-smi - Download the code. Grab the PiD code from GitHub and change into the folder. If it worked, you'll have a PiD folder with a pid subfolder.
git clone https://github.com/nv-tlabs/PiD.git cd PiD - Create a virtual environment. A virtual environment (venv) is an isolated space for Python packages, so nothing clashes with the rest of your system. Once activated, (.venv) appears at the start of the prompt.
python3 -m venv .venv source .venv/bin/activate - Install PyTorch and dependencies. PyTorch is the foundation the model runs on. This command installs the CUDA 12.9 build, as described in the related RAE project. We didn't have PiD's exact package list. If there's a requirements.txt file in the folder, install it with the second line. To be safe, read the README in the PiD folder too.
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu129 pip install -r requirements.txt - Log in to Hugging Face. The demo loads FLUX.1-dev. For that model, you need to be logged in and accept the license on its Hugging Face page (black-forest-labs/FLUX.1-dev). Then log in from the terminal with an access token, which you create in your Hugging Face account under Settings, Access Tokens.
pip install huggingface_hub hf auth login - Download the PiD weights. This command downloads the checkpoints folder directly into the PiD folder, skipping the README and the cover image. It's about 50 GB, so allow some time. You're done when the checkpoints folder contains ae.safetensors, among other files.
hf download nvidia/PiD --local-dir . --include "checkpoints/*" - Generate your first image. This command comes from the Model Card. FLUX first generates a cat image in the compressed intermediate format (latent). PiD then decodes it into a high-resolution image in just 4 steps. On the first run, FLUX.1-dev is downloaded automatically, which takes a while.
PYTHONPATH=. python -m pid._src.inference.from_ldm --backbone flux \ --prompt "A photorealistic half-body portrait of a brown tabby cat with bold stripes sitting attentively on a rustic wooden kitchen table, soft morning light streaming sideways through a large window, fine fur detail and stripe patterns sharply visible, intense amber-green eyes in razor-sharp focus, warm farmhouse kitchen softly out of focus, cinematic shallow depth of field, ultra-detailed fur texture, photorealistic" \ --ldm_inference_steps 28 --save_xt_steps 24 \ --output_dir ./results/official_demo/flux \ --pid_inference_steps 4 - View the result. Open the results/official_demo/flux folder. That's where the generated images are. If the program reports a memory error (out of memory), you don't have enough VRAM for FLUX.1-dev.
ls results/official_demo/flux - Result. The folder results/official_demo/flux contains a high-resolution image of a tabby cat on a kitchen table, decoded by PiD.
What to do after your first result
- Replace the text after --prompt with your own image description (ideally in English).
- With --pid_ckpt_type 2kto4k_v1pt5, PiD decodes images from FLUX, FLUX.2 and Qwen-Image up to 4K resolution. For SD3 and SDXL, use --pid_ckpt_type 2kto4k.
- Use --backbone to choose a different image model, for example zimage-turbo, flux2-klein-4b, qwenimage, sd3 or sdxl. Smaller models like flux2-klein-4b need less VRAM than FLUX.1-dev.
- The source doesn't mention a web interface. Everything is controlled from the command line.
Try it
Decode an SDXL image you generated yourself (1024×1024) directly to 4096×4096 with the SDXL PiD decoder and compare it with the regular VAE result.Encode an old, small vacation photo (512×512) with the FLUX.2 VAE and upscale it 4x with the description "forest path in autumn, sunlight through the leaves, detailed foliage".Enlarge a portrait render from any model via the FLUX.2 route with the description "close-up of a woman, natural skin texture, fine hair" and pay attention to skin and hair detail.With FLUX.1 [dev] at 28 steps, stop generation at step 24 and pass the intermediate latent to PiD, then compare with stopping at step 20 and at step 28.Upscale an image with text, such as a shop sign, and check whether the letters stay correct, to test the limits of the generative decoder.Tips for better results
- When upscaling, add a short, accurate image description. It steers what kind of details PiD adds, such as foliage, fabric or skin.
- If you use PiD directly during generation, stop the base generator three to five steps before the end. In the paper, that gave the best quality with FLUX.1 [dev] and saves time.
- Always compare the result with the original, especially for faces and text, because PiD invents details instead of restoring them.
- Use an input of about 512×512 as your starting point. That's what the paper's measurements are based on (2048×2048 output, 13 GB VRAM).
If something goes wrong
PiD only works with the matching latent space. Sending a latent from a model with a different VAE to the wrong decoder produces garbage. For other models, encode the image with the FLUX.2 VAE first.
For the upscaling trick, you need to download the FLUX.2 VAE separately and load it in the workflow.
The decoder is distilled to four steps. More steps don't automatically mean better results, so stick with the workflow's default values to begin with.
If ComfyUI doesn't recognize the PiD nodes, update ComfyUI and the required custom nodes. The community workflow usually flags missing nodes when you load it.
Check the license terms on the nvidia/PiD Hugging Face page before any commercial use. The sources don't mention the license.
If you hit memory errors, reduce the input size first: the output is four times as wide and tall and needs correspondingly more VRAM.
“out of memory,” a crash, or your computer freezes
The model is too big for your VRAM (video memory). Pick a smaller variant from the table above and close other programs, especially games and browsers with lots of tabs.
It runs, but extremely slowly
Your CPU is probably doing the work instead of your graphics card. In Ollama, the PROCESSOR column of ollama ps shows how much runs on the GPU. If it says "100% CPU", update your graphics driver or pick a smaller variant that fits entirely into VRAM (video memory).
“ollama” or “python” is not found
Close the terminal window and open it again so it recognizes the newly installed program. For Python on Windows: reinstall it and check Add python.exe to PATH.
Windows: "Running scripts is disabled on this system"
Run the script with exactly the command above: powershell -ExecutionPolicy Bypass -File try_windows.ps1. This allows execution for this one time only.
Error message when loading the model (version, "trust_remote_code", unknown architecture)
Brand-new models often need the latest software version. With Ollama: update Ollama. With Python: pip install -U transformers in the activated environment. The Model Card usually lists the exact version.
Tokenstrom generates the scripts and notebook automatically. Please take a quick look before running them – they're plain, readable text. Mind the model's license: custom license (see Model Card).
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.