Big coder MoE on an 8 GB GPU: Qwen3.8-Flash-Next with half its experts cut

What this is about

On the r/LocalLLaMA forum, a user shows how to run a heavily compressed coder version of Qwen3.8-Flash-Next on a graphics card with just 8 GB of VRAM, including 262K tokens of context. The underlying compression comes from the Deep Algorithms and Systems Lab (DASLab) at the Institute of Science and Technology Austria (ISTA). The user has published an additionally "vocabulary-pruned" GGUF variant of it on Hugging Face (Skyrion/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF-Pruned). GGUF is the file format used by llama.cpp and programs built on it, such as Ollama or LM Studio.

The model: removing experts instead of just compressing

Qwen3.8-Flash-Next is a Mixture-of-Experts (MoE) model: each layer has many independent sub-networks ("experts"), and a small router picks only a few of them per token. Here there are 512 experts per layer across 48 layers, with 10 active per token. The experts hold most of the parameters (the model's tunable values), but only a small fraction does any computing. The model has 176.9B parameters in total and takes up about 354 GB in BF16.

The coder variant combines two techniques: - Expert pruning: 50% of the experts are removed entirely (256 of 512 per layer). This shrinks the file and memory footprint and limits the router to fewer candidates. - Quantization: The remaining weights are stored at 3.5 bits per weight (bpw) instead of 16 bits, using the GSQ method.

That gives 58.4 GB, of which only 29.6 GB has to sit in memory: the second file (28.8 GB) is an n-gram lookup table that can also be served from disk. If you convert the removed experts back into the original parameter count, you get an effective 1.89 bits per parameter. Many people therefore wrongly took the model for a 1-bit quantization. In fact, no single weight sits at 1.89 bits; the kept weights have 3.5 bits.

How the experts were chosen

Selection used RCO (Riemannian-manifold-based optimization with exact budget constraints). Instead of ranking experts with a heuristic, RCO minimizes the KL divergence (a measure of how much the outputs differ) between the pruned and the full model on calibration data. Since GGUF stores only one uniform expert count per model, every layer has to keep the same number of experts. RCO enforces this budget exactly per layer, but optimizes all layers jointly.

The calibration data determines which abilities survive. A first search using only code and agent data removed the experts needed for images, and vision ability collapsed. The published version includes vision in the calibration data and keeps it. In exchange, other abilities are weakened. For general use, the authors recommend the unpruned GSQ-RCO versions.

Benchmarks (vendor figures)

All values were measured at "xhigh" reasoning effort.

According to the card, the October 17 update reduced endless loops and raised SWE-bench from 75.60 to 79.20 and LiveCodeBench from 86.28 to 87.43. Only the first shard changed; if you downloaded earlier, you need to re-download it. The authors explain the larger loss on SWE-bench by saying that long multi-step tasks accumulate errors and cope worse with a loss of capacity.

The vocabulary pruning

The user additionally removed tokens that are irrelevant for coding (non-English characters). Of 248,320 tokens, 130,020 remain (52.36%). The token_embd and output matrices shrink as a result, saving about 409 MB. That sounds small, but according to the user it can make the difference on systems with 8–16 GB of VRAM plus 32 GB of RAM. Automated checks confirm that the architecture, metadata, tokenizer order and all other 1,222 tensors remain byte-identical. This variant is not recommended for languages other than English. The IQ1_M variant's file is 57.993 GB.

What you need to try it

• Graphics card: about 8 GB of VRAM (NVIDIA or, if you run into problems, the Vulkan backend).
• Memory: around 24 GB of RAM; about 32 GB of total memory also works, but with a smaller context.
• Storage: about 58 GB, ideally on your fastest SSD, since the n-gram table is read from there.
• Mac/CPU/Colab: The source says nothing concrete about these; llama.cpp generally runs on macOS and Linux too.

The trick: most MoE experts stay in RAM, the smallest go into VRAM for more speed, and the n-gram table sits on the SSD. With selective MoE offloading via regex, the user gets about 70 T/s prompt processing and roughly 8 T/s generation, limited by his PCIe 3.0 connection. The --n-cpu-moe option is slower for the same purpose, he says. Because of mmap, the first run is slower, so run it twice for measurements. He also recommends the "qwen sharp" chat template, which he says reduces overthinking. The launch parameters are in the comments of the Reddit post.

Our take

The example shows how far a 177-billion-parameter model can be squeezed onto consumer hardware by combining expert pruning, low-bit quantization and memory offloading, with a loss of just over 4 points on SWE-bench. The caveats: it's an experimental model tuned for code, agents, vision and spatial reasoning, and general abilities suffer. The benchmarks come from the quant's creators, not an independent party. The speed figures (about 8 T/s generation) are rather slow for agent workflows and depend heavily on RAM, PCIe and SSD. One note from the user: CUDA has problems with 8 GB of VRAM, hence Vulkan. The model card text also mentions the engine "Strata," a third-party tool; there is no assessment of it. The license matches that of the base model, Qwen3.8-Flash-Next.