Project Maya runs the 321B-parameter GLM-5.3-Flash on a single GPU

The free, open-source tool Project Maya spreads huge AI models across your GPU, system RAM and SSD. That lets GLM-5.3-Flash, with 321 billion parameters, run on a single RTX 3090. Users report 26 to 35 tokens per second, as long as there's enough RAM.

Tools Nov 17, 2026 8 min read Source: Gigazine (JP)
Illustrative image, AI-generated

What it's about

Huge language models with several hundred billion parameters normally need a server with hundreds of gigabytes of VRAM (video memory). That's barely feasible at home. Project Maya aims to change that. The free, open-source program comes from developer mw00 and is available on GitHub. It combines VRAM, regular system RAM and SSD storage so that very large models can run on a single graphics card.

According to the project page, supported models include GLM-5.3 (744B parameters), GLM-5.3-Flash (321B), DeepSeek-V4.1-Flash (748B) and MiniMax-M3.1-Flash (428B). Maya ships with a web interface and offers an OpenAI- and Anthropic-compatible API, so you can plug in existing chat apps and coding agents directly. It handles up to eight requests at once, for example for multiple users or coding agents. Gigazine makes clear that this doesn't mean eight times the speed, just that several requests are processed in parallel.

Gigazine's editors introduced the tool, a user on note.com tested it in depth, and xenospectrum covered the latest major update.

The trick: experts on three memory tiers

GLM-5.3-Flash from zai-org is released under the MIT license and is a Mixture-of-Experts (MoE) model. It consists of many specialized subnetworks, the "experts," of which only a few are needed per token. According to the note.com tester, about 18B of the 321B parameters are active per token. The model writes text, codes and can analyze images.

Maya takes advantage of this design. Frequently used experts sit in the GPU's fast VRAM, the ones needed next in system RAM, and the rest on an NVMe SSD. They get swapped depending on the conversation. Think of a chef who keeps the most important ingredients on the counter, others in the fridge and the rest in the basement. According to the README, the engine grew out of the predecessor project Strata and was rewritten for GLM-5.3-Flash.

The second building block is quantization: the model is shrunk by storing its numbers with fewer bits. The variants are on Hugging Face:

Variant Size Accuracy vs. FP8 original*
Maya-XS 73.4 GB 92.4%
Maya-S (default) 96.5 GB 97.9%
Maya-S24 (for cards up to 24 GB) 94.7 GB 97.7%
Maya-M 116 GB 97.9%
Maya-L 156.3 GB 99.2%

*According to the developer, measured with zero-shot tasks, i.e. questions without prior examples. Gigazine points out that these are relative scores on specific test tasks, not a general accuracy rate.

Maya-S24 keeps the experts at 2 bits, as in Maya-S. But the parts every token passes through (attention and shared experts) are reduced from 6 to 4 bits. That saves about 1.5 GB on the GPU, leaving room for more experts.

Speed: what users are measuring

Gigazine cites two user measurements:

Setup GPU RAM Variant Speed
A 1× RTX 3090 (24 GB) 128 GB Maya-S24 26.0 tokens/s
B 2× RTX 5090 (32 GB) + 1× RTX 4090 (48 GB) 188 GB Maya-S 148.6 tokens/s

The conditions differed, so the number of GPUs alone doesn't explain the gap.

The note.com tester used an RTX 5090 (32 GB), a Ryzen AI MAX+ 395, about 94 GB of RAM and Linux with CUDA 13.3. The 5090 requires at least CUDA 12.8. Setup took about 45 minutes, roughly 40 of them for downloading the model at around 37 MB/s. The model takes up about 90 GB. It was up and running after 36 seconds: 23.7 GB of experts in VRAM, 64.5 GB in RAM, nothing on the SSD. The built-in benchmark measured:

  • Generating output (decode): 34.9 tokens/s on average
  • Reading input (prefill), 2,048 tokens: 261 tokens/s
  • Prefill, 8,192 tokens: 890 tokens/s

For comparison, the developer reports 19.1 tokens/s decode on a V100 (32 GB) with 64 GB of RAM. One detail from the logs stands out. The tester's GPU was connected via PCIe x4 only, so the data link between RAM and GPU was narrow. Maya therefore didn't bother sending the experts from RAM to the GPU and had the CPU compute them instead: 0.09 milliseconds on the CPU versus 1.04 milliseconds for the trip over PCIe. The experts on the GPU were hit 56 to 64% of the time; the CPU handled the rest using AVX-512, a fast instruction set on modern processors.

On his own coding benchmark, modeled on SWE-bench, GLM-5.3-Flash solved 54 of 60 basic tasks (90%) and 12 of 16 hard tasks (75%). The tester himself rates that as roughly on par with his previous Strata setup. The test conditions weren't identical, and with only 16 hard tasks, each one shifts the result by about 6 points.

Update 1.1.28: faster, but not everywhere

According to xenospectrum, version v1.1.28 was released on October 26, 2026. It changes how data moves from RAM to the GPU on Linux with a single NVIDIA card. The GPU now accesses system memory directly, skipping an intermediate copy. This also benefits DSpark, a speculative decoding feature that guesses several tokens ahead and then verifies them in a single step. On a PC with an RTX 3090, Ryzen 5 5600X and 125 GB of DDR4, the developer says DeepSeek with Maya-XS went from 12.9 to 27.8 tokens/s. After 128,000 tokens of input, it rose from 14.0 to 23.2 tokens/s, and after 256,000 tokens from 15.9 to 26.1. Reading long inputs, however, barely got faster.

Also new: context lengths of 512,000 and one million tokens for DeepSeek. That was tested on nine GPUs, though: reading in just over a million tokens took 23.8 minutes. Each context position costs about 1.1 KB of VRAM per GPU, space that's then unavailable for experts. On top of that, DeepSeek needs around 340 GB of memory in the Maya-S variant. One user even reported about 19% lower speed on a virtual L40S card (48 GB), dropping from 42.4 to 34.4 tokens/s. According to xenospectrum, the cause is still unclear. The update also adds optional features for AMD cards (RDNA), Windows (launched via START-MAYA.bat) and multiple GPUs.

What you need to try it

  • Memory: What matters is the combined total of VRAM and RAM. The documentation recommends 88 GB for Maya-XS, 128 GB for Maya-S and 192 GB for Maya-L. The official minimum is a supported GPU, 32 GB of RAM and an SSD. But Gigazine failed with 64 GB of RAM and an 8 GB card, getting a "not enough VRAM" error.
  • GPU: In practice, 24 GB (RTX 3090/4090) is a sensible starting point. The installer recommends Maya-S24 for cards up to 24 GB and Maya-S for 32 GB cards.
  • Storage: about 75 to 160 GB for the model, ideally on a fast NVMe SSD.
  • System: Linux with NVIDIA has the best support. Windows and AMD are partially supported. The sources don't mention Mac.
  • Pitfalls, according to the tester: Without --model Maya-S24, the installer picks Maya-S. Without --download-model, --yes on its own downloads nothing. The default port 8080 may already be in use. The context window has to be set large enough for long responses: the tester's output limit only fit at 65,536 tokens instead of 32,768. With GLM-5.3-Flash, the KV cache, which stores the text so far, only grows from 0.62 to 1.00 GB as a result. The model makers recommend a temperature of 1.0 and top_p of 0.95.

Our take

Project Maya shows how far clever offloading can push MoE models onto regular hardware. A 321-billion-parameter model responding at more than 25 tokens per second on a used RTX 3090 would have seemed unrealistic not long ago. If you have a well-equipped machine, it's a serious alternative to cloud services, especially for coding agents you want to run locally.

There are caveats, though. "One GPU" doesn't mean "modest hardware": without roughly 96 to 128 GB of RAM, you won't get far, and the large DeepSeek variant needs several times that. Whether it runs fast depends heavily on your hardware: PCIe connection, CPU performance and RAM capacity all play a role, as the conflicting measurements around the update show. All figures come from the developer and individual users; according to xenospectrum, there's no independent verification yet. The project is also moving very fast, with many releases in just a few weeks. If you use it, expect frequent changes.

Sources

#project maya#glm-5.3-flash#local ai#moe#offloading#open source
Read the original source

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.