A 177-billion-parameter model now runs on a 16 GB MacBook Air

The Ternii team has compressed the 177-billion-parameter MoE model Qwen3.8-Flash-Next so aggressively that it runs on a 16 GB MacBook Air M5 at around 12 tokens per second. The trick: the experts live on the SSD and get loaded just before they're needed. It's experimental and requires a custom build of llama.cpp.

Model releases Try it yourself Nov 16, 2026 6 min read Source: Hugging Face Posts
Illustrative image, AI-generated

What's going on

The Ternii team has published a heavily compressed version of Qwen3.8-Flash-Next on Hugging Face. It's a 177-billion-parameter language model from Alibaba's Qwen family. According to Ternii, this version runs on a MacBook Air M5 with just 16 GB of memory and generates about 12 tokens per second there, which is roughly a few words per second. Until now, models this size needed a machine with a huge amount of VRAM or RAM.

The file is called Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF and weighs in at 89.8 GB. If you want to try it, there's a one-liner for the terminal. It downloads everything and then launches a local chat in your browser:

curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen38-16gb.sh | bash

The project lives on GitHub at github.com/ternii-ai/tritq-vllm. Ternii explicitly calls it experimental. So far it has only been measured on a single Mac.

How a 90 GB model fits into 16 GB

Three ideas make it work.

First, the Mixture-of-Experts (MoE) architecture. The model consists of many sub-networks, the experts. For each token, meaning each word fragment, a router picks only a few of them. Think of a big office where only a handful of specialists work on any given request while everyone else takes a break. That's why the whole model never has to sit in memory at once. Ternii doesn't say how many parameters are active per token.

Second, extreme quantization. This shrinks the model by storing its numbers at lower precision. Ternii works in trits instead of bits. A trit is a digit with three possible states instead of two. The experts are stored at 2 trits per weight, the remaining layers at 3. The "TR2.31" in the file name suggests an average of about 2.31 trits per weight, though Ternii doesn't spell that out.

Third, streaming from the SSD. The experts stay on disk. RAM only holds a 6,750 MiB cache of the experts currently in heavy use. Ternii built its own llama.cpp version for this (llama.cpp-tr r3). llama.cpp is the popular open-source program for running language models locally. This version reads the next layer's experts from the SSD while the GPU is still busy with the current one. A small learned predictor decides which experts to fetch in advance. According to Ternii, it doesn't change the output at all; it only determines what gets prefetched.

There's also a trade-off: each token uses only 8 of the intended 10 experts. That saves compute and loading.

The numbers

Ternii measured on a 16 GB MacBook Air M5. Across six runs, including one independent retest, the model reached 11.9 to 12.6 tokens per second when generating its answer (decode). The previous file with the previous build managed 3.6. For comparison: the smaller Qwen3.6-35B-A3B in its TR2.25-G version runs at about 35 tokens per second on the same Mac with the same build.

Ternii also published numbers on whether 8 experts instead of 10 hurt quality:

Benchmark 8 experts 10 experts (original)
GSM8K (math word problems) 96.5 95.0
HumanEval (coding) 95.1 95.7
MBPP (coding) 91.1 91.1
IFEval (instruction following) 66.0 65.8

Ternii says none of the differences are statistically significant. All figures come from the developers. There are no independent tests yet.

The catches

Ternii itself lists several limitations:

  • Prompt processing is slow. It manages about 75 tokens per second. A 2,000-token prompt takes around half a minute before the answer even starts. That makes long documents or large codebases a slog.
  • Memory is tight. The setup uses about 13 GB. Close other programs first.
  • It needs special software. It won't work without Ternii's modified llama.cpp. LM Studio and Ollama can't load the file.
  • There's only one measurement. It was tested on exactly one Mac. How other devices, slower SSDs or older chips behave is an open question.

What you need to try it

  • Computer: a Mac with Apple Silicon and at least 16 GB of unified memory. Only the MacBook Air M5 has been tested. It will probably run slower on older M-series chips, but there are no figures.
  • Storage: just under 90 GB free on the internal SSD, ideally more. Because the experts are read from the SSD constantly, a fast internal SSD matters. External drives are likely much slower.
  • RAM: about 13 GB goes to the model, so close browser tabs and other apps.
  • Windows/Linux/Colab: The published launch script is explicitly meant for the Mac. Ternii doesn't say anything about other platforms in this post. On a regular PC with a graphics card, you'd need far more memory without this SSD trick.
  • Download: just under 90 GB. Depending on your connection, that can take hours.

Our take

Technically, this is a remarkable step. A 177-billion-parameter model on a fanless laptop costing under €2,000, at a speed that's still usable: not long ago, you'd have needed a workstation with 128 GB of memory for that. Combining MoE, very coarse quantization and predictive loading from the SSD shows that for big models, raw memory isn't everything. What counts is how cleverly you load data on demand.

Still, it pays to keep a level head. All the numbers come from Ternii and were measured on a single device. The benchmarks compare 8 versus 10 experts. The post doesn't make clear whether the heavy quantization itself lowers quality compared with the uncompressed original model. Because prompt processing is slow, the setup is better suited to short questions than long documents. And if you value convenience, this will put you off: no Ollama, no LM Studio, just a shell script you pipe straight from the web. If you don't need the size, smaller models like the Qwen3.6-35B-A3B mentioned above are often the better choice, since it runs almost three times as fast. But for tinkerers and anyone curious how far local AI can be pushed on everyday hardware, it's well worth a look.

Sources

#qwen#moe#apple silicon#quantization#llama.cpp#local
Read the original source

Try it yourself: Qwen3.8-Flash-Next (Ternii TR2.31-G GGUF)

Explained step by step, no prior knowledge needed – on your own computer or free in your browser.

What can I do with it?

Qwen3.8-Flash-Next is a large 177-billion-parameter language model that, in this version, runs fully offline on a 16 GB Mac. You can chat with it, write and edit text, solve math and logic problems, and have it write code. According to Ternii's benchmarks, it's strong at this: 96.5 on GSM8K (math word problems) and 95.1 on HumanEval (coding). It does less well at precisely following formal instructions (IFEval 66.0). Answers come out at about 12 tokens per second, a comfortable reading pace. Reading in long inputs, on the other hand, is slow at about 75 tokens per second, so long documents mean waiting. Compared with small local models, you get much more knowledge and reasoning power, but you pay for it in speed, roughly 90 GB of storage and an experimental setup.

Who is it worth it for?

A good fit for owners of Apple Silicon Macs with at least 16 GB who want to run a genuinely large model privately and offline and enjoy tinkering. Not a good fit if you need to process long documents quickly, rely on Ollama or LM Studio, or don't have 90 GB of free SSD space.

Ideas: What you can do with it

Private writing assistant

Paste in letters, job applications or emails and have them rephrased, shortened or adjusted in tone. Since everything runs locally, no personal data leaves your Mac.

Math tutoring

Enter word problems from school or university and have the solution explained step by step. With 96.5 on GSM8K, Ternii says this is one of the model's strengths.

Offline coding help

Have it write small scripts, explain error messages or refactor functions, even on a train without internet. Only paste the relevant code, because long inputs take time.

Ideas and planning

Have it draft travel itineraries, weekly meal plans or talk outlines, then refine them in conversation. Answers arrive at a comfortable reading pace.

Experiments with local AI

Compare the 177B model with smaller local models on the same questions. That way you can see for yourself what the extra size buys you day to day.

Size176.9 B parameters
Download106 GB (about 2.4 hr at 100 Mbit/s)
Licensecustom license (see Model Card)
TypeLanguage model (text chat)

What your computer needs

Graphics card (NVIDIA), fully in VRAM at least 118 GB VRAMservers only, with multiple data-center GPUs (e.g. 8× H100) – fastest; with less VRAM, see the next line
Graphics card + system memory (MoE offloading) from approx. 6 GB VRAM + 128 GB RAMdie Experten liegen im Arbeitsspeicher, nur der Kern auf der Grafikkarte (llama.cpp/LM Studio). Langsamer als komplett im VRAM, aber auf normalen PCs machbar
Mac (Apple Silicon M1–M5) from 192 GB unified memoryThe GPU and CPU share the memory – the model can use about 70% of it
CPU only (no graphics card) Possible; speed depends on the number of active expertsMoE-Modelle sind ohne Grafikkarte schneller, als ihre Größe vermuten lässt. Dafür mindestens 128 GB Arbeitsspeicher.
Storage approx. 108 GB freeModel download plus programs
Google Colab too big for Colabeven the largest Colab GPUs aren't enough

Values estimated from the model size – the Model Card may have more precise figures.

Which variant is right for me?

Many models come in several sizes or "quantizations." These are compressed versions: smaller and faster, but slightly less accurate. Rule of thumb: pick the largest variant whose memory requirement fits in your VRAM (video memory). The scripts below choose this for you automatically.

VariantDownloadNeeds approx.Compatible graphics cards
4-bit (Q4) Default 106 GB118 GBservers only, with multiple data-center GPUs (e.g. 8× H100)
8-bit (Q8) 189 GB210 GBservers only, with multiple data-center GPUs (e.g. 8× H100)
16-bit (Original) 354 GB390 GBservers only, with multiple data-center GPUs (e.g. 8× H100)

There's exactly one variant: Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF (89.8 GB, experts at 2 trits, other layers at 3 trits per weight), designed for 16 GB Macs and the qwen38-16gb.sh launch script. If you need speed more than size, go for Ternii's Qwen3.6-35B-A3B TR2.25-G, which reaches about 35 tokens per second on the same Mac.

  • Only tested on a MacBook Air M5 with 16 GB of unified memory. Other Macs haven't been measured.
  • The model uses about 13 GB of RAM, so close other apps and browser tabs first.
  • The model file is 89.8 GB and must sit on the SSD. The experts are read from there continuously, so a fast internal SSD matters.
  • A 6,750 MiB expert cache lives in RAM; everything else is loaded from the SSD as needed.
  • Prompt processing only manages about 75 tokens per second, so long inputs noticeably delay the answer.

Groß (176,9 B parameters, MoE): running entirely on the GPU takes a lot of VRAM (video memory). With MoE offloading, a regular graphics card plus plenty of RAM is enough (see table). Colab Free isn't enough.

How do I find out what my computer has?
  • Windows: Press Ctrl + Shift + Esc → Task Manager → Performance tab → click GPU on the left. At the bottom you'll see Dedicated GPU memory (that's your VRAM). You'll find system memory under Memory.
  • Mac: Apple menu at the top left → About This Mac. You'll see your chip (e.g. M2) and memory (e.g. 16 GB). On Apple chips, graphics and processor share this memory.
  • Linux: Type nvidia-smi in the terminal (shows your graphics card and its memory) and free -h for system memory.

Guide

Google Colab (in your browser, no installation)

Google Colab is a free service from Google. It gives you a computer with a graphics card in your browser for a few hours. All you need is a Google account. For this model: too big for Colab – even the largest Colab GPUs aren't enough.

  1. Download the notebook. Click the button and save the file.

    Download colab.ipynb

  2. Open Colab. Go to colab.research.google.com and sign in with your Google account.
  3. Upload the notebook. In the window that appears, click Upload on the left (or choose File → Upload notebook at the top) and select the file colab.ipynb.
  4. Turn on the GPU. In the top menu, go to Runtime → Change runtime type → under Hardware accelerator, select T4 GPU (with Colab Pro: L4 or A100) → Save.
  5. Start everything. Runtime → Run all (keyboard shortcut Ctrl + F9). If you see the warning “This notebook was not authored by Google,” click Run anyway.
  6. Wait. The cells run one after another. A green checkmark appears to the left of each cell when it's done. Installation and download take about 39 minutes here.
  7. Try it out. The result appears below the last cell. To test something else, change the text in the cell and run it again with Shift + Enter. The last cell starts a proper chat with an input field.
  8. Clean up. When you're done: Runtime → Disconnect and delete runtime. This saves your free quota.

Good to know: The free GPU is limited (usually a few hours per day). After about 90 minutes of inactivity, Colab disconnects, and you'll have to start everything over.

Windows 10 / 11

The easiest way is Ollama, a free program that launches AI models with a single command.

  1. Update your graphics driver (NVIDIA cards only): Install the NVIDIA app or the latest driver. For AMD cards, use the latest Adrenalin driver.
  2. Install Ollama. Download the file OllamaSetup.exe from ollama.com/download/windows, double-click it, and choose Install. Afterward, a small llama icon appears in the taskbar at the bottom right.
  3. Open a terminal window. Press the Windows key, type PowerShell and press Enter. A blue or black window opens.
  4. Start the model. Copy this command ("Copy" button), paste it into the window with a right-click, and press Enter:
    ollama run hf.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
    The first time, Ollama downloads the model (106 GB (about 2.4 hr at 100 Mbit/s)). After that, it's instant.
  5. Start typing. Once >>> Send a message appears, type your question and press Enter. Type /bye to end the chat.

Not enough memory? Pick a smaller variant from the table above, e.g. ollama run hf.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF.

Prefer automatic? A script that picks the right variant for you
  1. Download both files into the same folder: try_windows.ps1 demo.py
  2. Open the folder in File Explorer, right-click an empty spot and choose Open in Terminal.
  3. Paste this command and press Enter:
    powershell -ExecutionPolicy Bypass -File try_windows.ps1

If needed, the script installs Ollama via winget, measures your VRAM (video memory) and downloads the matching variant.

macOS

Macs with an Apple chip (M1 to M5) are ideal. Older Intel Macs are very slow.

  1. Install Ollama. Download it from ollama.com/download/mac, open the file, and drag Ollama into your Applications folder. Launch Ollama once – a llama appears in the menu bar at the top.
  2. Open Terminal. Press Cmd + Space, type Terminal, and press Enter.
  3. Start the model. Paste the command (Cmd + V) and press Enter:
    ollama run hf.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
    The first time, the model is downloaded (106 GB (about 2.4 hr at 100 Mbit/s)).
  4. Start typing. Type your question and press Enter. Quit with /bye.
Pick the right variant automatically

try_mac.sh

cd ~/Downloads
bash try_mac.sh

Linux

  1. Check your GPU. nvidia-smi should show your card. If it doesn't, install the driver on Ubuntu and restart:
    sudo ubuntu-drivers install
  2. Install Ollama:
    curl -fsSL https://ollama.com/install.sh | sh
  3. Start the model:
    ollama run hf.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
    Exit with /bye. Use ollama ps to check whether the model is running on the GPU.
  4. Or let it pick the right variant automatically:

    try_linux.sh

    bash try_linux.sh

Try it

Explain in simple terms why the sky is blue.
A train leaves at 9:40 a.m. at 120 km/h, and a second one follows on the same track at 10:10 a.m. at 160 km/h. When does it catch up? Work it out step by step.
Write a Python function that finds all duplicate files in a folder by their hash and explain the code.
Make this email more polite and shorter: 'I've been waiting three weeks for your reply, this is unacceptable.'
Compare the pros and cons of a heat pump versus a gas boiler for a 1975 single-family home and finish with a table.

Tips for better results

  • Keep your inputs as short and precise as possible. Prompt processing at about 75 tokens per second is the bottleneck, while generation runs smoothly at about 12 tokens per second.
  • For math and logic problems, explicitly ask for a step-by-step solution, since the model is especially strong on math benchmarks.
  • Keep the Mac plugged in and close memory-hungry apps like browsers with lots of tabs so the expert cache doesn't get pushed out.
  • If you need speed more than maximum knowledge, use Ternii's Qwen3.6-35B-A3B, which is almost three times as fast at about 35 tokens per second.

If something goes wrong

LM Studio and Ollama can't load the file. It only works with Ternii's modified llama.cpp (llama.cpp-tr r3), which the script brings along.

The one-liner runs a script straight from the internet. If you're cautious, open the URL in your browser first and read what it does.

Free up enough storage for the nearly 90 GB download beforehand, ideally with some headroom. Otherwise the download will fail.

If it slows down or crashes, memory is usually full. Close other programs, since the model needs about 13 GB.

The setup is explicitly experimental and has only been measured on one Mac. Speeds on other devices may differ significantly.

“out of memory,” a crash, or your computer freezes

The model is too big for your VRAM (video memory). Pick a smaller variant from the table above and close other programs, especially games and browsers with lots of tabs.

It runs, but extremely slowly

Your CPU is probably doing the work instead of your graphics card. In Ollama, the PROCESSOR column of ollama ps shows how much runs on the GPU. If it says "100% CPU", update your graphics driver or pick a smaller variant that fits entirely into VRAM (video memory).

“ollama” or “python” is not found

Close the terminal window and open it again so it recognizes the newly installed program. For Python on Windows: reinstall it and check Add python.exe to PATH.

Windows: "Running scripts is disabled on this system"

Run the script with exactly the command above: powershell -ExecutionPolicy Bypass -File try_windows.ps1. This allows execution for this one time only.

Error message when loading the model (version, "trust_remote_code", unknown architecture)

Brand-new models often need the latest software version. With Ollama: update Ollama. With Python: pip install -U transformers in the activated environment. The Model Card usually lists the exact version.

Tokenstrom generates the scripts and notebook automatically. Please take a quick look before running them – they're plain, readable text. Mind the model's license: custom license (see Model Card).

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.