Your 4-bit AI isn't the model that was safety-tested. Here's what rounding changes

If you run models locally with llama.cpp, Ollama or LM Studio, you're almost always using a heavily compressed copy, not the original the vendor safety-tested. Studies on whether these copies are more dangerous contradict each other. A new paper from Vancouver explains why. We laid the studies side by side.

Analysis Nov 18, 2026 10 min read Original analysis by our editors
Illustrative image, AI-generated

What this is about

The big model providers safety-test their models before release, checking for example whether they refuse to give instructions for weapons or malware. What gets tested is the original at full precision (BF16 or FP16). But if you load a model through llama.cpp, Ollama or LM Studio, you almost always get a quantized copy: the model's numbers are stored more coarsely, usually with 4 bits instead of 16, sometimes with just 2 to 3. With llama.cpp v1 and Ubuntu's Inference Manager, that's becoming the default, and extremely compressed giant models like Ternii's 177-billion-parameter model push it even further.

That raises a simple question: does the copy behave like the tested original? At first glance, the research contradicts itself sharply. A paper just published at the University of British Columbia (UBC) offers a plausible explanation. We laid the key studies side by side and checked what they actually measure.

What happens when you round

Quantization rounds a model's weights, in small groups, to a handful of allowed values. At 4 bits that's 16 levels per group, plus a scale factor that stretches or squeezes the group. Every weight ends up slightly off its original value. The sum of these deviations is the rounding error.

The key point: this error isn't random noise but a very specific pattern. It depends on the method, meaning how the groups are formed, how the scales are chosen and whether calibration data is involved. Two 4-bit copies of the same model therefore carry two different errors. llama.cpp's K-quants such as Q4_K_M use nested blocks and store some tensors at higher precision than others. According to the llama.cpp documentation, Q4_K_M comes to 4.89 bits per weight on Llama-3.1-8B, Q2_K to 3.16 and Q8_0 to 8.50.

Three studies, three answers

Study Models Formats Key finding
Kadadekar, June 2026 (arXiv 2606.10154) 6 models, 1–7B parameters GGUF Q8_0 to Q2_K, AWQ, GPTQ Refusals drop by 12 to 68 points in 9 cases while quality stays the same
Lobo Vaz, Univ. of Porto, July 2026 7 models (Llama 3.x, Qwen3), up to 8B FP16, Q4_K_M, Q3_K_M, Q2_K_L ToxiGen score drops in 20 of 21 variants
Kaya, UBC, Nov. 2026 (arXiv 2611.22186) Qwen2.5-3B, Llama-3.2-3B, Qwen3-4B MLX 4-bit, AWQ, MXFP4, Q4_K_M and others Effects with opposite signs, Q4_K_M changes nothing

Look closer and the contradictions shrink. The 12 to 68 percentage points from the June study come almost entirely from AWQ and GPTQ copies. Those are two other 4-bit methods, used mainly on servers. The extreme value of minus 68 points involves GPTQ on Llama-3.2-1B. Among the GGUF levels, only Q3_K_S on Llama-3.2-1B and Q2_K on Qwen2.5-7B fall into this danger zone. The author therefore recommends Q5_K_M as a cautious lower bound, and Q4_K_M only with model-specific testing. The same six models and the same agreement between two AI judges (89.9 percent, κ = 0.873) show up in the non-peer-reviewed report TR134 on the site Chimeraforge. Our take: both very likely come out of the same research program. That report also shows how much the measurement method matters. For GPTQ on Llama-3.2-1B, a simple text filter (regex) gives minus 68.2 points, while an AI judge gives only minus 44.3.

The “20 of 21” in the Porto paper by Flávio Lobo Vaz doesn't measure whether dangerous requests get refused. It refers to the ToxiGen benchmark, which is about toxic statements about minorities. On average, the score drops by 0.022 at Q4, 0.066 at Q3 and 0.099 at Q2. Many of the 20 drops are tiny: Llama-3.1-8B goes from 0.845 to 0.844. There are real collapses too, for example Llama-3.2-1B falling from 0.594 to 0.389 (Q3_K_M) and Qwen3-4B from 0.838 to 0.607 (Q2_K_L). Across the 39 attacks in the red-teaming tool garak, robustness stayed practically unchanged on average (+0.007). Some attack types did become more successful, but others balance that out overall. In the end, this paper also recommends Q4_K_M as the best compromise.

The UBC study's trick: perturbations of the same size

Ege Kaya of UBC asks a fundamental question. A 4-bit copy is just one of countless possible changes to the weights at the same magnitude. If its behavior changes, is that down to the specific rounding error, or would any perturbation of that size do it? The study was preregistered, meaning hypotheses and analysis were fixed before any data was collected. It uses paired tests, blind human ratings and 8-bit copies as a null control.

Two scenarios were measured. In a prefill attack, you hand the model the first words of a compliant answer, here 8 tokens, and see whether it finishes the harmful answer. As a 4-bit copy (MLX default, 4.5 bits per weight), Qwen2.5-3B did so on 24.2 instead of 18.0 percent of 562 requests. Human ratings even showed plus 19.4 points. On 400 held-out jailbreaks from WildJailbreak, however, Llama-3.2-3B behaved more safely: it gave in only 34.8 instead of 45.0 percent of the time. Qwen3-4B didn't change meaningfully. The 8-bit copies did flip 3 to 10 percent of individual ratings, but none of the rates.

Then comes the real trick. Kaya mirrors the actual rounding error, shuffles it within the groups, replaces it with random noise of the same strength, and scales it to half or double size. Shuffled error and noise do almost nothing on average (+0.4 and −1.1 points, respectively). The mirrored error, however, reverses the effect, and when scaled, the behavior changes almost linearly. Even at half the error, Llama gives in 5.8 points less often, even though its language quality, measured as the likelihood of its own answers, barely moves (+0.005 nats).

The most practically relevant line concerns llama.cpp: Q4_K_M shifted neither model (Qwen −1.2, Llama +2.5 points, neither significant). An MLX copy with exactly the same bit count (5.0 bits, group size 32), on the other hand, shifted Qwen by 4.1 points. Kaya's conclusion:

“Because each copy carries its own rounding error, audits of different copies can contradict each other even for the same model.”

Kaya explicitly warns against generalizing the Q4_K_M result. The study has clear limits: three small models, answers capped at 128 tokens, a single human rater (the author), and the AI judge is itself a quantized Qwen. Code and labels are at github.com/egekaya1/qrefusal. According to the README, the whole thing runs on a Mac with 16 GB, but takes several days of compute.

When “unsafe” really means “broken”

A second cause of apparent safety collapses is more mundane. Many measurements count whether an answer looks like a refusal (“I can't help with that”). If a damaged model produces gibberish or endless loops, that looks like “no refusal” and lands in the stats as a safety problem.

A preregistered audit by Gaurav Suresh Hadavale (NeurIPS 2026) tested NF4, the 4-bit format of the bitsandbytes library, on Qwen3-4B and Llama-3.1-8B in 10 languages. In the low-resource languages, 91.9 percent of Llama's NF4 answers were unusable, and none of the 945 answers contained usable harmful help under blind review. With Qwen, 40.2 percent of answers already ran into loops in the BF16 original, and 50.5 percent in NF4. After filtering out the broken outputs, the audit found no language-specific safety effect. Language ability itself, however, took a measurable hit, from minus 9.4 points in English to minus 13.7 in low-resource languages. The summary doesn't give figures specifically for German.

The same pattern shows up elsewhere. In the Chimeraforge report TR134, Llama-3.2-1B at Q2_K scores 36.8 percent refusals by text filter but 97.7 percent by AI judge. The report suspects “garbled outputs” as the reason. A factorial study by Conscious Engines (arXiv 2606.29581) considers standard quantization “approximately safety-neutral”: seven of eight models stayed within about 1.6 points with AWQ INT4. Along the way, the authors found that the standard export for Qwen3-4B and Llama-3.2-3B produced completely broken output and had to be regenerated.

When it actually gets risky

The effects are clearest at 2 bits. The non-peer-reviewed Chimeraforge reports (author listed as “Research Team”) cite a mean attack success rate of 83.8 percent for multi-turn jailbreaks on Qwen2.5-1.5B at Q2_K, versus 55.5 at Q4_K_M. For Llama-3.2-1B, the figures are 61.5 versus 5.8 percent. In many-shot attacks, where the model is fed dozens of example dialogues, Qwen2.5-1.5B at Q2_K hit 99 percent with 128 examples. But the same model already reached 40 percent at Q8_0, and Llama-3.2-3B stayed at 3.8 percent at Q2_K. Agreement between text filter and AI judge in TR139 was weak (κ = 0.10).

Lab studies using other methods show clearer effects. Q-resafe (ICML 2025) measures an increase in attack success on Llama-2-7B-Chat from 0.3 percent in the original to between 16.9 (LLM-QAT) and 42.4 percent (AWQ) at INT4. Targeted retraining of about 1.2 GPU hours pushed it back down to 1.8 percent. Seoul National University shows with CAQ that naive rounding at W4A4 (activations also in just 4 bits) drops Llama-3.1-8B's SafetyBench score from 62.6 to 37.5. Their method holds 60.1. W4A4, however, is far more aggressive than typical GGUF files.

An attack is a separate matter: in “Mind the Gap” (ICML 2025), researchers at ETH Zurich show that you can build a model that looks harmless at full precision and only turns malicious as a GGUF copy, for example producing 88.7 percent more insecure code. That's not a random effect, it's sabotage.

What you can check yourself

  • Calibration is a black box. Imatrix files, statistics from calibration texts that llama.cpp uses to round important weights more precisely, are geared toward chat and tool use at Unsloth and bartowski. According to a GitHub issue, Unsloth's dataset isn't public. We found no indication that safety behavior is taken into account. Kaya tested Q4_K_M without an imatrix.
  • Compare against the original. Ask your copy and a Q8_0 or BF16 version the same questions at temperature 0. Use questions from a public test set, plus harmless questions from XSTest that shouldn't be refused. According to Kaya, no 8-bit copy changed the rates significantly.
  • Read the answers. Loops and gibberish are a quality problem, not a jailbreak.
  • Don't rely on perplexity or KL divergence. Both studies show behavior changes without measurable quality loss.
  • Only download from trusted sources.

The bottom line

Our take: for everyday use from Q4_K_M to Q8_0, there's so far no evidence of a systematic safety loss. But there's no guarantee either, because each copy can tip in its own direction, sometimes less safe, sometimes overcautious. The big percentages in headlines come mainly from AWQ, GPTQ, 2-bit and very small models, and some of them trace back to broken output. Q2_K and below are risky for anything strangers use. Almost everything above 8 billion parameters remains an open question, as do MoE giants with 2-bit experts and the German language. Nobody has studied these yet.

#quantization#ai safety#llama.cpp#gguf#jailbreak#research

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.