Same model, different harness: What the 319 MB Qwen3.8 adapter really delivers
Hugging Face is promoting its LoRA adapter for Qwen3.8-27B with a jump from 39 to 65 percent on SWE-bench Pro. But the Model Card shows the add-on mainly makes the model more efficient and tunes it for the pi tool. It doesn't make it smarter. We explain what a "harness" is and why the same model can score very differently depending on the agent tool.

What this is about
On November 17 and 18, Hugging Face's "harness-hub" team released a small add-on for Alibaba's open model Qwen3.8-27B: Qwen3.8-27B-pi-adapted. It's a LoRA adapter, a small extra layer that sits on top of the unchanged base model. As a GGUF file for llama.cpp, it's only about 319 MB. It was trained for exactly one tool, the coding agent pi. The headline: On the 51 hardest SWE-bench Pro tasks, the success rate climbs from 39 to 65 percent.
That sounds like the model suddenly got much better at coding. The Model Card tells a more sober story. The adapter mainly makes Qwen3.8-27B more efficient and teaches it how to work with pi. It also illustrates a fundamental problem that usually gets lost in leaderboards: An AI model deployed as an agent is never just the model. It's always the model plus the tool.
What a harness is and why it flips leaderboards
Think of a harness the way you'd think of one on a draft horse. The term covers everything an agent program builds around the model: the system prompt, the tools (reading and writing files, running shell commands) and their format, how error messages get passed back to the model, timeouts, trimming of long histories and the cap on working steps. pi is an extreme case. According to eesel.ai, it ships with just four tools by default (read, write, edit, bash), and the system prompt plus tool descriptions stay under 1,000 tokens. Mario Zechner built pi, and it has belonged to the company Earendil since April 2026.
A team at Nanyang Technological University (NTU) measured in October just how much the harness shapes results (arXiv 2610.00917). It tested five models in four tools (OpenHands, DeepSeek Harness, pi, openJiuwen) on three benchmarks, plus the vendors' own tools Claude Code and Codex. That's 66 configurations in total. The most striking result on Terminal-Bench 4, according to a Besthub summary:
| Harness | Claude Opus 5 | GPT-6 Astra | Gap |
|---|---|---|---|
| OpenHands | 60.32% | 49.21% | Claude +7.94 |
| pi | 30.16% | 60.32% | GPT +30.16 |
Switching to pi, Claude loses 17 tasks and GPT gains 7. The leaderboard flips completely. The authors point mainly to error feedback as the reason. In 180 of 192 failure cases examined, the model initiated the fix itself. So what matters is whether the harness returns errors in a form the model can work with. pi, for example, sets no default timeout for shell commands. GPT-6 Astra set one itself for 56 to 67 percent of its commands, so the gap didn't hurt it. Kimi K3, on the other hand, got stuck for 39.8 minutes on a hanging GIMP script in one case study and scored zero. In openJiuwen, a timeout message came after 300 seconds, Kimi fixed the script and finished in 11 minutes. And a vendor's own tool isn't automatically the best: Codex with GPT trailed pi and openJiuwen with GPT on every benchmark.
What's in the Model Card and missing from a lot of coverage
The first numbers, shared on X by Hugging Face CEO Clément Delangue among others, were 29.4 → 62.7 percent. The card now carries a correction notice: A public tunnel between the sandbox and the model server had gone down, causing nine base model runs to fail. After reruns, the figures are 39.2 → 64.7 percent. The old values still show up on Digg, in explainx.ai's fact check and even in a user question on the model's discussion page. AI Times in Korea and MarkTechPost already cite the corrected numbers.
There was a second correction too. The first runs with thinking mode had a limit of 16,000 tokens per response, which cut off long chains of reasoning. With 32,000 tokens, the picture looks like this:
| SWE-bench Pro (hard 51), pi harness | Thinking off | Thinking xhigh |
|---|---|---|
| Base model | 39.2% | 74.5% |
| With adapter | 64.7% | 74.5% |
On Terminal-Bench 2.1 (83 tasks), the base model scores 41.0 and 54.2 percent respectively, while the adapter hits 56.6 percent in both modes.
More efficient, not smarter: Where the jump comes from
The key sentence on the card: On SWE-bench Pro, 20 of the base model's 31 failed attempts ended because it ran out of its 100 allowed model calls. That never happened once with the adapter. We ran the numbers: Of the 31 tasks where the base model finished before hitting the limit, it solved 20. That's 64.5 percent, almost exactly the adapter's rate. It's not proof, since the aborted tasks are likely the harder ones. But it shows where the leverage is.
That fits the training recipe. The adapter was trained with AsyncGRPO, a form of reinforcement learning where the model is rewarded for successful solutions. The reward consisted of the test harness result plus a bonus weighted at 0.1 for using few tool calls. In the blog post on the method, the formula is 0.1 × (1 − calls/15), and the bonus only applies to correct solutions. Training runs were capped at 50 steps, while testing allowed 100. On tasks both variants solved, the adapter needed a median of 23 turns instead of 59. Interestingly, it writes more text along the way: about 160 tokens of reasoning per turn instead of around 10. So it thinks briefly and visibly, rather than firing off tool calls without a plan like the base model.
With thinking mode on, the base model reaches the same score without the adapter. But it produces a median of 48,800 tokens instead of 9,400 and needs 61 model calls instead of 21. According to the card, the difference between the adapter with thinking off (64.7%) and the base model with thinking on (74.5%) isn't statistically significant with only 51 tasks (p = 0.33).
Our take: The adapter is an efficiency gain, not a capability gain. If you're running locally on slow hardware, that's still worth a lot, because five times fewer tokens means roughly five times less waiting. If you're paying per token through an API, it means money saved.
The card lists further limitations itself. Each task got only one attempt. The checkpoint (step 160) was selected based on these very SWE-bench Pro results and an 18-task slice of Terminal-Bench. So the cleaner test is the remaining 68 Terminal-Bench tasks, and there the gain holds at 39.7 → 55.9 percent. The numbers can't be compared with Qwen's official figures (61.7% on SWE-bench Pro in Claude Code, 73.0% on Terminal-Bench 2.1 in Terminus). Those used different harnesses, bigger budgets and the full benchmark.
pi, OpenCode, Hermes: What local users get out of it
There's now also an OpenCode adapter. Put both cards side by side and you can see the harness effect even in the base model:
| SWE-bench Pro (hard 51) | pi | OpenCode |
|---|---|---|
| Base, thinking off | 39.2% | 35.3% (corrected from 31.4%) |
| Adapter, thinking off | 64.7% | 51.0% (p = 0.057) |
| Base, thinking on | 74.5% | 47.1% |
| Adapter, thinking on | 74.5% | 54.9% |
On Terminal-Bench 2.1, the OpenCode adapter only improves the score across all 84 tasks from 46.4 to 52.4 percent (p = 0.38, so not significant), and on the 66 tasks not used for selection from 45.5 to 59.1 percent. The same base model with thinking on scores 74.5 percent in pi and 47.1 percent in OpenCode, a gap of 27 points. The card offers one possible cause: In OpenCode, the model ran text-only without web search, and sessions with image reads or search attempts aborted. The cards don't make clear whether the output limits were identical.
A Hermes adapter for Qwen3.8-27B doesn't exist yet. The organization page says "OpenCode and Hermes coming," and so far Hermes adapters only exist for smaller models like TwIL-8B. Hermes users are stuck with the base model for now.
The real question is whether the pi adapter helps in other tools. Nobody has measured that publicly, including us. On the discussion page, a user asks whether the gains carry over to omp.sh, with no answer. A GitHub issue plans such a test, but it's been put on hold. The only hint comes from an offshoot, xdannys9's Codemode adapter, trained on pi's JavaScript mode. In its tests, the pi adapter actually scored below the base model on 30 SWE-smith tasks (16.7% vs. 23.3%), but well above it on 10 greenfield tasks (60% vs. 20%). With so few tasks, that's noise, but it shows that nothing is guaranteed outside the adapter's own test environment. Even settings matter. The card warns that without "api": "openai-completions" in models.json, pi uses the Responses interface, which the adapter was never trained on.
Hardware: The adapter only works with Qwen3.8-27B. The recommended base is a Q8_0 build from ggml-org/Qwen3.8-27B-GGUF, because more aggressively quantized variants drift further from the training state (BF16). By our rough math, that's just under 29 GB, so on a 24 GB card part of it has to spill into system RAM. The card's MacBook example uses a Q4 variant at about 17 GB. Important: The GGUF version wasn't benchmarked separately. A Reddit test shows how slow things can get in the extreme: On an RTX 4060 with 8 GB of VRAM (video memory), it ran at 3.5 to 5.7 tokens per second. For agent sessions with dozens of turns, that's exactly where the adapter's efficiency makes the difference.
Training in the real tool is a trend
Hugging Face isn't alone in this approach. LEGO-RL (arXiv 2608.17393) trains Qwen3.5-35B-A3B directly in OpenHands, Claude Code and OpenCode, with a separate variant for each tool. On SWE-bench Verified, scores rose from 64.0 to 70.4, from 62.4 to 68.2 and from 57.2 to 66.6 percent. OpenForgeRL (arXiv 2607.21557) records model calls from any tool via a proxy and reports that reinforcement learning mainly improves reliability, while error recovery stays weak. RadixArk's DaisyRose-27B, also based on Qwen3.8-27B, was trained in agent sessions too. So far, there are only vendor numbers for it.
The bigger picture
The pi adapter is a well-documented, honestly corrected experiment. The card publishes corrections, significance tests and every run. But it's not a new, better model. It saves tokens and turns, and only in the tool it was trained in. Our verdict: If you work locally with pi, give it a try, with thinking mode off and the card's settings. If you use OpenCode, grab the matching OpenCode adapter. Dropping the pi adapter into Hermes or other tools is a gamble. The bigger lesson is in the NTU paper: A leaderboard ranking only ever applies to one specific combination of model, tool and task. If you're comparing models, do it in your own tool.
Sources
- Hugging Face: harness-hub/Qwen3.8-27B-pi-adapted (Modellkarte) huggingface.co
- Hugging Face: harness-hub/Qwen3.8-27B-opencode-adapted huggingface.co
- Hugging Face: harness-hub/Qwen3.8-27B-pi-adapted-GGUF huggingface.co
- Hugging Face Blog: Harness Adapters huggingface.co
- Hugging Face: Organisation harness-hub huggingface.co
- Hugging Face Collection: Qwen3.8-27B Harness adapters huggingface.co
- Harness-Adapters Live-Dashboard harness-hub-harness-adapters-rl.hf.space
- Hugging Face: Qwen/Qwen3.8-27B (Original-Modellkarte) huggingface.co
- arXiv 2610.00917: Finding the Right Fit – Model-Harness Interactions across Agent Tasks arxiv.org
- Besthub: NTU-Studie zu Modell-Harness-Passung besthub.dev
- Hugging Face: xdannys9/Qwen3.8-27B-pi-codemode-adapted huggingface.co
- Hugging Face Diskussion #8: omp.sh-Benchmark huggingface.co
- GitHub-Issue: pi-adapted Qwen3.8-27B als lokaler Worker github.com
- explainx.ai: Faktencheck zu Qwen3.8-27B und SWE-bench Pro explainx.ai
- Digg: Reaktionen zum pi-Adapter digg.com
- AI Times (KR): Hugging Face veröffentlicht Coding-Agenten-Adapter aitimes.com
- eesel.ai: Pi coding agent explained eesel.ai
- arXiv 2608.17393: LEGO-RL arxiv.org
- Hugging Face Papers: OpenForgeRL (2607.21557) huggingface.co
- dev.to: RadixArk DaisyRose-27B dev.to
- Reddit r/ollama: Qwen3.8-27B Q4 auf 8-GB-RTX-4060 reddit.com
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.