Same model, worse answers: what the Hugging Face router really hands you
Since November 16, OpenRouter picks the provider for every chat request on Hugging Face, based on price, speed and reliability. A week earlier, Artificial Analysis found that almost one in four endpoints serving open models scores worse than the reference. Here's where those gaps come from, how big they are and how to choose your provider yourself.

What this is about
When you call an open model like GLM-5.3 on Hugging Face, Hugging Face doesn't run the computation itself. Your request goes to one of many data center operators serving the model. Since November 16, OpenRouter decides which operator handles each chat request. According to the Hugging Face documentation, the factors are effective price, speed and reliability.
Answer accuracy isn't on that list. Yet a week earlier, Artificial Analysis measured that nearly one in four provider endpoints for open models performs measurably worse than the original. So our question is: how much do providers really differ, why, and what can you do about it?
The measurement: 24 of 103 endpoints below reference
For its Endpoint Accuracy Index v1.1, Artificial Analysis ran the official weights of six models itself, on Nvidia B200/B300/GB200 systems with vLLM or SGLang. It then ran the same tests against 103 serverless endpoints, meaning rented model access with no hardware of your own. Three areas were weighted equally: tool calling (BFCL-500, where the model has to call functions with correct arguments), hard expert knowledge (HLE-250) and retrieving information from roughly 100,000 tokens of input (AA-LCR-25). An endpoint only counts as “below reference” if its entire 95 percent confidence interval sits below 100. Measurements ran from October 8 to November 4.
| Model | Endpoints | Below reference | Share (our calculation) | Index range |
|---|---|---|---|---|
| GLM-5.3 | 26 | 10 | 38% | 67.2–99.3 |
| GLM-5.3 Flash | 22 | 5 | 23% | 78.6–104.2 |
| gpt-oss-120b | 16 | 5 | 31% | 78.2–104.0 |
| Kimi K3 | 12 | 2 | 17% | 91.0–101.5 |
| Muse Spark Open | 5 | 2 | 40% | 93.4–99.9 |
| DeepSeek V4.1 Flash 1017 | 22 | 0 | 0% | 97.5–101.4 |
So the spread depends heavily on the model. For DeepSeek V4.1 Flash, all 22 endpoints were within range. For GLM-5.3, nearly four in ten fell short. The article doesn't name the affected providers. It only describes the configurations it observed.
Where the differences come from
Capped output lengths. This is by far the biggest effect. Reasoning models write long chains of thought before answering. If a provider cuts output off at 16,384 tokens, GLM-5.3 only reaches 7.6 instead of 63.6 percent on HLE, roughly an eighth of the reference score. With a 32,000-token limit, it gets 24.8 percent. With Kimi K3, some endpoints aborted requests after about ten minutes, leaving around 9 percent of HLE tasks without an answer.
Less thinking. Two GLM-5.3 Flash endpoints produced about 30 percent fewer reasoning tokens with identical settings and scored 48.4 and 48.6 percent instead of 55.4, even though nothing was truncated.
Parser bugs in tool calling. For gpt-oss-120b, two endpoints managed only 21.3 and 22.3 percent instead of 37.1. One returned function calls as plain text rather than structured calls. The other filled optional arguments with placeholders like 0 or -1. Other failure patterns: object-type arguments came back as JSON strings, parameter names were mangled, and Muse Spark Open returned booking numbers as numbers instead of strings. One Kimi K3 endpoint reused tool call IDs across multiple conversation turns and dropped to 42.1 instead of 55.2 percent. The parser is the software that translates the model's raw output into the API format.
Quantization. This means the model computes with more coarsely stored numbers, such as FP4 instead of FP8. It's the usual suspect, and the data barely supports it: all three FP4 endpoints for GLM-5.3 Flash and five of six for GLM-5.3 were within range of the FP8 reference. That matches OpenRouter's findings. In its Auto Exacto post, the company writes:
“We have not seen a measurable impact of quantization alone on tool-call quality. The real culprit is usually: tool-call parsers.”
In its own test with Qwen3.8 27B, six endpoints ranging from fp16 to fp4 scored between 98.3 and 99.2 percent, with overlapping confidence intervals. OpenRouter itself concedes that this doesn't rule out a gap of one to two points. Only one model was tested, on tasks with exact answers.
Sampling defaults. According to the OpenRouter parameter docs, a parameter like temperature that you don't send isn't passed on at all. The provider's own default applies instead. So the same script can run with different settings depending on the provider.
A case on LessWrong shows that the choice of operator alone can skew research results. A paper had credited DeepSeek R1 with strikingly illegible chains of thought. But every request had landed with the provider Targon. On Novita, the mean illegibility score was 2.28 instead of 4.3, even though both apparently used fp8. The paper's author acknowledged that his results were probably distorted. The post “Not Pinning Your OpenRouter Provider Might Invalidate Your Research” also claims that 31 of 32 research codebases examined used OpenRouter in an unsafe way. That review, however, was generated by Claude and only spot-checked. Commenters therefore consider the figure shaky.
What Hugging Face's router catches
Neither OpenRouter's announcement nor the Hugging Face blog mentions accuracy, quantization or Exacto. Still, the routing isn't completely blind. According to the Hugging Face docs, “reliability” includes how often a provider returns broken tool calls. According to OpenRouter's post on its rebuilt routing, a provider flagged for broken tool calls or clearly too-low benchmark scores only receives a trickle of test traffic. Auto Exacto also re-scores providers for requests with tools roughly every five minutes.
The gap lies elsewhere. OpenRouter's telemetry checks whether a tool call is valid JSON, whether the tool name exists and whether the schema matches. A “-1” placeholder in an optional field may well pass that check, and a truncated chain of thought certainly will. On top of that, OpenRouter says an explicit sort order bypasses the new algorithm. Whether that also applies to :cheapest and :fastest on Hugging Face, thereby sidestepping the quality flags, isn't documented. Our take: the router weeds out gross outliers, but not the quiet quality losses Artificial Analysis measured. The new “Providers” tab on model pages shows price, speed and availability, but neither numerical precision nor accuracy scores.
Independent spot checks confirm the spread. On September 3, Openbenchmarks tested ten GLM-5.3 Flash providers with 600 structured tasks each. Success rates ranged from 99.5 percent (Fireworks, Telnyx) down to 84.8 percent (Nebius, a 14.8 percent error rate). The test did, however, use a 20-second timeout and 256 output tokens, so it also counts timeouts as failures.
The dark end: shadow APIs and malicious routers
With OpenRouter and Hugging Face, we're talking about configuration errors by legitimate providers. Unofficial middlemen, currently a hot topic in the Japanese tech press, are far worse. The CISPA study found 17 so-called shadow APIs that promise cheap access to GPT-5 or Gemini and were used in 187 research papers. As ITmedia summarizes, 11 of 24 tested endpoints (45.83 percent) didn't match the fingerprint of the model they claimed to be. Some “GPT-5” endpoints were actually GLM-4-9B or Qwen2.5-7B. Gemini-2.5-flash dropped from 83.82 to around 37 percent on the MedQA medical test.
The UCSB study “Your Agent Is Mine” examined 428 API routers, 28 paid and 400 free. Nine of them actively injected malicious code: one of the paid ones and eight of the free ones. Seventeen accessed planted AWS honeypot credentials, and one drained an Ethereum wallet. A figure of 68 malicious routers that turned up in our pre-briefing material appears neither in the paper nor in ITmedia's coverage. The documented number is 9. The 17 credential-access cases are a separate metric and shouldn't be added to it.
You can't read quality off the price either. Since November 12, China's Doubao has also offered third-party models, with cost multipliers from 0.01 to 1.48. That's the 148-fold range from the smzdm headline, but it's across different models. According to the report, the multiplier says nothing about quality or speed. On Doubao, the context length, meaning how much text the model can process at once, may also differ, and so may the system prompt compared with the official Kimi service. Users claim these are “stripped-down” versions, but that's unproven.
How to pin your provider and test it yourself
This article contains no measurements of our own. Here's how to check for yourself:
- Pin the provider: On Hugging Face, append the suffix to the model name, e.g.
deepseek-ai/DeepSeek-V4.1-Flash:together. According to the docs, a provider pinned this way is never replaced. If it goes down, you get an error. The suffixes are listed on the providers page. - Exclude providers: You can switch providers off in the Inference Providers settings. For organizations, the organization's settings apply.
- Directly on OpenRouter, you get more knobs:
provider.only,allow_fallbacks: false,quantizationsandrequire_parameters: true. The last one only sends requests to providers that support all your parameters (docs). Whether Hugging Face's router passes these fields through isn't documented. Note: the quantization info is self-reported by the provider. - Test it yourself, modeled on the Kimi Vendor Verifier: send the same tool-call request 20 times to each pinned provider. Then check whether a tool gets called at all, whether the arguments match the schema (watch the types!) and whether IDs are clean. Add a long output with
max_tokensset explicitly. Iffinish_reasonis “length” or missing entirely, the answer was truncated. Incidentally, Moonshot's results for K3.1 Pro show that reputable providers sit close together: ToolCall scores from 0.972 to 1.0. - Log everything: Record provider, date and parameters as routinely as you record temperature. The LessWrong post recommends this too.
The bigger picture
For casual users, Hugging Face's switch is mostly a win: fewer outages, automatic failover on errors, and gross tool-call outliers get filtered out. For agents, coding assistants and research, that's not enough. That's exactly where truncated reasoning chains and mistyped arguments do the most damage. The good news: as things stand, numerical precision is rarely the problem, and some models, like DeepSeek V4.1 Flash, run equally well across every provider measured.
Plenty remains open. Artificial Analysis names no providers and only captures snapshots, and OpenRouter's figures are vendor claims. How Hugging Face's routing weights quality signals is disclosed nowhere. Our take: if you need reproducible results, pin your provider, check it with a small test set of your own, and avoid middlemen whose operators you don't know.
Sources
- Artificial Analysis: Endpoint Accuracy Index v1.1 artificialanalysis.ai
- Artificial Analysis: Methodik Endpoint Accuracy Index artificialanalysis.ai
- Hugging Face Blog: OpenRouter huggingface.co
- Hugging Face Docs: Routing and Providers huggingface.co
- Hugging Face Docs: Providers huggingface.co
- Hugging Face Docs: Inference Providers huggingface.co
- OpenRouter: Hugging Face Inference Providers openrouter.ai
- OpenRouter: How we rebuilt provider routing openrouter.ai
- OpenRouter: Auto Exacto openrouter.ai
- OpenRouter: LLM Quantization openrouter.ai
- OpenRouter Docs: Provider Selection openrouter.ai
- OpenRouter Docs: Parameters openrouter.ai
- arXiv: Real Money, Fake Models (CISPA) arxiv.org
- arXiv: Your Agent Is Mine (UCSB) arxiv.org
- arXiv: Ventor-QTest arxiv.org
- ITmedia AI+: Schatten-APIs itmedia.co.jp
- ITmedia AI+: API-Router itmedia.co.jp
- GitHub: MoonshotAI/Kimi-Vendor-Verifier github.com
- GitHub: MoonshotAI/K2-Vendor-Verifier github.com
- LessWrong: Not Pinning Your OpenRouter Provider Might Invalidate Your Research lesswrong.com
- LessWrong: R1 CoT illegibility revisited lesswrong.com
- Openbenchmarks: GLM 5.3 Flash Provider Accuracy openbenchmarks.com
- AlphaSignal: 1 in 4 AI Endpoints Underperform alphasignal.ai
- smzdm: Doubao Drittmodelle und Kostenfaktoren post.smzdm.com
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.