Decision models like Jev put to the test: great gatekeepers, poor know-it-alls

What this is about

On September 15, 2026, US startup TypeSafe unveiled Jev. Jev doesn't write a single sentence. You give it a state, such as a customer request, a JSON document or a chat history, along with a question and fixed answer options. What comes back is a probability for each option, computed in a single forward pass. A language model, by contrast, generates its answer token by token. TypeSafe calls this "System One", a nod to fast, intuitive thinking. Since then, a whole new model class has emerged. In its November 5 snapshot, the BenchLM platform lists 23 entries in Artificial Analysis's Decision Index. Add to that Microsoft, Liquid AI, Perplexity and, as of today, Kakao with the open Kanana-2-v-decide.

The vendors make big promises: "43× faster than GPT-6 Luna", "can't hallucinate", "calibrated". For the first time, there are now independent numbers, including several arXiv studies, a hands-on test by Red Hat and the index itself. We put the two side by side.

How a decision model works

At its core, there's usually a regular language model that stops after reading the input. Instead of writing, it reads the answer distribution directly from its internals. Kakao lays this out in its Model Card: the question and input are combined into one vector, and each answer option into another. A vector is a long series of numbers that captures the meaning of a text. The similarity between these vectors yields the final probabilities. Since there are no output tokens, vendors bill only for input. Jev, Liquid d1-ultra and Microsoft Decision-1 each cost around $0.042 per million input tokens.

According to Microsoft, Microsoft Decision-1 is based on Qwen3.5-9B and is available as a public preview in Foundry, with no published weights. Liquid also offers d1-ultra via API only. Open weights come from the community: Kev (0.8B, 4B, 9B, Apache 2.0), decider-2b (Apache 2.0, about 3.8 GB), Laya and this-that-model. As of today, Kakao's Kanana-2-v-decide joins them: 3.67 billion parameters (3.18 billion for the language part, 0.49 billion for the vision part), up to 10 images per request, under the Kanana Open License, which also permits commercial use.

The claims, fact-checked

We ran the numbers on the 43× claim. Artificial Analysis measures 1.5 seconds per task for GPT-6 Luna as a regular language model. Divided by Kakao's 0.04 seconds, that's about 37×, roughly the ballpark of the marketing figure. But Luna is also listed there in decision mode, at 0.17 seconds. Against that, Kanana would be only a little more than four times as fast. On top of that, Kakao measured locally on an H200 GPU, while the index values include API calls over the network. Our take: the number isn't wrong, but it compares apples and oranges.

On hallucination, even sympathetic observers concede that the guarantee covers only the format. Jev can't invent an option that doesn't exist, but it can certainly pick the wrong one. According to modemguides.com, TypeSafe writes in its own launch post that the 0 percent figure is "not an empirical measurement". The Decision Index includes a fifth option, "I don't know". There, Microsoft Decision-1 achieves the best non-hallucination rate at 40 percent, while Jev 1.13 reaches 35 percent. No other decision model exceeds 17 percent.

What the independent studies show

The most thorough work so far comes from Amir Rafe and Subasish Das (Texas State University, arXiv 2610.00346, not yet peer-reviewed). They tested Jev, seven open decision models, two Qwen models and classic classifiers on the same requests:

• With training data, the classics win. On intent detection (CLINC-150, 150 categories), a fine-tuned BERT-base scores 0.940. A simple logistic regression on BGE-small embeddings even reaches 0.948, Jev only 0.913. Without training data, the picture changes: a zero-shot classifier, meaning one with no examples for the task, manages just 0.507.
• "Yes" and "no" swapped. Rename the options in yes/no questions, and Jev flips 50.5 out of 100 answers. The normal variance on repeated runs is 1.3. Kev-9B, Nimble-9B, decider-2b and this-that-model flip only 4.2 to 6.5. A second study (Sun et al., arXiv 2609.26758) confirms the pattern: the models follow an option's name, not the definition you give it.
• Out of scope, but waved through anyway. With a threshold tuned for normal requests, Jev accepts 31 percent of requests outside its domain (Kev-9B 1.5%, Nimble-9B 3%). An explicit "doesn't belong here" option lowers that to 19.5 percent.
• Calibration depends on conditions. Calibrated means that when the model says 80 percent, it's right in about 80 percent of cases. The calibration error ECE measures the deviation from that. For Jev, it's low on workflows at 0.033 and high on social science tasks at 0.251. For Kev-9B, it rises from 0.051 with 5 options to 0.377 with 150.

The authors themselves caution that there's no way to verify whether Jev saw the test data during training. Some open models were demonstrably trained on CLINC training data.

Delip Rao and Chris Callison-Burch (arXiv 2609.29769) pitted Jev and Perplexity's Decider against three low-cost LLMs acting as rubric-based judges, meaning language models that grade texts against fixed criteria. The result cuts both ways. The LLMs cost 16 to 357 times as much and take 5 to 350 times as long. The decision models fell significantly behind in no more than 6 of 27 comparisons. On the errors where a decision model was most confident, the LLMs repeated the mistake in over 85 percent of cases. With independent errors, you'd expect around 50 percent. The title sums it up:

"Cheaper, faster, and wrong in the same places."

Red Hat reached a similarly sober conclusion when testing guardrail filters. On prompt injection, meaning smuggled-in instructions, a small DeBERTa classifier scored 89.0 percent in 54 ms, Jev 86.4 percent in 348 ms. On content safety, Jev came out ahead at 86.2 percent. The authors' verdict: decision models don't reliably beat either LLM judges or pretrained classifiers.

The index: strong at routing, weak on knowledge

The Decision Index weights four tests. Knowledge questions and hallucinations (AA-Omniscience) count for 50 percent; hard reasoning tasks (HLE-MultiChoice), tool selection (When2Call) and safety classification (WildGuardTest) for 25 percent each. The November 5 leaderboard:

At its October 15 launch, Liquid promised a lead of at least 8 points. Today it's down to 0.2. Kanana-2-v-decide isn't in the official index yet. Kakao reports a self-measured score of 61.34. If that were comparable, Kanana would rank ahead of Jev and Microsoft, but behind the leaders. So the "best model" claim refers only to Kakao's own selection of comparison models.

Data scientist Randal Olson broke the scores down by task. On knowledge questions, LLMs lead by about 30 points; on reasoning, by about 19. On safety checks and tool selection, though, the medians are less than 2 points apart, and there, every decision model is faster than every LLM. According to Artificial Analysis, all decision models are overconfident, meaning more certain than their hit rate justifies. That's mainly down to the knowledge and reasoning tasks.

Where these models pay off

Our take: the models are a good fit as fast gatekeepers in agents, anywhere the input already contains everything needed for the decision. Examples include picking the right tool, triaging tickets, safety filters and the question "human or machine?". Rafe and Das demonstrate a tiered escalation: the small open this-that-model decides on its own and hands 25 percent of cases off to Jev. At full GPU utilization, that costs about 43 percent of running Jev alone and delivers 0.927 accuracy instead of 0.913. The gain, however, isn't statistically significant.

There are already real-world examples. The SAG2 search system uses Jev as a judge that scores passages. It achieves 86.6 percent recall in the top ten results on MuSiQue, compared with 54.4 percent for plain keyword search. But the authors openly state that questions and passages are sent to the hosted service and that fully on-premises operation isn't possible. For confidential data, that's a privacy problem. In the MiniCPM OCR series, a 2B verifier model, also called Jev, plays a similar role: it outputs confidence scores instead of text.

The models are not suitable for knowledge questions, multi-step reasoning, counting or date comparisons. TypeSafe itself lists the last two as weaknesses. According to t3n, the open models run locally from Ollama 0.35 and LM Studio 0.4.26 onward. Ollama offers nimble and tev1, among others.

The bigger picture

Decision models aren't a new miracle cure. Sebastian Raschka calls them the "plug-and-play version of specialized classifiers": they deliver usable results without training, but don't beat a fine-tuned BERT once training data exists. The vendors' marketing figures are hard to verify. TypeSafe has published neither a paper nor weights, and Kakao hasn't disclosed a baseline for its 43×.

What is established: in their niche, the models are about as good as LLMs while being much faster and cheaper. Their probabilities work as a threshold for escalation. But you need to validate that threshold yourself for every model, every task and every number of options, name options as neutrally as possible and offer an explicit "not responsible" option. With Kanana-2-v-decide, there's now an open model that also understands images and can be used commercially. Whether it lives up to Kakao's claims remains for independent tests to show.