“Mm-hmm,” “Hang on”: Why voice AI does worse in conversation than on benchmarks

What this is about

Within just a few days, several companies have unveiled “live conversations” with AI: ElevenLabs launched Eleven Realtime, ServiceNow released the open Apriel Voice 27B, and Japanese startup Shizuku AI introduced Shizuku-Duplex. Meta is rolling out video calls with its agent Muse, and the code editor Cursor is getting a voice chat built on Grok Voice. They all promise the same thing: you can interrupt the AI, and it says “mm-hmm” while you're talking.

Under the hood, though, the technology varies widely. And the number used in the marketing – up to 98.9 percent for “conversational dynamics” – measures less than it suggests. We explain how these systems are built, what the benchmark actually tests, and what independent studies have found.

Three architectures, one label

1. The chain (cascade). Speech recognition transcribes, a language model writes the reply, and speech synthesis reads it aloud. When the human has finished is decided by voice activity detection (VAD) – essentially a stopwatch that fires after a set amount of silence. According to Kyutai spin-off Gradium, 500 milliseconds is a typical value. Yet humans leave an average gap of only about 208 milliseconds between turns. Newer systems therefore add semantic turn detection, which uses the content to tell whether a sentence is still going. Kyutai's voice service Unmute works this way. Surprisingly, Eleven Realtime also belongs in this group. ElevenLabs' blog says the model is based on a cascaded architecture of speech recognition, turn-taking, language model and speech output that is “tuned end-to-end.” From the outside it's a single model; inside, it's still a chain.

2. Half-duplex with direct audio processing. The model hears audio directly, without going through text. But who speaks when is still controlled by an external switch. One example is Alibaba's Qwen Audio 3.0 Realtime. Vendor pages call it “full-duplex,” but the documentation shows that turn changes are detected via server_vad or smart_turn. Fillers like “hmm” deliberately don't trigger a turn change. So here, only the connection is full-duplex, not the model. Apriel Voice 27B goes one step less far: it understands speech directly but outputs only text and tool calls. According to its model card, speech output and turn-taking are “components of the application.”

3. True full-duplex. The template is Kyutai's Moshi from 2024. The model processes two audio streams at once: the user's and its own. On top of that comes a text stream, the so-called inner monologue: before speaking, the model writes down what it's about to say. The rhythm is set by the audio codec Mimi, which splits sound into 12.5 chunks per second – one every 80 milliseconds. At each tick, the model decides anew whether to speak, stay silent or drop in an “mm-hmm.” Turn-taking doesn't exist as a separate concept. Theoretical latency is 160 milliseconds, around 200 in practice. According to the Moshi paper, overlaps account for 10 to 20 percent of speaking time in real conversations. That's exactly what chains lose.

Shizuku-Duplex works on the same principle, with 80-millisecond ticks and an inner monologue. It hands off hard requests such as web searches to a background model via a DELEGATE command and keeps talking in the meantime. OpenAI's GPT-Live does something similar: according to VentureBeat, it decides “several times per second” and passes heavy tasks to GPT-5.5. Meta hasn't disclosed how Muse is built internally. All that's known is a response delay of just over 870 milliseconds. The sources also give no details on the architecture of Grok Voice, which Cursor uses.

What the industry standard measures

The most important leaderboard is run by Artificial Analysis. Its “Conversational Dynamics” score combines four parts of Full-Duplex-Bench: handling pauses and turn-taking (version 1), plus interruptions and backchannels (version 1.5). Each score is the share of test cases handled correctly. The original paper from National Taiwan University is explicitly English-only; the data comes from telephone corpora and synthetic dialogues generated with GPT-4o. Version 1.5 includes 200 interruptions and 99 backchannels. The authors themselves write that their metrics are “not tied to human preferences.”

Excerpt, as of November 9, 2026, 37 entries.

Three things stand out. First, the open “true” full-duplex models score worst of all, while a system with external turn detection sits in second place. Second, the leader, StepAudio 3 Realtime, hits 98.9 percent but, according to the same site, takes 8.83 seconds to produce its first sound. The percentage says nothing about speed. Third, the cheap Eleven Mini beats the bigger Max by 1.7 points.

Our take: a subtest with 99 cases and a 95 percent hit rate carries a statistical uncertainty of a little over ±4 points. Even averaged across four subtests, roughly ±2 points remain. That assumes Artificial Analysis uses the public test sets, since the site doesn't state sample sizes. Gaps between 95 and 99 percent therefore mean very little. Artificial Analysis itself dropped conversational dynamics from its overall index in August 2026, replacing it with a task success rate.

What independent studies find

Other benchmarks test harder situations and come to far more sobering results:

EchoChain is particularly revealing. The most common failure there is “interruption amnesia”: the system hears the interjection and forgets it anyway. Second is “contextual inertia,” where the model carries on with the old state. When the same content was supplied only after a complete answer, the error count dropped from 92 to 55, or 40.2 percent. So the problem is the interruption itself, not the task.

FLEXI also shows that for every model, the timing of “mm-hmm” has almost nothing to do with that of human listeners. The divergence, measured as Jensen-Shannon divergence, ranges from 0.947 to 0.986, where 1 is the maximum possible. Instruct-FD describes the typical failure as “right content at the wrong time”: the “mm-hmm” arrives only once the human has finished. That said, humans struggle too. In the same study, they matched only 63.7 percent of the reference decisions on backchannels. According to TurnBench, in a smooth handover people start speaking a median 151 milliseconds before the other person has finished. No system manages that yet without lots of false triggers.

One caveat applies to all these studies: they mostly tested older models such as Gemini 2.5 or 2025's GPT-realtime. Eleven Realtime, GPT-Live and Shizuku-Duplex aren't included yet.

Shizuku-Duplex itself shows how little a good “mm-hmm” says about content. In its own Japanese test, judged by a language model, it beats GPT-Live at continuing to talk after an “mm-hmm” (95 to 88%) and at falling silent when interrupted (89 to 82%). But it loses clearly on answer quality: 2.6 to 4.0 out of 5 points, and 79 to 95 percent correct answers on multi-step questions. The test data isn't public.

Can any full-duplex model speak German?

Provably, meaning with independent testing: we didn't find one.

• Moshi was trained on 2.1 trillion tokens of mostly English text. Nvidia's PersonaPlex builds on Moshi and is likewise available in English only.
• J-Moshi shows how much effort a new language takes: around 60,000 hours of Japanese conversations for pretraining, plus a new tokenizer (the component that splits text into computable chunks). Genuine stereo conversations amounted to just 344 hours. Shizuku-Duplex only speaks Japanese; more languages are planned.
• GPT-Live works in German ChatGPT accounts. But according to VentureBeat, OpenAI admits the model speaks with an accent or haltingly in some languages. We found no German real-world test of its interruption handling.
• Qwen Audio 3.0 Realtime officially lists German but handles turn-taking via turn detection (see above). Eleven Realtime cites more than 90 languages but is a chain. Berlin startup Deepslate is targeting European languages and German dialects with Opal. Opal scores 85.7 percent in the ranking, but the company doesn't call it a full-duplex model.

As far as we know, all the benchmarks mentioned test English only. That's explicitly documented for the original Full-Duplex-Bench paper, IHBench and Instruct-FD. We weren't able to run our own test.

The bigger picture

“You can interrupt it” is now a standard feature, not a mark of quality. Whether there's a fast pause stopwatch behind it or a model that genuinely listens and speaks at the same time is something marketing rarely tells you. And in practice, a chain with well-tuned turn detection scores at least as well on the standard test as true full-duplex.

Our take: the 95 to 99 percent on Artificial Analysis mainly proves that the top models handle the easy cases. They wait during pauses, keep talking after an “mm-hmm” and stop when you interrupt them. The hard cases – whether a model still knows what's current after being interrupted, and whether its “mm-hmm” lands at the right moment – even the best systems solve in independent studies at most every second or third time. If you want to deploy a voice agent in German, you currently have to test it yourself, because there are no reliable German measurements.