“Real-time” speech recognition: What 80 milliseconds actually means

What this is about

If you've ever turned on live captions in a video call, you know how it goes: a word appears, changes a second later, and half the sentence gets rewritten. Behind that is a trade-off every streaming speech recognizer has to make. Streaming ASR (“Automatic Speech Recognition”) transcribes while someone is still talking. It can guess early and correct itself later, or it can wait until it's sure and then stick with its decision.

The topic is getting attention right now because of Confucius4-R2T2 from NetEase Youdao. The open model appeared on Hugging Face in September, and the full paper is now out on arXiv. Its promise: once a word is emitted, it never gets rewritten. At the same time, ElevenLabs, Nvidia, Mistral and OpenAI all advertise “real time” with millisecond figures. But those numbers measure very different things. We explain what's behind them and work out how much accuracy each tenth of a second you save costs you, including in German.

Why words flip in the first place

A streaming recognizer receives audio in small pieces called chunks (blocks of, say, 80 or 160 milliseconds). At the end of a chunk, the model often doesn't know yet how a word ends. “Four” might turn into “forty”, “can” into “candidate”.

Many systems deal with this using provisional hypotheses. They show their best guess right away and overwrite it once more audio comes in. The well-known whisper_streaming project from Charles University in Prague turns Whisper, which wasn't built for streaming, into a live system with a rule called Local Agreement. Whisper reprocesses the growing audio buffer again and again. Text only counts as confirmed once two consecutive passes agree on it. The price: in the 2023 paper, latency is around 3.3 seconds for English and 4.1 to 4.4 seconds for German. In return, accuracy comes close to offline quality (German: 9.5 instead of 9.2 percent word error rate). The repository is now considered outdated and points to its successor, SimulStreaming.

Alibaba's Qwen3-ASR works much the same way. According to its technical report, the streaming mode uses 2-second chunks and reprocesses all the audio so far at every step. The last five tokens (word pieces) are generated fresh each time and can change. Those are exactly the words that flip.

There's evidence that the flicker bothers people. In a 2023 Google study, 123 participants rated captions for 20 YouTube clips. Stabilized captions scored better on five out of six statements: less distracting, less tiring, more pleasant to read. The researchers write:

“Text instability in live captions significantly impairs users' reading experience.”

For voice agents and interpreting pipelines, the problem is even worse. The R2T2 paper sums it up: an emitted token is an action, because downstream components start working with it right away. If it gets corrected later, the translation was wasted work, or the agent may already have called the wrong tool.

Three designs that don't backtrack

Cache-aware encoder (Nvidia Nemotron): Nemotron Speech ASR (English only) and Nemotron 3.5 ASR (40 language variants, including German) each have 600 million parameters. The encoder, the part that turns audio into features, keeps its previous states in a cache. According to the Model Card, this means every piece of audio is processed exactly once, with no overlap. The chunk size is an 80 ms frame plus look-ahead: the model may listen 0, 1, 3, 6 or 13 frames into the future before it decides. That gives you the settings 80 ms, 160 ms, 320 ms, 560 ms and 1.12 s. You choose the setting at startup, no retraining needed. The decoder is an RNN transducer that appends text piece by piece.

Delayed streams (Mistral Voxtral Realtime): The 4.4-billion-parameter model takes one step per 80 ms frame and outputs either a text token or a placeholder (“wait”). A word can only appear once it's acoustically complete and the configured delay has passed. You can set this delay in 80 ms steps up to 2.4 s. Mistral recommends 480 ms.

Stable prefix (NetEase Confucius4-R2T2): Here, NetEase post-trains an LLM-based model that originally rewrites its output: Qwen3-ASR-1.7B. The training data is 1,500 hours of Chinese and English audio. Only the language decoder is trained, while the audio encoder and the architecture stay unchanged. The model learns the longest stable prefix: the start of the text that won't change even with more audio. It also gets an explicit “wait” action for when it needs more audio. A parameter U sets how many provisional tokens are held back internally. With U=1, one token is held back, which improves quality. With U=0, nothing is held back, which cuts latency. Either way, only final text is emitted.

By the way, the R2T2 paper also lists Nemotron 3.5 and Voxtral Realtime as systems without rewritable intermediate output. The systems that rewrote in the table were the Qwen3 base model, WhisperRT and the three commercial APIs (AssemblyAI and two unnamed ones). So fixed output isn't new. What's new is that it works with a large LLM decoder at 160 ms.

Four numbers, all called “latency”

Nvidia's 24 ms cause the most confusion. According to the blog of Daily, which measured the figure, finalization only kicks in after 200 ms of silence. On top of that, the server appends 120 ms of artificial silence, because the model needs 320 ms of trailing audio to finalize the last words. The article doesn't say whether these times are included in the 24 ms. On an RTX 5090, the server-side values are 19 ms (median) and 23 ms (P90). For the whole voice agent, from when the speaker stops to when the bot starts talking, Daily gives 508 ms server-side. In the browser, you typically add 250 ms, and on the phone another 300 to 600 ms.

With ElevenLabs, there's no way to tell what the “under 150 ms” for Scribe v2 Realtime actually measures. The documentation distinguishes provisional “partial” transcripts from final “committed” ones. Text is committed automatically after a period of silence or manually by command, for example every 20 to 30 seconds. According to the documentation, processing only starts once the first two seconds of audio have been sent. OpenAI gives no latency figure at all for the new GPT-Live-Transcribe. According to a developer guide, the streamed text deltas can change earlier words.

Our take: the open models' numbers are at least defined. But they measure pure model time on the audio timeline. What actually reaches the user is always slower.

What every tenth of a second costs, including in German

For German, there are comparable numbers on the FLEURS test set. The table shows the word error rate (WER), meaning the share of wrong, missing or extra words. All figures come from the vendors, and text normalization may differ between the tables.

Run the numbers and a clear pattern emerges. Nemotron loses little: going from 1.12 s down to 80 ms raises the error rate by only 1.5 points, but it never drops below roughly 8 percent. Voxtral pays dearly for short delays: between 480 and 160 ms, it adds 3.3 points, a good point per tenth of a second saved. From just under a second upward, though, Voxtral is clearly more accurate than Nemotron in German. According to NetEase, R2T2 leads at 160 ms, even though German wasn't in its training data at all. With U=0, the figure rises to 12.35 percent by our reading of the appendix table. However, the column assignment isn't unambiguous in the text version of the paper. On the second German test, MLS, R2T2 scores 9.52 percent. NetEase itself says the model is optimized for Chinese and English.

An independent study (arXiv 2604.14493) confirms the trend for English on the CPU. Nemotron does best in streaming mode, while Qwen3-ASR-1.7B drops from 5.90 to between 9.69 and 10.45 percent when split into chunks. The authors' conclusion: offline error rate is a poor predictor of streaming quality.

What the jump from 21 to 5.8 percent really shows

NetEase highlights that the error rate at 160 ms falls from 21.01 percent (base model) to 5.76 percent (R2T2). That's true, but it's an unfair comparison, because the base model is built for 2-second chunks. In its intended setting (2 s, five rewritable tokens), it reaches 5.25 percent according to the same table, and 5.15 percent offline. So the real progress is this: one twelfth of the chunk size, no more rewriting, and all that for about half a percentage point more errors.

In the paper, R2T2 is clearly ahead of Nemotron 3.5 (average 14.21) and Voxtral Realtime (19.35). But those averages include Chinese test sets, and the table doesn't make clear which delay was set for Voxtral. Commercial services are close behind (6.01 and 6.23), but according to the paper, they rewrite their text or finalize it later. NetEase only shows emission lag values in charts. The text merely says that the mean lag is under half a second with 80 ms chunks and U=0. The paper explicitly did not measure how often other systems flicker.

What you need to try it

The R2T2 demo runs in your browser. To run R2T2 locally (around 2 billion parameters, BF16), you need an Nvidia GPU with vLLM. There are GGUF files for llama.cpp and an MLX version for Apple machines. One user on the openHAB forum runs it on an 8 GB vGPU. The code is licensed under Apache 2.0, the weights under NetEase's own model license. With 600 million parameters, Nemotron 3.5 is the least demanding. According to the independent study, an int4 version (0.67 GB) even runs on the CPU. Voxtral Realtime needs a GPU with at least 16 GB according to Mistral and is licensed under Apache 2.0. We didn't run our own test with German audio for this article.

The bottom line

For live captions, what matters most is that the text stays put. Half a second of delay is less annoying than flicker. That goes double for interpreting pipelines, because every correction triggers a new translation. For voice agents, on the other hand, what counts is the time after the end of a sentence, and that depends more on pause detection and the network than on the model.

Our take: “80 milliseconds” almost always just means the model works in 80 ms steps. It doesn't mean a word is fixed on screen 80 ms after it's spoken. In German, the shortest settings cost you roughly 1.5 to 5 percentage points of error rate compared with a one-second wait, depending on the model. The biggest gap is measurement itself. So far, neither the Model Cards nor the Open ASR Leaderboard track how often displayed text flips. The proposal for a streaming leaderboard in a fork of the leaderboard (as of November 1) includes streaming WER and a p95 finalization latency measured from the VAD-detected end of speech, with network time included for APIs. A flicker metric isn't defined yet. Until that changes, it pays to ask three questions about any real-time claim: measured from when, until when, and can the text still change afterwards?