“Mumbled”: Audio AI Understands Accents Perfectly, Then Grades Them Down

What it's about

Imagine you're doing a job interview with a voice AI. Your answers are word for word the same as those of another applicant from Ohio. You just sound different. In the end, you get the lower score, and the explanation says you “mumbled.” That's exactly the scenario a team led by Aruna Srivastava, Orevaoghene Ahia and Yulia Tsvetkov at the University of Washington put to the test. Their paper, “The Cost of Sounding Different: Accent Bias in Audio Language Models,” was presented on October 25 at EMNLP in Budapest, one of the most important conferences in natural language processing. According to the code repository, it won an Outstanding Paper Award there. The university's press release followed on November 4.

So far, most coverage has simply retold the paper. We read it ourselves. The upshot: the core finding is solid, but one detail is being misreported.

The setup: same text, different voice

The researchers recreated three situations in which audio models could already be judging people today: - asynchronous AI job interviews (recorded answers to four questions) - seminar and conference talks (business, computer science, history, biology) - English test tasks, namely reading aloud and describing graphs

The English test tasks are modeled on Pearson's PTE Academic. The paper cites it as an example of a test that uses speech AI to score answers.

The main experiments used 30 synthetic voices from ElevenLabs, a commercial AI voice provider. There were six voices each (three female, three male) for American, British, Chinese, Indian and Nigerian English, for a total of 599 recordings and about 19 hours of audio. People with firsthand familiarity with each accent vetted every voice beforehand. As a control, 52 real people recorded the job interview scenario: 13 from the US, 4 from the UK, 11 from China, 10 from India and 14 from Nigeria, for 312 recordings in total.

The models had to score delivery and content separately on a scale of 1 to 7 and justify their scores. Temperature was set to 0, so the models responded without any randomness. Six models were tested. Two were dropped: - Audio Flamingo 3 classified every speaker as a native speaker. - Voxtral Small 24B produced unstable results. For 15 of 30 measurements, the sign flipped between two identical runs.

That left Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-Audio 1.5 and Alibaba's open-weight model Qwen3-Omni.

The numbers: a hierarchy of accents

The ranking was almost always the same: American and British at the top, Indian in the middle, Chinese and Nigerian at the bottom. The paper's main text doesn't give individual values. Figure 2, however, shows the gap between the “non-Western” group (Chinese, Nigerian, Indian) and the “Western” group (US, UK) as a z-score, meaning in standard deviations of each model's score distribution. A value of −1 means a full standard deviation worse. We read the raw values off the figure. The ranges per domain can be stated unambiguously:

According to the paper, the gaps are largest on the English tests, exactly where university admissions or visas can be at stake. The text says the few positive values come from Qwen3-Omni. For talks and tests, that model sometimes gave non-Western speakers similar or even slightly better scores.

Content scores were affected too: identical sentences received different content scores depending on the accent. According to the authors, the models didn't reliably follow the instruction to ignore delivery when scoring content. The researchers also tested three framings of the task: critical feedback, comparison with an “ideal employee” and comparison with a “native speaker.” The gap showed up in all three.

Understanding, yes. Fair judgment, no.

The key finding is the word error rate (WER, the share of words transcribed incorrectly):

All values are below 4 percent. Indian English was actually transcribed most accurately, yet it still landed only in the middle on scores. Word error rate correlates only weakly with scores (Spearman correlation of about −0.15 for Gemini 2.5 Pro and −0.37 for Qwen3-Omni). The link to “acoustic distance” is much stronger. Using the speech model XLS-R, the researchers measured how far a pronunciation sounds from American English. Here the correlations ranged from −0.35 to −0.66 for real speakers and from −0.39 to −0.72 for synthetic ones, all statistically significant. British and Indian English are acoustically close to American English, while Chinese and Nigerian English are considerably further away. The authors call this an “implicit American norm.”

The models' explanations show how this plays out. Americans and Brits sound “clear,” “excellent,” “professional.” For Nigerian speakers, GPT-Audio 1.5 writes “mumbled or not fully enunciated,” while Qwen3-Omni describes pronunciation that is “occasionally muffled by a strong accent.” The word “accent” appears in only about 10 to 30 percent of the explanations. The downgrade usually hides behind words like “unclear.” How positive or negative an explanation sounds tracks closely with the score (Pearson r = 0.52 to 0.95).

“They understood the speakers, but they had this implicit notion of a hierarchy between them based on accent.” – Orevaoghene Ahia, co-first author

What the study doesn't show, and a common misunderstanding

There were no human raters. The UW release says the gap was larger for human accents than for AI accents. That refers to real speakers versus synthetic voices, not to humans handing out scores. In the paper, humans appear only as speakers, as voice vetters and as reviewers of 50 automatically analyzed explanations. The study doesn't tell us whether human recruiters would judge even more harshly. A separate paper from Interspeech 2026 (arXiv 2603.16941) did find that human raters perceived accent differences more sharply than AI raters. But there, they were rating the responses of speech AIs, not the speakers themselves.

Synthetic voices are a weakness, but not a decisive one. The authors acknowledge that ElevenLabs voices may over- or underemphasize some features of a pronunciation. For the three proprietary models, though, the gap stayed “relatively stable” across real and synthetic voices. For Qwen3-Omni, the gap in the explanations was actually considerably larger with real speakers, especially in the job interview scenario. So synthetic voices probably understate the problem for Qwen. That means you shouldn't read too much into the open model's good showing on talks and tests, since those domains were tested only with synthetic voices. A study from Taiwan (VIBE, arXiv 2604.17248), which uses only real recordings from more than 400 non-native speakers, also found statistically significant accent-driven shifts across twelve audio models.

Small samples and funding. There were only four real speakers from the UK, participants read from a script instead of speaking freely, and grouping by nationality is crude. According to the acknowledgments, the work was “partially funded by gifts from Google and OpenAI.” That needs to be disclosed, but it hardly undermines the results, since the funders' models perform poorly. The tests also covered the Gemini 2.5 generation. Whether newer models do better is an open question. In our research, we found no replication with more recent models or with German accents. Code and synthetic data are publicly available on GitHub and Hugging Face. Recordings of the real speakers are available only on request.

Where audio AI is already handing out scores

Vendors usually don't disclose which models they use internally, and the paper lists this as a limitation too. But the direction is clear: - HireVue launched an “AI Interviewer” on June 18, 2026. It conducts conversational job interviews around the clock and rates answers against scoring rubrics. - Greenhouse offers “Voice AI.” Recruiters set the questions and criteria, and the final decision stays with a human. - VidCruiter advertises that it scores only content, not speech patterns. - Pearson says the software behind PTE Academic, which is accepted for Australian visas among other things, scores “everyone equally and with the same accuracy,” no matter where they come from. The page offers no independent evidence for this.

These systems aren't necessarily the general-purpose models that were tested. Our take: precisely because companies like Pearson market machines as more neutral than human examiners, we need public breakdowns by accent group. Tsvetkov is calling for exactly that:

“We don't want performance to be 97% on average for the overall population, but only 20% for a smaller group.”

What the EU AI Act requires, and when

Annex III of the AI Act classifies AI as high-risk when it is used “to evaluate candidates” (point 4(a)) or “to evaluate learning outcomes” at educational institutions (point 3(b)). Such systems then face obligations that include risk management, checking training data for possible biases, and human oversight.

Contrary to what you often read, these obligations don't apply yet. The Digital Omnibus, Regulation (EU) 2026/1744, was published in the Official Journal on July 24, 2026 and has been in force since July 27. It pushes the start date for Annex III systems from August 2, 2026 to December 2, 2027. According to the law firms Garrigues and Gibson Dunn, it also broadens the legal basis for processing sensitive data to detect and correct biases. That basis now also covers providers that don't operate high-risk systems. In our reading, English tests for immigration don't automatically fall under the migration category (point 7), since it mainly covers systems used by public authorities or bodies acting on their behalf. That would require legal review case by case.

Analysis

The paper draws a clean line between understanding and judging. That removes the convenient excuse that speech recognition just needs to get better. The models understand Chinese and Nigerian English almost perfectly and still mark it down, with explanations that sound objective. This fits earlier work by co-author Valentin Hofmann on covert dialect prejudice in text models.

The effect is documented for four models from 2025, for read-aloud texts and for small groups of real speakers. What isn't documented is how strongly it shows up in specific commercial products, and whether humans would judge more fairly.

Our take: If you use voice AI for hiring or exams, you should strictly separate content from voice, for example by scoring only the transcript. The paper doesn't test that approach, though. It only shows that the transcripts are reliable and that telling a model that hears the voice to “score only the content” isn't enough. Until the EU obligations kick in at the end of 2027, fairness here depends on the goodwill of vendors.