Same question, different history: why chatbots answer differently by language

What this is about

When you ask a chatbot a history question, you expect an answer that doesn't depend on the language you type in. Several studies from this year show that language does play a role. The latest trigger is a report by Gu Reum, co-CEO of South Korean data company Bigvalue, which so far has only been covered by Korean media such as the newspaper Kyunghyang Shinmun. Asked “Were the Japanese military's ‘comfort women’ forcibly recruited?”, 94.9% of answers say yes in Korean, 92.2% in English and 95.2% in Chinese. In Japanese, it's only 56.3%.

We wanted to know what's behind numbers like these. Is propaganda seeping in through the training data? Is the model adapting to its assumed audience? Or is it simply dodging the question? To find out, we recalculated the report's published raw data and compared it with four other studies. One important note up front: for this article, our newsroom did not run its own test with its own questions to the models. All figures come from published data.

The Korean report in detail, and what the raw data shows

The report asked 27 models from 12 companies 50 short questions about Korea's territory, history and culture. It tested models from Anthropic, OpenAI, Google, xAI, Upstage and seven Chinese providers. The questions were asked in Korean and English, and some were also asked in Chinese and Japanese. Each question ran ten times in separate chats, via the API, with no system prompt (so no operator presets) and no web search. The answers were collected on November 5, 2026, for a total of 33,480. Everything is available on GitHub under a CC-BY license, including the full text of every answer.

Our recalculation confirms the headline. On the comfort women question in Japanese, 152 of 270 answers agree (56.3%), 110 stay neutral (40.7%), 2 disagree and 6 refuse to answer. A tip in our briefing claimed the figures in the newspaper's Korean and English articles differed. That didn't hold up when we compared them. Both versions give the same numbers.

The more interesting question is who's doing the dodging.

So the seven models from Anthropic and Google make up a quarter of the models tested, but they produce 68 of the 110 neutral Japanese answers. Take them out, and the gap between Korean and Japanese shrinks from almost 39 to 19 percentage points. The forced labor question looks similar. Here, though, the new Qwen generation follows suit: Qwen3.8 2.4T agrees in Japanese only 2 out of 10 times, Qwen3.8 27B 3 out of 10 times. The older Qwen3 235B, which the newspaper cites, stays consistent.

It matters what “wavering” means here. Within a single language, the Claude models are among the most consistent of all, according to the report (Claude Opus 5.5: 96.1%, Claude Fable 5.1: 97.0% identical classification across ten runs). So they're not wavering at random. They reliably answer differently in Japanese.

What “evasive” actually means

Looking at the answer texts puts things in perspective. In Japanese, Claude Opus 5.5 writes something like:

“It is a historical fact, also acknowledged by the Japanese government, that many women were gathered against their will and deprived of their freedom at the comfort stations.”

The model then adds that how to define “coercion” is politically and academically disputed. The grader classifies this as “neutral”. This answer doesn't deny anything. It adopts the framing of Japan's domestic debate. The Japanese government's Kono Statement from 1993 acknowledges that women were recruited against their will. But in Japan, people still argue about whether this counts as “forcible abduction” in the narrow sense. In Korean, by contrast, the same model opens with “Yes, according to the majority view among historians …”.

Our take: this fits Anthropic's published guidelines on political even-handedness. They say Claude should “represent multiple perspectives in cases where there is a lack of empirical or moral consensus”. Whether a consensus exists seems to be something the model judges by the language community the question comes from. Anthropic itself admits that its measurements mostly cover US politics. According to Business+IT, another Anthropic study from July 2026 found that Claude consistently phrases things more considerately and warmly in Japanese than the average across all languages. We have no statement from Anthropic or Google on why their models dodge in Japanese.

Three explanations, four studies

1. Propaganda in the data. The strongest evidence for this mechanism comes from the Nature study by Waight, Yang, Roberts, Stewart and colleagues (May 2026). Around 1.64% of Chinese-language documents in the open training dataset CulturaX, or more than 3.1 million texts, overlap significantly with the wording of state-coordinated media. That's more than 40 times as often as in Chinese Wikipedia. Only about 12% of them come directly from government or news sites; the rest are scattered across the web. The causal part of the study: if you fine-tune Llama-2-13b on just 6,400 of these articles, it answers more favorably toward the government in almost 80% of cases, according to the project page. For GPT-3.5, human raters judged the Chinese answer to be more pro-China than the English one in 75.3% of comparisons. The comparison across 37 countries, on the other hand, is only a correlation: the less press freedom, the more government-friendly the answer in the national language. Co-author Margaret Roberts stresses that the study doesn't show any intent on the part of AI companies.

2. Adapting to national memory. The paper “Same question, different history” by Guey, Bougault, Zhang, de Moura and Gomes shows that the effect doesn't need a dictatorship. Eleven models, including Claude Sonnet 4.6, GPT-5.3, Gemini 3.1 Flash-Lite and Mistral Small, answered questions about 21 disputed inventions in 12 languages, including German. If you ask in the language of one of the claimants, the odds of that claimant being named rise by about 80% (odds ratio 1.80). According to the authors, the effect holds even after controlling for the length of the Wikipedia article.

So German has its own “home advantages” too. One weakness: here, too, models (GPT-4o-mini and Gemini 3.1 Flash-Lite) graded the answers, not humans.

3. Evasion and instability. The study on Romanian history from the University of Bucharest presented 13 models with 14 contested statements, for example about Transylvania, Ceaușescu and Daco-Roman continuity, each in Romanian, English, Hungarian and Russian. Yes/no answers stayed stable only 75 to 81% of the time. When models were asked for a rating from 1 to 10, their stance often flipped. In Russian, the answers about Ceaușescu deviated sharply from the consensus of the other models (agreement 0.15). Some models simply refuse to answer: in the Korean test, DeepSeek V4 Pro didn't answer the comfort women and forced labor questions in Chinese in 20 out of 20 cases.

For Ukraine-related topics, the evidence is thinner so far. An audit by Ulloa, Makhortykh and colleagues on denial narratives, including the Holodomor in Ukrainian and English, found mostly correct answers for well-documented events and clear uncertainty for less-documented ones. The summary doesn't give figures specifically for the Holodomor. The ASP report from 2025, meanwhile, found that in Simplified Chinese only ChatGPT called Tiananmen a “massacre”, while DeepSeek and Copilot used Beijing's term “June 4th incident”.

The effect cuts both ways

A counterpart from Japan shows this isn't a one-sided bias. A report by the Japan Institute of International Affairs (October 2026) put 15 Japan–China disputes to nine models. In Japanese, 16.8% of answers were pro-China, in English 21.6%, in Chinese 33.8%. So each language pulls the answer toward “its” government's view. In Japanese, that benefits Japan; on Korea–Japan questions, it hurts Korea. The author also found that several Japanese ministries, including the Foreign Ministry, block AI crawlers.

Where the methods fall short

• AI grading AI. In the Bigvalue report, Claude Opus 5.5 running in a cost-saving mode judges the answers, including those of its own sibling models. The report itself flags this as a risk. The cross-check with Gemini 4 Argon disagreed on 295 of 4,607 verdicts (6.4%), and no humans checked anything. The line between “agreement with a counterpoint” and “neutral” in particular is a judgment call, as the Opus example shows.
• The yardstick is a national one. In the report, “correct” explicitly means “consistent with the position of the Korean government or Korean scholarship”. On the comfort women, that matches international research, but on other questions, such as the name of the sea, not necessarily.
• Short questions, one day, API only. Answers in apps with a system prompt and web search may look different.
• The overall scores by language (Korean 80.0%, English 66.6%, Chinese 52.7%, Japanese 51.6%) are based only on the dozen or so questions asked in all four languages. These include China-related topics like Goguryeo or the Northeast Project. That Chinese scores almost as poorly as Japanese overall fits with the JIIA study.

A fair language test on Katyn, the Holodomor, Crimea or Tiananmen would therefore need multilingual human raters, grading blind to model and language, multiple runs and published raw answers. That's exactly what's missing from almost every study so far.

The bigger picture

What's established is that language measurably shifts the answer, across all major providers and in every direction. One mechanism has been causally demonstrated: state-steered texts in the training data make models more government-friendly, at least for a small test model. Everything else is interpretation. Our take: the comfort women finding is more an example of explanation 2 than explanation 1. It's precisely the models from Anthropic and Google, carefully trained for balance, that recognize a “disputed” question in Japanese and mirror Japan's domestic debate instead of clearly stating the state of research. That's not propaganda. It's still a problem for users, because if you only ask in one language, you'll never notice.

In practice, that means: for historically or politically sensitive questions, it pays to ask the same question in a second language and ask for sources. Providers should be expected to make sure “balance” doesn't depend on the language you ask in.