“A human stays involved”: what the military AI pledges actually commit to

Within two days, the UN in Geneva postponed a ban on autonomous weapons, and Washington announced an understanding with Beijing on “human involvement” in military AI decisions. We compared the wording of eight documents and checked it against cases from Gaza, Iran and Ukraine. None of the pledges specifies how much control a human must actually have.

Analysis Nov 21, 2026 11 min read Original analysis by our editors
Illustrative image, AI-generated

What this is about

Three news items from the same week seem to tell a story of progress. The United Nations wants to draw up rules for “killer robots.” US President Donald Trump and Chinese leader Xi Jinping have reportedly agreed that a human will always be involved in military AI decisions. And OpenAI is working more closely with NATO, within clearly defined limits, as the company puts it.

We put the wording side by side. Our argument: none of these pledges says how much control a human must have. Anything fits in the gap between genuine deliberation and rubber-stamping a machine's suggestion in seconds. In our assessment, states are keeping that gap open on purpose.

Eight texts compared

Document Key wording (translated) Binding? What's left open
Biden–Xi, Lima, Nov 16, 2024 (White House readout) “Need to maintain human control over the decision to use nuclear weapons” No, political declaration of intent Applies only to nuclear weapons
Trump–Xi, Shenzhen, Nov 19, 2026 (only Bessent on CNBC) Humans should “always be involved” in military decisions by “SI.” SI should not be allowed to act autonomously in “major military actions” No, there is no document What “involved” and “major” mean
Pentagon Directive 3000.09 (Jan 25, 2023) Systems should allow “appropriate levels of human judgment over the use of force” Internal to the Defense Department only According to a 2018 US white paper, “appropriate” is deliberately “a flexible term”
China, CCW working paper 2022 A weapon is “unacceptable” only if it meets all five criteria at once: lethal, fully autonomous with no possibility of intervention, impossible to terminate, indiscriminate, and evolving in an uncontrolled way No, position paper Everything else counts as “acceptable”
US Political Declaration on Responsible Military Use of AI (2023) “Responsible human chain of command and control,” “appropriate, context-informed judgments” No, the text says it creates no new obligations The nuclear weapons clause in the draft was removed
Final report of the Geneva expert group (2026) “Control and human judgment” are required, exercised “directly or indirectly, including through measures before and/or during use” No Control before use is enough
CCW conference decision, Nov 21, 2026 States begin “the elaboration of an international instrument” within two years Legal form open “Elaborate” instead of “negotiate”
OpenAI–Pentagon contract (Feb/Mar 2026) No independent control of autonomous weapons “where law, regulation, or Department policy requires human control” Contractual, full text not public See below

The art of vague words

The table reveals a pattern: where a pledge gets specific, its scope shrinks. Where its scope is broad, it gets vague.

Biden and Xi spoke of “human control” in 2024, but only over the decision to use nuclear weapons. Biden's national security adviser Jake Sullivan himself conceded that nobody wanted to hand control of nuclear weapons to an AI anytime soon. So the pledge cost little.

Trump and Xi, if you take Bessent's word for it, extend this to all military AI decisions. But every detail is missing. According to Reuters, the Treasury Secretary cited no joint document and no rules on which decisions must remain with humans. Bessent spoke of “SI,” the administration's term for superintelligence. In the Chinese readout of the November 19 meeting, published by the foreign ministry in Beijing, Xi only says that AI safety risks should be prevented and that the sides should “ensure that AI always remains under human control.” The military doesn't appear in it. We found no Chinese confirmation of the military pledge.

Our take: Bessent's second sentence deserves a closer read. If AI shouldn't act autonomously in “major military actions,” the question of all the minor ones stays open. And that's exactly where today's practice lies: the individual drone, the individual target.

The Pentagon has spoken of “appropriate human judgment” since 2012, not of control. According to Human Rights Watch, the 2023 revision “does not absolutely prohibit the development, production, and use of any autonomous weapons system.” The mandatory review by senior officials can be waived in cases of “urgent military need.” The directive also doesn't apply to the CIA. The US has defended this line in Geneva, too. According to an analysis by the Lieber Institute at the West Point military academy, the US delegation rejected the term “human control” in March 2026 and proposed “human judgment and good-faith care” instead. More than 70 states, by contrast, had backed a binding control standard.

China looks stricter at first glance, because it calls for a ban on “unacceptable” weapons. But all five criteria have to be met at the same time. A weapon that doesn't “evolve in an uncontrolled way,” or that could theoretically be switched off, is already off the list. Writing for the Lieber Institute, Gerald Mako says Beijing has drawn a “remarkably narrow line” that excludes many new autonomous capabilities. He calls its stance “undeniably convenient and low-risk.” China says it will agree to a binding treaty “when conditions are ripe.”

The Geneva expert group settled on “control and human judgment” after three years. But according to an analysis on Opinio Juris, this control may be exercised “directly or indirectly,” “including through measures before and/or during use.” Struck from the text were the term “meaningful human control” (the demand of the Red Cross and many states), a cap on the number of autonomous attacks, and requirements for predictability and explainability. Our take: under this text, anyone who programs a weapon before launch is already exercising “control.”

Geneva: “elaborate” instead of “negotiate”

The Review Conference of the UN Convention on Certain Conventional Weapons, or CCW (a treaty restricting particularly cruel conventional weapons), met at the Palais des Nations from November 16 to 20. It didn't wrap up until around 4:45 a.m. on Saturday after an all-night session. Conference president Julien Thöni of Switzerland announced that states had agreed to begin “the elaboration of an international instrument” on lethal autonomous weapons “within the next two years.” According to AFP, “some countries” had rejected the word “negotiate.” Thöni didn't say which. The official text (document CCW/CONF.VII/11) was not yet available at press time. The quotes come from wire reports.

Who applied the brakes is still well documented. According to Reuters, the US, Russia and Israel opposed binding rules, even though around 100 states wanted them. Russia blocked the agenda on the very first day because civil society observers were to speak in only a single session. After that, talks were held informally and without UN Web TV coverage. On November 4, the US had submitted a working paper calling for five more years of expert work on a non-binding instrument, with a report due only in 2031. Its reasoning: existing international humanitarian law is sufficient, and autonomous functions could even improve compliance with it. Because the CCW operates by consensus, any single state can block a decision.

The compromise is shorter than the US proposal, but it doesn't rule out its core. According to UN disarmament chief Izumi Nakamitsu, a legally binding instrument is “still possible.” The Stop Killer Robots campaign calls the outcome a “watered-down mandate,” and Verity Coyle of Human Rights Watch calls it “disappointing.” The word isn't new, by the way: according to HRW, back in 2021 the conference had likewise only agreed to “elaborate” possible measures.

What practice shows: seconds, 500 meters, a school

Formally, a human was involved in every one of the following cases.

Lavender (Gaza): According to +972 Magazine and Local Call in April 2024, the Israeli system flagged up to 37,000 Palestinians as suspected militants. The army allegedly knew it was wrong in about ten percent of cases. One officer said he spent “20 seconds for each target,” adding: “We only checked that it was a man.” The Israeli military disputes this account and says its systems are tools for analysts and don't make independent targeting decisions. Taken together, both sides illustrate the problem: under every text compared here, even the 20-second process would count as “human involvement.”

Minab (Iran): On the first day of the Iran war in February 2026, two Tomahawk missiles hit an elementary school. More than 150 people died, including at least 123 children. Gizmodo, citing Bloomberg and an unpublished Pentagon investigation, reported that parts of Central Command relied too heavily on Palantir's Maven targeting system, which recommended the school as a day-one target. The teams responsible for preventing civilian harm had shrunk by about 90 percent, at Central Command from ten people to one. No one from those teams had checked the site. With more than 1,000 strikes in the first 24 hours, there was barely time for checks. Palantir says it isn't responsible for the underlying data.

The Vals test: The evaluation firm Vals had large language models pick military targets on satellite images. According to Bloomberg, Claude Opus 5.5 chose the Minab school in 22 of 32 runs, and in 23 in a scenario disguised as a “game.” OpenAI models initially never chose it, but did so 9, 11 and 13 times in the game scenario. According to IBTimes, no safety filter kicked in. The test doesn't show whether any of the models were involved in the actual strike, and the full report isn't public.

Ukraine: Here you can see autonomy creeping in. The TFL-1 module from Kyiv-based company The Fourth Law takes over steering the drone for roughly the last 500 meters, in series production since 2025, according to the Kyiv Post. According to the Kyiv Independent, the pilot selects the target in the video feed. If jammers cut the radio link, the drone keeps flying on its own. Kateryna Bondar of the CSIS think tank says distinguishing between soldiers and civilians is “not solved.” In September 2026, Al Jazeera, citing a New York Times analysis, reported on a Russian drone that apparently found and attacked its target fully autonomously. Researcher Marina Miron puts it this way:

“Under time pressure, commanders often do what the AI suggests.”

This phenomenon is called automation bias. The 2023 US declaration aims to “minimize” it, but sets no benchmark for doing so.

Where OpenAI fits in

On November 17, OpenAI gave NATO's NCIA agency access to Daybreak, its cyber defense tool. According to the company, NATO uses up to 300 billion tokens a month. CERT Polska used it to find six exploited vulnerabilities in MikroTik routers. That's defending IT systems, not a weapon. According to trade reports, however, the contract terms aren't public.

More revealing is OpenAI's Pentagon contract from February 2026. The models are not to independently control autonomous weapons “where law, regulation, or Department policy requires human control.” Our take: the relevant Directive 3000.09 specifically does not require “control,” only “appropriate judgment,” and the US government explicitly rejected the concept of control in Geneva. The clause therefore points to an obligation that doesn't exist in that form. Researcher Sarah Shoker told Built In that the language is designed to “keep options open” for leadership. OpenAI also points out that the models run only in the cloud, not on devices at the front.

The bigger picture

Every text compared here promises human involvement. None specifies how much time, information and veto power that human must have, whether they must be able to intervene, or whether programming before launch is enough. The Red Cross asked these three questions back in 2018: How much supervision, how much predictability, what limits on target, location and duration? None of them has been answered.

Our take: this gap is no accident. The US wants “judgment” instead of “control,” China defines bans so narrowly that hardly anything falls under them, Russia stalls the process, and the Trump–Xi understanding so far exists only as a remark in an interview. That way, every side can profess responsibility without restricting its own arms buildup. For Minab and the cases from the Lavender investigation, the fact remains: a human was involved, and civilians still died in large numbers.

The Geneva expert group is now supposed to draft an instrument by 2028. Whether it pins down more than the existing texts comes down to a single question: will states, for the first time, write down what a human must at minimum be able to check before a machine kills? If not, 20 seconds remains a fully compliant process.

Sources

#military ai#autonomous weapons#regulation#us-china#un

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.