Gemini refused the image – then the download button handed it over anyway

What happened

On November 5, 2026, the security firm Fog Security disclosed a flaw in Google's AI assistant Gemini that nobody would have expected. Gemini dutifully refused an image request because it broke the rules, then served the refused image through the download button anyway. The web version at gemini.google.com/app was affected. According to Cyber Press, researcher Jason Kao found the issue and reported it to Google in June 2026.

The twist is that nobody had to trick the model. There was no jailbreak, meaning no attempt to talk the AI into breaking its rules with clever wording. There was no prompt injection either, where hidden instructions change the AI's behavior. The safety filters made the right call. What failed was the infrastructure, the plumbing that delivers generated images to the user.

How the flaw worked

Every image Gemini generated could be reached two ways: as a smaller preview right in the chat, and as a full-size image behind the download icon that sits on top of the preview. Both were supposed to point to the same image. They didn't always.

Under the hood, both versions lived as signed URLs on Google's server lh3.googleusercontent.com. These links only work for the signed-in user and won't open in an incognito window. The display URL ended in =s1024-rj, the download URL in =s0-d-I. The safety filter's refusal only applied to what was shown in the chat. The download link was resolved separately, and that's where the blocked image slipped through.

The researchers' method was simple:

1. They asked Gemini for an image that triggers the safety filter, for example using an uploaded portrait photo of the researcher.
2. Then they asked for a different, harmless image.
3. The harmless image appeared in the chat, but the download button delivered the one that had been refused.

Results varied. Sometimes they got a half-finished intermediate image (blurry or misaligned), sometimes a fully finished refused image, and sometimes nothing unusual at all. The researchers say they reproduced complete prohibited images across several days and sessions.

A copyright case made it most obvious:

We asked Gemini to draw Mickey Mouse, and it politely declined. Then it gave us Mickey Mouse anyway.

Fog Security suspects a state management bug: the download served a different image from the one the safety filter had judged. That fits a race condition, where several processes run at the same time and compete. Here the competitors would be image generation, the safety abort and the deletion of intermediate files. CyberSecurityNews stresses that this explanation hasn't been confirmed.

Why this is a security problem

By classic standards, it's hard to say which boundary was crossed. No other users' data was exposed, nobody gained elevated privileges, and there was no classic victim. Fog Security argues that's exactly the problem. For categories like deepfakes, non-consensual intimate imagery, intellectual property and child safety, the model's refusal is often the only line of defense. If someone uploads a photo of another person and asks for a sexualized edit, the AI's "no" is the only thing in the way. If that no still ends up delivering an image, it protects nobody.

On top of that, the bug was invisible. In both cases, the user sees the same polite refusal. An enforced refusal and an unenforced one look identical. Even someone who never wanted the image only finds out once it's sitting in their downloads folder.

And because the model didn't make a mistake, Fog Security says better prompt filters, patched jailbreaks and stricter guardrails won't help. The flaw sat between the model and the user, in an area nobody really feels responsible for. Problems like this can be "triaged, rejected and closed without anyone taking ownership."

What was tested – and what wasn't

The researchers deliberately limited themselves to content with no affected third parties: a copyrighted character, an AI-generated fictional person and the researcher's own face. They say they did not test child safety categories under any circumstances, and they didn't use images of other real people either. Jason Kao documented the process in two videos in June 2026.

The symptom itself wasn't new. Since at least December 2025, users on Google's support forum and on Reddit had reported that the download button saved a different image from the one displayed. People dismissed it as user error, and nobody connected it to refused content. CyberSecurityNews notes that these older reports don't prove a safety filter bypass on their own. Fog Security sums it up: the UX bug was a security bug all along.

Google's sluggish response

The disclosure timeline doesn't make Google's Vulnerability Reward Program (VRP) look good. That's the program researchers use to report security flaws and potentially get paid for them:

Google didn't say exactly when the change was made. It also never responded to the draft, even though it had offered to share it with internal teams and possibly provide feedback.

What Fog Security recommends

The researchers have two concrete recommendations for Google:

• The download must be tied to exactly the finished image that's displayed, so unfinished intermediate states can't be retrieved.
• Files from refused or failed generations must be deleted so they can't be retrieved anymore.

Our take

This case is smaller than a data breach, but it matters more than it first seems. Most AI safety research focuses on talking models into breaking their rules. Here the model did everything right, and the prohibited output still reached the user. For anyone building image and video AI, the lesson is that safety decisions have to apply on every path content takes to the user: previews, downloads, stored files and links, not just the chat response.

Google's handling of the report deserves criticism. An issue that gets bounced for months between "not our area" and "routine maintenance" shows the very gap Fog Security describes: there's apparently no clear owner for bugs that sit between the model and the infrastructure. Several questions remain open. Did Google deliberately fix the root cause, or did the bug just happen to disappear? How long did the flaw exist? Was it actually exploited? There's no evidence of abuse. According to Fog Security's tests, the behavior no longer occurs.