Rolling dice, not scanning: why AI bug hunters almost never agree
Three frontier models searched the same code for vulnerabilities and agreed on just one of 121 findings. Even the same scanner returns something different every time on identical code. We pulled the measurements together and explain what this means for defenders, especially now that Anthropic is shortening its disclosure deadline.

What this is about
This week's headlines are all about hit counts: according to Forbes, GPT-6 Cyber found around 35 vulnerabilities in Apple devices. The startup Calif reports a serious Java flaw it calls “Java4Shell.” GitHub and Microsoft used AI to generate 9,791 alerts across six open-source projects. AI scanning is fast becoming standard practice. We're asking a different question: how often do these systems find the same vulnerabilities?
The answer is uncomfortable. Several independent measurements show that different models almost never agree on the same code. Even the same model rarely reaches the same result across multiple runs. An AI scan is more like a random sample than a survey. That has consequences for how defenders should use these tools.
Three models, 121 findings, one shared hit
The security firm Echo published a study today. In it, Anthropic's Claude Mythos 5.1, OpenAI's GPT-5.6 Sol (accessed through OpenAI's Daybreak program for vetted security teams) and Z.ai's GLM-5.3 ran through the same search harness and the same validation. The targets were two Java libraries from the Maven repository and an npm package that, according to Echo, has more than 350 million monthly downloads. The names are being withheld until the project maintainers have been notified.
The numbers, according to Infosecurity Magazine, AI Journal and Tecnología Geek:
- The nine runs produced 446 initial reports. After weeding out duplicates, unreachable code, documented behavior and false positives, 121 distinct potential issues remained.
- 1 of them was reported by all three models, 20 by two, and 100 (83%) by just one.
- Initially, all three models had jointly flagged 8 items. Only one of them survived validation.
- The models rated 45% of the 121 findings as high or critical; after testing, only 26% kept that rating. Automated proof-of-concept tests, small programs that actually trigger a vulnerability, confirmed 31 of 55 high-rated findings.
- One bug in the npm package was rated critical by GLM-5.3 and high by Mythos. GPT-5.6 Sol considered it documented behavior. Validation ultimately rated it low.
Mythos had the most validated findings (25, versus 9 for GLM-5.3 and 8 for GPT-5.6 Sol). But it was also the most expensive and slowest: about $303 and 198 minutes, compared with roughly $30 and 24 minutes for GPT-5.6 Sol and $22 for GLM-5.3. Divide cost by validated findings and a hit costs about $12 with Mythos and about $2.40 with GLM-5.3. That calculation is ours and it's rough. Still, GLM-5.3 found six confirmed issues in one library that both competitors missed entirely.
“The defender may never see the vulnerability the attacker is about to exploit.” – Yuval Saggie, Echo
Same scanner, three runs, three results
Echo ran each model only once. So some of the disagreement could simply be randomness within a single model. That's exactly what Contrast Security measured. In the first part of its study (August), three AI scanners agreed on only about 5% of findings on a two-million-line codebase. A single scanner reproduced around 17% of its own results on a rerun. Contrast puts the API cost of the scan at about $315 and the effort to triage the findings at around $128,000. The report on Help Net Security doesn't explain how that figure was calculated.
In the second part, Contrast CISO David Lindner scanned an 80,000-line application three times on the same day with Claude Security on Mythos 5, each time on the same commit:
| Run | Findings | Cost | Duration | of which HIGH |
|---|---|---|---|---|
| 1 | 29 | $250 | 37 min | 11 |
| 2 | 15 | $225 | 30 min | 4 |
| 3 | 21 | $255 | 50 min | 4 |
That adds up to 42 distinct candidates. Of those, 5 appeared in all three runs (12%) and 24 only once (57%). Just 2 showed up three times with the same severity rating. At $730 total cost, that works out to $146 per reproducible finding and $365 per truly stable one. For comparison, Contrast cites 17% agreement for Sonnet and 25% for Opus in part one, though on a different codebase. Important: none of the 42 findings was verified as real or false. Only consistency was measured.
The pattern shows up in other measurements too
| Study (vendor) | Setup | Result |
|---|---|---|
| Snyk VulnBench JS 1.0 | 10 JavaScript projects, 6 configurations, 5 runs each = 300 scans | Hits matching the reference findings appeared in all five runs 84.8% of the time. Additional reports only 13.7%, half of them just once |
| HoF-Bench (arXiv 2607.27030) | 95 real AI-discovered CVEs (publicly registered vulnerabilities), 10 smaller models, 4 passes each | Best model: 55% on the first pass, 68% after four. One run each from the four best models found 68 CVEs, more than four runs of the best model |
| Bright Security | 300-line app, 5 reviews with Claude Opus 4.6 | Only 32% of vulnerabilities found in all five scans, 60% of findings were false positives |
| Semgrep (2025) | Claude Code and Codex on Python apps | Three identical runs produced 3, 6 and 11 findings that only partially overlapped |
Snyk provides the most important distinction: real, known patterns are found fairly consistently. The noise is unstable. That fits a hobby experiment by a developer on dev.to, who had a local model review the same code 51 times. A majority vote across many runs selects what the model reports persistently, not what's true. Real bugs dropped out because the model gave the line number one way one time and another way the next. Harmless warnings, meanwhile, stuck around. Echo's data shows the same thing: of the eight items all three models initially flagged together, only one turned out to be real.
Why the scanners roll dice
Agentic scanners are AI systems that open files, call tools and pursue hypotheses on their own. They don't read a large codebase in full. They pick and choose. Lindner puts it this way:
“It picks a starting point, and every decision after that narrows what it will ever get to see. […] Three runs took three different paths through the same code.”
According to Lindner, lowering the temperature, meaning less randomness in word choice, doesn't fix this. The order of tool calls and the way context is built still change. Semgrep names two more causes. “Context rot” means the model gets worse at retrieving information from an overstuffed context window. “Compaction” is a lossy summary in which function names or paths get lost. Anthropic itself states in the Claude Security documentation that scans are “non-deterministic: two scans of the same code can produce different findings.” For large repositories, Anthropic recommends scanning areas individually, setting effort to “high,” or simply scanning again.
Our take: a single AI scan is one path through the code, not a map. A clean report only proves there was nothing along that one path.
What defenders should take from this
- Multiple runs, ideally multiple models. HoF-Bench shows that different models yield more than repeated runs of the same model, because their blind spots only partially overlap. Echo's cost figures suggest that cheap models are affordable as a supplement. That said, Echo sells exactly this multi-model approach.
- Not as a gate before merging. Contrast advises against using a non-deterministic scanner as a merge gate, a mandatory check before code is accepted. The decision would then hinge on random noise rather than the actual security posture. Snyk reaches the same conclusion. Even Anthropic, in its CI pipeline overview, still lists classic static analysis, rule-based checking tools that behave the same on every run. The AI is meant to complement them, not replace them. AI scans are better suited as a regular automated pentest.
- Budget for triage. In the GitHub and Microsoft pilot, 9,791 alerts became 99 reports, roughly 1 percent. Seven researchers needed about two days per project for manual review. That resulted in 55 fixes by November 15, 14 of them already released. One Netty maintainer said: “Finding vulnerabilities is no longer our bottleneck. Fixing them is the hard part.” According to a September analysis by Patrick Garrity for VulnCheck, Anthropic had reported 26,153 findings in Project Glasswing, of which 202 (0.8%) were considered fixed.
Attackers see different holes, and the clock is shrinking
Starting January 1, 2027, Anthropic will publish high and critical vulnerabilities 30 days after reporting instead of 90, and all others after 45 days. The rationale: a vulnerability one model finds is “increasingly likely to be found by another as well.” Anthropic doesn't provide numbers for this.
At first glance, that contradicts the Echo study. Our take: the two fit together. A single run draws only a small sample. But many actors running many models many times collectively cover a large share, as the rising rates in HoF-Bench show. The asymmetry lies elsewhere. An attacker needs only one vulnerability that the defender didn't see in their sample. So if you scan once with one model, you have no guarantee of knowing the hole an adversary is finding right now with a different model. For maintainers, the shorter deadline means more pressure. Anthropic acknowledges as much.
The bottom line
The data is more consistent than you might expect: Echo, Contrast, Snyk, Bright, Semgrep and the HoF-Bench team arrive at the same pattern using different methods. Still, there are three caveats. First, almost all of the sources sell competing or complementary products, such as runtime protection, classic code analysis or multi-model services. Second, the samples are small: three projects at Echo, one codebase at Contrast, a 300-line app at Bright. Third, Echo doesn't separate differences between models from randomness within a model, because each ran only once. The models' “1 of 121” disagreement is probably due in large part to dice-rolling, not different capabilities.
What's established is that AI scanners do find real vulnerabilities, just a different slice each time. What's open is how many runs and models you need before coverage is good enough, and who pays for the triage. Our verdict: an AI scan is a search party, not a seal of approval. Treat it like a seal of approval and you're lulling yourself into a false sense of security.
Sources
- Tecnología Geek: Modelos de IA solo coinciden en una de 121 vulnerabilidades tecnologiageek.com
- The AI Journal: Three Leading AI Models … Agreed on Just 1 in 121 aijourn.com
- Echo: Frontier Research (Berichtsseite) echo.ai
- Contrast Labs: Mythos 5 – The determinism problem (Teil 2) contrastsecurity.com
- Help Net Security: Contrast Security AI AppSec tools report helpnetsecurity.com
- Resultsense: AI AppSec scanners disagree resultsense.com
- Snyk Blog: VulnBench JS 1.0 – LLM security review repeatability snyk.io
- arXiv: HoF-Bench – Rediscovering Real AI-Discovered CVEs arxiv.org
- alphaXiv: HoF-Bench alphaxiv.org
- Bright Security: AI security review fails in practice brightsec.com
- Semgrep: Finding vulnerabilities using Claude Code and OpenAI Codex semgrep.dev
- dev.to: Local LLM reviewed the same code 51 times dev.to
- Claude Code Docs: Claude Security code.claude.com
- GitHub Blog: Maintainers don't need more AI bug reports. They need fixes. github.blog
- DiarioBitcoin: GitHub y Microsoft transforman 9.791 alertas de IA en 55 correcciones diariobitcoin.com
- Anthropic: Coordinated vulnerability disclosure timelines anthropic.com
- The Next Web: Anthropic cuts its bug disclosure window to 30 days thenextweb.com
- VulnCheck: Anthropic Glasswing receipts vulncheck.com
Ask the AI assistant
Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.
The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.
Comments
What do you think? Already tried it? Write it here, no sign-up needed.
Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.