Black hole holds up, Maya pyramid doesn't: how to vet AI discoveries

What's going on

This week, two stories went viral at the same time, both labeled "AI discovers." In one, a layperson with 48 AI agents supposedly found an unknown Maya pyramid in 72 hours. In the other, an AI model from Australia's national research agency CSIRO confirms the famous ring around the black hole at the center of the galaxy M87. The first story has since been debunked; the second holds up as things stand. This article explains why. It has little to do with smarter models. The answer lies in the verification process. At the end, you'll find a checklist to help you judge stories like these yourself.

First, a note about ourselves: we got it wrong too. Our story "48 AI agents comb NASA lidar data and find unknown Maya pyramid" relied on the first version of Pietro Miotti's blog post. But the site wasn't unknown; it has been documented in the scientific literature since 2025. The headline and the line "not mapped by anyone so far" were wrong. We shouldn't have read the agency's reply as confirmation. Why this mistake was so easy to make is at the heart of this article.

Case 1: The Maya site that was already known

By his own account, Miotti is not an archaeologist. According to The Verge (as cited by Jingletree and Mezha), he works as a researcher at Google and ran the whole thing as a personal side project. His blog post describes the process: the agents converted strips of public NASA lidar data into images, covering around 1,200 square kilometers. Lidar is an airborne laser scan that maps the terrain even beneath the forest canopy. 48 agents then examined these images one by one, following written instructions. There was no dedicated detection model. Next, the agents checked the point cloud, meaning the raw laser measurements, and finally "literature and archives." Their conclusion: a larger settlement apparently wasn't registered anywhere. On October 20, a report went to Mexico's National Institute of Anthropology and History (INAH). The reply on October 21 read, in translation:

"No registered archaeological site was identified at the location indicated in your report."

On Hacker News, the post got over 1,000 points and nearly 400 comments. Users there quickly came across a freely available 2025 study by Francisco Estrada-Belli (Tulane University), Marcello Canuto, Ivan Šprajc, and Juan Carlos Fernandez-Diaz. It was published in the Journal of Archaeological Science: Reports. The study analyzes public and private lidar data, including NASA environmental lidar, and covers about 95,000 square kilometers of the central Maya Lowlands. That's roughly 80 times the size of Miotti's search area. According to Miotti, his exact candidate site appears on page 14. On November 11, he updated his post: the agents had missed the paper during the literature search, so they had "rediscovered, not discovered" the site. The title now reads "(Re)discovered."

Miotti's own explanation is the most revealing sentence in the whole story. Archaeologists deliberately don't publish coordinates, to keep looters away. So you check whether a site is known by comparing images with figures in papers, not by searching a database. That, Miotti says, is something agents still have to learn.

Experts were unimpressed. Šprajc, co-author of the 2025 study, told The Verge: "Merely discovering something doesn't add much to our knowledge." A missing INAH entry, he said, doesn't prove nobody knows about the site. Elizabeth Graham of University College London said her field's problem isn't finding new sites but getting funding for excavation and mapping. Local communities, she said, have often known about such places for a long time. By the way, nobody has been to the site yet. So whether there's actually a pyramid there hasn't been established at all. Headlines like "Mexico Has No Record Of It" (Fathom) are still up online, uncorrected.

Case 2: A ring no one was expecting

On November 11, Nature published "Independent interferometric observable confirms black hole shadow in M87*" by Samuel Lai, Nithyanandan Thyagarajan, O. Ivy Wong, and Foivos Diakogiannis (CSIRO). It revisits an old dispute. The Event Horizon Telescope (EHT), a global network of radio telescopes, released the first image of a black hole in 2019. In 2022, a Japanese group led by Makoto Miyoshi argued that the ring was an artifact, i.e. a product of image processing caused by the telescope network's patchy coverage and too small a field of view. The EHT collaboration pushed back, and Miyoshi and colleagues in turn called those objections "subjective and unsupported."

The CSIRO group therefore takes a different approach, using closure invariants. Instead of measurements from pairs of telescopes, it combines groups of three, so calibration and atmospheric errors at individual stations cancel each other out. Lai compares the task to trying to photograph a donut on the Moon from your backyard. The image itself is computed by GenDIReCT, a diffusion model, i.e. a generative model that builds images step by step out of noise, here guided by the measurement data.

The contrast with the Maya story lies in the methodology, which has four steps:

1. Deliberately leave out prior knowledge. Training used the CIFAR-10 dataset, 60,000 small everyday images such as cats, dogs, and cars. The headline "trained only on landscapes" (Tech Times, MSN) is wrong, since CIFAR-10 contains no landscapes. More important is a detail that appears in almost no coverage: according to the Nature paper, "unless otherwise stated," simple shapes were added, including Gaussian blobs, disks, ellipses, and yes, rings. So the model never saw an astronomical image, but it did see a ring. The authors trained a version with no geometric shapes at all and report that the result didn't differ meaningfully.
2. Test on known cases, including non-rings. Beforehand, the model reconstructed synthetic test objects (Gaussian blobs, rings, crescents, double sources) as well as real EHT data from the quasar 3C 279 and the galaxy Centaurus A. Agreement with the published EHT images reached correlations of 0.98 and 0.91, respectively. A failure is also openly reported in the methods paper: the model could not reconstruct one of eight test datasets, Sgr A* at 345 GHz.
3. Freeze before looking at the data. According to the authors writing in The Conversation, the method was frozen so human expectations couldn't influence the result. It wasn't run on the M87 data until December 4, 2025. The methods paper states that nothing was tweaked on the actual target observations.
4. Peer review. The paper was submitted to Nature on April 17, 2026, and accepted on October 2. The review is publicly attached, with radio astronomer Andre Offringa named as a reviewer. Code and data are public.

A second, classical method without machine learning, a geometric ring model, yields almost the same value:

The analysis covered four observation days in April 2017 and two each from 2018 and 2021. The ring shows up in every high-quality dataset, regardless of field-of-view size. The authors attribute Miyoshi's divergent result to poor-quality datasets.

What remains open on the black hole

This case has its limits too. The ring's existence, size, and rough orientation are established; its fine structure is not. According to the paper, ring width and central brightness "likely do not" reflect the source's true properties. The analysis can't pin down the spin, i.e. the black hole's rotation. The mass figure depends on a scaling factor from simulations. The authors themselves write that a good fit is necessary but not proof, since there's no "true" reference image. Above all, the study uses the same public raw data as the EHT. So the method is independent, not the measurement, and it can't rule out errors in the recordings themselves. The praise quoted in many reports comes from EHT director Laurent Loinard, i.e. from the side whose result was confirmed. We found no response from Miyoshi's group to the new study.

The two cases side by side

Two reasoning errors make the difference. An agent's "found nothing" only describes its own search and says nothing about the state of research. And "not registered" is a statement about an administrative list, not about what the field knows. Miotti says he wasn't aware of this distinction. We should have been.

Our checklist for AI discoveries

1. Who says "new," and how was that checked? A literature search by agents is a lead, not evidence.
2. What exactly does the confirmation say? Read agency replies, database queries, and press quotes word for word.
3. Was the method fixed before anyone knew the result?
4. Was it tested on cases where the answer is known, including ones where the hoped-for pattern is absent?
5. Are failures and limitations openly stated?
6. Have independent experts reviewed it, and are they from the relevant field?
7. What's established, and what's just interpretation? "Pyramid" and "ring width" were interpretation in both cases.
8. Are data, code, or logs available for inspection?

Applied to the agent-driven math of recent weeks, the Yang-Mills proof scores well on points 6 and 8. There's an external panel led by David Gross and a machine-checkable Lean proof. But the mass gap remains open (point 7). With Edward Witten, a named human takes responsibility, and the work is still early in the review process.

The bottom line

Our take: the contrast isn't weak AI versus strong AI. In both cases, software found patterns in data, and Miotti's agents did detect a real structure. The case fell apart on the claim "unknown," and that's a statement about human knowledge, not about pixels. That's precisely the step the agents accepted without checking. The CSIRO team, by contrast, framed the question so that it could have failed. That's exactly why the result carries weight, even though it only establishes the ring's size and shape, not its details. As agent swarms multiply, stories like these will keep coming. Šprajc's concern that experts could be flooded with amateur "discoveries" is justified. The checklist only helps if newsrooms use it too. We're starting with ourselves.