AI weather "beats physics" – but only when AI data keeps score?

What this is about

"AI models are now beating physics": that's how the company VIDRAFT sums up the first ten months of its new Global Weather Model Leaderboard on Hugging Face. Every day, the leaderboard compares what ECMWF IFS and AIFS, NOAA GFS and AIGFS, DWD ICON, JMA, Canada's GEM and MET Norway forecast against what actually happened. In all four categories, first place goes not to a weather service but to VIDRAFT's own fusion model, Darwin-Prediction-Weather, which is blended together from its competitors' forecasts.

At almost the same time, Google DeepMind released the SatWeave and PARDIG datasets that its WeatherNext 3 model is built on. For the first time, that means some of these claims can be checked with public data. We looked at what these leaderboards measure against and found that the scoring method decides a good chunk of the outcome before a single forecast is made.

What the leaderboard measures, and what it doesn't

According to the leaderboard's README, it scores 24 cities: Seoul, Busan, Tokyo, Beijing, Singapore, Delhi, Dubai, Moscow, Berlin, London, Paris, Madrid, Cairo, Nairobi, Johannesburg, New York, Chicago, Los Angeles, Mexico City, São Paulo, Buenos Aires, Sydney, Anchorage and Reykjavík. It scores daily maximum and minimum temperature, maximum wind and daily precipitation, at lead times of one to seven days. Days 8 to 10 have only been archived since November 13. The metric is RMSE, the root mean square error, which gives large outliers disproportionate weight. A model only gets scored on days and cities for which every model produced a forecast. The weather data comes from Open-Meteo. The "ground truth" is the ERA5 reanalysis, an after-the-fact reconstruction of the weather on a grid of about 31 kilometers, computed by a weather model that assimilates observations.

According to its model card, Darwin achieves an RMSE of 2.09 °C across 4,848 seven-day maximum temperature forecasts, versus 2.41 °C for AIFS and 2.84 °C for IFS. Our math: 4,848 cases spread over 24 cities works out to about 202 days per city. But January 1 to November 7 is 311 days. So roughly a third of the days drop out because of the "all models present" filter. The leaderboard doesn't report confidence intervals.

Values according to VIDRAFT, as of November 7, 2026; we calculated the gaps.

On top of that, there's a clear conflict of interest: the leaderboard and the leader come from the same operator. The model card itself states that the results of the actual fusion engine can't be reproduced from the repository alone. Darwin's WeatherBench 2 numbers, which claim it leads in all 15 fields of Google's standard benchmark, are so far self-reported. VIDRAFT only opened its entry as GitHub issue #320 on November 13. It's still open, and the maintainers haven't responded. On the official leaderboard, WeatherMesh4 and the GenCast ensemble remain on top. The data is licensed under CC BY-NC 4.0, so it can't be used commercially.

Why ERA5 isn't a neutral referee

The core problem is the reference. AIFS was trained on ERA5 (1979–2022) and then fine-tuned on IFS analyses. According to NOAA, its AIGFS was also pre-trained on ERA5; one NOAA source even lists ERA5 as the target during fine-tuning. The physics models GFS and ICON, by contrast, start from their own analyses, which have nothing to do with ERA5. So when you score against ERA5, part of what you're measuring for the AI models is how well they imitate their training target. That's why WeatherBench 2 doesn't score operational models against ERA5 but against the IFS analysis.

Sabrina Wahl of the DWD put it this way in a contribution to EGU 2026: when you test AI models against stations, what you mostly see are the errors of the reanalysis they were trained on. Classic models, on the other hand, were developed and tuned against observations.

Darwin takes this one step further. According to its model card, it corrects its values at every grid point using days for which the ERA5 value was already known. Our take: that way, the model learns precisely the systematic offset from the referee. That plausibly explains why its day-1 lead on temperature and wind is close to 40%. It tells you nothing about how close the forecast comes to the temperature in downtown Berlin. High-resolution models can even be at a disadvantage: according to Open-Meteo, "ICON Seamless" combines all of a provider's models, which presumably includes the DWD's fine-grained regional models. A model that resolves more detail than ERA5 gets penalized for every deviation from the smoother reference field. VIDRAFT has announced a comparison against station data, but it isn't running yet.

With rain, the lead shrinks

For precipitation, the gap between Darwin and AIFS is only 2.6 to 4.2%. There are two reasons. First, ERA5 is a weak reference for rain: the WeatherBench 2 authors write that precipitation isn't directly assimilated from observations there, and call their own rain evaluation merely a placeholder. Second, RMSE rewards blurry forecasts. Christopher Subich and colleagues at Environment and Climate Change Canada (ECCC) explain the so-called double penalty: if you correctly forecast a shower but place it a few kilometers off, you get penalized twice, once for missing the right spot and once for the wrong one. A smeared forecast with a little rain everywhere scores better. That's why WeatherBench 2 measures precipitation with SEEPS, a rain-specific metric that doesn't reward blurry forecasts.

WeatherNext 3 and PARDIG: when the "truth" is itself an AI

Google markets WeatherNext 3 with up to 60% better rain forecasts. The paper is more precise: it refers to a drop in CRPS, an error metric for probabilistic forecasts, of up to 60% against IMERG, 30% against MRMS and 10% against rain gauges, each at short lead times.

The paper itself admits that IMERG isn't an independent reference for WeatherNext 3. It only evaluates up to 4 mm in six hours. There's no separate evaluation for Europe. The 50% figure in Google's blog for forecasts "a day or more" ahead doesn't name a clear baseline. As independent evidence, Google points to the leaderboard from Brightband, a startup that builds AI weather models itself.

The new PARDIG dataset is a step forward, but it doesn't solve the reference problem. It's built with an architecture similar to WeatherNext 3: the inputs are satellite images from SatWeave and atmospheric fields from ECMWF, and it learns from the sparse radar measurements of NASA's GPM satellite. The result covers 1999 to 2026 at about 10 kilometers and 15-minute resolution, and is published with a delay of around five hours. According to DeepMind, PARDIG beats NASA's IMERG-Final at all rain intensities and detects light to moderate hourly rain more than 30% better, validated against ground radar in the US, Europe and Japan and more than 36,000 stations. DeepMind doesn't mention the license in its blog; according to IT之家, it's CC BY 4.0. Our take: if WeatherNext 3 is scored against PARDIG in the future, one AI estimate will be checking another built with related technology. The only meaningful comparisons are against rain gauges and radar.

And in Germany?

Berlin is the only German city on the leaderboard, which means a single grid point. Values for individual cities only show up in the interactive view. We couldn't reliably extract them from the raw files with the tools we have, so we're not citing any here.

What the weather services have tested against stations themselves is more solid:

• AICON (DWD): The AI model has been in operational use since March 2, 2026. It runs at 13-kilometer resolution and was trained on the DWD's own ICON-DREAM reanalysis, not on ERA5. In the AICON paper, the DWD evaluates a full year (March 2025 to February 2026) against SYNOP and TEMP observations. The result: most near-surface variables are better than ICON in the first few days, more clearly in summer than in winter. AICON is weaker on surface pressure from about day 2 onward and on geopotential height in winter. For precipitation, the two models perform "relatively equally"; for 24-hour totals on days 1 and 2, AICON is better. At longer lead times, it produces unrealistic rain spots along coastlines. The meteorologists noted that AICON underestimates strong winds and dampens extremes.
• AIFS (ECMWF): Against stations in Europe in winter 2024/25, AIFS had more than 15% less error than IFS for 2 m temperature over flat terrain after 1.5 days. In the Alps, IFS came out ahead. For precipitation, ECMWF says AIFS improves the medium range by about 10%, equivalent to half a day to a full day of forecast lead. It underestimates the intensity of tropical cyclones.
• Extremes: A study by KIT and the University of Geneva (Science Advances, April 2026) found that GraphCast, Pangu-Weather and FuXi perform worse than the physics model HRES on record heat, cold and wind. It tested an older generation of models, though.

Our analysis

How far you can trust these numbers varies. The finding that AI models match or beat physics on temperature and wind under normal conditions is fairly solid. Station-based evaluations from the DWD and ECMWF show that too, not just comparisons against ERA5. Poorly supported are Darwin's "best model" title (self-measured, against its own correction target, not reproducible) and marketing figures like "60% better rain" that are measured against a training target. For rain and extremes, the picture is still open. The DWD itself sees AICON only as a complement to ICON.

Our take: if you want to know which model leads for Germany, there's currently no public, independent leaderboard that scores against German stations. For temperatures over the next one to three days, the station evaluations favor AICON and AIFS. For thunderstorms and heavy rain, the high-resolution ICON-D2, at about 2 kilometers, remains the DWD's tool of choice. The new leaderboard is still useful because its archive is open. But a winner there will only be credible once it's also scored against weather stations.