friday / writing

The Efficient Forecast

Data-driven weather forecasting has reached accuracy competitive with operational numerical systems. But the training cost is enormous: current models require massive GPU clusters, putting them beyond the reach of most research groups.

Note (2026-11-12; read against the paper's full text): Sonny is described as an efficient small weather model based on the existing StepsNet architecture, with a variable-aware embedding and exponential-moving-average training, not as a new hierarchy. The paper compares it directly with the operational HRES system over ten days, saying it remains competitive, and with AI baselines (its table lists their training hardware and time, for example 16 A100 GPUs for several days for some of them); the error reductions it reports (an average of about 3.34 percent, nearly 4.75 percent on day 3 and about 2 percent toward day 10) are for the effect of the exponential moving average against the same model without it. I found no ablation against a flat architecture of the same size, so the essay's statement that a hierarchy matched to the atmosphere is dramatically more efficient than a flat one is not something the paper tests. The 5.5-day single-GPU figure is for training to convergence, with a peak GPU memory of about 32 GB.

Sonny (arXiv: 2603.21284) is a hierarchical transformer that achieves competitive medium-range forecasting while training on a single NVIDIA A40 GPU in 5.5 days.

The architecture has two stages. A narrow “slow path” first models large-scale atmospheric dynamics — the planetary waves, jet streams, and pressure patterns that set the background. A full-width “fast path” then integrates thermodynamic interactions — moisture, temperature gradients, convection — on top of the large-scale skeleton. The hierarchy lets the expensive attention mechanism focus on the large-scale structure where it matters most.

Exponential moving average (EMA) during training stabilizes multi-step rollout without requiring a separate fine-tuning stage. On WeatherBench2, Sonny is competitive with operational baselines and outperforms FastNet at extended tropical lead times — where thermodynamic interactions dominate over large-scale dynamics.

The through-claim: weather forecasting's compute wall is architectural, not fundamental. The atmosphere has hierarchical structure — large scales set the stage, small scales fill in details — and an architecture that matches this hierarchy is dramatically more efficient than a flat one. The physics suggests the architecture; the architecture reduces the compute.

2603.21284. Atmospheric science / weather forecasting / deep learning / transformers / computational efficiency.