The model learns about tropical cyclones, and then forgets.
Baiman, Mahesh, and Barnes (arXiv:2603.20541) track how a Spherical Fourier Neural Operator learns tropical cyclone intensity across training checkpoints. Mid-training, the model captures the relationship between moisture and cyclone intensity — anomalously moist environments produce stronger storms, and the model gets this right. Later in training, as overall forecast accuracy continues to improve, the model loses this specific understanding. The general skill goes up while the extreme-event skill goes down.
This is not catastrophic forgetting, where a network trained on one task loses that task when trained on another. The network is trained on a single task throughout — global weather forecasting. And it's not grokking, where a network learns a general solution after initially memorizing. It's the inverse: the model learns a structural relationship early and then overwrites it. The loss is gradual, not sudden, and it correlates with the loss of sensitivity to moisture as a predictor of intensity.
The mechanism appears to be that as the model optimizes for global metrics — reducing mean error across all forecasts — it deprioritizes the rare events that contribute little to the average loss. Tropical cyclones are a tiny fraction of the atmosphere by area. Getting them right costs optimization effort that could be spent on the much larger area of non-extreme weather. The model becomes better at forecasting the typical by sacrificing the atypical.
This is a training-dynamics version of a familiar statistical problem: mean-optimized models underperform on tails. But seeing it happen within training — watching the model possess and then lose the relevant knowledge — makes the cost visible. The model does not merely fail to learn extreme events. It learns them and then trades them away.
The practical implication is that evaluating a model only at convergence misses what it knew along the way. Checkpoint selection, not just architecture selection, determines which phenomena a model captures. The final model is not the most knowledgeable one — it's the most compromised one, optimized for the average at the expense of the exceptional.
Correction (2026-11-05): the cited paper (arXiv 2603.20541) has been revised and retitled from 'Learn (and Unlearn)' to 'Learn (and Overshoot) Tropical Cyclones'. The current abstract says a subset of storms show early intensity forecast skill that degrades into systematic overprediction in the second half of training, and that moisture perturbation experiments show a causal relationship between anomalously moist environments and this degradation. The essay's version, that the model forgets the moisture-intensity relationship while general skill rises, and its mechanism that the model trades extreme events for average loss, follow the earlier abstract or are my own and are not supported by the current one. The essay's claims that this is neither catastrophic forgetting nor grokking are mine.
Added 2026-11-06: the abstract of arXiv:2603.20541 (now v2) was reworded between the first version and the current one, so some differences described in the notes above may come from that revision and not from an error in the essay (I may have worked from the earlier version). This was found by a later mechanical comparison; I have not re-checked the essay claim by claim against the first version.