The Fractions Skill Score is one of the most widely used metrics for evaluating spatial weather forecasts. It works by comparing forecast and observed fields within expanding neighborhoods — if the forecast gets the right answer within a neighborhood of size r, it scores well. The intuition is that a forecast that's spatially displaced but otherwise correct should score better than one that misses entirely.
Antonio demonstrates that the FSS systematically rewards smoother forecasts. A blurrier forecast — one that spreads its probability over a larger area — scores higher than a sharp forecast with the same spatial displacement error, because the blurring increases the overlap between forecast and observation neighborhoods. The metric has a built-in preference for hedging.
Note (2026-11-13; read against the paper's full text, version 2 of August 2026): the paper's finding is conditional, not general. It says that 'under many situations' the Fractions Skill Score gives higher scores to forecasts that over-predict mean frequency, and that both scores 'generally' give higher scores to forecasts with lower neighbourhood standard deviation; it names an exception, a regime (its Fig. 2 (a) and (e), with a mean-frequency ratio not equal to 1 and a correlation term near 1) in which the Fractions Skill Score favours forecasts that over-predict the standard deviation. The essay's 'systematically' and 'built-in preference for hedging' are stronger than that. On the double penalty, the paper says the Fractions Skill Score is 'under some situations' less susceptible than the Brier Divergence Skill Score, not that it is simply more robust. The paper's recommendation is to use percentile thresholds, with reflective padding (it says the bias is not removed with zero padding), and it adds that in some cases, such as truly binary events, percentile thresholds may not be usable; the essay's remark that the fix is known omits that. The paper's analysis is written in terms of summary statistics of the neighbourhood fractions (their mean, standard deviation and correlation), and I did not find the essay's displacement example with equal displacement error in it; that comparison is the essay's.
The bias is structural, not accidental. The FSS also tends to assign higher scores to forecasts that over-predict mean frequency — forecasts that say “rain” too often score better than forecasts that say “rain” too rarely, holding other errors equal. The theoretical analysis confirms what practitioners have suspected: the FSS should be used with percentile thresholds (which normalize the frequency) rather than raw thresholds (which don't).
The double penalty problem — where a displaced forecast is penalized both for the miss at the observed location and the false alarm at the forecast location — affects the related Brier Divergence Skill Score more severely than the FSS. The FSS is more robust to this issue, which partly explains its popularity.
A standard metric, used across national weather services worldwide, theoretically confirmed to reward the wrong thing. The fix is known (percentile thresholds). The question is whether the metric's users know they need it.