Monitor a deployed model long enough, and any fixed statistical test will eventually raise a false alarm — even if nothing has changed. The longer you watch, the more certain the false alarm becomes. This is the fundamental trap of model monitoring: the act of continuous observation violates the assumptions of the test being applied.
Note (2026-11-13; read against the paper's full text): the guarantee is Type I error control, not a monitor that 'never makes a false claim': the paper says PITMonitor provides Type I error control over an unbounded monitoring horizon through a mixture e-process, which bounds the probability of a false alarm, and it adds Bayesian changepoint estimation separate from the alarm time. The evidence on detection is from one synthetic benchmark, river's FriedmanDrift regression stream with three scenarios, where the paper reports detection rates 'competitive with the strongest baselines' but detection delay 'substantially longer under local drift'; the essay's comparison with 'aggressive baselines' and the statement that this is the correct trade-off for production systems are the essay's. The paper's criticism of existing methods is that they lack formal error guarantees, conflate alarm time with changepoint location, and monitor indirect signals, and it says some classical detectors (DDM, HDDM) rely on heuristic thresholds while ADWIN bounds the false alarm probability only within a single window and not at the stream level.
Farran addresses this with PITMonitor, which uses a mixture e-process to provide Type I error control over an unbounded monitoring horizon. The key insight is that probability integral transforms — checking whether a model's predicted probabilities are calibrated — can be combined with anytime-valid sequential testing to create a monitor that never makes a false claim regardless of how long it runs.
The distinction matters because current practice ignores it. Most deployed model monitoring uses tests designed for fixed sample sizes, applied repeatedly at arbitrary intervals. The resulting false alarm rate is not the nominal one — it accumulates, and eventually the monitor cries wolf on a model that hasn't moved. The response is either to ignore the monitor (defeating its purpose) or to recalibrate models that were fine (wasting effort and potentially degrading performance).
The trade-off is honest: detection delay under local drift is “substantially longer” than aggressive baselines. You pay for the guarantee of no false alarms with slower detection of real problems. But this is the correct trade-off for production systems where false alarms erode trust in the monitoring infrastructure itself. A monitor nobody believes is worse than no monitor at all.