The attack works in the paper. It fails in the deployment.
A systematic evaluation of federated learning poisoning attacks (arXiv:2603.20615) finds that idealized research conditions consistently overstate security risk. Attacks that appear effective in standard benchmarks lose their potency — or cause unacceptable collateral damage to benign tasks — when tested under real-world constraints.
Reader note (2026-11-17): Checked against the full text of arXiv:2603.20615 (Chen, Zhao, Pu, Zhou, Feng, Li and Ji; v1), which I had earlier held as matching on an abstract-level read. The description of the paper's findings matches: unrealistic attacker participation (the paper gives 20 to 50 percent malicious clients and fixed participation as the assumptions it criticizes), omitted heterogeneity, and metrics that overstate peak attack success; its takeaway is that final-round success can overstate risk and even reverse the ranking between attacks once stability and benign-task degradation are counted. Three things in the essay's last paragraphs are the essay's own. I searched the full text for publish, incentive and novelty and found no hits, so the claim that security research optimizes for publishable threat demonstrations is not from the paper. The paper's remark about defenses is that attack studies assume sophisticated, idealized defenses while most deployed platforms use simple ones; the essay's statement that the defense literature inherits the same bias and is solving the wrong problem is not stated in the paper. And the paper's scope is poisoning attacks in cross-device and asynchronous settings, with privacy risks left to future work, so the essay's closing remarks about the security of federated learning in general go beyond poisoning attacks, the only risk the paper measured.
The mismatches are specific. Research assumes unrealistic attacker participation rates (higher fractions of compromised clients than any practical system would allow). Studies ignore hybrid heterogeneity (real FL systems have clients with different data distributions, hardware, and connectivity). Metrics emphasize peak attack success at the final training round while ignoring temporal stability (an attack that works once but fails in surrounding rounds isn't operationally useful).
The most damaging finding: several attacks appearing consistently effective in idealized settings incur clear benign-task degradation once practical constraints are enforced. The attack “works” only if you ignore what it breaks. Joint evaluation — effectiveness, stability, and collateral damage — reverses the conclusions of papers that report each metric in isolation.
The structural problem is that security research optimizes for publishable threat demonstrations, not for accurate risk assessment. A dramatic attack in an idealized setting is more publishable than a nuanced analysis showing the attack is impractical. The field selects for alarming results and against calibrated ones.
This doesn't mean FL is secure. It means we don't know how insecure it is, because the measurement instrument (idealized evaluation) is systematically biased toward overestimation. The defense literature inherits the same bias — if the threats are overstated, the defenses are solving the wrong problem. Accurate risk requires matching the evaluation to the deployment, and the gap between lab and field is where the real security posture hides.