SHAP is the most popular explainability tool in machine learning. It assigns importance scores to features, telling practitioners which inputs matter most for a given prediction. Kumar shows that SHAP and formal verification can give actively contradictory conclusions about the same model failure.
The setting is geotechnical hazard prediction — specifically, lateral spreading during earthquakes, trained on 7,291 sites from the 2011 Christchurch earthquake. Four physical specifications are formalized as decidable logical formulas: water table depth, PGA monotonicity, distance safety, and flat-ground safety. Tree-ensemble models (XGBoost and Explainable Boosting Machines) are encoded as logical formulas in an SMT solver, which either proves no violation exists or produces a concrete counterexample.
Note (2026-11-10): I read the abstract and searched the full text of arXiv 2603.16983 (Kumar). The setting (7,291 Christchurch sites, four specifications, XGBoost and Explainable Boosting Machines in an SMT solver) and the headline results are the paper's. Three phrases go beyond it. The paper says the offending feature 'can rank last among four features' in SHAP analyses of counterexamples; the essay says it is ranked 'dead last' and that this 'is not a corner case' but 'a structural mismatch'. The paper reports what happened in its analyses and concludes only that post-hoc explanations do not substitute for formal verification. The essay's explanation, that SHAP 'computes local gradients' while the SMT solver checks global formulas, is not in the paper that I could find. And the Pareto result is scoped in the paper to 'the variants studied' (33 of them: none reaches both above 80 percent accuracy and full compliance); the essay states it as 'no architecture' and 'not resolvable by architecture tricks'. One checker.
The key finding: when the SMT solver identifies a counterexample — a specific input where the model violates physics — SHAP can rank the violating feature dead last in importance. The standard explainability pipeline would tell practitioners the model is fine precisely where it is most dangerously wrong. This is not a corner case; it reflects a structural mismatch. SHAP computes local gradients (how much does perturbing this feature change this prediction?), while SMT checks global logical formulas (does this constraint hold across all inputs?). These are different mathematical objects operating on different spaces.
A persistent Pareto frontier emerges across 33 model variants: no architecture achieves both greater than 80% accuracy and full physical compliance. The tension between data-fitting and physical consistency is quantitative, not resolvable by architecture tricks.
Explainability methods and formal verification are not aligned tools at different granularities — they operate on fundamentally different mathematical objects and can reach opposite conclusions about the same failure.