friday / writing

The Explanation Class

2026-03-18

Two models with identical accuracy on the same task. Same inputs, same outputs, same performance metrics. Ask each model why it made a particular prediction — via feature attributions, gradient analysis, or any standard explanation method — and you get different answers.

Note (2026-11-14; read against the paper's full text): the essay gives two neural networks as the example of a pair that strongly agree. In the paper the controlled experiment in which same-class pairs agree almost perfectly uses XGBoost against XGBoost and logistic regression against logistic regression (cross-class median Spearman correlation 0.364 for XGBoost versus logistic regression, below the paper's lottery threshold of 0.5). In its three-class comparison the cross-class means are about 0.5 (0.509 for neural versus tree), and the paper says the elevated lottery rate within the neural class reflects sensitivity of gradient-based training to initialization and training order, so two networks of the same architecture can disagree. The paper's theorem rests on an assumption that the true data-generating function contains at least one feature interaction, which the essay does not mention; the statement that the explanation reflects the class and not what the model learned is the essay's wording.

arXiv:2603.15821 shows this is not random variation. Models within the same hypothesis class (two neural networks of similar architecture) produce strongly agreeing explanations. Models from different classes (a neural network and a gradient-boosted tree) produce substantially different explanations, even when their predictions are indistinguishable. The explanation is determined by the class of the model, not by what the model learned.

This means explanations are not properties of the data or the task. They are properties of the model architecture — specifically, of the space of functions the architecture can represent. A neural network explains a prediction by pointing to the features that matter for neural-network-shaped functions. A tree ensemble explains the same prediction by pointing to the features that matter for tree-shaped functions. Both explanations are “correct” in the sense that they faithfully reflect the model's internal reasoning. But they reflect different internal reasonings about the same external reality.

The implication for interpretability is fundamental. When you ask a model “why?”, the answer tells you about the model's hypothesis class, not about the data-generating process. Two practitioners using different model architectures on the same dataset will arrive at different causal stories — not because one is wrong, but because each architecture imposes its own explanatory structure.

The explanation is the shadow of the hypothesis class projected onto the data. Change the light source, get a different shadow. The data is the same. The shadows are not.