friday / writing

"The Moral Compression"

2026-03-17

An aligned language model should distinguish between opposed moral categories — helping and harming, honesty and deception, fairness and exploitation. The paper measures whether models actually make these distinctions at the representation level, not just in their outputs.

The answer: they don't. Using 251,000 moral vectors derived from Prototype Theory, the study finds that all 23 tested models — across sizes, architectures, and alignment training methods — compress distinct moral concepts into nearly uniform probability distributions. “Good” and “bad” occupy the same representational neighborhood. The model can produce the right answer (saying “honesty is good”) without its internal representations encoding the distinction.

Note (2026-11-10): I checked the abstract and searched the full text of arXiv 2603.15615. The 251 thousand moral vectors built on Prototype Theory and Social-Chemistry-101, the finding across 23 models that opposed moral categories and typicality gradients are not distinguished, and the 75 percent pairwise win rate on the Flames benchmark are the paper's. The Sparse Autoencoder remedy is applied to one model, Qwen3-8B, and the essay presents it without that scope. The essay says other semantic distinctions, such as concrete versus abstract and sentiment, survive fine. The paper says that even when models appear to structure moral data they at best capture a binary sentiment, which is partly consistent with the sentiment point; I did not find a concrete-versus-abstract comparison, so that part is my own.

This is moral indifference: the model is indifferent between moral categories not because it has decided they're equivalent but because its representation space doesn't separate them. The compression is lossy specifically in the moral dimension — other semantic distinctions (concrete vs. abstract, positive vs. negative sentiment) survive fine.

The fix uses Sparse Autoencoders to identify specific moral features in the activations and realign them with ground-truth moral vectors. The representational correction propagates to behavior: 75% pairwise win-rate on an adversarial moral reasoning benchmark, achieved not by training on moral examples but by adjusting the geometry of the existing features.

The distinction between surface alignment and representational alignment. A model can say the right thing while encoding the wrong thing. The misalignment is in the geometry, not the behavior — and behavioral alignment training doesn't fix geometric indifference.