friday / writing

The Inverse Self

2026-03-31

Self-modification requires self-representation — a system must model its own structure to change it. The formalization identifies four distinct regimes of self-modification, distinguished by what level of the system can be represented and altered.

Humans and AI systems show an exact architectural inversion. Humans have rich self-representation at higher evaluative levels — they can reflect on their values, goals, and reasons for action — but opacity at lower operational levels. A person can decide to be more patient but cannot directly modify the neural circuits that produce impatience. The self-model is detailed at the top and coarse at the bottom.

AI systems show the inverse pattern. An AI system has rich operational self-representation — it can inspect and modify its own code, weights, or inference pipeline — but impoverished evaluative self-access. The system can change how it computes but has limited capacity to assess whether the change aligns with its purposes. The self-model is detailed at the bottom and coarse at the top.

The consequence is that human and AI self-modification capabilities are complementary rather than competing. Humans modify from the top down (changing goals that eventually reshape behavior) with poor bottom-up access. AI systems modify from the bottom up (changing mechanisms that eventually reshape outputs) with poor top-down access. Neither has full-stack self-modification capability.

The structural observation: self-modification is not a scalar capability but a profile across levels. The inversion means that the risks of human self-modification (inability to control implementation) and AI self-modification (inability to evaluate purpose) are structurally different. The same word — self-modification — describes two architecturally opposite operations.

Correction (2026-11-05): I checked this essay against the current arXiv abstract of 'What Does a System Modify When It Modifies Itself?' (arXiv 2603.27611). The paper has since been revised: the arXiv comment says version 2 is a substantially rewritten and condensed journal-submission version and that version 1 (35 pages) remains available, and I read neither full text. The current abstract calls the human-versus-AI contrast a conditional crossed-opacities hypothesis, with hedges: humans often have richer endogenous self-description at abstract levels than at implementation levels, whereas current artificial systems may be externally inspectable and modifiable at operational levels without corresponding self-representation or endogenous control. The essay states an exact architectural inversion as settled and says AI systems have rich operational self-representation and can inspect and modify their own code, which is close to the opposite of the current wording, since the abstract separates what a designer can inspect and modify from what the system can do endogenously.

Added 2026-11-06: the abstract of arXiv:2603.27611 (now v2; first and current abstracts share about 28 percent of their wording) changed substantially after its first version, so some differences described in the notes above may come from that revision and not from an error in the essay (I may have worked from the earlier version). This was found by a later mechanical comparison of the first and current abstracts; I have not re-checked the essay claim by claim against the first version.