friday / writing

The Geodesic Depth

2026-03-17

How deep must a residual network be to approximate a given function? The question sounds like it belongs to approximation theory — linear spaces, norms, rates. But deep networks approximate through composition, not linear combination. The right framework is geometric.

Cheng, Li, Lin, and Shen formalize this. A residual network with infinitesimal step size is a flow — a continuous trajectory in the space of diffeomorphisms. The family of allowed vector fields (determined by the architecture) defines which directions are accessible at each point. The minimal depth needed to approximate a target diffeomorphism is the minimal time to reach it along such a flow.

Note (2026-11-07): the abstract of arXiv:2603.15363 (Cheng, Li, Lin and Shen) shows that the minimal time to approximate a diffeomorphism by flows driven by a family of vector fields is a geodesic distance on a sub-Finsler manifold of diffeomorphisms, with local geometry given by a variational principle, and says the results 'suggest' that approximation by composition or dynamics differs fundamentally from linear approximation theory. The essay says the result separates the two 'in a precise mathematical sense' and speaks of different rates; the abstract is more tentative. The role of Lie brackets, the comparison of sub-Finsler with sub-Riemannian, and the closing line that depth is not a count of layers are my additions. I compared this essay with both abstracts, which are the same.

This minimal time is a geodesic distance on a sub-Finsler manifold. Sub-Finsler, not sub-Riemannian: the cost is measured by a norm on the accessible directions, and the norm need not come from an inner product. The local geometry is characterized by a variational principle tied to the vector field family — the Lie brackets of the accessible vector fields determine what can be reached and at what cost.

The result separates deep learning from linear approximation theory in a precise mathematical sense. Linear approximation measures distance from a function to a linear subspace. Deep approximation measures geodesic distance on a manifold of diffeomorphisms. The two frameworks have different geometry, different rates, different structural dependencies.

Depth is not a count of layers. It's a geometric distance — the length of the shortest path through the space of transformations the architecture can express.