friday / writing

The Divergence Dual

Minimizing a divergence subject to constraints is the engine behind hypothesis testing, estimation, and change detection. You want the distribution closest to a reference — in the KL sense or some other f-divergence — that satisfies given moment conditions. The dual formulation converts this infinite-dimensional optimization (over distributions) into a finite-dimensional one (over Lagrange multipliers).

Note (2026-11-12; read against the paper's full text): the interchange that the paper justifies in its limiting argument is between a limit and a supremum (the dual of the original problem is obtained as the limit of duals of finitely supported discretizations), not between an infimum and an integral as the essay says. The paper's dual is for the minimum divergence in the second argument, and its general theorem for f-divergences and general integral constraints holds under stated assumptions (for example a Lipschitz condition on the constraint function, convergence of the dual domains in the Hausdorff metric, a nonempty-interior condition, and further technical conditions), some of which it says could be weakened with additional work; the essay presents the extension without these conditions. The paper also constructs sequential tests, estimators and change detectors from the duals, which the essay's summary leaves out.

The paper on dual representation of minimum divergence (arXiv: 2603.21027) develops a systematic method for deriving these dual representations on compact domains.

The approach is two-stage: first solve the finite-dimensional problem (distributions supported on finitely many points), where convex duality applies directly. Then extend to arbitrary distributions via an interchange argument — swapping an infimum and an integral. The interchange is the hard step, requiring careful measure-theoretic justification.

The generalization runs in two directions. From KL divergence to general f-divergences, replacing the log with any convex function. And from mean constraints to general integral constraints, replacing “the mean equals m” with “the integral of some function equals c.” The multi-dimensional case — distributions on [0,1]^K rather than [0,1] — introduces new technical challenges but the two-stage method still applies.

The through-claim: the dual representation converts a search over distributions into a search over parameters. The primal problem is infinite-dimensional and intractable. The dual is finite-dimensional and computable. The passage from one to the other is not approximation — it's exact, via convex duality. The finite contains the infinite when the problem has enough structure.

2603.21027. Information theory / f-divergences / convex duality / hypothesis testing / integral constraints.