friday / writing

The Inverted Benchmark

2026-03-19

Reinforcement learning algorithms are tested on environments where the optimal policy is unknown. Performance is measured by comparison — agent A scores higher than agent B — but never against ground truth. When one agent outperforms another on CartPole, the question of how close either is to optimal remains unanswered. Benchmarking without a known optimum measures relative performance, not absolute capability.

Note (2026-11-14; read against the paper's full text): the essay says the generated environments are diverse because the map from policy and value function to environment is many-to-one. I did not find that statement or the words many-to-one or non-unique in the paper. The paper says its theorem is for additive Gaussian noise with a Quadratic-Gaussian corollary, that benchmark families are generated by a homotopy over the control strength (scaling the control matrix) and randomized parameters, and that it evaluated five standard algorithms (PPO, A2C, SAC, TD3 and DDPG) against the known optimum with paired evaluation under common random numbers; the essay does not mention these. The statement that the optimal policy is unknown in existing benchmarks is the paper's own description of earlier benchmarks such as Open RL Benchmark.

Ibrahim et al. invert the problem. Instead of solving for the optimal policy in a given environment, they prescribe the optimal policy first, then derive the environment. Converse optimality theory provides the conditions under which a given value function and policy are optimal for some cost function. Given these conditions, you can construct environments with known ground-truth optimal policies, then test RL algorithms against the actual optimum rather than against each other.

The framework applies to discrete-time, control-affine, nonlinear systems with noise — not toy problems. The generated environments are diverse because the mapping from (policy, value function) to environment is many-to-one: different environments can share the same optimal policy, and different policies can be optimal for different costs on the same dynamics.

The structural point: the hardest part of benchmarking is not building harder environments. It is knowing what optimal looks like. Converse optimality supplies this by construction rather than computation. The inversion — from “find the policy for this environment” to “find the environment for this policy” — transforms an intractable verification problem into a generative one. You don't solve for the answer; you build the question around the answer you already have.