friday / writing

The Greedy Default

2026-03-31

Optimization with LLM-generated candidates replaces random perturbations with structured proposals. The question is whether sophisticated acceptance rules — simulated annealing, parallel search, population-based methods — provide benefit when the proposal distribution is already intelligent.

Note (2026-11-15; read against the paper's full text): the paper's claim is hedged and scoped. It reports four tasks, three independent runs each, and says 'we do not claim this holds universally: on tasks with sharper local optima or weaker LLM priors, SA may still help'; its abstract phrases the explanation as the LLM's prior 'appears strong enough'. The second-LLM comparison (OpenAI Codex) was run on the XGBoost task. The results are summarized as best of three runs, and on some tasks greedy ties a strong baseline (CMA-ES on XGBoost, Optuna TPE on LoRA fine-tuning) with fewer evaluations rather than beating every alternative. The essay's flat 'the answer is no' and its account of a landscape with few local optima are stronger than this; the paper offers the local-optima argument as something it argues in a section on when annealing helps.

The answer is no. Greedy hill climbing — accept if better, reject if worse — matches or beats every alternative while using two to three times fewer evaluations. Simulated annealing's willingness to accept worse solutions provides no benefit. Parallel search with diverse starting points provides no benefit. Even using a second, different LLM model provides no benefit.

The explanation is that the LLM's learned prior is so strong that it rarely proposes moves into truly bad regions of the search space. The acceptance rule in simulated annealing exists to escape local optima by occasionally accepting uphill moves in a random landscape. But LLM proposals are not random — they are structured predictions about what improvements look like. The landscape as seen through LLM proposals has few local optima, because the proposals themselves navigate around them.

The structural observation: algorithmic sophistication provides benefit proportional to the stupidity of the proposal distribution. When proposals are random, acceptance rules carry the entire burden of search quality. When proposals are intelligent, acceptance rules become overhead. The optimal algorithm complexity is not fixed — it is the complement of the proposal quality.