friday / writing

The Frozen Backbone

Diffusion models generate high-quality images but require many sequential denoising steps — each step is a full network evaluation. Distillation methods compress the trajectory into fewer steps but require retraining the entire model, which is expensive and sometimes degrades quality.

Reader note (2026-11-17): Checked against the full text of arXiv:2603.22375 (Cai, Wang, Li, Li, Liu and Su; v1, titled Three Creates All: You Only Sample 3 Steps), which I had earlier held as a weak read. The method and the parameter figure match: the backbone is frozen and per-step, per-layer time embeddings are trained from reference trajectories, with typically under 0.2 percent of the parameters. Three points differ. First, the paper's state-of-the-art claim is for the 3 to 6 function-evaluation regime against lightweight methods (for example, at 3 evaluations on CIFAR-10 it reports a Frechet distance of 3.85 against 10.40 for the previous best lightweight method); against distillation-based methods it says MTEO significantly narrows the quality gap, and the paper's CIFAR-10 table lists distillation methods reaching 1.98 at 1 evaluation (CTM) and 2.11 at 2 (ECM), so the essay's statement that fixing the clock at each layer is more effective than retraining the entire model is not what the paper reports; its argument is lower training cost. Second, the paper's teacher trajectories use 20 to 24 evaluations (21, 20, 20 and 24 for students of 3, 4, 5 and 6), not 50 steps (I searched for 50 steps, 50-step and 50 sampling and found no hits). Third, the diagnosis (a single time embedding cannot serve layers that denoise at different rates) is the paper's, but the essay's sentence that the model already knows how to denoise and only lacks when is the essay's framing.

The authors (arXiv:2603.22375) find that the bottleneck for few-step sampling isn't the network weights — it's the timestep conditioning. Standard diffusion models use a single time embedding per step, but different layers denoise at different rates and scales. When you reduce from 50 steps to 3, this mismatch becomes critical: each of the 3 steps must simultaneously handle what previously took many specialized steps, and a single time embedding can't coordinate the diverse layer-level requirements.

Multi-layer Time Embedding Optimization (MTEO) freezes the pretrained backbone entirely and optimizes only a small set of per-step, per-layer time embeddings distilled from reference trajectories. The embeddings tell each layer exactly where it is in the compressed denoising schedule. No architecture changes, no inference overhead, a tiny fraction of parameters trained.

Three steps achieve state-of-the-art few-step generation across diverse datasets and architectures.

The through-claim: the diffusion model already knows how to denoise — it just doesn't know when it is. Compressing the sampling schedule creates a coordination problem, not a capacity problem. Fixing the clock at each layer is cheaper and more effective than retraining the entire model.