Vector quantization turns continuous representations into discrete tokens — a necessary step for language models that process images, audio, or protein structures. The standard approach: train an encoder and a codebook simultaneously, snapping continuous vectors to their nearest codebook entry at every training step.
Reader note (2026-11-17): Checked against the full text of arXiv:2603.22304 (Zhao, Zou, Lin and Liu; v1 of March 2026), which I had earlier held as a weak read. The problem name, the curriculum idea, the ImageNet results and the protein-structure result match. Three points differ. First, the mechanism: the paper's first stage trains a plain continuous autoencoder with no quantization (manifold warmup); the second stage uses a hybrid latent that interpolates between the continuous encoder output and its nearest-codebook-entry quantization, with a coefficient annealed on a cosine schedule. It does not give every codebook vector a weighted influence, so the essay's soft assignments describes a different mechanism. Second, the paper's account of the failure is a cycling co-adaptation in which the encoder collapses toward the nearest, sub-optimal codebook entries and the code stagnates; the essay's sentence that the codebook cannot reorganize later is the essay's wording (I searched for reorganiz, cannot and can't: no hits). Third, the closing claim that forcing discrete structure too early is worse than no discretization at all is not in the paper (I searched worse than, without quantization and no quantization: no hits); the ablation the paper reports compares a cosine schedule with a hard schedule (reconstruction FID 3.23 against 3.40), and on protein structures the paper describes a new performance ceiling on the StructTokenBench leaderboard, an existing benchmark, not new benchmarks.
The authors (arXiv:2603.22304) identify a problem they call Premature Discretization. The codebook crystallizes before the encoder has learned the data's structure. Early in training, the encoder's representation is rough — the manifold it maps onto is incomplete. Forcing discretization onto an immature representation locks the codebook into a bad partition of latent space. Later, as the encoder improves, the codebook can't reorganize to match.
Progressive Quantization (ProVQ) treats discretization hardness as a training axis. Start with soft, continuous assignments — every codebook vector gets weighted influence. Gradually anneal toward hard, discrete assignments as the encoder matures. The codebook expands to cover the manifold because it has time to explore before committing.
The results span modalities: improved reconstruction on ImageNet, and new benchmarks for protein structure tokenization on StructTokenBench. The same fix works for images and proteins because the problem is in the quantization, not the data.
The through-claim: discretization is a curriculum problem, not an architecture problem. Forcing discrete structure too early is worse than no discretization at all. The codebook needs to learn the landscape before it can carve it into regions.