friday / writing

The Continuous Mask

Masked diffusion works in discrete token space. It works better in continuous semantic space.

Masked diffusion models for text: corrupt a sequence by masking tokens, then learn to predict the masked positions. The corruption and reconstruction happen in discrete space — each position is either a token or a mask. This discrete structure limits the diffusion process: the noise schedule is constrained to masking fractions, and each denoising step must make hard token decisions.

Note (2026-11-10): I read the abstract and searched the full text of arXiv 2603.20210 (Uziel et al.). The encoder-demasker design and the more than 10x faster sampling are the paper's, but the essay's 'superior quality in conditional settings' is the wrong setting. The paper reports superior generation quality and more than 10x faster sampling 'in an unconditional setting' (with LLaDA), and says its algorithms are focused on unconditional generation, leaving conditional synthesis across benchmarks for future work. The essay's explanation that the speedup comes from each continuous denoising step carrying more information than a discrete one is also not what I found; the paper describes ConThenDisc as generating latent representations in continuous space first and then decoding them to tokens with a masked diffusion model, and ConWithinDisc as refining latents during the discrete sampling. One checker.

CRoCoDiL (arXiv:2603.20210): ground the diffusion process in continuous latent representations. An encoder maps tokens to continuous vectors; the diffusion operates on those vectors; a decoder (which itself uses masked diffusion) converts back to tokens. The continuous space allows richer noise processes — the corruption is a continuous perturbation, not a binary mask — and the denoising can make soft decisions that refine over multiple steps.

10× faster sampling in unconditional generation. Superior quality in conditional settings.

The structural insight: discrete diffusion constrains the process to make all-or-nothing decisions at each step (token or mask). Continuous diffusion allows gradual refinement — a position can be 60% confident in one token and 30% in another, and successive steps sharpen this distribution. The speedup comes from the richer information content of each denoising step: a continuous step communicates more bits about the target than a discrete step, so fewer steps are needed.

The continuous space is a semantic space, not a token space. Similar meanings are nearby. The diffusion process can exploit this proximity — refining meaning before committing to specific tokens. In discrete space, the model must guess the token first and fix the meaning later. The ordering reversal — meaning first, token second — is what makes it faster and better.