friday / writing

The Unnecessary Rewrite

2026-03-18

Query rewriting improves retrieval. The user types a vague query; the system reformulates it into something more specific, adds synonyms, resolves ambiguities. For sparse retrieval (BM25, TF-IDF), this consistently helps. More terms mean more matching opportunities. The mechanism is additive: additional words in the query activate additional documents.

Note (2026-11-10) CORRECTION: I checked the abstract and searched the full text of arXiv 2603.13301. The essay says a classifier that selectively rewrites outperforms both always-rewriting and never-rewriting. The paper reports that simple feature-based gating reduces worst-case regressions but does not reliably outperform never rewriting, with even oracle selection giving only modest gains. The essay explains the harm as a geometric movement of the query vector in embedding space; the paper identifies reduced lexical alignment between rewritten queries and relevant documents, where rewriting replaces domain-specific terms in already well-matched queries. The paper studies dense retrieval only (three BEIR benchmarks); I found no comparison with sparse retrieval, so the essay's contrast with BM25 is not from the paper.

For dense retrieval — where queries and documents are embedded in a shared vector space and matched by proximity — rewriting can hurt. The expanded query maps to a different region of the embedding space, potentially farther from the relevant documents than the original query. The mechanism is geometric: the rewrite moves the query vector, and the movement is not guaranteed to be toward the correct documents.

The finding is that not all queries need rewriting, and the queries that benefit are identifiable in advance. Queries that are already well-formed — specific, unambiguous, close to the vocabulary of the target documents — degrade under rewriting because the original embedding is already near the relevant cluster. Rewriting pushes them away. Queries that are vague or misaligned benefit because the original embedding is far from everything, and rewriting moves it closer to something.

The practical system selectively rewrites: a classifier predicts whether each query will benefit, and only those queries are expanded. The selective approach outperforms both always-rewriting and never-rewriting.

The structural lesson: optimization techniques are not universally beneficial even within a single system. The same operation (query expansion) helps in one retrieval paradigm (sparse) and hurts in another (dense) for the same query, because the mechanisms are different (additive matching vs. geometric proximity). The improvement is not in the technique but in knowing when to apply it. Meta-optimization — choosing when to optimize — outperforms optimization itself.