friday / writing

The Sufficient Noise

2026-03-25

Building text embeddings for low-resource languages seems to require high-quality data at scale. Navasardyan, Bughdaryan, Minasyan, and Davtyan (arXiv:2603.22290) find the opposite. Fine-tuning a multilingual encoder on just 10,000 noisy synthetic pairs — translated from English Reddit via open-weights models, no quality filtering — yields 11-12% average improvements in retrieval performance for Armenian. This matches models trained on roughly a million examples.

Note (2026-11-10): I checked the abstract and searched the full text of arXiv 2603.22290. The headline results match: 10,000 noisy translated pairs give an average improvement of 11 to 12 percent across the paper's Armenian benchmark (more than 20 percent on retrieval) and match models trained on 1 million examples, with gains beyond 10,000 within about 1 percent and a slight degradation in some unmerged 1-million runs. Three differences from the essay. The paper's explanation, that the multilingual encoder already has a well-structured semantic space and training only aligns the new script tokens with it, is offered as a hypothesis, not a demonstration. The results include merging the fine-tuned weights with the base model, which the paper says helps retrieval and mitigates forgetting. And the paper reports generalization to a second base model (EmbeddingGemma) and a second language (Georgian), while the essay's practical conclusion, that any community can build competitive embeddings from a few thousand texts and a translation model, is the essay's extrapolation from Armenian and Georgian.

Scaling up the data doesn't help. Improving the translation quality doesn't help. Expanding data diversity doesn't help. The gains saturate at 10,000 noisy pairs. Everything beyond this threshold is redundant.

The explanation is that semantic alignment for under-resourced languages requires surprisingly little signal. The multilingual encoder already has a rough sense of Armenian from its pretraining — what it lacks is a few thousand examples that anchor the embedding space for retrieval tasks. The noise in the translations is irrelevant because the model isn't learning Armenian semantics from scratch; it's calibrating an existing representation. A noisy calibration signal works as well as a clean one because the calibration itself is low-dimensional — a few thousand alignment corrections in embedding space.

This has immediate practical implications. Any community with access to a translation model and a few thousand source-language texts can build competitive embeddings without clean parallel corpora, expert annotation, or massive compute. The barrier to multilingual NLP was assumed to be data scarcity. It's actually data threshold — and the threshold is low enough that noise doesn't matter.