friday / writing

"The Artificial Language"

2026-03-25

You can't study how models learn unfamiliar languages if every language is already in the training data.

Bafna, Shim, Plank, Yarowsky, and Sirin (arXiv:2603.22497) solve this with a trick: cipher a high-resource language. Take English, apply a consistent substitution cipher, and you have a language the model has never seen — but whose structure is known perfectly, because it IS English, transformed. The grammar is identical. The semantics are identical. Only the surface forms are new.

Note (2026-11-12; read against the paper's full text): the paper's cipher is a bijective character-level cipher, and it says this cipher maintains all structural linguistic properties of the base language and its language family apart from surface forms, including the usage patterns of ciphered words and morphemes and the morphosyntax; I found no ciphers that change morphological structure or a scale of cipher complexity, so those parts of the essay are not the paper's. Its example ciphers French into a language it calls Gerestal. For machine translation it reports chrF, xCOMET and the LLM-based GEMBA-MQM; BLEU appears only in its references, as a metric it cites as correlating worse with human judgement, and the essay's list of BLEU, accuracy and parse quality is not the paper's. The paper also tests strategies on downstream tasks beyond machine translation and includes manual analysis.

The Rashid framework makes this systematic. Reversible ciphers let researchers construct languages at controlled distances from familiar ones — simple substitution changes only characters, more complex ciphers change morphological structure. The key property: because the underlying language is high-resource, all standard NLP evaluation tools still work. You can measure BLEU, accuracy, parse quality — metrics that would require building evaluation infrastructure from scratch for a genuinely unseen language.

This unlocks experiments that were previously impossible. Want to know whether a model learns grammar from few examples? Cipher the grammar and test. Want to compare how different in-context learning strategies handle morphologically complex versus isolating languages? Cipher both types and compare. Want to systematically vary the distance between the unseen language and the model's training languages? Vary the cipher complexity.

The deeper insight: studying language learning requires controlling for prior knowledge, and the only way to guarantee zero prior knowledge is to construct the language after training. Real low-resource languages are low-resource for researchers but may have trace representation in training data through multilingual corpora, missionary dictionaries, or linguistic descriptions. A ciphered language has exactly zero representation — it didn't exist until the experiment started.

The limitation mirrors its strength. A ciphered language inherits the structure of its source. It can't test how models handle genuinely alien structures — click consonants, polysynthetic morphology, tonal semantics — because those structures aren't in the source language to cipher. The framework tests surface-form novelty, not structural novelty.