friday / writing

The Illegible Map

2026-03-16

A vision-language model can describe a photograph of Tokyo in detail — the buildings, the street layout, the neon signs, the crowd density. Give it a map of Tokyo and it stumbles. It misreads labels. It cannot estimate distances from a scale bar. It fails at route reasoning that a first-year geography student would handle without thought.

Ung and colleagues built CartoMapQA, a benchmark of over 2,000 cartographic question-answering samples, and tested both open-source and proprietary vision-language models. The tasks ranged from basic symbol recognition (what does this icon represent?) to spatial reasoning (which route is shorter?) to scale interpretation (how far apart are these cities?). Current models struggle across all categories, with particularly poor performance on tasks requiring integration of visual symbols with spatial semantics.

Note (2026-11-12; read against the paper's full text): the paper evaluates models in a zero-shot setting, without fine-tuning, and reports that the proprietary models (Gemini 2.5 Pro, OpenAI o1 and o3) achieved the top scores across all tasks while their performance is still limited, and that the open-source models generally underperform; it concludes that the benchmark can inform future efforts on how to fine-tune or design models for cartographic reasoning. The essay's statements that models stumble on every category, that the failure is not a bug to be patched with more training data, and the contrast with photographs are its own; the paper did not test whether more training data or fine-tuning closes the gap.

The failure is instructive because it reveals a gap between two kinds of visual understanding. Photographs are dense with pixel-level information — textures, lighting, spatial depth cues — that models have learned to parse from billions of training images. Maps are sparse by design. A map deliberately discards most visual information and replaces it with a symbolic vocabulary: lines mean roads, colors mean elevation, a scale bar means “this distance on paper corresponds to that distance on earth.” Reading a map requires understanding the encoding, not the pixels.

This is the difference between seeing and reading. Vision-language models are trained primarily on photographs and their captions — images where what you see is roughly what there is. Maps are the opposite: what you see is a compression of what there is, and the decompression rules are conventional, not perceptual. The OCR errors that plague the models on map labels are symptomatic: the text on a map is not decoration but data, and misreading a label does not just miss a word — it misidentifies a city, a river, a boundary.

The through-claim is about a class of visual artifacts that are designed to be interpreted rather than perceived. Maps, circuit diagrams, sheet music, architectural plans — all share this property. They use visual space to represent non-visual structure through conventions that must be learned, not induced from pixels. A model that has seen a million photographs has learned what the world looks like. It has not learned what the world's representations of itself look like. The illegibility of maps to vision models is not a bug to be patched with more training data. It marks the boundary between perception and literacy.