How many photons does it take to recognize a shirt? In conventional machine vision: millions. Each pixel needs enough photons to estimate intensity with reasonable signal-to-noise. A 28x28 grayscale image at modest bit depth means hundreds of thousands of detected photons minimum. Ma et al. (arXiv: 2603.23974) achieve 73% accuracy on FashionMNIST with 4.9 total detected photons per inference.
Reader note (2026-11-17): Checked against the full text of arXiv:2603.23974 (Ma, Laydevant, Sohoni, Wright, Wang and McMahon; v1), which I had earlier held as a weak read. The headline numbers match the abstract: 73 percent on FashionMNIST with an average of 4.9 detected photons in total per inference (82 percent at 17), and 86 percent on MNIST at 8.6 photons (97 percent at 29). The mechanism in the essay's third paragraph is not the paper's design. The essay describes an image of pixels that each detect zero or one photon, a sparse binary pattern. In the paper, that is the direct-imaging comparison; the PANS experiment projects a small number of learned nonnegative illumination patterns onto the object (the feature dimension is varied from 3 to 32), and a single-pixel single-photon detector gives one single-shot binary outcome per pattern, so the data the classifier sees are those few binary measurements, not a sparse image. The paper also calls the experiment a proof-of-principle demonstration, reports the photon count as an average over the test set, and adds simulations for other tasks (for example through multimode fibers, where passive PANS reaches about 90 percent on MNIST at about 10 photons while direct imaging needs hundreds of photons to exceed 50 percent).
Not 4.9 photons per pixel. Not 4.9 photons per feature. 4.9 photons total, across the entire image. The system makes a classification decision from fewer photons than a digit has pixels.
The method, photon-aware neuromorphic sensing (PANS), jointly optimizes the optical front-end and the computational back-end while incorporating photon statistics into the loss function. At extreme photon scarcity, each pixel detects zero or one photon according to a Poisson process. The image is not an intensity map but a sparse binary pattern — mostly zeros, with a handful of ones at random locations weighted by the object's reflectance. Training the system to classify from this sparse stochastic input means the optical design must concentrate information into the few photons that arrive.
On MNIST (simpler shapes), 86% accuracy at 8.6 photons. On FashionMNIST (shirts, bags, shoes — harder): 73% at 4.9 photons, 82% at 17 photons. Orders of magnitude below what conventional imaging requires.
The through-claim: the bottleneck in machine vision is not information but convention. Standard imaging systems collect far more photons than classification requires because they aim to reconstruct the image first and classify second. Reconstruct-then-classify wastes photons on spatial details irrelevant to the decision. When you optimize the entire pipeline — optics plus computation — for the classification task directly, the photon budget collapses. Most of what a camera sees is irrelevant to what it needs to decide.
Ma, Laydevant, Sohoni, Wright, Wang & McMahon, 2603.23974. Neuromorphic sensing / photon-limited imaging / machine vision / optical computing / photon statistics.