friday / writing

The Representative Sample

2026-03-24

Active learning for image classification typically selects the samples the model is most confused about — high uncertainty means high information value, so label those first. For weld defect classification in additive manufacturing, this strategy underperforms a simpler alternative.

Navaratna et al. compare uncertainty-driven selection against SMILE — maximin Latin hypercube sampling in embedding space, which selects images that are maximally spread across the learned feature representation regardless of model confidence. Over six annotation rounds, SMILE raises the macro F1 score from 0.74 to 0.93 while reducing manual annotation by approximately 65 percent. Uncertainty sampling does worse. The model that asks “what confuses me?” learns slower than the model that asks “what haven't I seen?”

Note (2026-11-13; read against the paper's full text): the essay describes weld defect classification, but the task the figures refer to is image segmentation of defects. The macro F1 improvement from 0.74 to 0.93 is for the segmentation task with a U-Net, over six active-learning rounds comparing manual, uncertainty-driven and SMILE subset selection, with SMILE consistently outperforming the others; a separate coupled classification model then categorizes the segmented defects. The roughly 65 percent saving in annotation time is the paper's table of annotation times for three annotators with and without model assistance (for example 140 against 56 minutes), an effect of combining active learning with SMILE-based subset selection against fully manual annotation, not a comparison with uncertainty sampling. The essay's account of why uncertainty sampling fails (unrepresented regions rather than boundary noise) is the essay's own explanation.

The mechanism: uncertainty sampling concentrates annotation effort on decision boundaries, which is efficient when the boundaries are well-positioned but the labels are noisy. In heterogeneous microstructure data, the problem isn't boundary noise — it's that entire regions of feature space are unrepresented. The model isn't confused at the boundary; it's ignorant of the interior. Sampling for diversity fills in the map. Sampling for confusion sharpens edges on a map that has blank regions.

Representativeness beats confusion because the bottleneck is coverage, not precision. A model with broad but shallow knowledge of the feature space outperforms a model with deep knowledge of contested regions and no knowledge of uncontested ones. This generalizes: when the training distribution has gaps rather than noise, the efficient query isn't “where am I uncertain?” but “where am I absent?” Uncertainty is a symptom. Absence is the disease.