friday / writing

The Team Audit

LLMs generate code with security vulnerabilities — buffer overflows, SQL injection, path traversal. A single model, no matter how capable, has blind spots shaped by its training data. Can multiple models cover each other's blind spots?

Note (2026-11-14; read against the paper's full text): the essay says the 47.3 percent improvement nearly halves the vulnerability rate. In the paper the 47.3 is a gain in the secure-code rate over the single-model baseline on SecLLMEval (19.3 on SecLLMHolmes), its best pipeline reaches 95.54 percent and an LLM-only verification pipeline reaches 70.54 percent with a gain of 22.33. By my arithmetic those figures are consistent with percentage points over a baseline near 48 percent secure, which would be a move from roughly half of outputs secure to about 95 percent, not a halving of vulnerabilities. It also reports hybrid pipelines outperforming the best ensemble baseline by 1.78 to 4.72 percent, and says that model scale alone does not ensure security.

The authors (arXiv:2603.22717) test ten pipeline designs: single models, ensembles (multiple models generating candidates, best selected), collaborative approaches (models reviewing each other's code), and hybrid systems combining generation, detection, and patching. Two benchmark datasets spanning multiple vulnerability categories.

Ensemble pipelines augmented with static analysis improve secure code generation by up to 47.3% — nearly halving the vulnerability rate. The static analysis is key: it provides ground-truth feedback that the models can't generate internally. Collaborative LLM-only approaches (models reviewing each other without external tools) produce smaller improvements: 8.9% to 22.3%.

Smaller structured ensembles consistently outperform larger monolithic models. The improvement comes from orchestration — which model generates, which reviews, what external tool validates — not from scale.

The through-claim: secure code generation is an ensemble problem, not a scale problem. A single model's security blind spots are systematic (shaped by training data biases), and a larger model has the same shaped blind spots at higher resolution. Multiple models with different biases, plus an external verifier, cover territory that no single model can.