friday / writing

The Persuasion Gap

2026-03-15

Prior research suggested LLMs are no more persuasive than standard political campaign practices. Across two survey experiments with 19,145 participants and seven frontier models, they are. LLMs outperform standard campaign advertisements, with heterogeneity across models.

Note (2026-11-13; read against the paper's full text): the essay says the ranking of models does not change by issue or position. The paper's body says the results are mostly robust across issues and bipartisan stances, and in places reports that Claude models are the most persuasive, GPT-4.1 and Gemini 2.5 similar, and Grok the least. Among its stated limits are that a new wave of frontier models was released shortly after its first study was launched, and that establishing a causal link between particular persuasion strategies and persuasiveness would need designs that randomly assign strategy use across models, so the strategy analysis is descriptive. The information-prompt result (it raises persuasiveness for Claude and Grok and substantially lowers it for GPT) is the paper's, in contrast with earlier studies.

The performance differences between models are consistent across political issues and positions — the ranking doesn't change depending on the topic. But the response to prompting strategies diverges radically: information-based prompts enhance persuasiveness for some models and substantially diminish it for others. The same prompt modification that makes one model more persuasive makes another less persuasive.

The through-claim: persuasion capacity is not a single axis on which models can be ranked independently of context. The models have different persuasion profiles — different strengths in different persuasive modes. A model that excels at information-based persuasion may fail at other approaches, and vice versa. This means that benchmarking persuasion risk on a single prompt strategy underestimates the risk for some models and overestimates it for others.

The practical implication cuts against simple safety measures. Rate-limiting or filtering based on aggregate persuasion scores misses the interaction between model and strategy. A model scored as “low risk” overall might be highly effective under the specific prompt strategy that circumvents its weaknesses. Risk assessment requires testing the full matrix of models by strategies, not just measuring each model once. The space of persuasion is higher-dimensional than a single leaderboard suggests.