Knowledge distillation compresses a large model into a smaller one by training the small model to match the large model's outputs. The knowledge lives in the weights. The process requires training — gradient descent, data, compute — and the result is a model whose reasoning is encoded in parameters that humans cannot inspect.
Note (2026-11-13; read against the paper's full text): the method is not 'no training'. The paper describes a framework that uses a labeled training dataset: instructions are extracted from training examples with a teacher model, then clustered, merged and checked in a conflict-resolution phase; what it avoids is updating the student model's parameters. The 57 to 90 and 67 to 83 percent figures are macro F1 for Gemma-3 4B against its own zero-shot baseline. In the paper's table the fine-tuning baseline for the same model reaches 0.89 on StereoSet (against 0.90) and 0.76 on Contract-NLI; for Mistral Small 3.1 the StereoSet fine-tuning figure is 0.96 against 0.97, and LogiQA accuracy moves from 0.67 to 0.70 for Gemma and stays at 0.56 for Mistral. The evaluation is on reasoning-heavy classification tasks with static decision boundaries, and the paper says the method may face limitations where runtime computation is needed, such as complex arithmetic or symbolic proofs. The paper reports the method ahead of fine-tuning on Contract-NLI, so calling the instruction path 'lower-capacity' is the essay's framing.
Badhe and Shah (arXiv:2602.21103) propose a different kind of distillation: extract the large model's reasoning patterns and encode them as structured natural-language instructions in a system prompt for the smaller model. No training. No weight modification. The knowledge lives in the prompt, not the parameters.
The results are substantial: a 4B-parameter model jumps from 57% to 90% on StereoSet and from 67% to 83% on Contract-NLI using distilled instructions. The smaller model doesn't learn to reason — it receives instructions about how to reason, and follows them.
The advantage is interpretability. The decision logic is human-readable. In regulated industries — law, finance, healthcare — this matters: an auditor can inspect the prompt and verify the reasoning strategy without reverse-engineering neural network weights. The reasoning is transparent because it was never opaque — it was extracted as text.
The general observation: knowledge can be transferred between systems in two forms: as parameters (opaque, high-capacity, requires training) or as instructions (transparent, lower-capacity, zero-shot). When the reasoning pattern is describable in natural language — when it has structure that words can capture — the instruction path is not just cheaper but better: it preserves interpretability that parameter transfer destroys.