Distillation

A large teacher model is run over a set of inputs, and a smaller student is trained on what the teacher produced rather than on the original labels. The teacher’s full distribution over possible outputs carries more information than a single correct answer does — the relative weight it gives to the wrong answers says something about the structure of the problem, and the student learns from that too.

It is not the same as quantization. Quantization keeps the same model and stores its numbers less precisely; distillation produces a genuinely different, smaller model. The two compose — a distilled model can then be quantized.

When one task matters and generality does not. A distilled model that only has to classify support tickets or extract fields from invoices can be dramatically smaller than the model it learned from, because it never has to be good at anything else.

It is also the standard answer to latency floors that no amount of infrastructure will fix. If a task must return in tens of milliseconds, a smaller model is the only lever with enough range.

Expecting the student to hold up outside the distribution it was trained on. It inherits the teacher’s behaviour on the inputs it saw and has no obligation to match it anywhere else, so a distilled model that is excellent on your evals can be poor on a category nobody thought to include.

The other mistake is skipping the eval set. Distillation trades quality for cost, and without a fixed set of cases measured before and after there is no way to know how much quality was actually traded.