Chain of thought

A model does a fixed amount of computation per token. Asked for an answer immediately, it has to reach a multi-step conclusion in one shot. Asked to work through it, each step it writes becomes part of the context for the next, so the intermediate results are available to build on rather than having to be held implicitly.

In practice this is as simple as adding "think step by step" or asking for the working before the conclusion. Newer reasoning models do this internally by default and may hide the steps, but the mechanism is the same one.

It turns a class of problems from unreliable into reliable. Arithmetic, multi-hop questions, planning and anything with constraints to satisfy improve substantially, often without changing the model or the data at all.

The written steps are also the cheapest debugging tool available. When an answer is wrong you can see which step went wrong, which is far more actionable than a wrong answer with no visible route to it.

Reading the steps as an account of what the model did. The reasoning is generated text, produced by the same process as the answer, and a model can produce plausible steps that do not correspond to how it reached its conclusion — including correct steps followed by a wrong answer, and wrong steps followed by a right one.

The second is using it everywhere. On simple extraction or classification, chain of thought adds latency and tokens for no accuracy gain, and it can hurt by giving the model room to talk itself out of an obvious answer.