Inference

Two numbers matter. Time to first token is dominated by how much input the model has to read, which is why long prompts feel slow before they are expensive. Tokens per second then governs how quickly the answer appears, which is why streaming output matters so much to how fast a product feels even when total time is unchanged.

Cost tracks tokens in both directions, with output typically priced several times higher than input. A prompt that is re-sent on every turn of a long conversation is paid for on every turn.

Match the model to the task — routing simple classification to a small model and reserving the large one for hard cases is usually the largest single saving available. Cache aggressively, both your own repeated results and, where the provider supports it, the unchanging prefix of your prompts. Batch anything that is not interactive.

Where models run on your own hardware, quantisation reduces numeric precision to cut memory and increase throughput, usually with modest quality loss — worth measuring on your own evals rather than trusting a benchmark.

Budgeting from a prototype. Costs scale with usage, conversation length and retry behaviour all at once, and a feature that is trivially cheap for a team of ten can be the largest line item at ten thousand. Instrument token usage per request from the start.

The other is optimising the model choice while ignoring the prompt. Trimming a bloated system prompt that is sent on every call often beats any amount of model shopping.