Quantization stores a model at lower numerical precision than it was trained at, so each weight takes fewer bits. The model gets substantially smaller and faster, with some loss of quality.
Also called: model quantization, INT8, INT4 · Reviewed
Weights are typically trained at 16 or 32 bits each. Quantization maps them onto a smaller range — 8 bits, or 4 — by recording the range each group of weights covers and expressing each one as a step within it. Going from 16-bit to 4-bit cuts memory roughly fourfold.
The subtlety is that not all weights matter equally. Good quantization keeps the sensitive parts at higher precision and compresses the rest hard, which is why two 4-bit versions of the same model can differ noticeably in quality.
It decides what you can run and where. A model that will not fit in available memory at full precision may fit comfortably quantized, which is the difference between needing several accelerators and needing one — and between running on a server and running on a laptop or a phone.
Smaller weights also move faster. Much of generation is limited by moving data rather than by arithmetic, so a smaller model is often quicker in direct proportion to how much smaller it got.
Assuming the loss shows up evenly. Quantized models often score almost identically on general benchmarks while degrading noticeably on the things that need precision — long reasoning chains, code, careful instruction following and rare languages. The benchmark says fine; your actual task may not be.
The second is quantizing before checking whether you needed to. A smaller model at full precision frequently beats a large model heavily quantized, at the same memory budget, and is the option people skip past.