Hallucination

A model is trained to produce likely text, not true text, and it has no internal signal separating "I know this" from "this is the shape an answer would take". Asked for a citation, it generates something with the cadence of a real citation: plausible authors, a plausible journal, a plausible year. Fluency is preserved end to end; accuracy is not, because accuracy was never the objective being optimised.

Rare, specific and recent facts are the danger zone. Anything thinly represented in training data — a small company, a niche API, last month's release — is where invention is most likely and hardest to spot.

Grounding beats instruction. Retrieval-augmented generation, where the model answers from documents you supply and cites them, removes most of the guesswork for factual questions. Letting the model call a tool — a search, a database query, a calculator — is the same principle: replace recall with a lookup.

Then verify. Ask for quotes rather than summaries so claims can be checked against the source, keep a human between the model and any irreversible action, and build an eval set from the failures you have actually seen so you can tell whether a change helped.

Believing the problem can be prompted away. "Do not make anything up" and "only answer if you are certain" help at the margin and fail exactly where they are needed most, because a model that could reliably detect its own uncertainty would not have hallucinated in the first place.

The other error is treating confidence as evidence. Tone carries no information about correctness — a fabricated answer reads exactly like a sound one, which is precisely why hallucination is dangerous rather than merely annoying.