Guardrails are the checks placed around a model — before the request, after the response, and around any tool it can call — that keep a system safe, in scope and predictable when the model does something unexpected.
Also called: safety layer, output validation · Reviewed
At the input, screen for content the system should not act on and strip or mark anything that arrived from an untrusted source. At the output, validate structure before anything downstream consumes it — a schema check that rejects malformed JSON prevents a whole class of failure that no amount of prompting will.
At the tool layer, apply least privilege. An agent with read-only credentials cannot delete a table however it is manipulated, and permissions enforced in code hold in situations where instructions in a prompt do not.
Models are probabilistic and their inputs are often supplied by strangers. A system whose safety depends entirely on the model behaving as instructed has one layer, and that layer can be talked out of its instructions. Defence in depth assumes the model will sometimes be wrong and arranges for that to be survivable.
The bar rises sharply with autonomy. Anything irreversible — sending, paying, deleting, publishing — deserves an explicit human confirmation rather than a model's judgement.
Putting the rules only in the system prompt. A prompt is a request, not an enforcement mechanism; the checks that hold are the ones outside the model, in code that runs regardless of what the model decided.
The opposite failure is smothering the product. Guardrails so aggressive that ordinary requests get refused train users to route around the system entirely, which is worse for safety than a narrower, well-placed set of checks.