A jailbreak is a prompt crafted to make a model ignore its own safety training and produce something it was tuned to refuse. The user is the attacker, which is what separates it from prompt injection, where the attacker is content the model reads.
Also called: jailbreaking, safety bypass · Reviewed
The two get used interchangeably and are not the same attack. In a jailbreak the person typing is trying to get past the model’s own rules. In prompt injection a third party has hidden instructions inside content the model reads, and the user may be the victim rather than the attacker.
The distinction matters because the defences differ. Jailbreaks are mitigated in training and by output filtering; injection is mitigated by never letting retrieved content reach the model with the authority of an instruction. A system can be well defended against one and wide open to the other.
| Jailbreak | Prompt injection | |
|---|---|---|
| Who is the attacker? | The person typing | A third party, through content the model reads |
| What is subverted? | The safety training the model was given | The instructions the developer set |
| Who is harmed? | Usually the model provider, or nobody | Usually the user, who never sees the hidden instruction |
| Where does the defence belong? | In training, and in filtering what comes out | In never granting retrieved content the authority of an instruction |
Safety training shapes what a model is likely to say, not what it is able to say. The capability is still in the weights, and a refusal is a learned tendency rather than an enforced rule — which is why phrasing a request as fiction, as a translation task, or as a hypothetical has historically been enough to move the model off the refusal.
Worse, attacks generalise. Research has shown that adversarial suffixes found against one open model transfer to other models, including closed ones the attacker cannot inspect, so patching a specific string does not close the class.
Treating a system prompt as a security boundary. Instructions in the system prompt are strong suggestions, not permissions, and they are the first thing a jailbreak targets. Anything that genuinely must not happen belongs in code that runs regardless of what the model decided.
The other mistake is assuming this only matters for chatbots. A jailbroken model with tools is a jailbroken model that can act — which is why the risk scales with capability rather than with how public the interface is.