Jailbreak

The two get used interchangeably and are not the same attack. In a jailbreak the person typing is trying to get past the model’s own rules. In prompt injection a third party has hidden instructions inside content the model reads, and the user may be the victim rather than the attacker.

The distinction matters because the defences differ. Jailbreaks are mitigated in training and by output filtering; injection is mitigated by never letting retrieved content reach the model with the authority of an instruction. A system can be well defended against one and wide open to the other.

Safety training shapes what a model is likely to say, not what it is able to say. The capability is still in the weights, and a refusal is a learned tendency rather than an enforced rule — which is why phrasing a request as fiction, as a translation task, or as a hypothetical has historically been enough to move the model off the refusal.

Worse, attacks generalise. Research has shown that adversarial suffixes found against one open model transfer to other models, including closed ones the attacker cannot inspect, so patching a specific string does not close the class.

Treating a system prompt as a security boundary. Instructions in the system prompt are strong suggestions, not permissions, and they are the first thing a jailbreak targets. Anything that genuinely must not happen belongs in code that runs regardless of what the model decided.

The other mistake is assuming this only matters for chatbots. A jailbroken model with tools is a jailbroken model that can act — which is why the risk scales with capability rather than with how public the interface is.