RLHF

A base model trained only to predict text will happily continue your question with more questions, because that is what the internet looks like. RLHF fixes the mismatch between predicting text and being useful.

Human annotators are shown several candidate answers to the same prompt and rank them. Those rankings train a separate reward model to predict which answer a person would prefer, and the language model is then optimised against that reward. The humans never write the answers — they only judge them, which is far cheaper and scales further.

It is the difference between a model that is impressive in a research paper and one that is usable in a product. Instruction following, refusing harmful requests, admitting uncertainty and keeping to a format are all learned here rather than during pretraining.

It also explains a great deal of model behaviour that otherwise looks arbitrary — the hedging, the eagerness to please, the tendency to give an answer rather than say it does not know. Those are preferences somebody expressed, learned and amplified.

Assuming it adds knowledge. RLHF changes what a model tends to produce, not what it knows. A fact absent from pretraining is still absent afterwards, which is why alignment does not reduce hallucination about things the model never learned.

The second mistake is treating it as a safety guarantee. It shifts likelihoods rather than enforcing rules, which is precisely the gap a jailbreak exploits.