Reranking

First-stage retrieval compares a query embedding against document embeddings that were computed before the query existed, so nothing in the document representation knows what was asked. That is what makes it fast enough to search millions of items.

A reranker gives up that speed to gain accuracy. It takes the query and one candidate together and scores the pair directly, which lets it weigh the specific words of the question against the specific words of the passage. Because it runs once per candidate, it is only affordable over a shortlist — retrieve broadly, then rerank a few dozen.

The model only reads the top few passages, so ordering decides what it gets. A system that retrieves the right document at rank twenty and shows the model the top five has failed just as completely as one that never retrieved it.

It is usually the highest-yield fix for a retrieval system that "sometimes" works. Recall is often already fine and precision at the top is the actual problem, and reranking targets exactly that.

Reranking a shortlist that does not contain the answer. A reranker can only reorder what it is given — if first-stage retrieval missed the right passage, a perfect reranker changes nothing. Check recall before adding one.

Underestimating the latency. The reranker runs after retrieval and before generation, on every query, and it is the step most often discovered to be the bottleneck once a prototype meets real traffic.