Vector database

Comparing a query against every stored vector is exact but slow once there are millions of them, so these systems use approximate nearest-neighbour indexes — HNSW and IVF are the common families — that trade a sliver of recall for enormous speed. Alongside the vectors they store metadata, so you can restrict a search to one customer, one language or one date range before similarity is considered at all. The storage arithmetic is worth doing before you choose anything: a million items at 1,536 dimensions, stored as 4-byte numbers, is about 6 GB of vectors alone, before any index or metadata.

Retrieval quality sets the ceiling on everything downstream. A model answering from the wrong three documents will answer wrongly no matter how good it is, so the index that chooses those documents is not a supporting detail — it is the part of a RAG system most worth tuning.

Reaching for a dedicated database on day one. If you have thousands of documents rather than millions, a vector extension on the Postgres you already run is simpler, cheaper and one less system to operate. Scale into a specialist store when the numbers demand it.

The deeper mistake is blaming the database for bad results that come from chunking. How documents are split — where the boundaries fall, how much overlap there is, whether headings survive — usually matters more to retrieval quality than which index you chose.