The idea
Comparing a query with every stored vector works for thousands of chunks, not millions. Vector databases store embeddings with their metadata and build an index that finds approximate nearest neighbours in milliseconds.
They also handle filtering by metadata, updates and deletes, replication and multi-tenancy. Options range from embedded libraries (Chroma, LanceDB) to self-hosted servers (Qdrant, Weaviate, Milvus) and managed services (Pinecone), and extensions to databases you already have (pgvector for Postgres).
Choose on scale, filtering needs, operations burden and whether you want vectors next to your relational data. For most products under tens of millions of chunks, Postgres with pgvector is a sensible default.
It is the storage layer of every RAG system and the main operational decision in it.
Words to know
- ANNApproximate nearest neighbour search: fast, very slightly imperfect.
- Collection / indexA named set of vectors with the same dimensions.
Try it without code
Compare three options
- Read the quick-start of Chroma, Qdrant and pgvector.
- Write down for each: how it runs, how filtering works, and how you would back it up.
Use it in code
An in-process vector store with Chroma in a few lines.
# pip install chromadb
import chromadb
client = chromadb.PersistentClient(path="./chroma")
col = client.get_or_create_collection("faq", metadata={"hnsw:space": "cosine"})
col.add(ids=["1", "2", "3"],
documents=["Refunds take 5 days.", "Reset your password in Settings.", "We ship to the EU."],
metadatas=[{"topic": "billing"}, {"topic": "account"}, {"topic": "shipping"}])
res = col.query(query_texts=["how long until I get my money back"], n_results=2)
print(res["documents"], res["distances"])Load 500 chunks into Chroma and into pgvector, and compare query speed and how filters are written.
Common mistakes
- Picking a managed vector DB before you know your scale.
- Keeping vectors in one system and permissions in another with no sync.