What you will have at the end
- An ingestion script that turns a folder of PDFs into searchable chunks
- A chat UI or CLI with page-level citations
- An eval report: Recall@5 and MRR for three pipeline variants
Step by step
- Run Postgres with pgvectordocker run pgvector/pgvector:pg16, create the extension and a chunks table with doc, page, heading, body, embedding and a tsvector column.
- ParseConvert each PDF to page-tagged Markdown. Inspect three outputs by eye before going further.
- Chunk and embedHeading-aware chunks of ~800 tokens with overlap. Embed with a local model, insert with metadata. Add an HNSW index after loading.
- Vector + keyword searchWrite two search functions, one using <=> and one using ts_rank, each returning ranked chunk IDs.
- Fuse and rerankMerge with RRF (top 30 each), rerank the merged list with a cross-encoder, keep the top 5.
- Answer with citationsNumber the 5 chunks in the prompt, require [n] citations, validate them, and render "file, page" links.
- Build the eval setWrite 30 questions with labelled relevant chunk IDs, including 5 that the documents cannot answer.
- Measure and write upReport Recall@5 and MRR for vector-only, hybrid, and hybrid + rerank. Put the table in the README.
DONE WHEN
- Every answer shows clickable page citations
- Unanswerable questions get a clear "not in the documents"
- The README shows Recall@5 for three variants
Stretch goals
- Add contextual retrieval and measure the gain
- Add query rewriting for follow-ups
- Per-user document collections with a tenant filter