Building AI Agents That Survive Production · Part 20 of 44

Knowledge Graphs for Agent Grounding: Worth the Build?

Microsoft measured GraphRAG beating vector RAG on 72-83% of sense-making questions. Microsoft then shipped LazyGraphRAG, which indexes at 0.1% of that cost. When the vendor builds the cheap version, the expensive version's price was the finding. Here is where a graph actually earns its keep, and the hybrid most teams land on instead.

The question is not graph versus vectors

Every knowledge-graph discussion starts in the wrong place. It is framed as a retrieval bake-off — embeddings against triples, cosine against Cypher — when the only thing that decides the answer is which failure mode your agent actually has.

Vector search has one structural limit, and it is not accuracy. It returns the k passages most similar to the query. If the answer does not exist in any single passage, no value of k produces it. Ask which of our enterprise customers were onboarded by someone who has since left, and the corpus may contain onboarding records and a departures list and never a sentence that joins them. Top-k over a corpus that does not contain the answer returns k near misses, confidently.

That is the shape a graph fixes: the answer is a join across documents, not a passage inside one. If your production failures are instead wrong chunk, stale document, or bad reranking, a graph fixes nothing and adds an extraction pipeline you now have to run forever.

What the graph actually buys, measured

The reference numbers come from Microsoft Research's GraphRAG paper. Two corpora — 1,669 podcast chunks at about 1M tokens, 3,197 news chunks at about 1.7M — extracted into an entity graph (8,564 nodes and 20,691 edges for the podcast set), partitioned hierarchically with Leiden, with an LLM summary written for every community at every level.

On global sense-making questions, GraphRAG beat naive vector RAG on comprehensiveness 72-83% of the time on the podcast corpus and 72-80% on news, all at p<.001, with diversity wins in the same band. These are real effects, not rounding.

The underrated result is the token economics at query time. Answering from root-level community summaries cost 26,657 tokens against 1,014,611 for stuffing the source text — over 97% fewer. A graph is a precomputed index over relationships, and like every index it trades build cost for query cost. Note also that the paper tested 8k, 16k, 32k and 64k context windows and the 8k window won on comprehensiveness at 58.1%. Bigger windows did not make the structure unnecessary.

The multi-hop benchmark that should cool you down

Now the part the vendor decks omit. On actual multi-hop QA benchmarks, graph methods do not dominate a strong embedding model — they frequently lose to it.

From the HippoRAG 2 evaluation, F1 on MuSiQue: NV-Embed-v2, a plain dense retriever, scores 45.7. GraphRAG scores 38.5. RAPTOR scores 28.9. On HotpotQA it is 75.3 dense against 68.6 for GraphRAG. A well-chosen embedding model, with no graph at all, beat the graph pipeline on the benchmark the graph pipeline is supposed to own.

And LightRAG — a popular, heavily starred graph-RAG implementation — scores 1.6 on MuSiQue, 11.6 on 2Wiki and 1.0 on HotpotQA. That is not a tuning gap, that is a pipeline that does not work on this task. The lesson is the one worth taking into an interview: "graph" is not a quality tier. The topology does not confer correctness, and a graph built by a weak extraction step is worse than no graph.

# The query shape that decides the architecture, not the vendor's demo.

# Answerable by top-k: the answer lives in one passage.
"What is our refund window for enterprise plans?"
#   -> one policy doc, one chunk, cosine finds it. Ship vectors.

# NOT answerable by top-k at any k: the answer is a join.
"Which enterprise accounts were onboarded by a CSM who has since left?"
#   -> onboarding records name the CSM.
#      the HR export names the leavers.
#      no single chunk contains the intersection.

# Bucket 200 real production queries this way before you build anything.
# If fewer than ~15% are joins, the graph is a quarter you will not get back.
HOPS = {"single_passage": 0, "join_two_docs": 1, "traverse_n": 2}

Where the win concentrates

Read the same table by benchmark instead of by method and the picture sharpens. On 2WikiMultiHopQA — the set whose questions are explicitly compositional, built from Wikipedia relations — the original HippoRAG scores 71.8 against 61.5 for NV-Embed-v2. A ten-point win for the graph on the same table where it lost MuSiQue by seven.

The difference is what the question is made of. 2Wiki questions are traversals over typed relations that exist in the data as relations. MuSiQue questions need semantic interpretation across passages, which is what embeddings are for. The graph wins where the edge is real and named, not where the connection is a matter of meaning.

So the honest scoping rule: build the graph only over the relations you can extract reliably and would recognise as correct in a spot check. Entities with stable identity — accounts, parts, people, services, versions — and edges a domain expert would name out loud. Extracting "relates_to" between fuzzy noun phrases produces a graph whose edges mean nothing, and then PageRank over it means nothing either.

The cost Microsoft quietly admitted

The GraphRAG paper reports its own build cost plainly: indexing the 1M-token podcast corpus took 281 minutes on a 16GB VM against gpt-4-turbo. Every entity extraction and every community summary at every level is an LLM call, and that scales with corpus size, not with query volume.

Then Microsoft shipped LazyGraphRAG, and the numbers in that announcement are the real argument. Its data indexing cost is "identical to vector RAG" and "0.1% of the costs of full GraphRAG", with global query cost more than 700 times lower at comparable answer quality. It gets there by deleting the expensive part: no LLM summarization at index time at all. Noun-phrase extraction builds a concept co-occurrence graph with classical NLP, and all LLM use defers to query time, bounded by a relevance-test budget.

When the team that published the method ships a version that indexes for a thousandth of the price, the original indexing cost was the finding. Read it as the correction it is: the community-summary layer is the optional, expensive luxury, and the graph structure is the cheap part.

The maintenance nobody budgets for

Build cost is a one-time line item people will approve. Maintenance is the one that kills these projects, and it follows from a single property: extraction is an LLM call, so it is not deterministic.

Run the same document through your extractor twice and you can get Acme Corp and Acme Corporation as two nodes. Re-ingest a corrected document and you have added a parallel subgraph rather than updated one. Vector stores are forgiving here — re-embed a chunk, overwrite by id, done. A graph has no such idempotency unless you build it, because the identity of a node is a decision your pipeline makes on every run.

That means entity resolution is not a nice-to-have you add in phase two; it is the thing that makes the graph a durable asset instead of a snapshot that rots. And it means the deletion path matters: an edge sourced from a document that has since been retracted will keep answering questions until something explicitly removes it. Budget for a resolution strategy, stable keys, and provenance on every edge, or budget for a quarterly rebuild and admit that is what you are doing.

# Idempotent upsert: the only thing that makes re-ingest safe.

def entity_key(name: str, etype: str) -> str:
    """Stable key, computed deterministically — never the LLM's spelling."""
    canon = CANONICAL.get(normalize(name), normalize(name))  # alias table
    return f"{etype}:{canon}"

MERGE_NODE = """
MERGE (e:Entity {key: $key})
  ON CREATE SET e.name = $name, e.type = $type, e.first_seen = $ts
  ON MATCH  SET e.last_seen = $ts
"""

MERGE_EDGE = """
MATCH (a:Entity {key: $src}), (b:Entity {key: $dst})
MERGE (a)-[r:REL {type: $rel, doc_id: $doc_id}]->(b)
  ON CREATE SET r.ts = $ts, r.confidence = $conf
"""

# Provenance on the edge is what makes retraction possible at all:
RETRACT = "MATCH ()-[r:REL {doc_id: $doc_id}]->() DELETE r"

# Re-ingesting a document = RETRACT its edges, then re-MERGE.
# Without doc_id on every edge, a corrected document can only add.

The hybrid most teams end up with

Here is where the evidence actually points, and it is not "replace your vector store". HippoRAG 2 is the one structure-augmented method in that evaluation that does not pay for its multi-hop gains with losses elsewhere — 48.6 on MuSiQue and 71.0 on 2Wiki, while also holding 63.3 on Natural Questions against 61.9 for the dense baseline. Every other graph method in the table degrades on simple single-hop QA: GraphRAG drops to 46.9 on NQ, LightRAG to 16.6, against 61.9 dense.

Its architecture is the lesson. Dense retrieval stays the backbone: passages are nodes, linked to the phrases they contain. The query matches graph triples, an LLM filter discards the irrelevant ones, surviving phrase nodes plus all passage nodes become seeds for Personalized PageRank, and passages are ranked by the resulting score. Critically, if no triples survive the filter, it falls back to plain embedding ranking.

That fallback is the whole design philosophy in one branch. The graph is an expansion and reranking signal layered over retrieval that already worked, not a replacement for it — which is also why it does not collapse on the easy questions that are most of your traffic.

def retrieve(query: str, k: int = 10) -> list[str]:
    # 1. Dense retrieval is the backbone and the floor.
    passages = vector_store.search(query, k=50)

    # 2. Graph layer: match triples, then FILTER before trusting them.
    triples = graph.match_triples(embed(query), top_k=25)
    kept = llm_relevance_filter(query, triples)   # recognition memory

    if not kept:
        return [p.id for p in passages[:k]]       # pure-vector fallback

    # 3. Seeds: surviving entities + every passage, weighted by similarity.
    seeds = {entity_key(e): score for t in kept for e, score in t.entities}
    seeds.update({p.id: p.score * PASSAGE_WEIGHT for p in passages})

    # 4. Personalized PageRank spreads relevance across real edges.
    ranked = graph.personalized_pagerank(reset=seeds, damping=0.5)
    return [n for n in ranked if n.startswith("passage:")][:k]

The decision rule, and the interview answer

Do not decide this from a benchmark table, including the ones above. Decide it from your own query log, because the answer is entirely a property of your traffic. Pull a few hundred real queries, label each one by how many documents the answer requires, and measure recall for your current retriever on each bucket. If single-passage queries dominate and your recall there is the problem, you have a retrieval-quality project, not a graph project.

If a meaningful slice are joins, there is still a cheaper rung on the ladder before the graph: a tool that queries your actual relational data. Most enterprise relations already live in Postgres with real foreign keys, resolved entities and a deletion path. Re-extracting them into a probabilistic graph with an LLM is paying to make reliable data unreliable. Extraction is for relations that only exist in prose.

The interview answer is three sentences. Vector search fails structurally when the answer is a join across documents, and that failure is invisible — it returns confident near misses rather than nothing. A graph is a precomputed index over those joins, so it earns its keep in proportion to how many of your queries are traversals, and it costs an extraction pipeline with an entity-resolution problem you own forever. Microsoft's own LazyGraphRAG result says buy the structure and skip the LLM-summary layer, and HippoRAG 2 says keep dense retrieval as the backbone with the graph as an expansion signal and a fallback to plain embeddings.

# Run this before the design doc, not after the quarter.
from collections import defaultdict

def should_build_graph(queries, retriever, gold) -> dict:
    buckets = defaultdict(list)
    for q in queries:
        buckets[n_docs_required(q, gold)].append(q)

    report = {}
    for n_docs, qs in sorted(buckets.items()):
        hits = sum(recall_at_k(retriever(q), gold[q], k=10) for q in qs)
        report[n_docs] = {"share": len(qs) / len(queries),
                          "recall@10": hits / len(qs)}
    return report

# The pattern that justifies the build looks like this:
#   {1: {'share': 0.71, 'recall@10': 0.93},
#    2: {'share': 0.22, 'recall@10': 0.31},    # <- the join cliff
#    3: {'share': 0.07, 'recall@10': 0.08}}
#
# Flat recall across buckets means your problem is chunking or reranking.
# A cliff at n_docs >= 2 that covers real traffic is the graph's case.
PRACTISE IT · SYSTEM DESIGN LABS

Work the same ideas as a full interview case study, with pressure questions and model answers.

Keep reading

Related articles