Ask two engineers how to build legal AI and you can start a fight about acronyms. One camp says retrieval augmented generation is all you need: embed the documents, retrieve relevant passages, let the model write. The other says only a knowledge graph can be trusted with legal structure. Both camps are right about the other's weakness, and serious systems now use both layers.
This piece explains what each layer actually does, where plain retrieval genuinely works, where it breaks on legal corpora, and how the combined architecture fits together in practice.
What vanilla RAG actually does
The recipe is short. Split documents into chunks of a few hundred words each. Run every chunk through an embedding model, which maps text into a vector space where similar meanings sit near each other. At question time, embed the question, pull the nearest chunks, and hand them to a language model with instructions to answer from the provided text.
Notice what the recipe throws away. Chunking strips a passage from its document, its version, its parties, and its place in a negotiation. The embedding preserves meaning in the small and loses identity in the large. Every strength and every failure of the approach follows from that trade.
Still, the approach deserves credit before criticism. Embedding similarity is a real advance over keyword search: it finds "assignment restrictions" when you asked about "transfer limitations," which two decades of enterprise search could not do. For questions whose answer lives inside one or two passages that resemble the question, plain RAG is fast to build, cheap to run, and genuinely good.
What does the termination clause in this agreement say. Summarize the indemnity in this draft. Find our standard force majeure language. If your workload is dominated by questions like these, a plain retrieval pipeline serves you well, and you should be suspicious of anyone selling you more.
Where similarity search breaks on legal corpora
The trouble is that the questions carrying real weight at a firm have a different shape. Five patterns recur.
- Multi-hop questions. "Which of our funds have side letters with most favored nation provisions that the fee terms we just offered would trigger?" The answer requires connecting a new fact to funds, through side letters, to specific elections. No single chunk contains it, so no similarity search retrieves it.
- Aggregation. "What survival periods did we accept across our last forty sponsor deals?" This is a query over a population, not a passage. Retrieving the twelve most similar chunks answers a smaller question and silently drops the other twenty-eight deals.
- Version confusion. A deal file holds nine drafts of the SPA. Chunks from draft three and from the executed agreement embed almost identically. A retriever with no version awareness will quote abandoned language as if it were the deal, with complete confidence.
- Definitional dependency. A clause reading "upon a Change of Control, the Lenders may accelerate" means nothing without the definition forty pages away, which may sweep in a minority stake sale. The passages that matter are not textually similar to the question; they are structurally connected to the answer.
- Permissions. Retrieval has to respect ethical walls and matter-level security for the specific person asking, at the moment they ask. A vector index with no concept of identity returns whatever is nearest, which inside a law firm is a confidentiality incident waiting to happen. We describe permission-aware retrieval on our security page.
None of these are prompt engineering problems. They are structural: the index holds isolated passages, and the questions are about connected facts.
What a knowledge graph layer adds
A knowledge graph stores entities and typed relationships instead of loose text. That one change supplies exactly what similarity lacks.
Entity resolution. MERIDIAN CAP HOLDINGS in the billing system, Meridian Capital Holdings, L.P. in the engagement letter, and plain Meridian in an associate's email become one node. Questions about the client mean the client, not a string.
Typed relationships. Governs, amends, cites, adverse to, due by. Multi-hop questions become traversals: start at the fee terms, walk to the elections, return the affected funds. The path is deterministic and inspectable, not a similarity guess.
Time and version awareness. The graph knows draft three was superseded, when the amendment took effect, and which agreement controls today. Questions phrased "as of closing" stop depending on luck.
Aggregation. Populations become queries: all sponsor deals since 2022, their survival periods, grouped and counted, each figure tied to its source clause.
Provenance. Every edge carries pointers back to the passages that support it, so an answer can show its work. Why that matters for citation integrity is the subject of our piece on hallucinated citations.
What the graph cannot do is write. It holds structure, not prose. Ask it to summarize a negotiation history and it hands you nodes. The layers are complementary, which is the entire point. For the broader case for the graph itself, see what a legal knowledge graph actually does for a law firm.
The combined architecture
The production pattern is graph-guided retrieval. Parts of the literature call a version of this GraphRAG; the mechanics matter more than the label.
A question arrives. First, resolve the entities it mentions against the graph: which client, which matters, which agreements, and which versions currently control. Second, traverse the relationships the question implies: from the client to its executed agreements, from those agreements to the clauses and defined terms that govern the topic, from a clause to the definitions it depends on. Third, collect the source passages behind those nodes and edges, filtered by what the asking user is allowed to see. Only then does generation happen: the model writes from that curated, permissioned, version-correct context, citing the passages it was given.
Each layer does the work it is built for. The graph decides which material is legitimate: right entities, right versions, right permissions. Retrieval finds the sharpest passages within that legitimate set. The model turns grounded passages into prose. Remove any layer and a specific failure returns: remove the graph and version confusion comes back; remove retrieval and you lose the text itself; remove generation and you are back to search results.
Adoption can be incremental. Firms usually stand up grounded passage retrieval first, because it is immediately useful, then add graph structure where the failure modes bite: version control for active deals, entity resolution across the client base, deadline aggregation for the docketing function. The architecture tolerates being built in stages. What it does not tolerate is pretending the second layer is optional forever.
This is how Reframe is built: the per-tenant Context Graph supplies structure and permissions, and retrieval and generation operate inside the same boundary. The platform overview shows how the pieces connect.
Which layer answers which question
A rough dispatch table for the questions firms actually ask.
Plain retrieval is enough when the answer lives in a document you can already name: what does this clause say, summarize this brief, find the standard confidentiality rider. If a first-year could answer it with the document open, retrieval can too.
The graph must lead when the question spans entities, versions, or populations: our exposure to a counterparty across every open matter, the positions we conceded across a deal history, everything due in the next thirty days, every draft where we held a position later abandoned.
Both layers together whenever the answer must be written and trusted: memos, client alerts, negotiation summaries, anything an attorney will sign or send. The graph selects and grounds; the model composes; the citations survive checking.
The argument between RAG and knowledge graphs dissolves once you see them as layers with different jobs. Passages persuade; structure verifies. Legal work needs answers that are both readable and right, and delivering that combination is the design brief Reframe was built against.