Why relationships between documents matter
Fund work depends on how documents relate to one another. A lawyer needs to know which amendment governed a term on a given date, which side letter belongs to which investor, and whether a document is a signed agreement or an unsigned form. That information is spread across hundreds of files. How a system organizes the information determines answer quality, the number of steps an agent takes, and the cost of each answer.
What Reframe does
Reframe files each document to the client and matter the document belongs to, reads each document once, and records the funds, investors, provisions and dates in the document, together with the connections between those records. Any agent can then retrieve that record, with citations to the source, under the firm's access rules.
The graph has three parts. Records are the subjects of the work: funds, investors, agreements and provisions. Relationships record how two records connect: an amendment changes a provision, a side letter binds one investor, a restatement replaces an earlier agreement. Every fact also stores the quoted passage the fact came from. Because Reframe records the effective date of each document, an agent can retrieve from the graph the agreement that governed on any date.
For each limited partnership agreement, Reframe reads the agreement against the fund ontology and extracts every declared term: fees, carried interest, the distribution waterfall, key person rules, borrowing limits and investor consents, each with the sentence that states the term.
The test corpus
The benchmark uses a simulated fund formation practice of 1,000 documents across 25 sponsors, 139 matters and 91 funds. From these documents, Reframe built more than 28,000 records, including more than 21,000 extracted provisions, and more than 70,000 relationships between those records. The corpus includes 700 documents and 66 funds that the system had never seen before the test. On those 66 funds, Reframe extracted the fund-level terms with 99.5 percent accuracy.
How we measured
We ran 144 questions across 12 question types, using the same model for every system. We ran each question 8 times on each of three systems: an agent reading the raw files, advanced retrieval search, and Reframe. An independent model graded every answer against the answer key for the question.
One question
One question asked how many days a fund had to deliver its quarterly reports, as of 26 January 2023. The agent reading the raw files opened the documents individually and gave the wrong answer in all 8 runs. With Reframe, the same model found the terms in force on that date, answered correctly in all 8 runs, and showed a 62 percent improvement in turns over the agent reading raw files.
Where the graph helped most
| Measure | Improvement over retrieval search | Improvement over the agent reading raw files |
|---|---|---|
| Accuracy overall | 9 percent | 13 percent |
| Errors | 45 percent | 53 percent |
| Terms as of a past date | 67 percent | 51 percent |
| What a redline changed | 88 percent | 7 percent |
| Which funds lack a provision | 12 percent | 45 percent |
| Questions that require 6 or more exact facts | 25 percent | 92 percent |
The graph helped most on questions that depend on dates and on many funds at once. A question about a past date requires the agreement that governed on that date, and Reframe records the effective date of every amendment and restatement. A question about which funds lack a provision requires a review of every fund, and Reframe holds the terms of all 91 funds in a structured record.
Cost
| Measure | Improvement over retrieval search | Improvement over the agent reading raw files |
|---|---|---|
| Cost per question | 32 percent | 75 percent |
| Text read per question | 42 percent | 80 percent |
| Cost per rubric criterion met | 46 percent | 85 percent |
On portfolio questions that compare many funds, the improvement in cost per criterion met exceeded 90 percent over the agent reading raw files.
As the document set grows
We ran the same test on 300 documents and then on 1,000 documents. On portfolio questions at 1,000 documents, Reframe held its accuracy and showed a 9 percent improvement over retrieval search and 36 percent over the agent reading raw files, because retrieval search lost 10 percent of its accuracy and the agent reading raw files lost 16 percent as the document set grew.
Build once, reuse
Reframe builds the graph once, when each document is added, and every later question reuses the graph. An agent without the graph repeats the search and the reading for every question and pays for that work each time. With the graph, each question starts from work that is already complete, so each question costs less, and the savings grow with every question the firm asks.