WorksComputer Science
Reading Between the Chapters: A Graph-Free RAG Rebuilt With AI and Human Feedback
Abstract
Many questions about a book can only be answered by connecting passages chapters apart, such as a cough heard three times across a horse's life. We introduce CrossChapterQA, 1,853 such questions over 14 public-domain books. Each question lists the two to six chapters it needs (spanning half the book at the median) and the relation linking them, cites evidence from each, and has two independent reference answers graded blind. We use it to build LadderRAG, which indexes a book as passages, summaries and book-level cards for about $0.41 and answers a question for under a cent. Its first version failed mostly at retrieval. We redesigned it four times, steered by an LLM judge's verdicts and our analyses of failed answers: a design-level counterpart of RLAIF and RLHF in which feedback changes the architecture, not model weights. We also report the two redesigns that made answers worse. On the 191 questions its first version got wrong, the final version retrieves over twice as many required chapters on held-out books (recall 0.36 to 0.79), and ten annotators rate its answers 6.5/10 on average (764 ratings).
Introduction
In Anna Sewell’s Black Beauty, the narrator hears a cough three times: in a burning stable, after a hard season of hunting, and years later on a London cab stand, where it has become the sign of a ruined horse. A reader asked which cough recurs, and what it comes to mean, has to connect chapters 16, 24 and 40 and see that the three moments form a single decline. No passage answers the question on its own, and a passage that merely mentions a cough can lead an answer astray (Figure 1).

Figure 1. A CrossChapterQA question about Black Beauty. The thread joins the three chapters where Ginger’s cough is heard. v1 retrieved six passages from five other chapters (×), two of them about another character’s cough, and answered about the wrong character. v4b retrieved all three required chapters (dark green) among its passages.
Questions like this are common whenever people read a long text closely, and they are where retrieval-augmented generation (Lewis et al., 2020) is weakest. A retriever returns the passages most similar to the question, and those tend to cluster in one place. Graph-based systems such as GraphRAG [5] were built for questions about a whole corpus: they extract an entity graph and summarize its communities, at the cost of many model calls per document, which later work has tried to reduce [6] (Guo et al. 2024). Trees of summaries offer another route (Sarthi et al., 2024). What has been missing is a way to check, question by question, whether a system found the places an answer depends on.
Long-book benchmarks already show that models struggle when evidence is spread across a novel (Wang et al., 2025; Gupta et al., 2025; Karpinska et al., 2024), and some attach supporting evidence to each answer (Wang et al., 2025). None that we know of ensures that every question spans distant parts of the book, or records the relation that connects them, so it is hard to tell whether a wrong answer failed at retrieval or at reasoning. CrossChapterQA does both: every question lists two or more far-apart sections with evidence from each, and is typed by one of eight relations, from setup and payoff to the rationale behind the book’s ordering.
We then use the benchmark to build LadderRAG, which keeps GraphRAG’s goal of answering questions about a whole book but drops the graph. LadderRAG indexes each book once as a ladder of passages, summaries and five book-level cards in a single index. Its first version answered 1,119 sampled questions for $0.0031 each, and an LLM judge found 83% of the answers consistent with the references. The remaining 17% were mostly retrieval failures. We improved the system with a loop that borrows the logic of RLAIF and RLHF [2] (Lee et al. 2024; Ouyang et al. 2022) but applies it to design rather than weights. Each version answered a development book’s questions; an LLM judge flagged answers that contradicted the references; we wrote analyses of the failed answers; and the next version was designed against that feedback. No model was trained.
Our contributions are:
CrossChapterQA: 1,853 cross-chapter questions over 14 books (986k words), labelled with their required sections, relation type and per-section evidence, each with two reference answers from different frontier models graded blind (§ 3).
LadderRAG, a low-cost retrieval system for whole-book questions, and a documented redesign loop driven by AI and human feedback, including the two versions that made answers worse and why (§ 4–§ 5).
A human evaluation of the final system on the 191 questions its first version failed, with 764 ratings from ten annotators and full agreement statistics (§ 6).
We will release the questions, references, judgments, the code and outputs of every version, and the human ratings.1
Related Work
Questions over whole books. NarrativeQA asks questions about books and films, written from their summaries (Kociský et al., 2018). NovelQA includes long-dependency questions over novels longer than 200k tokens and gives evidence for its answers (Wang et al., 2025), NovelHopQA varies the number of reasoning hops over novel excerpts (Gupta et al., 2025), NoCha pairs true and false claims about recent novels (Karpinska et al., 2024), and DetectiveQA and ∞Bench probe long-context reasoning (Xu et al., 2024; Zhang et al., 2024). Xiang et al. (2025) compare graph and vector retrieval, partly on novels. CrossChapterQA differs in two ways: every question is built to span distant sections and lists them, and every question is typed by the relation that joins them. Retrieval can therefore be scored apart from the answer, and results can be broken down by the kind of connection a question asks for.
Retrieval over long documents. Beyond flat chunk retrieval, systems add structure. RAPTOR builds a tree of recursive summaries (Sarthi et al., 2024); GraphRAG, its lighter variants and HippoRAG build entity graphs (Edge et al., 2024a,b; Guo et al., 2024; Jiménez Gutiérrez et al., 2024); contextual retrieval prefixes each chunk with a sentence that places it in its document [1]. LadderRAG combines summary tiers and contextual prefixes with book-level cards and with section structure read from the text itself, and builds no graph.
Improving systems from feedback. RLHF and RLAIF train models on human or AI preferences [2] (Ouyang et al. 2022; Lee et al. 2024), and both have been applied to parts of RAG pipelines (Zhang et al., 2025a; Nguyen et al., 2024). Another line improves pipelines without training: DSPy tunes prompts against a metric (Khattab et al., 2024), TextGrad passes textual critiques back through a pipeline (Yuksekgonul et al., 2024), AutoRAG searches over RAG modules (Kim et al., 2024), and ADAS and AFlow search over agent designs (Hu et al., 2025; Zhang et al., 2025b). These methods optimize automatically against a score. Our loop changes the architecture by hand, combines an LLM judge (Zheng et al., 2023) with human error analyses, and records what each change did, including the regressions. Shankar et al. (2024) show how people refine their criteria while checking LLM evaluators; we bring the same attention to the system under evaluation.
CrossChapterQA
Books. We use 14 public-domain English books: Black Beauty, The Call of the Wild, A Christmas Carol, The Memoirs and The Return of Sherlock Holmes, The Hound of the Baskervilles, The Invisible Man, Little Women, Persuasion, The Prisoner of Zenda, The Railway Children, The Sign of the Four, A Study in Scarlet and Tarzan of the Apes. Together they hold 986k words in 290 sections: chapters, the five staves of A Christmas Carol, and the stories of the two Holmes collections. We cleaned each text and marked every section with an explicit header, so that a label such as “Chapter 27” or “Stave III” points to an exact span (Appendix § 9).
Questions. Every question can be answered only by combining two or more places in the book that are far apart, and its answer depends on how those places relate. The questions were written by an LLM, Claude Opus 5.5, working through each book chapter by chapter in an agentic session and writing questions from the text it was reading. Each question carries a list of required sections, a relation type and supporting evidence from every required section, mostly as verbatim quotations. Automatic checks confirm that every question requires at least two sections, that the first and last are at least two sections apart (one stave apart in A Christmas Carol, whose five staves run to about 6,000 words each), that no question names a chapter, stave or story, since that would hand the retrieval target to the system, and that no question duplicates another. We also reviewed the wording of all 254 Black Beauty questions.
Table 1 lists the eight relation types. Most questions need three sections (53%), 36% need two and 11% need four to six. The first and last required sections lie a median of 9 sections apart, which at the median is 49% of the book. Questions average 17 words. Figure 2 draws the questions as threads between the sections they join: 79% reach past a quarter of their book and half reach past its middle, so the evidence is rarely on neighbouring pages.
Figure 2. CrossChapterQA questions as threads. (a) Each grey arc joins two chapters that one Black Beauty question requires; four questions of different relation types are highlighted, one of them the cough of Figure 1. (b) For all 1,853 questions, the distance between the first and last required section as a share of the book: answering usually means connecting parts of a book that lie far apart.
| Relation type: what the answer must connect | n |
| Recurring device: a thing, phrase or event that returns, and what changes between returns | 360 |
| Setup and payoff: an early detail and the later scene that pays it off | 317 |
| Arc: how something develops across three or more stages | 240 |
| Local element, global function: a small detail and its larger role elsewhere | 209 |
| Document-wide technique: a narrative technique and its instances across the book | 201 |
| Framing symmetry: two scenes that mirror or contrast | 187 |
| Multi-hop chain: a chain in which each link leads to the next | 170 |
| Ordering rationale: why the book places one part before another | 169 |
Table 1. Relation types in CrossChapterQA (1,853 questions).
Reference answers. Each question has two reference answers, written independently. Claude Opus 5.5 wrote one while composing the question, with the evidence in view. GPT-5.7 Sol wrote the other from the question, its required sections and the book text, without seeing the first. The two styles differ (31 and 51 words on average), so before grading we normalized typography, removed every field that could identify the source, and shuffled each pair under neutral labels, balancing the positions within each book. A third model, OpenAI’s GPT-6 Astra, then graded every answer blind, using only the text of the required sections and a 100-point rubric: accuracy (30), coverage of the required sections (20), the relation between them (20), completeness (15), specificity (10) and discipline (5), with caps for invented events and for denying the question’s premise (Appendix § 12). Neither source dominated. Opus’s answer won 578 questions and GPT’s 561, and 714 were ties (totals within two points); both averaged 91.4. We take the winning answer, or both answers on a tie, as the gold reference. Gold answers average 97.2, and 88% score at least 90.
LadderRAG
Building the ladder. LadderRAG indexes a book once. It reads the sections from their headers, with no model calls, and splits each section into chunks of about 400 tokens with 50 tokens of overlap, never crossing a section boundary. A model then writes one sentence placing each chunk in the book, in one batched call per section ([1]). Next it summarizes each section (splitting sections longer than 12 chunks) and adds upper tiers of summaries over groups of three, as in RAPTOR ((Sarthi et al., 2024)). From the section summaries it writes five book-level cards: a structural card that includes a one-line map of every chapter, and cards for characters, terms, chronology and recurring themes. Passages (L1), summaries (L2) and cards (L3) share one index, searched with dense vectors (bge-small; Xiao et al., 2024) and BM25 (Robertson and Zaragoza, 2009) fused by reciprocal rank (Cormack et al., [4]). Every indexing call uses gpt-5-mini. Building the 14 ladders took 1,196 calls and cost $5.78 as billed, including retries and failed calls: $0.41 per book.
Answering (v1). A router (gpt-5-nano) labels each question LOCAL, MID or GLOBAL and picks extra cards. LOCAL retrieves six passages; MID retrieves three summaries and then four passages from the sections they cover; GLOBAL retrieves two upper-tier summaries and no passages. The structural card is always in the prompt, as a cached prefix. gpt-5-mini answers in one paragraph of about 120 words or fewer and cites the sections it uses.
Where v1 fails. We sampled 80 questions per book, stratified by relation type (79 for A Christmas Carol, which has 79), and v1 answered all 1,119 for $0.0031 each. Word-overlap metrics say little on this task: ROUGE-L (Lin, 2004) between v1’s answers and the references is 0.16, but between the two references themselves, both graded correct, it is only 0.20 (Appendix § 16). We therefore asked an LLM judge, Claude Sonnet 5.5, a narrower question about each answer: is it completely different from the reference, that is, does its main point contradict the reference or concern other events or characters? Differences in wording, missing or extra detail, partial answers and refusals do not count.
The judge flagged 191 answers (17.1%) and found the other 928 consistent with the references.
The flagged answers are mostly retrieval failures. We measure location recall, the share of a question’s required sections that appear among the retrieved passages (0 when no passages are retrieved). It averages 0.34 for the flagged answers and 0.46 for the rest (Mann–Whitney ), and 36% of the flagged answers retrieved none of their required sections. The router sent 55% of all questions to LOCAL, whose six passages come from a single similarity ranking, so near-duplicate passages from one chapter crowd out the other end of the link (Figure 1). Failure rates also follow the relation types. They are highest for local elements with a global function (24%) and for setup and payoff (23%), whose answers hinge on two specific scenes, and lowest for document-wide techniques, arcs and ordering (10–11%), which the book-level cards are built to cover. In 127 answers (11%) v1 said that the retrieved passages did not contain the information. Appendix § 10 breaks these failures down further.
Redesign from AI and Human Feedback
We improved LadderRAG on a single development book, Black Beauty, which both authors know well enough to judge an answer without rereading. Every version answered the same 80 sampled questions from the question text alone. Three signals shaped the next design:

Figure 3. LadderRAG at a glance. (A) Each book is indexed once, in four steps, into a ladder with three levels: passages (L1), section summaries (L2) and five book-level cards (L3). (B) The final version answers in four steps: it plans the question, finds evidence for each part among the passages, adds the book-level context, and answers in a single call (retrying with more passages if a part is missing). The coloured tags show which level of the ladder each step reads. (C) How v4b came about: each version answered the development questions, an AI judge flagged wrong answers, we analysed the failures, and the design changed (§ 5).
AI feedback. gpt-oss-120b ((OpenAI, 2025)), at high reasoning effort, applied the completely-different criterion to every answer. Re-judging identical answers moved the count by two, so differences of about two questions in 80 are noise.
Human feedback. We wrote analyses of the failed answers to 13 annotated questions, classifying each error and stating what the system should have done.
Diagnostics. Location recall, questions with no required section retrieved, refusals, and cost.
We proposed each change from these signals, implemented it with the help of an AI coding assistant, and kept it only after reviewing its results. No weights changed, and no answer was chosen by its judge score: each version answered once and was judged afterwards. In RLAIF, an AI judge’s preferences update a model’s weights; here the judge’s verdicts, together with our written analyses, update the system’s design, and each update is a code change that a person reviews. Table 2 follows the five versions, and Figure 4 plots their trajectory.
Figure 4. The redesign trajectory on the development book. Retrieval (top) improved almost every round, but answers (bottom, share flagged by the AI judge) did not follow: v3 and v4 retrieved more and answered worse. Below each version: cost per answer.
| Feedback that led to the version | Design change | Flagged↓ | Rev.↑ | Recall↑ | None↓ | $/ans. |
| v1 (first version) | router picks one route; one similarity ranking per route | 28/79 | 1/13 | 0.34 | 37 | 0.0029 |
| v2 AI judge and diagnostics: questions linking distant chapters routed as local; passages crowded into one chapter; two-part questions searched as one string | planner routes by the scope an answer needs; up to 3 sub-queries fused by RRF; at most 2 passages per chapter | 17/78 | 5–7/13 | 0.52 | 16 | 0.0033 |
| v3 Our review of 12 answers: a scene with the right words but the wrong event; first and last swapped; another character’s story credited to the narrator; categories merged; “three times” answered from one chapter | constraint planner; stemmed BM25, rare-word lookup, position masks; a verifier that can reject passages; “not found” allowed | 31/78 | 6/13 | 0.69 | 4 | 0.0161 |
| v4 AI judge: the verifier vetoed correct evidence. Our review of 13 answers: wrong anchor passage (8 of 13); objects read as symbols; one hop where two were asked; characters lost across re-names | literal anchors from the question’s own words; names via the characters card; reserved slots per part; a checklist in one answer call; one retry with 14 passages | 24/79 | 8/13 | 0.72 | 4 | 0.0050 |
| v4b AI judge: 13 answers that v2 had right went wrong: interpretive questions without book-level context; back-references and speakers | restore v2’s cards and summaries; rules for back-references and speakers | 17/80 | 8/13 | 0.71 | 7 | 0.0053 |
Table 2. The redesign loop on the development book (80 Black Beauty questions). Each row gives the feedback that prompted a version, the change made, and the result. Flagged: answers the AI judge found completely different from the reference, over the questions it gave a verdict on (35%, 22%, 40%, 30%, 21%). Rev.: of the 13 questions we analysed by hand, those not flagged (v2 was judged twice). Recall: mean location recall. None: questions with no required section retrieved.
v2: plan before retrieving. The first round of feedback named three problems: questions linking distant chapters were routed by their wording, the passage slots went to one chapter, and a two-part
question was searched as one string. In v2 a planner (gpt-5-nano) routes by how much of the book an answer needs, splits the question into up to three sub-queries that are searched separately and fused, and admits at most two passages per chapter. Recall rose from 0.34 to 0.52, and flagged answers fell from 35% to 22%. Letting the planner name chapters from the chapter map lowered recall to 0.49, so we switched that part off. Across all books, v2 also answered 114 of the 127 questions v1 had refused (22 of them only with twelve passages instead of six); the 13 that remained were exact-word misses, such as a “blistered heel” that the text describes as a blister on the heels.
What we found by hand. Our review of 12 failed answers found that eight were retrieval failures of specific kinds: a scene with the same vocabulary but the wrong event, a “last meeting” answered with the first, a story another character tells credited to the narrator, two of the mother’s “sorts of men” merged into one, and something “heard three times” answered from a single chapter. The common cause was that nothing checked the evidence against the question’s own constraints.
v3: a verifier, and a regression. v3 added that check as a separate step. A planner extracts each part’s constraints (early or late in the book, a count, the speaker), retrieval adds stemmed BM25, rare-word lookup and position masks, a verifier tests every passage against every constraint, and the answer may say “not found” instead of substituting another scene. Recall rose to 0.69, but flagged answers rose to 40% and the cost per answer to five times v2’s. The verifier often rejected correct evidence because the book did not use the question’s own words (it looked for a “third sort” of man, for example), and 16 of the 31 flagged answers contained “not found”.
v4: anchors first. A second review of 13 answers, sharpened the goal: correct and relevant answers at low cost, not stylistic parity with frontier models. Wrong anchor passages explained 8 of the 13 failures; other errors included reading literal objects as symbols, answering two-hop questions with one hop, confusing first and last, crediting the wrong speaker, and losing a character across re-names (Darkie, Black Beauty, Black Auster, Jack, Blackie, Old Crony). v4 moves the fixes into retrieval and a soft checklist. A planner (gpt-5-mini) splits the question into parts, turns descriptions into character names with the help of the characters card, and proposes anchors, literal words from the question such as “little brown book”. Code discards any anchor not made of the question’s own words, after we saw the planner steer search toward scenes it had guessed. Each part gets reserved passage slots, filled from exact anchor matches, rare terms, and dense plus stemmed-BM25 search, with one passage per chapter when a part asks for several instances. A single answer call receives the passages, a checklist of parts and a few rules (objects are literal; answer every part; say “not found” only for a part with no plausible passage), and the system retries once with 14 passages if a part is missing.2 Recall reached 0.72 and v4 answered 8 of the 13 reviewed questions acceptably, but flagged answers rose back to 30%. Without the cards and summaries, v4 lost the interpretive questions about why the book is arranged as it is, and it mixed up speakers and back-references.
v4b: anchors and the ladder. v4b restores the cards and summaries selected by v2’s plan, and adds two rules: a later part that refers back (“that horse”) keeps the same character, and a line belongs only to the speaker the passage names. It ties v2 for the fewest flagged answers (21%), keeps v4’s recall and its 8 of 13 review questions, and costs $0.0053 per answer. A further version, v5, added three more fixes and was flagged on 16 and 18 of 80 questions in two judgings, within noise of v4b, so we kept v4b.
What the loop taught us. Better retrieval did not guarantee better answers: v3 and v4 retrieved more of the required sections than v2 but answered worse, once because a separate step could veto good evidence and once because the book-level context was gone. The two kinds of feedback also did different jobs. The judge told us how many answers were wrong, and the written reviews told us why.
Human Evaluation
Setup. For the final evaluation we ran v4b on the 191 questions whose v1 answers had been flagged: 178 from the 13 held-out books and the 13 Black Beauty questions we had analysed. We re-answered only where v1 had gone wrong; the other 928 questions keep their v1 answers, which the judge had found consistent with the references. On the held-out questions, v4b’s location recall is 0.79 against v1’s 0.36, and only 6 of the 178 retrieved no required section, against 58 for v1. Each v4b answer here cost $0.0066.
Ten annotators, paid for their time, scored v4b’s answers. They worked in three pools. Pools A and B each had three members who know the stories well; pool A covered six books and pool B the other eight, and each member scored every answer in the pool’s books independently.
Pool C was a panel of four annotators from a university English department, who know all 14 books and scored every answer together, producing one score per answer. Every answer therefore received four scores, 764 in all. Annotators saw the question, the gold reference and v4b’s answer, and scored v4b’s answer against the book text on a scale from 1 to 10. The rubric (Appendix § 13) has them split the question into parts and mark each part correct, partly correct, missing or wrong. The answered parts set a ceiling of (parts answered / parts asked), from which they deduct for wrong parts, minor errors, invented details, a missing or thinly stated link, vagueness and padding. A refusal counts as a missing part, and length earns nothing.
Results. Figure 5 shows how the scores spread over the rubric’s bands. Averaged over the four scores, v4b’s answers to the questions v1 got wrong score 6.48, in the rubric’s “fair” band, where every part is touched but with weaknesses. Of the 191 answers, 116 (61%) average 6.0 or more, 71 (37%) average 7.0 or more, the “good” band in which every part is answered, and only 4 fall below 4.0. The held-out books average 6.44 and the development book 7.11. Scores follow retrieval: answers whose passages covered every required section average 6.88, against 5.99 for the rest (Spearman between location recall and score), while answer length is unrelated to score (; Appendix Figure 8). Among the relation types, arcs (6.86) and setup and payoff (6.74) score highest, and framing symmetry (6.18) and ordering rationale (6.19) lowest.
Figure 5. Human scores for v4b’s answers to the 191 questions v1 answered wrongly, by the rubric’s score bands (mean in brackets). “Pool member” scores come from pool A or B, depending on the book; the panel is pool C. Numbers inside the bars are percentages.
Where v4b still falls short. The three lowest-scored answers show the failures that remain, and none of them is a simple retrieval miss. Asked in what order four remembered episodes of The Prisoner of Zenda occur, v4b retrieved three of the four required chapters and still replied that the passages did not contain the narrator’s list. Asked how The Sign of the Four quotes the same author twice, it found a different book that Holmes quotes twice, a gazetteer rather than Winwood Reade. And asked why Persuasion dismisses Dick Musgrove so harshly before Wentworth arrives, it retrieved both required chapters but explained a different dismissal, after Louisa’s fall. A refusal despite the evidence, a similar scene chosen over the right one, and a “why” question answered with plot: the last two also remained open on the development book, and all three mark where the next round of feedback should go.
Agreement. The four scores order the answers similarly but sit at different levels: the two most lenient pool members average 7.7 and the panel 4.7. That offset pulls absolute agreement down. Krippendorff’s (interval; (Hayes and Krippendorff, 2007)) over all four scores is 0.22 (95% CI 0.16–0.28), and the single-score ICC(2,1) is 0.29 ((Shrout and Fleiss, 1979)). Consistency, which ignores the offsets, is higher at ICC(3,1) = 0.46, and the mean of the four scores, the quantity we report, is reliable: ICC(2,4) = 0.62 and ICC(3,4) = 0.77. Agreement is strongest between the third pool member and the panel ( = 0.67, 95% CI 0.58–0.74; quadratic-weighted = 0.64, [3]) and weakest between the two lenient members ( = 0.28). Fleiss’ ([7]) over the rubric’s eight score bands is 0.07, because band agreement counts every offset as a disagreement; Appendix § 15 gives all pairwise statistics. The offset is largest on the clearest failures: on the Persuasion answer above, the two lenient members gave 6.25 and 7.5, and the third member and the panel gave 1.0. The annotators thus rank the answers alike and differ mainly in how much they deduct for incomplete or wrong parts. Under the strictest reading, the panel’s, 20% of these answers are good or better and 39% are poor or failing.
Conclusion
CrossChapterQA asks questions whose answers live in more than one chapter, and it records where. That record was the most useful tool we had while building a system for such questions. It showed that our first system failed at retrieval, let us measure each redesign’s effect on retrieval and on answers separately, and warned us when the two came apart. The final version of LadderRAG answers questions about a whole book without a graph for under a cent each, and on the questions its first version got wrong, human annotators now rate its answers as fair on average and good in more than a third of cases. We hope the benchmark, the version history and the ratings help others build and test systems that can read between the chapters.
Limitations
Data. The questions and the reference answers were written by LLMs. Automatic checks guard the structure of every question, and we reviewed the wording of the Black Beauty questions, but some questions may admit valid answers that neither reference gives, and the gold answer is the better of two model answers rather than a human one. The books are English classics from the nineteenth and early twentieth centuries and are almost certainly in the pretraining data of the models we use, so some answers may draw on memory rather than on retrieved text.
Development and evaluation. We developed every version on Black Beauty and report it separately; the held-out evidence comes from the other 13 books. The screen of v1’s answers rests on one LLM judge, and both judges give a coarse yes-or-no verdict; re-judging the development book moved the count by about two questions in 80. We applied v4b to the 191 questions where v1 disagreed with the references and kept v1’s answers where it already agreed. Running v4b on the full sample is a natural next step to confirm that it keeps those answers; on the development book, where every version answered all 80 questions, v4b disagreed with the references less often than v1 (21% against 35%).
Human study. Annotators knew which answer came from the system and could see the reference while scoring. Absolute agreement is low because the annotators differ in leniency, so we report the mean of four scores alongside every agreement statistic. The panel’s scores are joint decisions rather than independent ones. We did not collect human scores for v1 on the same questions.
Comparisons. We do not compare LadderRAG head to head with GraphRAG, other structured retrieval systems or long-context models; our cost figures are measured on our own runs and are not matched comparisons.
Ethics Statement
Data and licences. The 14 books were published between 1817 and 1912 and are in the public domain. We took the texts from Project Gutenberg and other public-domain editions, and Appendix § 9 credits their authors. We will release CrossChapterQA, with its questions, evidence, reference answers, grades and human ratings, under CC BY 4.0, and our code under the MIT licence, for research on question answering and retrieval over long documents. We used the proprietary OpenAI and Anthropic models through their providers’ services and within their terms of use, and the open models under their licences: bge-small-en-v1.5 (MIT), gpt-oss-120b (Apache 2.0), roberta-large (MIT) and all-mpnet-base-v2 (Apache 2.0). The questions and reference answers were generated with LLMs and are labelled as such in the release.
Personal information and offensive content. The data concerns the events of the books and contains no personal information, and the human ratings are released without the annotators’ names. The books reflect the attitudes of their time and include dated and offensive portrayals, for example of Native Americans and Mormons in A Study in Scarlet, of Africans in Tarzan of the Apes and of an Andaman Islander in The Sign of the Four. Questions, evidence and references about these passages sometimes repeat the books’ terms. We keep the texts unaltered as historical documents.
Annotators. We recruited the ten annotators through our academic and personal networks: four members of a university English department (pool C) and six readers who know the books well (pools A and B). All ten live in Bangladesh, and we collected no other demographic information. They were compensated fairly for their time, were told how their scores would be used, and agreed to take part. Appendix § 13 gives their instructions. The task, scoring system answers about public-domain fiction, collected no personal data and carried no foreseeable risk, so it was not submitted to an ethics review board.
foreseeable risk, so it was not submitted to an ethics review board.
Risks. The gold references are written by models, so a system tuned to them can inherit their errors; the blind grading, the second independent reference and the human evaluation reduce this risk but do not remove it. LadderRAG can give fluent but wrong answers about a book, so it should support reading rather than replace it; its answers cite the sections they draw on, so readers can check them. Once released, the questions may leak into training data and inflate later results on the benchmark. The experiments used little compute (Appendix § 14).
AI assistance. Besides the models studied in this paper, we used AI assistants to write code and to help draft and edit the paper and its figures. The authors checked every claim and number and take full responsibility for the content.
Books and Sections
Table 3 gives per-book statistics and Figure 6 the make-up of the questions.
| Book | Words | Sections | Questions | Sampled | v1 flagged | Human mean |
| Black Beauty (development) | 59,886 | 49 | 254 | 80 | 13 | 7.11 |
| The Call of the Wild | 31,833 | 7 | 88 | 80 | 9 | 6.33 |
| A Christmas Carol | 29,975 | 5 | 79 | 79 | 8 | 7.77 |
| The Memoirs of Sherlock Holmes | 87,063 | 11 | 120 | 80 | 25 | 6.54 |
| The Return of Sherlock Holmes | 112,336 | 13 | 122 | 80 | 13 | 6.49 |
| The Hound of the Baskervilles | 59,193 | 15 | 130 | 80 | 6 | 6.99 |
| The Invisible Man | 48,469 | 29 | 134 | 80 | 5 | 6.34 |
| Little Women | 186,127 | 47 | 143 | 80 | 21 | 6.10 |
| Persuasion | 86,419 | 24 | 131 | 80 | 16 | 6.49 |
| The Prisoner of Zenda | 53,666 | 22 | 131 | 80 | 18 | 6.08 |
| The Railway Children | 59,377 | 14 | 138 | 80 | 10 | 5.64 |
| The Sign of the Four | 43,077 | 12 | 125 | 80 | 12 | 6.32 |
| A Study in Scarlet | 43,416 | 14 | 126 | 80 | 14 | 6.63 |
| Tarzan of the Apes | 85,523 | 28 | 132 | 80 | 21 | 6.62 |
| Total | 986,360 | 290 | 1,853 | 1,119 | 191 | 6.48 |
Table 3. The 14 books. Sampled: questions answered by v1. v1 flagged: v1 answers judged completely different, which v4b then answered for human scoring. Human mean: mean of the four scores over those answers.
Figure 6. What CrossChapterQA asks. (a) Questions per relation type. (b) Distance between the first and last required section, as a share of the book. (c) Number of required sections per question.
Every cleaned text marks each section with an explicit header. For indexing we also removed the table of contents and illustration captions from A Christmas Carol and the trailing Project Gutenberg licence from Persuasion. The two Holmes collections are divided into stories rather than chapters. Question labels use the book’s own numbering style (Arabic or Roman), and every label in the dataset resolves to exactly one marked section.
Sources and coverage. We took the texts from Project Gutenberg and other public-domain editions. All 14 books are English fiction by seven British and three American authors, published between 1817 and 1912: Jane Austen (Persuasion, 1817), Charles Dickens (A Christmas Carol, 1843), Louisa May Alcott (Little Women, 1868–69), Anna Sewell (Black Beauty, 1877), Arthur Conan Doyle (A Study in Scarlet, 1887; The Sign of the Four, 1890; The Memoirs of Sherlock Holmes, 1894; The Hound of the Baskervilles, 1902; The Return of Sherlock Holmes, 1905), Anthony Hope (The Prisoner of Zenda, 1894), H. G. Wells (The Invisible Man, 1897), Jack London (The Call of the Wild, 1903), E. Nesbit (The Railway Children, 1906) and Edgar Rice Burroughs (Tarzan of the Apes, 1912). They span detective fiction, adventure, the novel of manners, children’s fiction, animal stories, a ghost story and scientific romance. The questions and answers are also in English, and the release documents every field of the data.
Where v1 Fails, in Detail
Figure 7 breaks down v1’s failures. Questions that hinge on two specific scenes fail most often (a). Flagged answers are more likely to have retrieved none of their required sections and less likely to have retrieved all of them (b). The GLOBAL route, which answers from the book-level cards and summaries alone, fails least (c), even though it retrieves no passages.
Figure 7. v1 on the 1,119 sampled questions. (a) Share of answers the screening judge flagged, by relation type. (b) How many of a question’s required sections v1 retrieved, for flagged and unflagged answers. (c) Flag rate by the route v1’s router chose (number of questions in brackets).
Example Questions
Table 4 shows one question of each relation type. Questions describe characters and scenes by content (“the worn-out mare”, “a black beard”), so finding the right sections is part of the task.
| Relation type | Book | Question | Requires |
| Setup and payoff | Black Beauty | Which wish of the worn-out mare does the narrator afterwards repeat as his own? | Ch. 40, 47 |
| Recurring device | The Hound of the Baskervilles | How does a black beard recur as a false clue? | Ch. IV, V, VII |
| Arc | The Call of the Wild | How do the dog’s masters range from fair to fatal to ideal? | Ch. 1, 5, 6 |
| Local element, global function | The Sign of the Four | Why does the detective bother to have a green lamp removed from a boat? | Ch. VIII, IX, X |
| Document-wide technique | The Railway Children | How does the keeping and breaking of secrets organise the book? | Ch. I, VI, XI, XIII, XIV |
| Framing symmetry | The Invisible Man | How is the hidden face of the opening matched at the close? | Ch. I, XXVIII |
| Multi-hop chain | Tarzan of the Apes | How does Kala’s death lead, step by step, to Tarzan becoming king? | Ch. IX, X, XI |
| Ordering rationale | Persuasion | Why is Mrs Smith’s revelation held back until after the concert? | Ch. XVII, XX, XXI |
Table 4. One CrossChapterQA question of each relation type. Questions describe scenes by their content and never name the sections they need.
Rubric for Grading the References
The blind judge scored every reference answer out of 100, using only the text of the question’s required sections. Accuracy (30) starts at 30 and loses 10 for each major error on something the question asks about, 4 for each minor error and 2 for each plausible detail the text does not contain. Coverage (20) is proportional to the required sections the answer actually draws on. Relationship (20) asks whether the answer states the connection the question’s type calls for (for example, how a payoff answers its setup, or how each link of a chain leads to the next). Completeness (15) is divided among the parts of the question. Specificity (10) rewards concrete, identifiable detail over generic statements, and discipline (5) deducts for outside knowledge, speculation stated as fact, and material that does not address the question. Caps override the total: 20 for answering a different question, 30 for rejecting the question’s premise, and 40 for inventing an event, character or quotation. Length, naming style and tone are explicitly excluded, and a worked example on a book outside the set calibrates the scale.
Human Annotation Rubric
Annotators scored each answer from 1.0 to 10.0 with one decimal, in four steps. (1) Split the question into parts: each what, which or who it asks, each item in a list, and each named thing in a comparison. (2) Mark each part Correct, Partly, Missing (including “the book does not say”) or Wrong, and set the ceiling to : 10.0 when all parts are answered, 6.0 for half, 2.0 for none. (3) Deduct 1.0 for each wrong part, 0.5 to 2.0 for each partly answered part, 0.5 for each minor error, 0.3 for each invented detail, 1.0 for a missing link between the places (0.5 if thin), 0.3 to 1.0 for vagueness where the book is concrete, and at most 0.5 for padding; never below 1.0. (4) Check the result against the score bands (10.0 flawless, 9.0–9.9 excellent, 8.0–8.9 very good, 7.0–7.9 good, 6.0–6.9 fair, 4.0–5.9 weak, 2.1–3.9 poor, 1.0–2.0 fails), with an adjustment of at most 0.5 that must be explained in a note. Annotators worked alone, kept the book text at hand, used no AI tools, and were told that the book is the only evidence and that length and style earn nothing.
Screening Criterion and Models
Both AI judges (Claude Sonnet 5.5 for the screen of all 1,119 v1 answers, gpt-oss-120b for every version on the development book) saw the question, the gold reference answer or answers, and the system’s answer, and returned completely different or not. An answer is completely different only if its main point contradicts the reference or is about different facts, events or characters. Differences in wording or detail, partial answers, and answers saying that the passages do not contain the information are not completely different. When two references tied, an answer was flagged only if it differs from both. For gpt-oss-120b we capped reasoning at about 4k tokens and re-ran calls that returned no verdict.
All LLM calls made by LadderRAG went through the OpenRouter API. The parameter counts of the proprietary models are not public; gpt-oss-120b has 117B parameters (5.1B active per token), and bge-small-en-v1.5, which runs locally, has 33M. Token counts, including the 400-token chunk size, use tiktoken’s cl100k_base encoding. Indexing and every answering run of LadderRAG, including the versions we did not adopt, cost about $14 in API fees.
Agreement Details
Table 6 gives pairwise agreement. Rank correlations exceed the weighted kappas for every pair involving a lenient pool member, which is the signature of a level offset rather than a disagreement about order. Over all four scores, Krippendorff’s is 0.22 on the interval scale; ICC(2,1) = 0.29, ICC(2,4) = 0.62, ICC(3,1) = 0.46 and ICC(3,4) = 0.77. Confidence intervals come from 2,000 bootstrap resamples of the 191 answers.
| Pair | Pearson | Spearman | QW |
| Member 1 – Member 2 | 0.28 | 0.31 | 0.28 |
| Member 1 – Member 3 | 0.35 | 0.43 | 0.26 |
| Member 1 – Panel | 0.57 | 0.60 | 0.24 |
| Member 2 – Member 3 | 0.36 | 0.42 | 0.24 |
| Member 2 – Panel | 0.47 | 0.51 | 0.18 |
| Member 3 – Panel | 0.78 | 0.79 | 0.64 |
Table 6. Pairwise agreement over the 191 answers. QW : quadratic-weighted kappa over the rubric’s eight bands.
| Stage | Model (reasoning effort) |
| Situating sentences | gpt-5-mini (minimal) |
| Summaries, all tiers | gpt-5-mini (medium) |
| Book-level cards | gpt-5-mini (high) |
| Router (v1), planner (v2) | gpt-5-nano (low) |
| Planners (v3, v4, v4b) | gpt-5-mini (low) |
| Verifier (v3), answers | gpt-5-mini (medium) |
| Embeddings | bge-small-en-v1.5 |
| Screening judge | Claude Sonnet 5.5 |
| Development judge | gpt-oss-120b (high) |
| References | Claude Opus 5.5, GPT-5.7 |
| Reference grading | GPT-6 Astra |
Table 5. Models used at each stage. Situating sentences for two books were made at gpt-5-mini’s default effort before we set it to minimal.
Figure 8 shows the link between retrieval and human scores (a) and the spread across relation types (b). The intervals in (b) overlap widely: with 11 to 44 answers per type, the differences between types are suggestive rather than settled.
Figure 8. Human scores for v4b’s 191 answers. (a) Mean of the four scores against v4b’s location recall (jittered); bars mark the mean for answers that found fewer than half, at least half, or all of their required sections. (b) Mean score by relation type, with bootstrap 95% confidence intervals.
Word-Overlap Metrics
Two correct answers to the same question share few words (Table 7), so overlap metrics cannot separate right answers from wrong ones on this task. This is why we rely on an LLM judge for screening and on human annotators for the final evaluation.
| Metric | v1 vs. refs. | Ref. vs. ref. |
| ROUGE-1 | 0.26 | 0.28 |
| ROUGE-2 | 0.05 | 0.06 |
| ROUGE-L | 0.16 | 0.20 |
| BERTScore F1 | 0.08 | 0.15 |
| Embedding cosine | 0.62 | 0.54 |
Table 7. Overlap between v1’s 1,119 answers and the two references (mean over the references), and between the two references. ROUGE uses F-measures with stemming, BERTScore uses roberta-large with base-line rescaling ((Zhang et al., 2020)), and the embedding is all-mpnet-base-v2.
v4b Answer Instructions
After the cards, summaries, passages and question (the same layout as v1), v4b’s answer prompt ends with the following text, in which {checklist} lists the planner’s parts and their anchors:
CHECKLIST - the question’s parts and conditions: {checklist}
Before writing, pick for each part the passage whose scene matches the question’s own description (the exact object, person, place, first/last, who is speaking, which category). If two scenes are similar, use the one that matches the description, not the more general one. Objects in the question are literal objects in the story, not symbols. When a later part refers back (“that horse”, “the same way”, “that description”), it is asking exactly what you found for the earlier part - keep the same character. Attribute a line only to the speaker the passage names. Answer every part, each from the passage that fits it; if a part needs instances from several places, give each one. If no passage plausibly fits a part, say briefly that it was not found, for that part only.
Answer in a single paragraph of about 120 words or fewer, not counting bracketed citations. No headings or bullet points.
References
- [1]Anthropic. 2024. Introducing contextual retrieval. Anthropic News.
- [2]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional AI: Harmlessness from AI feedback. Computing Research Repository, arXiv:2212.08073.arxiv.org/abs/2212.08073
- [3]Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220.pubmed.ncbi.nlm.nih.gov/19673146
- [4]Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759.DOI
- [5]Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazawa Ness, and Jonathan Larson. 2024a. From local to global: A graph RAG approach to query-focused summarization. Computing Research Repository, arXiv:2404.16130.arxiv.org/abs/2404.16130
- [6]Darren Edge, Ha Trinh, and Jonathan Larson. 2024b. LazyGraphRAG: Setting a new standard for quality and cost. Microsoft Research Blog.
- [7]Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382.DOI





