betaFeedback

WorksComputer Science

GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs

Abstract

The needle-in-a-haystack (NIAH) benchmark was introduced as a test for measuring a model's ability to find small relevant facts within a very large body of input text. However, existing NIAH benchmarks each suffer from one or more of the following limitations: (1) reliance on synthetic data, so their evaluations may not generalize to real-world settings; (2) reuse of previously published datasets, leading to the risk of benchmark contamination; (3) dependence on manual annotation, which requires human effort to create and update data and is prone to future benchmark contamination; and (4) unrealistic simplification of long-context multi-hop reasoning where each next step is trivial given previous steps, while real-world reasoning often requires planning over multiple possible proof paths. To address these limitations, we introduce \Benchmark, a dynamic benchmark built from real citation relationships among scientific papers published after the evaluated models' knowledge cutoff date, which utilizes real-world information while ensuring no benchmark contamination. The fully automated benchmark construction pipeline samples representative subgraphs of the citation graph formed by these papers and converts each subgraph into a long-context QA example with a verifiable answer. We use GraphNIAH to evaluate eight LLMs from the GPT-5.6, Gemini-3, and Qwen3.5 families. We find that accuracy falls as questions require longer reasoning paths: at question depths of 16 and 32, the GPT-5.6 and Qwen3.5 models answer YES/NO questions no better than random guessing, mostly by answering NO; four of the eight models answer no counting example correctly, and the others model get worse as the subgraph size increases. These findings suggest that current LLMs often stop searching the citation relationships in their context before they reach the answer, even though every fact they need is in the context.

Introduction

With the rapid evolution of large language models (LLMs) in recent years, their native context length has also increased significantly from 2,048 tokens of GPT-3 (Brown et al., [6]) to 128k tokens of LLAMA-3.2 (Grattafiori et al., [23]), and now, to 1M tokens of DEEPSEEK-V4 (DeepSeek-AI, [13]). With context extension techniques, such as RoPE-based scaling (Liu et al., [44]; Ding et al., [14]) and long-context training (Hu et al.,

2025; Gao et al., 2025), context length can be extended to upwards of 4M tokens. With this ability to receive extremely long input text sequences, LLMs have been adopted for many complex long-context tasks, such as deep research ([15]), long-context summarization ([34, 65]), and long-horizon reasoning and planning ([7, 17]). Furthermore, as downstream tasks increase in complexity and input prompts increase in length, it is becoming increasingly important for LLMs to be able to process key information in very long contexts, and then combine it with other information to solve such tasks ([30, 48, 1]). The needle-in-a-haystack (NIAH) test ([28]) evaluates whether LLMs can retrieve a small relevant fact, called a needle, from a long context containing unrelated information. However, single-hop NIAH tests do not accurately reflect real-world scenarios where relevant facts are often derived from multiple parts of the context or conversation. Hence, several multi-hop NIAH benchmarks have been proposed, including RULER ([25]), Sequential-NIAH ([69]), and U-NIAH ([19]), which require models to connect multiple facts.

Despite these advances, existing NIAH benchmarks face at least one of four limitations (see Table 1). First, many of them are built on synthetic data ([58, 56, 25, 69, 45] or LLM-generated content ([19]), which may poorly reflect real-world use cases. Second, some benchmarks, including Zero-SCROLLS ([54]), LongBench ([4]), InfBench ([37]), LongGen-Bench ([43]), and ICR² ([50]), are built on previously published NLP datasets. Relying on these published datasets leads to benchmark contamination when the data appears in the model’s training corpus ([53, 67, 11, 10]). Third, some NIAH benchmarks rely on human effort to create new datasets ([19] (Yuan et al. 2025)). Expanding or updating these benchmarks requires additional human effort, making them difficult to scale; once the data is published, it is invariably incorporated into later LLMs’ training data, resulting in future benchmark contamination. Finally, most current NIAH benchmarks require LLMs to connect retrieved facts along a single fixed logical path, where the next retrieval step is unambiguous given the previous step, to derive the final answer ([25, 69, 56, 46]). However, downstream tasks and applications often involve interconnected information in the context, where the next retrieval step is not always unambiguous and the model must perform search over multiple potential proof paths ([16, 62] (Wang et al. 2026a)). We use non-linear reasoning to describe this kind of reasoning, which requires proof search to find the correct proof in a large space of possible proofs (or to find that no such proof exists). Most NIAH benchmarks (see Table 1) do not fully evaluate this kind of proof search; Figure 1 illustrates the difference.

Figure 1. Three kinds of NIAH tests. In each row, the left column shows the LLM input context with its needles, and the yellow boxes show the text of needles; the right column shows the question and its search space from ss to tt. Single-hop NIAH (Kamradt, [28]) needs one lookup, and in RULER (Hsieh et al., [25]) and BABILong (Kuratov et al., [32]) the next step is unambiguous. In GraphNIAH, needles describe papers and their references; ss and tt lie in different connected components (dashed line), so the model must search all paths from ss to answer NO.

BenchmarkMulti-hop ReasoningReal DataUpdatableConstruction without Human AnnotationNon-linear Reasoning
Naive NIAH ([28])✗✗✓✓✗
RULER ([25])✓✗✗✓✗
BABILong ([32])✓✗✓✓✗
LV-Eval ((Yuan et al., 2025))✓✓✗✗✗
NeedleBench ((Li et al., 2025a))✓✗✓✓✓
Counting-Stars ([56])✓✗✓✓✗
Sequential-NIAH ([69])✓✗✗✓✗
NeedleChain ([46])✓✗✓✓✗
NoLiMa ([45])✓✗✓✓✗
U-NIAH ([19])✓✗✓✗✗
GraphNIAH (Ours)✓✓✓✓✓

Table 1. Comparison of NIAH benchmarks across five desiderata: (1) support for multi-hop reasoning, (2) use of real data, (3) the ability to update benchmark data in the future, (4) construction without manual annotation, and (5) support for non-linear reasoning. Here, non-linear reasoning requires searching among possible proof paths to find a valid proof or establish that none exists, which is in contrast with linear reasoning, where each next proof step is unambiguous given the previous steps.

To address these gaps, we propose GraphNIAH, a dynamic NIAH benchmark built from real scientific papers and their citation relationships. For each example, the benchmark construction pipeline samples a representative subgraph of the real citation graph from Semantic Scholar (Kinney et al., [29]), in which each node is a paper published after the evaluated LLM’s training cutoff date and nodes are connected by cites and cited_by edges. The pipeline then converts each subgraph into a long-context QA example, where the papers in the subgraph and their bibliographies comprise the context, and the ground truth answer is computed directly from the subgraph. This process requires no manual annotation, so new examples can be generated automatically as new papers are published. GraphNIAH focuses on reachability: whether a target paper is reachable from a source paper, i.e., connected to it by a sequence of citation relationships stated in the input text. Each sampled subgraph contains several connected components, so that some papers are reachable from the source paper while others are not. GraphNIAH supports three tasks: (1) YES/NO reachability, (2) multiple-choice reachability, and (3) reachability counting (see Figure 2). To measure and analyze LLMs’ ability to perform non-linear reasoning over facts from very long contexts, GraphNIAH is designed with many different variables as “control knobs”, including question depth (i.e., the number of nodes on the path from the source paper to the target paper in the component), the subgraph size, the target context length, and various types of distracting content, i.e., information not supporting the correct reasoning or the answer.

Figure 2. Abstract visualization of three reachability tasks in GraphNIAH. Each task is visualized with the question and ground truth answer from an example, together with the connected components constructed for that task. Each paper node in the diagram represents a paper, while each edge between two paper nodes represents the citation relationship between the two papers (i.e., cites or cited_by relationships). Orange and blue paper nodes indicate source and target papers, respectively. For the YES/NO reachability task shown in panel (b), the source and target paper nodes are unreachable from each other because they belong to different connected components, resulting in the NO label for this example. Green highlights around some paper nodes and citation edges in the diagrams indicate the correct paths the model must take to travel from the source paper to the target paper. For the reachability counting task (c), the answer counts all other papers starting from the source paper, so the entire source component is highlighted in green.

We evaluate eight LLMs on GraphNIAH: GPT-5.6-SOL and GPT-5.6-LUNA (OpenAI, [47]), GEMINI-3.1-PRO (Google, [20]), GEMINI-3.8-FLASH (Google, [22]), GEMINI-3.6-FLASH (Google, [21]), and three QWEN3.5 models (Qwen Team, [51]): QWEN3.5-397B-A17B, QWEN3.5-122B-A10B, and QWEN3.5-35B-A3B. Our analysis shows three main findings. First, accuracy decreases as question depth grows, and at large depths, GPT-5.6 and QWEN3.5 models answer YES/NO questions no better than random guessing, mostly by answering NO (§4.2).

Second, reachability counting is the hardest task: half of the models fail to correctly answer any counting question, and the other models struggle as the component size grows (§4.2 and § 9.1). Third, neither the type of distracting content nor a higher level of reasoning effort reliably changes accuracy (§ 9.3 and § 9.4). We make the code and benchmark data publicly available at https://github.com/trminhnam/GraphNIAH.

Related Work

Long-Context Benchmarks. The context length of LLMs has grown in recent years, leading to the emergence of long-context benchmarks. Broad benchmarks, such as ZeroSCROLLS (Shaham et al., [54]), LongBench (Bai et al., [4]), InfiBench (Li et al., [37]), L-Eval (An et al., [3]), and LongICLBench (Li et al., [39]), evaluate LLMs’ ability to handle long-context inputs across different downstream tasks, including question answering (QA), summarization, and code understanding. These benchmarks are highly important as the input sequences of LLMs in real-world scenarios grow longer. However, many existing long-context benchmarks are built on previously published datasets (Shaham et al., [54]; An et al., [3]), making them vulnerable to benchmark contamination and difficult to update over time. In contrast, our work focuses on constructing a scalable and automatically updatable long-context evaluation using real-world data.

Needle-in-a-haystack Benchmarks. Kamradt [28] introduced the NIAH test, which hides one or a few needles in very large block of unrelated text and asks single-hop questions about the embedded information. Later benchmarks extend this setting: LV-Eval (Yuan et al., 2025) and LooGLE (Li et al., [36]) cover several long-context tasks, RULER (Hsieh et al., [25]) adds multi-hop tasks with synthetic needles, U-NIAH (Gao et al., [19]) compares retrieval-augmented generation (RAG) systems with LLMs, and Sequential-NIAH (Yu et al., [69]) asks the LLM to extract multiple needles and list them in their correct temporal or logical order. As Table 1 shows, most compared NIAH benchmarks rely on synthetic data, some are difficult to update, and all except NeedleBench (Li et al., 2025a) evaluate only linear reasoning, in which every next step is unambiguous given the previous steps (Hsieh et al., [25]; Moon and Lim, [46]; Yu et al., [69]; Song et al., [56]; Modarressi et al., 2025). GraphNIAH keeps the NIAH setting but uses real citation relationships among papers published after the evaluated LLMs’ knowledge cutoff date and requires non-linear reasoning.

Dynamic Benchmarks. Unlike static benchmarks, whose examples are fixed after creation, dynamic benchmarks can be updated over time ([55, 12, 24, 8]). They follow three main strategies. The first strategy builds new examples from data published after the knowledge cutoff date of the evaluated LLMs, which lowers the risk of benchmark contamination ([8]): for example, recent arXiv papers in AcademicEval ([72]), post-cutoff knowledge in AntiLeak-Bench ([66]), recent programming contests in LiveCodeBench ([27]), and recent math competitions in LiveBench ([63]). The second strategy generates examples with rules, for example, from directed acyclic graphs in DyVal ([75]), from the rules of the Mathador game in Mathador-LM ([33]), or from randomly generated databases in S3Eval ([35]). The third strategy

uses LLMs to paraphrase examples of static benchmarks ([9, 68, 76]) or to generate new ones ([49, 60]). Some benchmarks combine these strategies, such as LatestEval ([41]), DARG ([73]), and C²LEVA ([40]). GraphNIAH combines the first two: it samples subgraphs from a citation graph of papers published after the knowledge cutoff date and converts each subgraph into an example with rules, so new examples can be generated automatically as new papers appear.

GraphNIAH Benchmark

Each example in GraphNIAH is generated using a subgraph which we sample from the citation graph in the Semantic Scholar Academic Graph Datasets ([29]). To sample these subgraphs, we use a preferential-attachment-based sampling procedure inspired by the scale-free nature of real-world citation graphs ([59, 74]).

Task Definition

Task Definition

In GraphNIAH, each example consists of a question, a context, and a ground truth answer. The context contains descriptions of scientific papers and their references, which indicate the relationships between the papers. Given a source paper, our task asks one of the three types of questions: (1) whether a target paper is reachable via a path of citation relationships, (2) which paper out of a list of candidates is reachable, or (3) how many other papers are reachable. In this subsection, we define the citation subgraph of each example and the reachability tasks.

Citation Subgraph

The Semantic Scholar citation graph is far too large to fit into the context window of an LLM, so for each example qq, we sample a subgraph Gq=(Vq,Eq)G_q = (V_q, E_q) of the Semantic Scholar graph. Each node in VqV_q is a paper, and each directed edge (u,v)∈Eq(u, v) \in E_q means that paper uu cites paper vv, or equivalently, that vv is cited by uu.

Reachability Tasks

In all three reachability tasks, the fundamental subtask is to determine whether a paper is reachable from another paper via a sequence of edges, where each edge is either cites or cited_by; Or equivalently, whether there exists an undirected path between the corresponding source and target nodes. To construct examples in which some papers are reachable from a given source paper and others are not, we sample subgraphs containing multiple connected components. For a paper uu, let KuK_u denote the connected component of GqG_q that contains uu; then a paper vv is reachable from uu if and only if v∈Kuv \in K_u. Importantly, reachability is defined with respect to the subgraph GqG_q, rather than the Semantic Scholar citation graph. Thus, two papers in different connected components of GqG_q are considered unreachable from each other, even if they are connected in the Semantic Scholar citation graph. The model must therefore answer each question using only the citation relationships stated in the context. Based on this definition of reachability, we formulate three reachability tasks: YES/NO reachability, multiple-choice reachability, and reachability counting.

We use qYNq_{\mathrm{YN}}, qMCq_{\mathrm{MC}}, and qcountq_{\mathrm{count}} to denote an example of the YES/NO, multiple-choice, and counting tasks, respectively, and yYNy_{\mathrm{YN}}, yMCy_{\mathrm{MC}}, and ycounty_{\mathrm{count}} for their ground truth answers. Additionally, qYESq_{\mathrm{YES}} and qNOq_{\mathrm{NO}} denote examples of the YES/NO reachability task with YES and NO labels, respectively.

YES/NO reachability

Given a source paper ss and a target paper tt, the model has to answer whether tt is reachable from ss, i.e., whether t∈Kst \in K_s. The ground truth answer is yYN=YESy_{\mathrm{YN}} = \mathrm{YES} if t∈Kst \in K_s and yYN=NOy_{\mathrm{YN}} = \mathrm{NO} otherwise.

Multiple-choice reachability

Given a source paper ss and NN candidate papers t1,t2,…,tNt_1, t_2, \ldots, t_N, the model has to select the candidate that is reachable from ss. In our implementation, there is exactly one reachable candidate tit_i from ss (i.e., ti∈Kst_i \in K_s for some 1≤i≤N1 \le i \le N and tj∉Kst_j \notin K_s for all j≠ij \ne i), and the ground truth answer is tit_i.

Reachability counting

Starting from a source paper ss, the model has to determine the number of distinct papers that are reachable from ss, i.e., the component size of KsK_s, but excluding ss itself. The ground truth answer is a non-negative integer, formulated as ycount=∣Ks∣−1y_{\mathrm{count}} = |K_s| - 1.

Example Construction Pipeline

The generation of each example in GraphNIAH is dependent on the following variables:

  • Task: the task of the example (i.e., YES/NO reachability, multiple-choice reachability, or reachability counting).

  • Question depth kk: the number of nodes on the path from the source paper to the target paper, including both endpoints (§ 7.2).

  • Component size: the number of nodes in each connected component (§ 7.2).

  • Distractor type: code, essays, books, wikipedia, or disjoint-component (Step 4).

  • Target context length: the maximum number of tokens in the context (Step 4).

In the subgraph of each example, the number of connected components is determined by its reachability task. For qNOq_{\mathrm{NO}}, we need one component that contains ss and another that contains tt. For qYESq_{\mathrm{YES}}, we also generate two components, and ss and tt are in the same component, so that YES-label and NO-label examples differ only in the location of tt. For qMCq_{\mathrm{MC}}, we generate NN components: one contains ss and the reachable candidate tit_i, and each other component contains one of the other candidates. For qcountq_{\mathrm{count}}, we generate two components, one of which contains ss while the other prevents the model from simply counting all papers in the context. It is important that these components are sampled i.i.d. (independent and identically distributed), so that there are no shortcuts that would allow the model to answer the question without searching through the citation relationships. We require that no two components share a paper.

Below, we describe the generative process for the connected components (Steps 1 and 2), and then how we convert the subgraph into the input prompt (Steps 3 and 4). Details are in Appendix § 12.

Step 1: Sampling the backbone. Our goal is to generate each example such that at least d=k−1d = k - 1 steps of search are required to find the solution. To this end, we want the subgraph to contain at least two disjoint paths with dd edges starting from the source paper ss: one true path, which leads to the target paper tt, and one distractor path. Similarly, since the model may search backward from tt, we want at least two disjoint paths with dd edges starting from tt: the true path and another distractor path. Thus, to generate such a connected component, we first sample an undirected path of 3k−23k - 2 distinct papers, all published on or after April 1, 2026. We partition this path into three paths of kk nodes each, where consecutive paths share one endpoint: a distractor path that ends at scs_c, the “true path” from scs_c to tct_c, and a distractor path that starts at tct_c, where cc indicates the component. The sampling of every connected component cc in the subgraph of an example starts from this backbone structure. Note that scs_c and tct_c are not necessarily identified with the source and target paper specified in the question (ss and tt); for example, in qNOq_{\mathrm{NO}}, ss and tt belong to two different components. Details are in Appendix § 12.2.2.

Step 2: Sampling the subgraph. Next, we repeatedly add papers from the Semantic Scholar citation graph to the subgraph. At each sampling step, we select one eligible node, i.e., a node that has neighboring nodes in the citation graph which have not yet been added to the subgraph, and whose component has not yet reached the target component size. To select this node, we aim to mimic the scale-free degree distribution of citation graphs ([5]), we use a preferential-attachment rule akin to the one-parameter Chinese restaurant process ([2]): a node is selected with probability proportional to the number of children it has received in Step 2, or proportional to the concentration parameter α>0\alpha> 0 if it has received none. For the selected node vv, we sample the direction of the new edge ee uniformly from {\text{cites},\text{cited_by}}.

Next, we uniformly sample a candidate child cc of vv that is connected via an edge with label ee in the Semantic Scholar citation graph and has not yet been added to the subgraph (if no such neighbor exists, we consider the other edge direction). We reject a candidate child if it would create an undirected cycle in the subgraph, or if the candidate child already belongs to another component in the subgraph. Each step adds one new leaf, so every component is a tree and its true path never changes. Sampling stops when each component reaches the termination criteria. Details are in Appendix § 12.2.2.3.

Step 3: Rendering the subgraph in text. We render each paper in the subgraph as blocks of natural language text, each containing the title, authors, and publication information of that paper, followed by an APA-style reference list restricted to its citations within the component (see Appendix § 12.§ 10 for the template). A paper that cites no other paper in its component has no block of its own and appears only in the reference lists of the papers that cite it.

Step 4: Distractors and prompt assembly. As in other NIAH benchmarks, the context also contains distracting content. Each example uses one of five distractor types: code, essays, books, and wikipedia. For each type, unrelated text is drawn randomly from, respectively: source files of the Linux kernel1, Paul Graham’s essays2, PG-19 books (Rae et al., [52]), and Wikipedia passages (Lin et al., [42]). The disjoint-component type adds connected components sampled using the same procedure described in Steps 1–3 with the same question depth and component size. We add distractors until no additional distractors can fit within the target context length, shuffle them randomly with the blocks from Step 3, and insert everything into a task-specific prompt template (Appendix § 19).

Complexity Metrics

We characterize each example with two complexity metrics:

  • Question depth kk: The number of nodes on the true path from the source paper to the target paper, including both endpoints. For qYESq_{\mathrm{YES}} and qMCq_{\mathrm{MC}}, this is the number of nodes on the unique path from ss to tt or tit_i. For qNOq_{\mathrm{NO}} and qcountq_{\mathrm{count}}, this is the number of nodes on the path from scs_c to $t_c

in the backbone of each component cc. In both cases, kk is the number of BFS steps required to solve each example.

  • Connected component size: The number of nodes in a connected component, which is the same for all components of an example. Especially, for qNOq_{\mathrm{NO}} and qcountq_{\mathrm{count}}, a correct answer requires the model to explore all nodes in Ks\mathbb{K}_s, so the component size directly measures the difficulty of these tasks.

Experiments and Analysis

In this section, we aim to answer the following research questions:

RQ1 Complexity scaling (§4.2). Can frontier LLMs find papers in large in-context citation graphs via a sequence of citation relations (i.e., the reachability task) as the complexity of the questions scales with question depth and component size?

RQ2 Task-specific performance (§ 9.1). How does the LLM’s performance vary in different reachability tasks?

RQ3 Context length scaling (§ 9.2). How does the performance of LLMs on this task vary as context length increases?

RQ4 Distractor content (§ 9.3). Are LLMs distracted more by some types of distracting content than others when answering long-context questions about citation relations?

RQ5 Reasoning effort (§ 9.4). Does accuracy on this task depend on the reasoning effort of the LLMs?

Experimental Setup

Models. We evaluate the following frontier models on the three reachability tasks in GraphNIAH: two models from the GPT-5.6 model family ([47]) (GPT-5.6-SOL and GPT-5.6-LUNA), GEMINI-3.1-PRO ([20]), GEMINI-3.8-FLASH ([22]), GEMINI-3.6-FLASH ([21]), and three QWEN3.5 models ([51]) (QWEN3.5-397B-A17B, QWEN3.5-122B-A10B, and QWEN3.5-35B-A3B). While we test all aforementioned models in the complexity scaling experiment, we choose to test a select subset of the models on other experiments to mitigate the high cost of inference. Appendix § 15 provides the model specifications and inference parameters.

Complexity Scaling Experiments

Data and variables. Each experiment varies exactly one of the following variables: question depth k∈{4,8,16,32}k \in\{4, 8, 16, 32\}, component size ∈{100,150,200,250}\in\{100, 150, 200, 250\}, reachability task (i.e., YES/NO reachability, multiple-choice reachability, or reachability counting), distractor type (code, essays, books, wikipedia, or disjoint-component), target context length (200,000, 500,000, or 922,0003 tokens), or reasoning effort (low, medium, or high). For each value of the tested variable, we sample all other variables independently and uniformly at random: kk from {4,8,16,32}\{4, 8, 16, 32\}, the component size from the integers in {100,250}\{100, 250\}, the task from the three reachability tasks, and the distractor type from the set of all distractor types. All examples use a target context length of 200,000 tokens (except for the context-length experiment) and the concentration parameter α=0.1\alpha= 0.1 (see Appendix § 14 for details on the selection of the concentration parameter). The number of examples for each experiment is reported in Section A.3.

Evaluation metrics. We extract the model answer from the last answer{} block of each response and report accuracy (the fraction of examples whose extracted answer matches the ground truth answer). For each configuration, we always include a random guessing baseline as a gray dashed line (see Appendix § 16 for details). Error bars and shaded bands show 95% Wilson confidence intervals (Wilson, [64]), and Appendix § 17 details all accuracy quantities with their intervals.

Complexity Scaling Experiments

We vary the question depth over k∈{4,8,16,32}k \in\{4, 8, 16, 32\} in one experiment and the component size over {100,150,200,250}\{100, 150, 200, 250\} in another.

Overall results. Accuracy decreases as question depth increases for every model, but the point at which this decrease occurs differs across models (Figure 3). GPT-5.6-LUNA and the QWEN3.5 models drop from 63-66% at k=4k = 4 to 39-48% at k=8k = 8 and then seem to level off at 34-42%. GEMINI-3.1-PRO and GEMINI-3.6-FLASH decline steadily, from 71% and 75% at k=4k = 4, respectively, to 52% and 41% at k=32k = 32. GPT-5.6-SOL falls from 75% at k=8k = 8 to 54% at k=32k = 32. GEMINI-3.8-FLASH stays at 83-87% up to k=16k = 16 and drops to 59% only at k=32k = 32. Component size mainly affects the stronger models: as it grows from 100 to 250, GPT-5.6-SOL and the three GEMINI models fall from 69–88% to 43–67%, whereas GPT-5.6-LUNA and the QWEN3.5 models stay between 36% and 54%, which are uniformly below the performance of GPT-5.6-SOL and the three GEMINI models at every component size.

Figure 3. Accuracy by question depth (top row) and component size (bottom row). The dashed gray line denotes random guessing accuracy. Due to cost limitations, GPT-5.6-SOL was not run for question depth 4.

Breakdown by task. On YES/NO and multiple-choice reachability, accuracy declines with question depth for every model (Table 5 and Table 6). For k≥16k \ge16, most models fall back to one answer in YES/NO reachability (Table 11): GPT-5.6-LUNA, GPT-5.6-SOL, and the QWEN3.5 models answer NO to almost every example, which is no better than random guessing; GEMINI-3.1-PRO and GEMINI-3.6-FLASH also lean towards NO, and only GEMINI-3.8-FLASH tends to answer YES. Component size has a weaker, model-dependent effect on these two tasks (Table 8 and Table 9): as component size increases from 100 to 250, YES/NO accuracy drops for all models except GPT-5.6-SOL and GEMINI-3.8-FLASH (e.g., GEMINI-3.1-PRO: 86.84% to 64.29%), and multiple-choice accuracy shows no consistent trend.

Model481632Overall
GPT-5.6-SOL–96.43 [82.29, 99.37]57.50 [42.19, 71.49]53.49 [38.92, 67.49]65.77† [56.54, 73.93]
GPT-5.6-LUNA94.12 [80.91, 98.37]50.00 [32.63, 67.37]50.00 [35.20, 64.80]51.16 [36.75, 65.38]60.69 [52.56, 68.27]
GEMINI-3.1-PRO100.00 [89.85, 100.00]89.29 [72.80, 96.29]67.50 [52.02, 79.92]67.44 [52.52, 79.51]79.31 [72.00, 85.11]
GEMINI-3.8-FLASH97.06 [85.08, 99.48]100.00 [87.94, 100.00]85.00 [70.93, 92.94]67.44 [52.52, 79.51]85.52 [78.87, 90.33]
GEMINI-3.6-FLASH100.00 [89.85, 100.00]82.14 [64.41, 92.12]67.50 [52.02, 79.92]58.14 [43.33, 71.62]75.17 [67.55, 81.49]
QWEN3.5-397B-A17B94.12 [80.91, 98.37]71.43 [52.94, 84.75]50.00 [35.20, 64.80]51.16 [36.75, 65.38]64.83 [56.76, 72.13]
QWEN3.5-122B-A10B94.12 [80.91, 98.37]60.71 [42.41, 76.43]50.00 [35.20, 64.80]51.16 [36.75, 65.38]62.76 [54.66, 70.20]
QWEN3.5-35B-A3B88.24 [73.38, 95.33]57.14 [39.07, 73.49]50.00 [35.20, 64.80]55.81 [41.11, 69.57]62.07 [53.96, 69.56]
Random guessing50.0050.0050.0050.0050.00
#Examples34284043145

Table 5. Accuracy (%) of each model on the YES/NO reachability examples of the question depth experiment (§4.2), with reasoning effort low. Overall pools all question depths. The number of examples differs between columns because the task of each example is sampled at random. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column. †GPT-5.6-SOL has fewer examples than the other models in this cell (111 Overall), so it is not bold. A dash marks a missing result.

Model481632Overall
GPT-5.6-SOL–100.00 [90.82, 100.00]86.21 [69.44, 94.50]69.23 [50.01, 83.50]87.10† [78.79, 92.46]
GPT-5.6-LUNA100.00 [89.85, 100.00]65.79 [49.89, 78.79]58.62 [40.74, 74.49]46.15 [28.76, 64.54]69.29 [60.80, 76.65]
GEMINI-3.1-PRO100.00 [89.85, 100.00]92.11 [79.20, 97.28]89.66 [73.61, 96.42]65.38 [46.22, 80.59]88.19 [81.43, 92.71]
GEMINI-3.8-FLASH100.00 [89.85, 100.00]100.00 [90.82, 100.00]100.00 [88.30, 100.00]80.77 [62.12, 91.49]96.06 [91.11, 98.31]
GEMINI-3.6-FLASH100.00 [89.85, 100.00]73.68 [57.99, 85.03]62.07 [44.00, 77.31]46.15 [28.76, 64.54]72.44 [64.10, 79.47]
QWEN3.5-397B-A17B97.06 [85.08, 99.48]73.68 [57.99, 85.03]75.86 [57.89, 87.78]61.54 [42.53, 77.57]77.95 [69.98, 84.28]
QWEN3.5-122B-A10B97.06 [85.08, 99.48]73.68 [57.99, 85.03]68.97 [50.77, 82.72]57.69 [38.95, 74.46]75.59 [67.44, 82.24]
QWEN3.5-35B-A3B97.06 [85.08, 99.48]68.42 [52.54, 80.92]58.62 [40.74, 74.49]57.69 [38.95, 74.46]71.65 [63.27, 78.77]
Random guessing25.0025.0025.0025.0025.00
#Examples34382926127

Table 6. Accuracy (%) of each model on the multiple-choice reachability examples of the question depth experiment (§4.2), with reasoning effort low. Overall pools all question depths. The number of examples differs between columns because the task of each example is sampled at random. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column. †GPT-5.6-SOL has fewer examples than the other models in this cell (93 Overall), so it is not bold. A dash marks a missing result.

Model100150200250Overall
GPT-5.6-SOL75.00† [46.77, 91.11]57.14† [25.05, 84.18]76.19 [61.47, 86.52]71.43 [52.94, 84.75]73.03† [63.00, 81.16]
GPT-5.6-LUNA65.79 [49.89, 78.79]57.14 [39.07, 73.49]59.52 [44.49, 72.96]53.57 [35.81, 70.47]59.56 [51.16, 67.43]
GEMINI-3.1-PRO86.84 [72.67, 94.25]82.14 [64.41, 92.12]78.57 [64.06, 88.29]64.29 [45.83, 79.29]78.68 [71.05, 84.72]
GEMINI-3.8-FLASH86.84 [72.67, 94.25]85.71 [68.51, 94.30]78.57 [64.06, 88.29]89.29 [72.80, 96.29]84.56 [77.55, 89.67]
GEMINI-3.6-FLASH71.05 [55.24, 83.00]71.43 [52.94, 84.75]76.19 [61.47, 86.52]64.29 [45.83, 79.29]71.32 [63.22, 78.26]
QWEN3.5-397B-A17B73.68 [57.99, 85.03]57.14 [39.07, 73.49]61.90 [46.81, 75.00]57.14 [39.07, 73.49]63.24 [54.87, 70.87]
QWEN3.5-122B-A10B65.79 [49.89, 78.79]57.14 [39.07, 73.49]57.14 [42.21, 70.88]53.57 [35.81, 70.47]58.82 [50.42, 66.74]
QWEN3.5-35B-A3B60.53 [44.72, 74.40]57.14 [39.07, 73.49]57.14 [42.21, 70.88]53.57 [35.81, 70.47]57.35 [48.95, 65.35]
Random guessing50.0050.0050.0050.0050.00
#Examples38284228136

Table 8. Accuracy (%) of each model on the YES/NO reachability examples of the component size experiment (§4.2), with reasoning effort low. Overall pools all component sizes. The number of examples differs between columns because the task of each example is sampled at random. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column. †GPT-5.6-SOL has fewer examples than the other models in these cells (12 at 100, 7 at 150, and 89 Overall), so they are not bold.

Model100150200250Overall
GPT-5.6-SOL87.50† [52.91, 97.76]90.91† [62.26, 98.38]78.57 [60.46, 89.79]84.38 [68.25, 93.14]83.54† [73.85, 90.12]
GPT-5.6-LUNA57.58 [40.81, 72.76]64.71 [47.91, 78.51]60.71 [42.41, 76.43]68.75 [51.43, 82.05]62.99 [54.33, 70.89]
GEMINI-3.1-PRO96.97 [84.68, 99.46]91.18 [77.04, 96.95]78.57 [60.46, 89.79]84.38 [68.25, 93.14]88.19 [81.43, 92.71]
GEMINI-3.8-FLASH96.97 [84.68, 99.46]94.12 [80.91, 98.37]100.00 [87.94, 100.00]100.00 [89.28, 100.00]97.64 [93.28, 99.19]
GEMINI-3.6-FLASH81.82 [65.61, 91.39]76.47 [60.00, 87.56]60.71 [42.41, 76.43]78.12 [61.24, 88.98]74.80 [66.60, 81.55]
QWEN3.5-397B-A17B78.79 [62.25, 89.32]70.59 [53.83, 83.17]60.71 [42.41, 76.43]75.00 [57.89, 86.75]71.65 [63.27, 78.77]
QWEN3.5-122B-A10B75.76 [58.98, 87.17]70.59 [53.83, 83.17]60.71 [42.41, 76.43]65.62 [48.31, 79.59]68.50 [59.98, 75.94]
QWEN3.5-35B-A3B60.61 [43.68, 75.32]64.71 [47.91, 78.51]64.29 [45.83, 79.29]68.75 [51.43, 82.05]64.57 [55.93, 72.35]
Random guessing25.0025.0025.0025.0025.00
#Examples33342832127

Table 9. Accuracy (%) of each model on the multiple-choice reachability examples of the component size experiment (§4.2), with reasoning effort low. Overall pools all component sizes. The number of examples differs between columns because the task of each example is sampled at random. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column. †GPT-5.6-SOL has fewer examples than the other models in these cells (8 at 100, 11 at 150, and 79 Overall), so they are not bold.

ModelLabel481632Overall
GPT-5.6-SOLYES-label–92.86 [68.53, 98.73]15.00 [5.24, 36.04]4.76 [0.85, 22.67]30.91{sup}†` [20.28, 44.03]
GPT-5.6-SOLNO-label–100.00 [78.47, 100.00]100.00 [83.89, 100.00]100.00 [85.13, 100.00]100.00 [93.58, 100.00]
GPT-5.6-LUNAYES-label88.24 [65.66, 96.71]7.14 [1.27, 31.47]0.00 [0.00, 16.11]0.00 [0.00, 15.46]22.22 [14.17, 33.09]
GPT-5.6-LUNANO-label100.00 [81.57, 100.00]92.86 [68.53, 98.73]100.00 [83.89, 100.00]100.00 [85.13, 100.00]98.63 [92.64, 99.76]
GEMINI-3.1-PROYES-label100.00 [81.57, 100.00]78.57 [52.41, 92.43]35.00 [18.12, 56.71]33.33 [17.19, 54.63]58.33 [46.81, 69.01]
GEMINI-3.1-PRONO-label100.00 [81.57, 100.00]100.00 [78.47, 100.00]100.00 [83.89, 100.00]100.00 [85.13, 100.00]100.00 [95.00, 100.00]
GEMINI-3.8-FLASHYES-label100.00 [81.57, 100.00]100.00 [78.47, 100.00]90.00 [69.90, 97.21]100.00 [84.54, 100.00]97.22 [90.43, 99.23]
GEMINI-3.8-FLASHNO-label94.12 [73.02, 98.95]100.00 [78.47, 100.00]80.00 [58.40, 91.93]36.36 [19.73, 57.05]73.97 [62.89, 82.66]
GEMINI-3.6-FLASHYES-label100.00 [81.57, 100.00]64.29 [38.76, 83.66]35.00 [18.12, 56.71]19.05 [7.67, 40.00]51.39 [40.07, 62.57]
GEMINI-3.6-FLASHNO-label100.00 [81.57, 100.00]100.00 [78.47, 100.00]100.00 [83.89, 100.00]95.45 [78.20, 99.19]98.63 [92.64, 99.76]
QWEN3.5-397B-A17BYES-label88.24 [65.66, 96.71]42.86 [21.38, 67.41]0.00 [0.00, 16.11]0.00 [0.00, 15.46]29.17 [19.94, 40.51]
QWEN3.5-397B-A17BNO-label100.00 [81.57, 100.00]100.00 [78.47, 100.00]100.00 [83.89, 100.00]100.00 [85.13, 100.00]100.00 [95.00, 100.00]
QWEN3.5-122B-A10BYES-label88.24 [65.66, 96.71]21.43 [7.57, 47.59]0.00 [0.00, 16.11]0.00 [0.00, 15.46]25.00 [16.44, 36.09]
QWEN3.5-122B-A10BNO-label100.00 [81.57, 100.00]100.00 [78.47, 100.00]100.00 [83.89, 100.00]100.00 [85.13, 100.00]100.00 [95.00, 100.00]
QWEN3.5-35B-A3BYES-label76.47 [52.74, 90.45]14.29 [4.01, 39.94]0.00 [0.00, 16.11]9.52 [2.65, 28.91]23.61 [15.30, 34.60]
QWEN3.5-35B-A3BNO-label100.00 [81.57, 100.00]100.00 [78.47, 100.00]100.00 [83.89, 100.00]100.00 [85.13, 100.00]100.00 [95.00, 100.00]
Random guessing50.0050.0050.0050.0050.00
#ExamplesYES-label1714202172
#ExamplesNO-label1714202273

Table 11. YES/NO reachability accuracy (%) of each model on YES-label and NO-label examples of the question depth experiment (§4.2), with reasoning effort low. Overall pools all question depths. Below each accuracy value is its 95% Wilson confidence interval. {sup}†`GPT-5.6-SOL has fewer examples than the other models in these cells: YES-label, 55 Overall; NO-label, 56 Overall. A dash marks a missing result.

Reachability counting shows the opposite pattern (Table 7 and Table 10). For the four models with non-zero accuracy (GPT-5.6-SOL, GEMINI-3.1-PRO, GEMINI-3.8-FLASH, and GEMINI-3.6-FLASH), question depth has no consistent effect, but accuracy falls sharply as the component size grows: from 100 to 250, GEMINI-3.8-FLASH drops from 79% to 25%, and the other three drop to 5% or less.

Model448816163232Overall
GPT-5.6-SOL–29.41 [16.83, 46.17]25.81 [13.70, 43.25]41.94 [26.42, 59.23]32.29† [23.78, 42.17]
GPT-5.6-LUNA0.00 [0.00, 10.72]0.00 [0.00, 10.15]0.00 [0.00, 11.03]0.00 [0.00, 11.03]0.00 [0.00, 2.91]
GEMINI-3.1-PRO9.38 [3.24, 24.22]5.88 [1.63, 19.09]12.90 [5.13, 28.85]19.35 [9.19, 36.28]11.72 [7.23, 18.44]
GEMINI-3.8-FLASH62.50 [45.25, 77.07]50.00 [34.07, 65.93]77.42 [60.19, 88.61]29.03 [16.10, 46.59]54.69 [46.05, 63.05]
GEMINI-3.6-FLASH21.88 [11.02, 38.76]14.71 [6.45, 30.13]6.45 [1.79, 20.72]12.90 [5.13, 28.85]14.06 [9.08, 21.14]
QWEN3.5-397B-A17B0.00 [0.00, 10.72]0.00 [0.00, 10.15]0.00 [0.00, 11.03]0.00 [0.00, 11.03]0.00 [0.00, 2.91]
QWEN3.5-122B-A10B0.00 [0.00, 10.72]0.00 [0.00, 10.15]0.00 [0.00, 11.03]0.00 [0.00, 11.03]0.00 [0.00, 2.91]
QWEN3.5-35B-A3B0.00 [0.00, 10.72]0.00 [0.00, 10.15]0.00 [0.00, 11.03]0.00 [0.00, 11.03]0.00 [0.00, 2.91]
Random guessing0.660.660.660.660.66
#Examples32343131128

Table 7. Accuracy (%) of each model on the reachability counting examples of the question depth experiment (§4.2), with reasoning effort low. Overall pools all question depths. The number of examples differs between columns because the task of each example is sampled at random. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column. †GPT-5.6-SOL has fewer examples than the other models in this cell (96 Overall), so it is not bold. A dash marks a missing result.

Model100150200250Overall
GPT-5.6-SOL71.43† [35.89, 91.78]25.00† [7.15, 59.07]20.00 [9.50, 37.31]5.00 [1.38, 16.50]17.65† [11.00, 27.10]
GPT-5.6-LUNA0.00 [0.00, 11.70]0.00 [0.00, 9.18]0.00 [0.00, 11.35]0.00 [0.00, 8.76]0.00 [0.00, 2.73]
GEMINI-3.1-PRO31.03 [17.28, 49.23]7.89 [2.72, 20.80]0.00 [0.00, 11.35]2.50 [0.44, 12.88]9.49 [5.63, 15.56]
GEMINI-3.8-FLASH79.31 [61.61, 90.15]55.26 [39.71, 69.85]50.00 [33.15, 66.85]25.00 [14.19, 40.19]50.36 [42.10, 58.61]
GEMINI-3.6-FLASH51.72 [34.43, 68.61]15.79 [7.44, 30.42]16.67 [7.34, 33.56]0.00 [0.00, 8.76]18.98 [13.29, 26.36]
QWEN3.5-397B-A17B0.00 [0.00, 11.70]0.00 [0.00, 9.18]0.00 [0.00, 11.35]0.00 [0.00, 8.76]0.00 [0.00, 2.73]
QWEN3.5-122B-A10B0.00 [0.00, 11.70]0.00 [0.00, 9.18]0.00 [0.00, 11.35]0.00 [0.00, 8.76]0.00 [0.00, 2.73]
QWEN3.5-35B-A3B0.00 [0.00, 11.70]0.00 [0.00, 9.18]0.00 [0.00, 11.35]0.00 [0.00, 8.76]0.00 [0.00, 2.73]
Random guessing0.660.660.660.660.66
#Examples29383040137

Table 10. Accuracy (%) of each model on the reachability counting examples of the component size experiment (§4.2), with reasoning effort low. Overall pools all component sizes. The number of examples differs between columns because the task of each example is sampled at random. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column. †GPT-5.6-SOL has fewer examples than the other models in these cells (7 at 100, 8 at 150, and 85 Overall), so they are not bold.

79% to 25%, and the other three drop to 5% or less.

Overall, we conclude that the two YES/NO and multiple-choice reachability tasks are more negatively affected by question depth than component size, while component size has a greater negative effect on the counting task than question depth.

Performance by Reachability Tasks

Figure 4 compares the three reachability tasks. Reachability counting is the hardest task for all three models: GPT-5.6-LUNA and QWEN3.5-35B-A3B do not correctly answer any counting example, and GEMINI-3.8-FLASH reaches only 55%, its lowest accuracy across the three tasks (Table 26). The complexity scaling experiments show the same phenomenon: GPT-5.6-LUNA and all three QWEN3.5 models do not correctly answer any counting example at any question depth or component size (Table 7 and Table 10).

Figure 4. Accuracy by reachability task; each dashed gray line denotes random guessing accuracy.

ModelYES/NOMultiple choiceCountingOverall
GPT-5.6-LUNA66.00 [56.28, 74.54]68.00 [58.34, 76.33]0.00 [0.00, 3.70]44.67 [39.14, 50.32]
GEMINI-3.8-FLASH87.00 [79.02, 92.24]97.00 [91.55, 98.97]55.00 [45.24, 64.39]79.67 [74.75, 83.83]
QWEN3.5-35B-A3B64.00 [54.24, 72.73]72.00 [62.51, 79.86]0.00 [0.00, 3.70]45.33 [39.79, 50.99]
Random guessing50.0025.000.6625.22

Table 26. Accuracy (%) of each model in the task experiment (§ 9.1), with reasoning effort low. Each task has 100 examples, and Overall pools all three tasks. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column.

The other two tasks differ with respect to each model’s performance as compared to random guessing: on YES/NO reachability, GPT-5.6-LUNA, QWEN3.5-35B-A3B, and GEMINI-3.8-FLASH exceed random guessing accuracy by 16, 14, and 37 percentage points, respectively, but on multiple-choice reachability by 43, 47, and 72 points. On YES/NO reachability, the models are also biased toward one answer: GPT-5.6-LUNA and QWEN3.5-35B-A3B answer NO on 84 and 86 of the 100 examples. This is even more apparent for high question depth examples (k≥8k \ge8; Table 14), resulting in 100% accuracy on NO-label examples but only 32% and 28% on YES-label examples, respectively (Table 13).

ModelYES-labelNO-labelOverall
GPT-5.6-LUNA32.00 [20.76, 45.81]100.00 [92.86, 100.00]66.00 [56.28, 74.54]
GEMINI-3.8-FLASH96.00 [86.54, 98.90]78.00 [64.76, 87.25]87.00 [79.02, 92.24]
QWEN3.5-35B-A3B28.00 [17.47, 41.67]100.00 [92.86, 100.00]64.00 [54.24, 72.73]
Random guessing50.0050.0050.00
#Examples5050100

Table 13. YES/NO reachability accuracy (%) of each model on YES-label and NO-label examples of the task experiment (§ 9.1), with reasoning effort low. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column.

Model448816163232Overall
Ground truth43.4864.2943.4846.1550.00
GPT-5.6-LUNA43.48 [25.63, 63.19]89.29 [72.80, 96.29]100.00 [85.69, 100.00]100.00 [87.13, 100.00]84.00 [75.58, 89.90]
GEMINI-3.8-FLASH43.48 [25.63, 63.19]64.29 [45.83, 79.29]39.13 [22.16, 59.21]15.38 [6.15, 33.53]41.00 [31.87, 50.80]
QWEN3.5-35B-A3B47.83 [29.24, 67.04]92.86 [77.35, 98.02]100.00 [85.69, 100.00]100.00 [87.13, 100.00]86.00 [77.86, 91.47]
#Examples23282326100

Table 14. Percentage of the YES/NO reachability examples of the task experiment (§ 9.1) that each model answers NO, overall and by question depth kk, with reasoning effort low. In this experiment, the question depth of each example is sampled uniformly at random. Below each percentage is its 95% Wilson confidence interval. The first row is the percentage of NO-label examples, that is, the percentage of NO answers of a model that is always correct. Every answer is YES or NO, so the percentage of YES answers is 100 minus the value shown.

We hypothesize that multiple-choice reachability is easier because there is always a “correct” option, so the model never has to conclude that no target is reachable. To test this, we append a fifth option, “None of these targets is reachable from the source.”, to the same multiple-choice examples and keep all other aspects of each example unchanged, so the new option is never correct. With this option, GPT-5.6-LUNA and QWEN3.5-35B-A3B drop from 68% and 72% to 14% and 15%, respectively, no better than random guessing (20%), because they choose the new option in 85 and 84 of the 100 examples, whereas GEMINI-3.8-FLASH stays at 97% (Table 17). The two weaker models also produce roughly half as many output and reasoning tokens, while GEMINI-3.8-FLASH is unaffected (Table 18). This suggests that, once a negative answer is available, these models stop searching earlier and fall back to it, as in YES/NO reachability.

Adding “None of the above”
Model#ExamplesBeforeAfterChange (pp)Gained / lost
GPT-5.6-LUNA10068.00 [58.34, 76.33]14.00 [8.53, 22.14]−54.00+0 / −54 p=<0.01p = <0.01
GEMINI-3.8-FLASH10097.00 [91.55, 98.97]97.00 [91.55, 98.97]+0.00+3 / −3 p=1.00p = 1.00
QWEN3.5-35B-A3B10072.00 [62.51, 79.86]15.00 [9.31, 23.28]−57.00+0 / −57 p=<0.01p = <0.01
Random guessing25.0020.00−5.00

Table 17. Multiple-choice reachability accuracy (%) of each model on the multiple-choice examples of the task experiment (§ 9.1), with reasoning effort low, before and after we append a fifth option, “None of these targets is reachable from the source.” The context, the four original options, and the ground truth answer are unchanged, so the new option is never correct. #Examples counts the examples that the model answered both before and after, and both accuracies use only these examples. Below each accuracy value is its 95% Wilson confidence interval. Change is After minus Before, in percentage points. Gained / lost gives the number of examples that become correct (+) and wrong (−) after the option is added, and below it the pp-value of the exact McNemar test. Random guessing is the expected accuracy of uniformly random answers (Appendix § 16), with four options before and five after.

ModelTokensBeforeAfterMedian changepp
GPT-5.6-LUNAOutput638.95 ±300.07306.26 ±167.83−318.00<0.01
Reasoning621.85 ±296.96288.22 ±163.40−318.00<0.01
GEMINI-3.8-FLASHOutput5,271.84 ±5,039.825,702.91 ±7,893.90+45.500.42
Reasoning4,595.61 ±4,774.194,981.18 ±7,575.28+34.500.68
QWEN3.5-35B-A3BOutput19,619.60 ±9,206.7610,954.93 ±3,664.83−8,295.00<0.01
Reasoning18,064.96 ±6,741.7310,505.45 ±3,679.49−8,477.00<0.01

Table 18. Mean output tokens and reasoning tokens per example of each model before and after adding the “None of these targets is reachable from the source.” option, on the same examples as Table 17, with reasoning effort low. Below each mean is its standard deviation. Output tokens are the completion tokens that the provider reports, and reasoning tokens are the tokens that it reports as reasoning. Median change is the median over the examples of After minus Before, and pp is the two-sided Wilcoxon signed-rank test of this difference; the pp-values are not adjusted for multiple tests.

Overall, reachability counting is the hardest task, and for GPT-5.6-LUNA and QWEN3.5-35B-A3B, the advantage of multiple-choice over YES/NO reachability disappears once a negative answer is available.

Effects of Context Length

We vary the target context length over 200,000, 500,000, and 922,000 tokens for GPT-5.6-LUNA and GEMINI-3.8-FLASH. GPT-5.6-LUNA stays at 38–44%, while GEMINI-3.8-FLASH falls from 71% at 200,000 tokens to 64% and 60% at 500,000 and 922,000 tokens, respectively (Figure 5; Table 25). Neither change is significant (Fisher’s exact test, all p>0.1p > 0.1; Table 22).

Figure 5. Accuracy by context length.

Model200k → 500k200k → 922k500k → 922k
GPT-5.6-LUNA38.00 → 44.00 p = 0.4938.00 → 40.00 p = 0.8644.00 → 40.00 p = 0.84
GEMINI-3.8-FLASH71.00 → 64.00 p = 0.4671.00 → 60.00 p = 0.2064.00 → 60.00 p = 0.84

Table 22. Fisher’s exact test of whether accuracy in the target context length experiment (§ 9.2) differs between two target context lengths, with reasoning effort low. Each cell gives the accuracy (%) at the two context lengths and below it the two-sided pp-value. Each target context length has its own examples, so the test treats the answers at two lengths as independent samples. Examples per target context length: 100 (200k), 50 (500k), 50 (922k).

Model200k500k922k
GPT-5.6-LUNA38.00 [29.10, 47.79]44.00 [31.16, 57.69]40.00 [27.61, 53.82]
GEMINI-3.8-FLASH71.00 [61.46, 78.99]64.00 [50.14, 75.86]60.00 [46.18, 72.39]
Random guessing25.2225.2225.22
#Examples1005050

Table 25. Accuracy (%) of each model in the target context length experiment (§ 9.2), with reasoning effort low. The 200k results reuse the examples and answers of the reasoning effort experiment. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column.

Effects of Distractor Type

We run experiments where we vary the type of the distractors for GPT-5.6-LUNA, GEMINI-3.8-FLASH, and QWEN3.5-35B-A3B. The distractor type has no obvious effect on accuracy (Figure 6; Table 27): GPT-5.6-LUNA stays between 41% and 50%, QWEN3.5-35B-A3B between 38% and 46%, and GEMINI-3.8-FLASH between 72% and 88%. Only GEMINI-3.8-FLASH demonstrates a minor difference in performance across the different distractor types: an overall χ2\chi^2 test across the five distractor types gives p=0.03p = 0.03, mainly because it is more accurate with essays (88%) than with code (72%) and wikipedia (73%). However, no pair of distractor types differs significantly after correcting for the ten pairwise comparisons (Fisher’s exact test with Holm correction, all adjusted p>0.05p > 0.05; Table 20).

Figure 6. Accuracy by distractor type.

Distractor typesGPT-5.6-LUNAGEMINI-3.8-FLASHQWEN3.5-35B-A3B
books vs code+0.00 p = 1.00−11.00 p = 0.63+4.00 p = 1.00
books vs disjoint-component+0.00 p = 1.00−6.00 p = 1.00+2.00 p = 1.00
books vs essays+5.00 p = 1.00+5.00 p = 1.00+4.00 p = 1.00
books vs wikipedia+9.00 p = 1.00−10.00 p = 0.74+8.00 p = 1.00
code vs disjoint-component+0.00 p = 1.00+5.00 p = 1.00−2.00 p = 1.00
code vs essays+5.00 p = 1.00+16.00 p = 0.07+0.00 p = 1.00
code vs wikipedia+9.00 p = 1.00+1.00 p = 1.00+4.00 p = 1.00
disjoint-component vs essays+5.00 p = 1.00+11.00 p = 0.49+2.00 p = 1.00
disjoint-component vs wikipedia+9.00 p = 1.00−4.00 p = 1.00+6.00 p = 1.00
essays vs wikipedia+4.00 p = 1.00−15.00 p = 0.11+4.00 p = 1.00
All five, χ2\chi^2 testp=0.61p = 0.61p=0.03p = 0.03p=0.84p = 0.84

Table 20. Pairwise comparison of the distractor types in the distractor type experiment (§ 9.3), with reasoning effort low. All tasks are pooled, because the task of each example is sampled at random, independently of its distractor type. Each cell gives the accuracy with the second distractor type minus the accuracy with the first, in percentage points, and below it the pp-value of Fisher’s exact test after the Holm correction for the ten pairs of each model. The last row gives the pp-value of the χ2\chi^2 test of whether accuracy is the same for all five distractor types.

Modelbookscodedisjoint-componentessayswikipediaOverall
GPT-5.6-LUNA41.00 [31.87, 50.80]41.00 [31.87, 50.80]41.00 [31.87, 50.80]46.00 [36.56, 55.74]50.00 [40.38, 59.62]43.80 [39.51, 48.18]
GEMINI-3.8-FLASH83.00 [74.45, 89.11]72.00 [62.51, 79.86]77.00 [67.85, 84.16]88.00 [80.19, 93.00]73.00 [63.57, 80.73]78.60 [74.79, 81.97]
QWEN3.5-35B-A3B38.00 [29.10, 47.79]42.00 [32.80, 51.79]40.00 [30.94, 49.80]42.00 [32.80, 51.79]46.00 [36.56, 55.74]41.60 [37.36, 45.97]
Random guessing25.2225.2225.2225.2225.2225.22

Table 27. Accuracy (%) of each model in the distractor type experiment (§ 9.3), with reasoning effort low. Each distractor type has 100 examples, and Overall pools all distractor types. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column.

Effects of Reasoning Effort

We vary the reasoning effort of GPT-5.6-LUNA, GEMINI-3.8-FLASH, and QWEN3.5-35B-A3B: low, medium, and high (Figure 7). More effort does not reliably help the LLMs (Table 28): GEMINI-3.8-FLASH reaches 71%, 75%, and 66% at low, medium, and high effort, respectively, and QWEN3.5-35B-A3B stays around 39% and 40% for all three reasoning effort levels. Only GPT-5.6-LUNA improves from 38% at low effort to 48% at high effort, where it correctly answers 10 examples which were incorrectly answered at low effort, without producing any incorrect answers on an example which was answered correctly (Table 21); however, the tradeoff is that, on average, high reasoning effort results in eight times as many output tokens, a longer response time by factor of 4.7, and 30.26% higher API cost as compared to the low setting (Table 33).

Figure 7. Accuracy by reasoning effort.

Modellow → mediumlow → highmedium → high
GPT-5.6-LUNA+3 / −2 p = 1.00+10 / −0 p = <0.01+10 / −1 p = 0.01
GEMINI-3.8-FLASH+12 / −8 p = 0.50+7 / −12 p = 0.36+2 / −11 p = 0.02
QWEN3.5-35B-A3B+3 / −4 p = 1.00+4 / −4 p = 1.00+5 / −4 p = 1.00

Table 21. Paired comparison of the reasoning efforts in the reasoning effort experiment (§ 9.4). Each model answers the same 100 examples at every effort. Each cell gives the number of examples that become correct (+) and that become wrong (−) when the effort changes as in the column header, and below it the pp-value of the exact McNemar test.

Modellowmediumhigh
GPT-5.6-LUNA38.00 [29.10, 47.79]39.00 [30.02, 48.80]48.00 [38.46, 57.68]
GEMINI-3.8-FLASH71.00 [61.46, 78.99]75.00 [65.70, 82.45]66.00 [56.28, 74.54]
QWEN3.5-35B-A3B40.00 [30.94, 49.80]39.00 [30.02, 48.80]40.00 [30.94, 49.80]
Random guessing25.2225.2225.22

Table 28. Accuracy (%) of each model in the reasoning effort experiment (§ 9.4). All reasoning efforts use the same 100 examples. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column.

ModelReasoning effort#ExamplesOutput tokensResponse time (s)Cost per example (¢)Total cost ($)
GPT-5.6-LUNA*low100823.21±539.75823.21 \pm539.7513.18±8.2713.18 \pm8.271.95±0.681.95 \pm0.681.95
medium1001,937.24±1,427.191{,}937.24 \pm1{,}427.1925.75±18.3225.75 \pm18.322.26±0.262.26 \pm0.262.26
high1006,672.60±6,128.726{,}672.60 \pm6{,}128.7261.73±50.5061.73 \pm50.502.54±0.452.54 \pm0.452.54
GEMINI-3.8-FLASHlow10023,086.40±23,049.8623{,}086.40 \pm23{,}049.86137.27±127.62137.27 \pm127.6222.46±8.8222.46 \pm8.8222.46
medium10029,207.72±25,517.0229{,}207.72 \pm25{,}517.02171.27±141.94171.27 \pm141.9424.75±9.6824.75 \pm9.6824.75
high10037,624.94±29,513.4837{,}624.94 \pm29{,}513.48204.32±154.93204.32 \pm154.9327.11±10.4227.11 \pm10.4227.11
QWEN3.5-35B-A3Blow10015,219.30±5,487.9715{,}219.30 \pm5{,}487.97223.67±89.98223.67 \pm89.984.92±0.844.92 \pm0.844.92
medium10016,866.72±9,704.8816{,}866.72 \pm9{,}704.88195.15±95.51195.15 \pm95.515.14±1.355.14 \pm1.355.14
high10016,423.57±6,641.7716{,}423.57 \pm6{,}641.77172.53±65.77172.53 \pm65.775.07±0.965.07 \pm0.965.07
All models96.20

Table 33. Cost and response time of each model in the reasoning effort experiment (§ 9.4). #Examples counts the examples that the model answered, and each cell shows mean±std_{\pm\mathrm{std}} over these examples. Output tokens are the completion tokens reported by the provider, including reasoning tokens. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total. *Sent at the flex service tier, which has lower prices and can respond more slowly.

Conclusion

In this paper, we introduce GraphNIAH as an updatable, realistic needle-in-a-haystack benchmark. By automatically constructing examples from the Semantic Scholar citation graph using post-cutoff papers, we prevent benchmark contamination. GraphNIAH supports multi-hop, non-linear reasoning, with diverse families of distracting content, while reducing reliance on synthetic data, human annotation, and static published datasets. Our experiments with eight LLMs show that this search remains difficult. Accuracy falls as question depth grows, and at large depths, most models do not outperform random guessing, often falling back to choosing the negative option. Half of the models do not correctly answer any counting example, and the weaker models significantly underestimate the number of reachable papers. Neither the type of distracting content nor a higher level of reasoning effort reliably closes this gap. These results suggest that current LLMs often stop exploring the context before they reach the answer, even though every required fact is present.

There are multiple promising directions for future work. Firstly, we use a limited set of templates for the questions in the reachability tasks. Future work can therefore utilize a more diverse set of templates, or utilize LLMs to paraphrase the questions to produce more naturalistic language. Secondly, since GraphNIAH is limited to the domain of citation graphs, we can further expand this work to other domains with different kinds of relational data. Thirdly, the experiments are performed over a limited number of examples per configuration due to the high cost of inference, so future work can evaluate LLMs with a higher number of examples to increase the statistical power of our analysis, strengthening the claims and comparisons. Finally, the findings show that when complexity increases, models perform no better than random guessing and fallback to choosing the negative option. We can therefore explore methods to encourage models to search more deeply when given a highly complex query and to avoid falling back to a default response.

Limitations

GraphNIAH has several limitations. First, it covers one domain and one kind of relationship, citations between scientific papers, and all prompts are in English, so our findings may not transfer to other relational data or other kinds of long-context reasoning. Second, because long-context inference is expensive, we evaluate eight models in total, and some experiments only test two or three models, and each model is evaluated on at most 100 examples per setting, so many confidence intervals are fairly wide. Third, using papers from Semantic Scholar published after the training cutoff dates of the evaluated models reduces the risk of benchmark contamination but does not remove it: an earlier version of a paper, such as a preprint, may have been public before its recorded publication date, and a model may still guess an answer from how closely related two papers appear. The reachability counting task is one possible solution, since the model must find all reachable papers to answer correctly.

Acknowledgments

We thank Abdulrahman Mahmoud for his valuable insights and constructive discussion. We gratefully acknowledge Google’s support through Gemini API credits.

Wang, Yuxiang; Dai, Xinnan; Fan, Wenqi; Ma, Yao | 2026 | Exploring graph learning tasks with pure LLMs: A comprehensive benchmark and investigation | Findings of the Association for Computational Linguistics: ACL 2026 | | | 7909–7942 | | | raw: Yuxiang Wang, Xinnan Dai, Wenqi Fan, and Yao Ma. 2026b. Exploring graph learning tasks with pure LLMs: A comprehensive benchmark and investigation. In Findings of the Association for Computational Linguistics: ACL 2026, pages 7909–7942, San Diego, California, United States. Association for Computational Linguistics.

White, Colin; Dooley, Samuel; Roberts, Manley; Pal, Arka; Feuer, Benjamin; Jain, Siddhartha; Shwartz-Ziv, Ravid; Jain, Neel; Saifullah, Khalid; Dey, Sreemanti; Shubh-Agrawal; Sandha, Sandeep Singh; Naidu, Siddartha Venkat; Hegde, Chinmay; LeCun, Yann; Goldstein, Tom; Neiswanger, Willie; Goldblum, Micah | 2025 | Livebench: A challenging, contamination-limited LLM benchmark | The Thirteenth International Conference on Learning Representations | | | | | | raw: Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. Livebench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations.

Wilson, Edwin B. | 1927 | Probable inference, the law of succession, and statistical inference | Journal of the American Statistical Association | 22 | 158 | 209–212 | | | raw: Edwin B Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212.

Wright, Dustin; Mujahid, Zain Muhammad; Wang, Lu; Augenstein, Isabelle; Jurgens, David | 2025 | Unstructured evidence attribution for long context query focused summarization | Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing | | | 1839–1867 | | | raw: Dustin Wright, Zain Muhammad Mujahid, Lu Wang, Isabelle Augenstein, and David Jurgens. 2025. Unstructured evidence attribution for long context query focused summarization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1839–1867, Suzhou, China. Association for Computational Linguistics.

Wu, Xiaobao; Pan, Liangming; Xie, Yuxi; Zhou, Ruiwen; Zhao, Shuai; Ma, Yubo; Du, Mingzhe; Mao, Rui; Luu, Anh Tuan; Wang, William Yang | 2025 | AntiLeakBench: Preventing data contamination by automatically constructing benchmarks with updated real-world knowledge | Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) | | | 18403–18419 | | | raw: Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, and William Yang Wang. 2025. AntiLeakBench: Preventing data contamination by automatically constructing benchmarks with updated real-world knowledge. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18403–18419, Vienna, Austria. Association for Computational Linguistics.

Xu, Ruijie; Wang, Zengzhi; Fan, Run-Ze; Liu, Pengfei | 2024 | Benchmarking benchmark leakage in large language models | arXiv preprint | | | | | | 2404.18824 raw: Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824.

Ying, Jiahao; Cao, Yixin; Bai, Yushi; Sun, Qianru; Wang, Bo; Tang, Wei; Ding, Zhaojun; Yang, Yizhe; Huang, Xuanjing; YAN, Shuicheng | 2024 | Automating dataset updates towards reliable and timely evaluation of large language models | The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track | | | | | | raw: Jiahao Ying, Yixin Cao, Yushi Bai, Qianru Sun, Bo Wang, Wei Tang, Zhaojun Ding, Yizhe Yang, Xuanjing Huang, and Shuicheng YAN. 2024. Automating dataset updates towards reliable and timely evaluation of large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Yu, Yifei; Zhang, Qian-Wen; Qiao, Lingfeng; Yin, Di; Li, Fang; Wang, Jie; Xi, Chen Zeng; Zheng, Suncong; Liang, Xiaolong; Sun, Xing | 2025 | Sequential-NIAH: A needle-in-a-haystack benchmark for extracting sequential needles from long contexts | Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing | | | 29450–29468 | | | raw: Yifei Yu, Qian-Wen Zhang, Lingfeng Qiao, Di Yin, Fang Li, Jie Wang, Chen Zeng Xi, Suncong Zheng, Xiaolong Liang, and Xing Sun. 2025. Sequential-NIAH: A needle-in-a-haystack benchmark for extracting sequential needles from long contexts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29450–29468, Suzhou, China. Association for Computational Linguistics.

Tao, Yuan; Ning, Xuefei; Zhou, Dong; Yang, Zhijie; Li, Shiyao; Zhuang, Minghui; Tan, Zheyue; Zhu, Zhuyu* | | | | | | | | | raw: Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu

Yao, Dahua; Lin, Boxun; Dai, Guohao; Yan, Shengen; Wang, Yu | 2025 | LV-eval: A balanced long-context benchmark with 5 length levels up to 256k. | Second Conference on Language Modeling | | | | | | raw: Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2025. LV-eval: A balanced long-context benchmark with 5 length levels up to 256k. In Second Conference on Language Modeling.

Zhang, Haozhen; Feng, Tao; Han, Pengrui; You, Jiaxuan | 2025 | Academiceval: Live long-context LLM benchmark | Transactions on Machine Learning Research | | | | | | raw: Haozhen Zhang, Tao Feng, Pengrui Han, and Jiaxuan You. 2025. Academiceval: Live long-context LLM benchmark. Transactions on Machine Learning Research.

Zhang, Zhehao; Chen, Jiaao; Yang, Diyi | 2024 | Darg: Dynamic evaluation of large language models via adaptive reasoning graph | Advances in Neural Information Processing Systems | 37 | | 135904–135942 | | | raw: Zhehao Zhang, Jiaao Chen, and Diyi Yang. 2024. Darg: Dynamic evaluation of large language models via adaptive reasoning graph. Advances in Neural Information Processing Systems, 37:135904–135942.

Zhong, Xiaoshi; Liang, Huizhi | 2024 | On the scale-free property of citation networks: an empirical study. | Companion Proceedings of the ACM Web Conference 2024 | | | 541–544 | | | raw: Xiaoshi Zhong and Huizhi Liang. 2024. On the scale-free property of citation networks: an empirical study. In Companion Proceedings of the ACM Web Conference 2024, pages 541–544.

Zhu, Kaijie; Chen, Jiaao; Wang, Jindong; Gong, Neil Zhenqiang; Yang, Diyi; Xie, Xing | 2024 | Dyval: Dynamic evaluation of large language models for reasoning tasks. | The Twelfth International Conference on Learning Representations | | | | | | raw: Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024a. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations.

Zhu, Qin; Cheng, Qinyuan; Peng, Runyu; Li, Xiaonan; Peng, Ru; Liu, Tengxiao; Qiu, Xue; Huang, Xuanjing | 2024 | Inference-time decontamination: Reusing leaked benchmarks for large language model evaluation. | Findings of the Association for Computational Linguistics: EMNLP 2024 | | | 9113–9129 | | | raw: Qin Zhu, Qinyuan Cheng, Runyu Peng, Xiaonan Li, Ru Peng, Tengxiao Liu, Xue Peng Qiu, and Xuanjing Huang. 2024b. Inference-time decontamination: Reusing leaked benchmarks for large language model evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9113–9129, Miami, Florida, USA. Association for Computational Linguistics.

Appendix for “GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs”

The appendix is structured as follows:

  • Sections A, B and G describe the benchmark: its data sources, construction details, dataset statistics, and templates; how we select the concentration parameter α\alpha;

  • Sections C and D describe the evaluation setup: the models and their inference settings, and the random guessing baseline.

  • Sections E and F report additional results: accuracy tables, results by task and by ground truth answer, statistical tests, and the inference cost and response time of each experiment.

Benchmark Details

This section provides the experimental configurations, dataset statistics, concentration-parameter selection procedure, question templates, and APA-style template for the papers.

Data Sources and Preparation

Data sources and snapshot. We use scientific paper metadata and citation relationships from release 2026-10-06 of the Semantic Scholar S2AG Datasets [29], and we use them for research purposes only, following their license and terms of use4.

For the wikipedia distractors, we retrieve Wikipedia passages with BM25, using the pre-built index of pyserini [42].

Benchmark Construction Details

Citation Path Sampling

In Step 1 (§ 7.1), each connected component starts from an undirected path of 3k−23k - 2 papers. We sample these paths from the citation graph of all eligible papers, without starting from seed papers.

Citation graph. A paper is eligible if it has a complete publication date on or after the cutoff date (April 1, 2026, in this study) and a non-empty title without MathML markup. We build an undirected graph whose nodes are the eligible papers and whose edges are the citation relationships between them.

Pruning. For question depth kk, each path has 3k−23k - 2 papers. Before sampling, we remove the papers that cannot lie on any simple path of 3k−23k - 2 papers. Because finding the longest simple path is NP-hard, we use two upper bounds on the length of such a path: we remove connected components with fewer than 3k−23k - 2 papers, and we use the biconnected components of the graph to bound the length of a path through each paper. We repeat both steps until no paper is removed.

Sampling. We sample each path with a randomized depth-first search. We choose a start paper uniformly at random from the papers that can be an endpoint of a (3k−2)(3k - 2)-paper path. At each step, we move to a neighboring paper chosen uniformly at random among those that are not on the path and can still complete a path of the required length. If no such neighbor remains, we backtrack to the previous paper on the path and try another neighbor. If a search does not find a path within 100,000 steps, we stop it and draw a new start paper. We keep only distinct paths, counting a path and its reverse as the same path, and we allow each paper to appear in at most 100 paths to reduce the overlap between paths. We discard papers without usable metadata and replace the paths that contain them. For each question depth, we sample a pool of 1,000 paths.

Paths for each example. For each example, we draw the paths of its components uniformly at random from the pool of the required question depth, independently of other examples. The drawn paths must be disjoint: no two of them share a paper or a title, and no citation relationship connects them. Different examples may use the same path, but the components of one example never share a paper. We use each path in a random direction.

Backbone Structure

This part provides detail information about the initial structure of a component, which is described in § 7.1.

For question depth k≥2k \ge2, the sampled path (p1,…,p3k−2)(p_1,\ldots,p_{3k-2}) of component cc forms its backbone, with source node s=pks = p_k and target node t=p2k−1t = p_{2k-1}. The true path is (pk,…,p2k−1)(p_k,\ldots,p_{2k-1}), and the two distractor paths are (p1,…,pk)(p_1,\ldots,p_k), which ends at ss, and (p2k−1,…,p3k−2)(p_{2k-1},\ldots,p_{3k-2}), which starts at tt. Each of the three paths has kk nodes and k−1k - 1 edges, and the distractor paths share only ss and tt with the true path.

Component Sampling

To mimic the scale-free, heavy-tailed degree distribution of citation graphs [74], Step 2 (§ 7.1) uses a sampling rule akin to the one-parameter Chinese restaurant process [2], which favors nodes that have already received children.

We sample all components of an example in one loop. Let Fℓ\mathbb{F}_{\ell} be the set of nodes that are eligible to receive a new child at the ℓ\ell-th sampling step, over all components of the example. A node vv is eligible if (1) its component has not yet reached the component size of the example, and (2) vv still has untried neighboring papers that pass the publication date filter. At the first step, F0\mathbb{F}_0 contains the eligible nodes of all backbones.

At each sampling step ℓ=0,1,…\ell= 0, 1, \ldots, let gℓ(v)g_{\ell}(v) be the number of children that node vv has received before step ℓ\ell; edges of the backbone do not count. We select a node v∈Fℓv \in\mathbb{F}_{\ell} to receive a new child with probability

Pℓ(v)=wℓ(v)∑u∈Fℓwℓ(u)P_{\ell}(v) = \frac{w_{\ell}(v)}{\sum_{u \in F_{\ell}} w_{\ell}(u)}

where

wℓ(v)={α,gℓ(v)=0,gℓ(v),gℓ(v)>0,(1)w_{\ell}(v) = \begin{cases} \alpha, & g_{\ell}(v)=0,\\ g_{\ell}(v), & g_{\ell}(v)>0, \end{cases} \tag*{(1)}

where α>0\alpha> 0 is the concentration parameter. Each node without children has its own weight α\alpha, and smaller values of α\alpha favor nodes that have already received children.

Sampling stops when every component reaches the component size of the example. The additional components that Step 4 adds as distractors are sampled in the same way, and no two components of an example share a paper.

Prompt Assembly

This part details Step 4 of § 7.1.

One prompt per example for all models. Every evaluated model receives the same prompt for each example, so that all models are compared on the same input. However, the models use different tokenizers, so the same text has a different number of tokens for each model. We therefore treat the target context length LL as a limit on the whole prompt: the system prompt, the question, the context, the response-format reminder, and the tokens added by the chat template. A prompt satisfies this limit if it has at most LL tokens under each of the following tokenizers:

  • GEMINI: GEMMA-4 uses the same tokenizer as GEMINI (Team et al., [57]), so we use the GEMMA-4 tokenizer to estimate the token counts of the GEMINI models.

  • QWEN3.5: the tokenizer shared by the evaluated QWEN3.5 models (Qwen Team, [51]).

  • GPT-5.6: the o200k_base encoding of tiktoken5, with a fixed overhead of 3 tokens per message and 3 tokens per prompt.

We use the same three tokenizers for every target context length.

Filling the context. We build the prompt of each example as follows.

  1. If the needles alone exceed LL under any tokenizer, we construct the example again. We choose the component sizes and target context lengths so that this is unlikely.

  1. We draw distractors one at a time. A distractor is one source file, one essay, one book file, one Wikipedia passage, or one additional component. We split text distractors into blocks at blank lines, and we render an additional component as in Step 3, with one block for each paper that cites another paper in the component.

  1. Each needle and each distractor block receives an independent random number, and the context lists them in increasing order of these numbers, so the needles appear at random positions among the distractors.

  1. After each distractor, we count the tokens of the whole prompt with each tokenizer. Once the prompt reaches LL, we keep the longest sequence of blocks, in the order they were drawn, for which the largest of the three counts is at most LL. We stop when the next block does not fit, or when the prompt is within 100 tokens of LL.

The last distractor may therefore be cut. In that case, we keep its blocks from the beginning, in their original order, up to the last block that fits; we never split a block. For an additional component, this order is the order in which component sampling added its papers, so the papers added last are dropped. Because a block is never split, the prompt can fall short of LL by up to the size of the first block that does not fit.

The prompt is close to LL only under the tokenizer that gives the largest count, and it is shorter under the other two. For example, with wikipedia distractors and L=200,000L = 200{,}000, a prompt has about 189,000 tokens under the GPT tokenizer (Table 2).

Value#ExamplesQuestion (words)Context (k words)Input (k tokens)NeedlesNeedle words (%)#Nodes
Question depth kk
410072.65±20.8772.65 \pm20.87101.78±25.55101.78 \pm25.55174.79±16.52174.79 \pm16.52256.79±111.50256.79 \pm111.5027.30±15.5327.30 \pm15.53691.30±508.34691.30 \pm508.34
810073.55±20.1973.55 \pm20.1998.41±28.6898.41 \pm28.68171.80±26.64171.80 \pm26.64266.35±129.73266.35 \pm129.7330.85±22.4630.85 \pm22.46809.74±586.53809.74 \pm586.53
1610068.76±19.3968.76 \pm19.39101.48±26.39101.48 \pm26.39171.55±22.59171.55 \pm22.59249.62±128.76249.62 \pm128.7628.52±22.5528.52 \pm22.55737.17±566.70737.17 \pm566.70
3210068.30±18.7368.30 \pm18.73101.59±27.15101.59 \pm27.15169.96±25.30169.96 \pm25.30239.49±108.88239.49 \pm108.8827.86±22.1627.86 \pm22.16671.86±525.55671.86 \pm525.55
Component size
10010070.94±20.8570.94 \pm20.85102.27±27.98102.27 \pm27.98171.38±17.52171.38 \pm17.52145.67±52.83145.67 \pm52.8315.41±7.6115.41 \pm7.61530.73±565.42530.73 \pm565.42
15010073.72±20.0173.72 \pm20.0199.19±29.3699.19 \pm29.36171.15±25.48171.15 \pm25.48222.87±84.02222.87 \pm84.0226.69±19.5326.69 \pm19.53631.39±509.17631.39 \pm509.17
20010069.49±19.7369.49 \pm19.7392.56±28.0292.56 \pm28.02168.00±25.78168.00 \pm25.78282.40±103.96282.40 \pm103.9635.14±21.7735.14 \pm21.77777.08±519.18777.08 \pm519.18
25010071.59±18.2371.59 \pm18.2395.69±29.4295.69 \pm29.42166.83±33.27166.83 \pm33.27367.29±133.00367.29 \pm133.0045.21±26.8445.21 \pm26.84797.78±412.91797.78 \pm412.91
Task
YES/NO10054.29±6.0954.29 \pm6.09104.77±26.06104.77 \pm26.06175.14±17.60175.14 \pm17.60183.88±54.32183.88 \pm54.3219.61±9.8719.61 \pm9.87634.58±565.27634.58 \pm565.27
Multiple choice10097.24±9.9297.24 \pm9.9286.75±27.4486.75 \pm27.44163.65±32.74163.65 \pm32.74387.19±112.43387.19 \pm112.4350.41±24.3850.41 \pm24.38856.47±409.24856.47 \pm409.24
Counting10063.93±3.6263.93 \pm3.6297.04±29.0897.04 \pm29.08171.72±22.95171.72 \pm22.95189.88±53.94189.88 \pm53.9422.70±14.5122.70 \pm14.51671.65±587.54671.65 \pm587.54
Distractor type
books10071.40±20.2871.40 \pm20.28113.84±10.71113.84 \pm10.71176.48±9.13176.48 \pm9.13243.38±110.58243.38 \pm110.5822.23±12.7622.23 \pm12.76438.84±197.92438.84 \pm197.92
code10070.32±19.9770.32 \pm19.9759.09±7.2359.09 \pm7.23155.13±8.25155.13 \pm8.25257.72±121.82257.72 \pm121.8241.95±14.9441.95 \pm14.94468.08±213.22468.08 \pm213.22
disjoint-component10070.30±19.1470.30 \pm19.1490.93±1.8590.93 \pm1.85186.08±1.26186.08 \pm1.26232.10±90.69232.10 \pm90.6925.71±11.1525.71 \pm11.151,674.75±92.831,674.75 \pm92.83
essays10073.09±19.5373.09 \pm19.5395.58±40.3995.58 \pm40.39138.95±42.07138.95 \pm42.07268.15±127.40268.15 \pm127.4044.38±40.2544.38 \pm40.25482.44±218.73482.44 \pm218.73
wikipedia10075.73±19.4675.73 \pm19.46121.72±5.80121.72 \pm5.80188.70±0.61188.70 \pm0.61280.18±113.73280.18 \pm113.7323.08±10.6323.08 \pm10.63502.76±200.20502.76 \pm200.20
Reasoning effort
low/medium/high10073.76±20.5573.76 \pm20.5599.23±27.9899.23 \pm27.98171.39±19.43171.39 \pm19.43262.73±126.45262.73 \pm126.4529.72±18.5729.72 \pm18.57628.93±464.16628.93 \pm464.16
Target context length
200k10073.76±20.5573.76 \pm20.5599.23±27.9899.23 \pm27.98171.39±19.43171.39 \pm19.43262.73±126.45262.73 \pm126.4529.72±18.5729.72 \pm18.57628.93±464.16628.93 \pm464.16
500k5073.22±21.7173.22 \pm21.71262.21±82.65262.21 \pm82.65442.57±37.30442.57 \pm37.30273.32±132.35273.32 \pm132.3512.15±7.9612.15 \pm7.961,150.66±1,453.651,150.66 \pm1,453.65
922k5074.48±20.5274.48 \pm20.52490.65±165.54490.65 \pm165.54821.01±70.67821.01 \pm70.67268.92±127.14268.92 \pm127.146.68±4.816.68 \pm4.811,798.20±2,843.101,798.20 \pm2,843.10
Overall1,76071.95±19.9271.95 \pm19.92113.48±81.37113.48 \pm81.37196.32±119.39196.32 \pm119.39256.72±121.56256.72 \pm121.5629.35±22.2229.35 \pm22.22752.38±762.13752.38 \pm762.13

Table 2. Statistics of the evaluated examples. Each row is one value of the tested variable of an experiment; the reasoning effort experiment uses the same examples for all three efforts, and these are also the 200k examples of the target context length experiment. Overall counts each distinct example once. Cells show mean±std over examples. Words are whitespace-separated. Context covers the needles and distractors, without the question and instructions. Input tokens cover the whole prompt, including the system prompt, counted with the GPT tokenizer (o200k_base). Needles is the number of needles, one for each paper of the components generated for the task that cites another paper in its component; Needle words is the percentage of context words that belong to needles. Nodes is the number of papers of the subgraph that appear in the context, including the additional components of disjoint-component. GraphNIAH is an evaluation benchmark and has no training or development split.

Answer Balance

Within each value of the tested variable, we go through the YES/NO examples in order and assign their labels in pairs, one YES and one NO in random order. Similarly, we assign the correct option letters to consecutive groups of four multiple-choice examples, one per letter in random order. Because the task of each example is sampled, the last pair or group may be incomplete (Table 3).

Value#ExamplesYESNOABCD#Examples
Question depth kk
41001717898932
8100141491010934
161002020787731
321002122676731
Component size
1001001919888929
1501001414898938
2001002121777730
2501001414888840
Task
YES/NO1005050–––––
Multiple choice100––25252525–
Counting100––––––100
Distractor type
books1002019878731
code1001616777740
disjoint-component1001616777740
essays1001516999933
wikipedia10011121111111133
Reasoning effort
low/medium/high1001414988839
Target context length
200k1001414988839
500k5066455420
922k5066455420
Overall1,760288287144147145146603

Table 3. Ground truth answers of the evaluated examples in Table 2: the number of YES-label and NO-label examples, of multiple-choice examples whose correct option has each letter, and of counting examples. Within each value of the tested variable, YES/NO labels are assigned in pairs and option letters in groups of four (Appendix § 12.2.5). The ground truth count of a counting example is its component size minus one. A dash marks a task that the row does not contain.

Dataset Statistics

Table 2 summarizes the evaluated examples of each experiment, and Table 3 gives their ground truth answers.

Question Templates

Figure 8 shows the question templates of the three tasks; the system and user prompts that contain them are in Appendix § 19. The multiple-choice template has four candidate papers. When the none option is included in the multiple-choice question, the last entry in the option list is E. None of these targets is reachable from the source.

Question templates for the three reachability tasks

Figure 8. Question templates for the three reachability tasks. Placeholders in braces denote example-specific values.

APA-Style Template for Papers

Figure 8 shows the question templates of the three tasks; the system and user prompts that contain them are in Appendix § 19. The multiple-choice template has four candidate papers. For the negative option test of the multiple-choice reachability task in § 9.1, we append a fifth option, E. None of these targets is reachable from the source., after option D and keep the rest of the question unchanged.

Dates are rendered as a month name, day, and year (e.g., September 1, 2025). For an arXiv paper without another publication venue, the publication information line Published in {Venue of the paper} on {Publication date}. is replaced with Uploaded to arXiv on {Publication date}.

In the References part, a reference source is a publication venue, arXiv:{arXiv ID}, or arXiv, depending on available metadata. Missing metadata is omitted. For example, a publication line may contain only the venue or date, or may be omitted entirely. Similar to the main block, Reference entries omit unavailable authors, year, or source.

As in Step 3, a paper with no selected outgoing citation has no block of its own and appears only in the reference lists of the papers that cite it. Such a paper is still a node of the subgraph, so the model must include it when it decides reachability or counts reachable papers.

Selection of the Concentration Parameter

Step 2 aims to mimic the degree distribution of citation graphs (§ 7.1). We therefore select the concentration parameter α\alpha in Equation (1) by comparing the in-degree distribution across generated examples with the in-degree distribution of the filtered Semantic Scholar citation graph, which contains the papers published on or after the cutoff date (April 1, 2026, in this study). We measure the

difference with the Kullback-Leibler (KL) divergence (Kullback and Leibler, [31]) from the first distribution to the second.

Selection Setup

We test ten values, α∈A={0.1,0.2,…,1.0}\alpha\in\mathcal{A} = \{0.1, 0.2, \ldots, 1.0\}. For each value, we generate 2,000 examples of each task, which gives 6,000 examples per value and 60,000 examples in total. We sample the question depth and component size of each example uniformly at random, as in the experiments (§ 8.1), and use a target context length of 200,000 tokens. We add no distractors to these examples, because they make the search slower and four of the five distractor types add no papers to the subgraph.

Degree Distribution Calculation

An edge u→vu \to v means that paper uu cites paper vv. The in-degree of paper vv is the number of that cite it, i.e., din(v)=∣{u:u→v}∣d_{\mathrm{in}}(v) = |\{u : u \to v\}|. For a generated example, we count only the citation edges inside its subgraph; for the filtered Semantic Scholar citation graph, we count only the citation edges between papers published on or after the cutoff date.

APA-style template used to render each paper

Figure 9. APA-style template used to render each paper. Placeholders in braces denote example-specific values.

In both distributions, we keep only in-degree values of at least 1 and normalize the remaining counts. This filtering affects only the comparison, not the generation of examples. We select this in-degree filtering because all papers in the subgraph must have at least one citation edge (due to the nature of the component sampling procedure). Hence, we only consider degrees greater than or equal to 1 to avoid the mismatch between the two distributions at low degrees.

For a fixed concentration parameter α\alpha and reachability task τ\tau, let hi(a)h_i(a) count papers with in-degree aa in example ii; we pool these counts and normalize over degrees greater than or equal to 1 to obtain the conditional degree distribution across generated examples:

Pα,τ(a)=Hα,τ(a)∑a′≥1Hα,τ(a′)(2)P_{\alpha,\tau}(a) = \frac{H_{\alpha,\tau}(a)}{\sum_{a'\geq1} H_{\alpha,\tau}(a')} \tag*{(2)}

where Hα,τ(a)=∑ihi(a)H_{\alpha,\tau}(a) = \sum_i h_i(a) and a≥1a \geq1. Let H∗(a)H^*(a) count papers with in-degree aa in filtered Semantic Scholar citation graph. Its conditional degree distribution is

Q(a)=H∗(a)∑a′≥1H∗(a′),a≥1.(3)Q(a) = \frac{H^*(a)}{\sum_{a' \ge1} H^*(a')}, \qquad a \ge1. \tag*{(3)}

Selection of the Concentration Parameter with KL Divergence

We form a common support SS from all in-degree values of at least 1 observed in the filtered Semantic Scholar citation graph or across generated examples for any candidate value of α\alpha and any reachability task. Unobserved degree values within this support receive zero probability. Let M=∣S∣M = |S| denote the number of support values. To prevent the KL divergence from becoming infinite when a degree value has zero probability, we smooth both conditional distributions with ε=10−8\varepsilon= 10^{-8}

P~α,τ(a)=(1−ε)Pα,τ(a)+εM,\widetilde{P}_{\alpha,\tau}(a) = (1-\varepsilon)P_{\alpha,\tau}(a) + \frac{\varepsilon}{M},
Q~(a)=(1−ε)Q(a)+εM.(4)\widetilde{Q}(a) = (1-\varepsilon)Q(a) + \frac{\varepsilon}{M}. \tag*{(4)}

For each value α\alpha and task τ\tau, we calculate KL divergence from the conditional degree distribution across generated examples to that of the filtered Semantic Scholar citation graph:

Dα,τ=∑a∈SP~α,τ(a)⋅log⁡P~α,τ(a)Q~(a).(5)D_{\alpha,\tau} = \sum_{a \in S} \widetilde{P}_{\alpha,\tau}(a) \cdot\log\frac{\widetilde{P}_{\alpha,\tau}(a)}{\widetilde{Q}(a)}. \tag*{(5)}

We average the three KL divergences, one for each task, to obtain the mean KL divergence of each candidate:

S(α)=13∑τDα,τ(6)S(\alpha) = \frac{1}{3}\sum_{\tau} D_{\alpha,\tau} \tag*{(6)}

Finally, we select the value of α\alpha with the minimum mean KL divergence:

α∗=arg⁡min⁡α∈A S(α).(7)\alpha^* = \underset{\alpha\in\mathcal{A}}{\arg\min}\ S(\alpha). \tag*{(7)}

Concentration Parameter Selection Results

Among the ten values, α∗=0.1\alpha^* = 0.1 gives the smallest mean KL divergence, S(0.1)=0.0089S(0.1) = 0.0089 (Figure 10), so we use it to generate the examples of all experiments (§ 8.1). To check whether this choice is stable, we draw 500 bootstrap replicates, resampling the examples with replacement within each task and value of α\alpha, and report pointwise 95% percentile bootstrap intervals. The value α=0.1\alpha= 0.1 is selected in 98.4% of the replicates, and α=0.2\alpha= 0.2 in the remaining 1.6%.

Figure 10. Concentration parameter selection results. Top: In-degree distributions of the filtered Semantic Scholar citation graph and generated examples using the selected value α⋆=0.1\alpha^\star= 0.1, shown on linear (left) and logarithmic (right) axes. Bottom: Mean KL divergence S(α)S(\alpha) with 95% bootstrap intervals (left) and bootstrap selection frequencies (right).

Models and Inference Settings

Table 4 lists the evaluated models. We evaluate all models with temperature = 1.0 and reasoning_effort = low; the reasoning effort experiment (§ 9.4) also uses medium and high. We set top_p = 0.95 for every model except the GPT-5.6 models, which do not accept top_p while reasoning is enabled. We set no limit on the number of output tokens, except for the QWEN3.5 models. We call all models through LiteLLM6.

Model#ParamsRelease dateCutoff dateContext lengthI/O pricingProvider
Proprietary Models
GPT-5.6-SOLN/AJul. 9, 2026Feb. 16, 20261,050,0004.00 / 20.00OpenAI
GPT-5.6-LUNAN/AJul. 9, 2026Feb. 16, 20261,050,0000.20 / 1.20OpenAI
GEMINI-3.8-FLASHN/ASep. 2, 2026Mar. 20261,048,5760.75 / 3.75Google
GEMINI-3.6-FLASHN/AJul. 21, 2026Mar. 20261,048,5760.75 / 3.75Google
GEMINI-3.1-PRON/AFeb. 19, 2026Jan. 20251,048,5762.00 / 12.00Google
Open-source Models
QWEN3.5-397B-A17B397B (17B active)Feb. 16, 2026Feb. 16, 2026262,1440.39 / 2.34OpenRouter
QWEN3.5-122B-A10B122B (10B active)Feb. 24, 2026Feb. 24, 2026262,1440.26 / 2.08OpenRouter
QWEN3.5-35B-A3B35B (3B active)Feb. 24, 2026Feb. 24, 2026262,1440.16 / 1.30OpenRouter

Table 4. Model specifications. Qwen has not published a training cutoff date for the QWEN3.5 models, so we use their release dates as their cutoff dates. I/O pricing is the standard price in US dollars per million input and output tokens; for the QWEN3.5 models, it is the price of Alibaba, the first provider in our OpenRouter routing order. For GEMINI-3.1-PRO, the listed price applies to prompts of at most 200,000 tokens; above this length, the standard price is 4.00 / 18.00. Its flex prices are 1.00 / 6.00 up to 200,000 tokens and 2.00 / 9.00 above. N/A: not published.

We send the GPT-5.6 and GEMINI requests at the flex service tier of OpenAI and resend a request at the standard tier when the flex tier has no capacity.

For the QWEN3.5 models, we add top_k = 20, min_p = 0, and presence_penalty = 1.5, the values that Qwen recommends for thinking mode on general tasks, and allow at most 40,000 output tokens. We access these models through OpenRouter, with Alibaba as the first provider and DeepInfra (FP8 weights) as the second.

Random Guessing Baseline

We compare each model with random guessing, which ignores the context and chooses an answer uniformly at random from the possible answers of the task. We compute its expected accuracy exactly from the sampling probabilities of the benchmark construction pipeline, rather than measuring it on the evaluated examples, so the baseline does not depend on which examples were drawn.

Accuracy on one task. Let AτA_\tau denote the set of possible answers of task τ\tau. For any ground truth answer y∈Aτy \in A_\tau, a uniform random guess equals yy with probability 1/∣Aτ∣1/|A_\tau|. This probability is the same for every yy, so the expected accuracy of random guessing on task τ\tau is Acc⁡τ=1/∣Aτ∣\operatorname{Acc}_\tau= 1/|A_\tau| for any distribution of ground truth answers:

  • YES/NO reachability: Aτ={YES,NO}A_\tau= \{\mathrm{YES}, \mathrm{NO}\}, so Acc⁡τ=1/2=50.00%\operatorname{Acc}_\tau= 1/2 = 50.00\%.

  • Multiple-choice reachability: four options, so Acc⁡τ=1/4=25.00%\operatorname{Acc}_\tau= 1/4 = 25.00\%.

  • Reachability counting: the component size is an integer in [100,250][100, 250] and ycount=∣Ks∣−1y_{\mathrm{count}} = |K_s| - 1, so AτA_\tau contains the 151 integers from 99 to 249 and Acc⁡τ=1/151≈0.66%\operatorname{Acc}_\tau= 1/151 \approx0.66\%.

We use these answer sets in every experiment.

Accuracy on a mix of tasks. In every experiment except the task experiment, the task of each example is sampled uniformly from the three tasks. By the law of total probability, the expected accuracy of random guessing is

Acc⁡mix=∑τPr⁡(τ)Acc⁡τ=13(12+14+1151)≈25.22%.(8)\operatorname{Acc}_{\mathrm{mix}} = \sum_{\tau} \Pr(\tau)\operatorname{Acc}_{\tau} = \frac{1}{3}\left(\frac{1}{2}+\frac{1}{4}+\frac{1}{151}\right) \approx25.22\%. \tag*{(8)}

Task experiment. In the task experiment, each value of the tested variable is one task, so the baseline for each value is Acc⁡τ\operatorname{Acc}_{\tau}: 50.00%, 25.00%, and 0.66%. Pooled over the three tasks, which have the same number of examples, the baseline is again 25.22%.

Extra Experimental Results

This section breaks the results of § 8 down by task and by ground truth answer, and it reports the statistical tests used in the analysis.

Complexity Scaling

Table 5 to Table 10 split the accuracy of the question depth and component size experiments (§4.2) by task. Table 11 and Table 12 further split their YES/NO reachability accuracy into YES-label and NO-label examples.

ModelLabel100150200250Overall
GPT-5.6-SOLYES-label50.00† [18.76, 81.24]25.00† [4.56, 69.94]52.38 [32.37, 71.66]42.86 [21.38, 67.41]46.67† [32.93, 60.92]
GPT-5.6-SOLNO-label100.00† [60.97, 100.00]100.00† [43.85, 100.00]100.00 [84.54, 100.00]100.00 [78.47, 100.00]100.00† [91.97, 100.00]
GPT-5.6-LUNAYES-label31.58 [15.36, 53.99]14.29 [4.01, 39.94]19.05 [7.67, 40.00]7.14 [1.27, 31.47]19.12 [11.53, 30.01]
GPT-5.6-LUNANO-label100.00 [83.18, 100.00]100.00 [78.47, 100.00]100.00 [84.54, 100.00]100.00 [78.47, 100.00]100.00 [94.65, 100.00]
GEMINI-3.1-PROYES-label73.68 [51.21, 88.19]64.29 [38.76, 83.66]57.14 [36.55, 75.53]28.57 [11.72, 54.65]57.35 [45.52, 68.40]
GEMINI-3.1-PRONO-label100.00 [83.18, 100.00]100.00 [78.47, 100.00]100.00 [84.54, 100.00]100.00 [78.47, 100.00]100.00 [94.65, 100.00]
GEMINI-3.8-FLASHYES-label100.00 [83.18, 100.00]92.86 [68.53, 98.73]100.00 [84.54, 100.00]100.00 [78.47, 100.00]98.53 [92.13, 99.74]
GEMINI-3.8-FLASHNO-label73.68 [51.21, 88.19]78.57 [52.41, 92.43]57.14 [36.55, 75.53]78.57 [52.41, 92.43]70.59 [58.89, 80.08]
GEMINI-3.6-FLASHYES-label57.89 [36.28, 76.86]42.86 [21.38, 67.41]52.38 [32.37, 71.66]28.57 [11.72, 54.65]47.06 [35.67, 58.76]
GEMINI-3.6-FLASHNO-label84.21 [62.43, 94.48]100.00 [78.47, 100.00]100.00 [84.54, 100.00]100.00 [78.47, 100.00]95.59 [87.81, 98.49]
QWEN3.5-397B-A17BYES-label47.37 [27.33, 68.29]14.29 [4.01, 39.94]23.81 [10.63, 45.09]14.29 [4.01, 39.94]26.47 [17.45, 38.01]
QWEN3.5-397B-A17BNO-label100.00 [83.18, 100.00]100.00 [78.47, 100.00]100.00 [84.54, 100.00]100.00 [78.47, 100.00]100.00 [94.65, 100.00]
QWEN3.5-122B-A10BYES-label31.58 [15.36, 53.99]14.29 [4.01, 39.94]14.29 [4.98, 34.64]7.14 [1.27, 31.47]17.65 [10.39, 28.36]
QWEN3.5-122B-A10BNO-label100.00 [83.18, 100.00]100.00 [78.47, 100.00]100.00 [84.54, 100.00]100.00 [78.47, 100.00]100.00 [94.65, 100.00]
QWEN3.5-35B-A3BYES-label21.05 [8.51, 43.33]21.43 [7.57, 47.59]14.29 [4.98, 34.64]7.14 [1.27, 31.47]16.18 [9.28, 26.69]
QWEN3.5-35B-A3BNO-label100.00 [83.18, 100.00]92.86 [68.53, 98.73]100.00 [84.54, 100.00]100.00 [78.47, 100.00]98.53 [92.13, 99.74]
Random guessing50.0050.0050.0050.0050.00
#ExamplesYES-label1914211468
#ExamplesNO-label1914211468

Table 12. YES/NO reachability accuracy (%) of each model on YES-label and NO-label examples of the component size experiment (§4.2), with reasoning effort low. Overall pools all component sizes. Below each accuracy value is its 95% Wilson confidence interval. †GPT-5.6-SOL has fewer examples than the other models in these cells: YES-label, 6 at 100, 4 at 150, and 45 Overall; NO-label, 6 at 100, 3 at 150, and 44 Overall.

Reachability Tasks

For the task experiment (§ 9.1), Table 13 and Table 14 report YES/NO reachability accuracy by ground truth answer and the percentage of NO answers by question depth. Table 15 and Table 16 compare the counting answers with the ground truth counts and report counting accuracy by component size. Table 17 and Table 18 report the accuracy and the number of tokens in the model’s answer for the multiple-choice task with a "None of the above" option.

ModelExactToo lowToo highNo numberWithin 10%Median ratio
GPT-5.6-LUNA0.00 [0.00, 3.70]98.00 [93.00, 99.45]2.00 [0.55, 7.00]0.00 [0.00, 3.70]1.00 [0.18, 5.45]0.30
GEMINI-3.8-FLASH55.00 [45.24, 64.39]12.00 [7.00, 19.81]15.00 [9.31, 23.28]18.00 [11.70, 26.67]64.00 [54.24, 72.73]1.00
QWEN3.5-35B-A3B0.00 [0.00, 3.70]99.00 [94.55, 99.82]0.00 [0.00, 3.70]1.00 [0.18, 5.45]0.00 [0.00, 3.70]0.16

Table 15. Outcomes of the reachability counting answers of each model in the task experiment (§ 9.1), with reasoning effort low, as a percentage of its 100 counting examples; below each percentage is its 95% Wilson confidence interval. Exact, Too low, Too high, and No number add up to 100%: Too low and Too high compare a wrong numeric answer with the ground truth count, and No number covers responses without a number in the final answer block, including responses without an answer block. Within 10% includes exact answers. Median ratio is the median of the predicted count divided by the ground truth count over the numeric answers.

Model100–149150–199200–250Overall
GPT-5.6-LUNA0.00 [0.00, 10.43]0.00 [0.00, 10.72]0.00 [0.00, 9.89]0.00 [0.00, 3.70]
GEMINI-3.8-FLASH78.79 [62.25, 89.32]59.38 [42.26, 74.48]28.57 [16.33, 45.06]55.00 [45.24, 64.39]
QWEN3.5-35B-A3B0.00 [0.00, 10.43]0.00 [0.00, 10.72]0.00 [0.00, 9.89]0.00 [0.00, 3.70]
Random guessing0.660.660.660.66
#Examples333235100

Table 16. Reachability counting accuracy (%) of each model in the task experiment (§ 9.1) by component size, with reasoning effort low. In this experiment, the component size of each example is sampled uniformly at random from the integers in [100, 250]. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column.

Distractor Type

Table 19 splits the accuracy of the distractor type experiment (§ 9.3) by task, and Table 20 compares all five distractor types with a χ2\chi^2 test, as well as compare each pair of distractor types with Fisher’s exact test, corrected for the ten pairs with the Holm method.

ModelTaskbookscodedisjoint-componentessayswikipediaOverall
GPT-5.6-LUNAYES/NO56.41 [40.98, 70.70]65.62 [48.31, 79.59]59.38 [42.26, 74.48]67.74 [50.14, 81.43]56.52 [36.81, 74.37]61.15 [53.34, 68.42]
GPT-5.6-LUNAMultiple choice63.33 [45.51, 78.13]71.43 [52.94, 84.75]78.57 [60.46, 89.79]69.44 [53.14, 82.00]84.09 [70.63, 92.07]74.10 [66.94, 80.16]
GPT-5.6-LUNACounting0.00 [0.00, 11.03]0.00 [0.00, 8.76]0.00 [0.00, 8.76]0.00 [0.00, 10.43]0.00 [0.00, 10.43]0.00 [0.00, 2.12]
GEMINI-3.8-FLASHYES/NO89.74 [76.42, 95.94]90.62 [75.78, 96.76]90.62 [75.78, 96.76]96.77 [83.81, 99.43]69.57 [49.13, 84.40]88.54 [82.61, 92.62]
GEMINI-3.8-FLASHMultiple choice96.67 [83.33, 99.41]96.43 [82.29, 99.37]96.43 [82.29, 99.37]100.00 [90.36, 100.00]97.73 [88.19, 99.60]97.59 [93.97, 99.06]
GEMINI-3.8-FLASHCounting61.29 [43.82, 76.27]40.00 [26.35, 55.40]52.50 [37.50, 67.06]66.67 [49.61, 80.25]42.42 [27.24, 59.19]51.98 [44.65, 59.22]
QWEN3.5-35B-A3BYES/NO56.41 [40.98, 70.70]71.88 [54.63, 84.44]59.38 [42.26, 74.48]67.74 [50.14, 81.43]56.52 [36.81, 74.37]62.42 [54.63, 69.61]
QWEN3.5-35B-A3BMultiple choice53.33 [36.14, 69.77]67.86 [49.34, 82.07]75.00 [56.64, 87.32]58.33 [42.20, 72.86]75.00 [60.56, 85.43]66.27 [58.78, 73.02]
QWEN3.5-35B-A3BCounting0.00 [0.00, 11.03]0.00 [0.00, 8.76]0.00 [0.00, 8.76]0.00 [0.00, 10.43]0.00 [0.00, 10.43]0.00 [0.00, 2.12]
Random guessingYES/NO50.0050.0050.0050.0050.0050.00
Random guessingMultiple choice25.0025.0025.0025.0025.0025.00
Random guessingCounting0.660.660.660.660.660.66
#ExamplesYES/NO3932323123157
#ExamplesMultiple choice3028283644166
#ExamplesCounting3140403333177

Table 19. Accuracy (%) of each model on each task in the distractor type experiment (§ 9.3), with reasoning effort low. Overall pools all distractor types. The number of examples per task differs between columns because the task of each example is sampled at random. Below each accuracy value is its 95% Wilson confidence interval.

Reasoning Effort

Table 21 compares each pair of reasoning efforts on the same 100 examples (§ 9.4): for each pair, it gives the number of examples that become correct or wrong and the pp-value of the exact McNemar test.

Context Length Tests

Table 22 compares the accuracy at each pair of target context lengths (§ 9.2) with Fisher’s exact test. Because the examples differ between context lengths and share at most their settings, the test treats the two sets of answers as independent samples instead of pairs.

Full Experimental Results by Experiments

Details of the experimental results are provided in Table 23 to Table 28. Each table shows the accuracy of each model for each value of the studied variable, along with 95% Wilson confidence intervals. The tables also include the expected accuracy of random guessing, which serves as a baseline for comparison.

Model481632Overall
GPT-5.6-SOL–75.00 [65.70, 82.45]56.00 [46.23, 65.33]54.00 [44.26, 63.44]61.67{sup}† [56.05, 66.99]
GPT-5.6-LUNA66.00 [56.28, 74.54]39.00 [30.02, 48.80]37.00 [28.18, 46.78]34.00 [25.46, 43.72]44.00 [39.22, 48.90]
GEMINI-3.1-PRO71.00 [61.46, 78.99]62.00 [52.21, 70.90]57.00 [47.22, 66.27]52.00 [42.32, 61.54]60.50 [55.63, 65.17]
GEMINI-3.8-FLASH87.00 [79.02, 92.24]83.00 [74.45, 89.11]87.00 [79.02, 92.24]59.00 [49.20, 68.13]79.00 [74.74, 82.71]
GEMINI-3.6-FLASH75.00 [65.70, 82.45]56.00 [46.23, 65.33]47.00 [37.51, 56.71]41.00 [31.87, 50.80]54.75 [49.85, 59.56]
QWEN3.5-397B-A17B65.00 [55.25, 73.64]48.00 [38.46, 57.68]42.00 [32.80, 51.79]38.00 [29.10, 47.79]48.25 [43.39, 53.14]
QWEN3.5-122B-A10B65.00 [55.25, 73.64]45.00 [35.61, 54.76]40.00 [30.94, 49.80]37.00 [28.18, 46.78]46.75 [41.91, 51.65]
QWEN3.5-35B-A3B63.00 [53.22, 71.82]42.00 [32.80, 51.79]37.00 [28.18, 46.78]39.00 [30.02, 48.80]45.25 [40.44, 50.15]
Random guessing25.2225.2225.2225.2225.22

Table 23. Accuracy (%) of each model in the question depth experiment (§4.2), with reasoning effort low. Each question depth has 100 examples, and Overall pools all question depths. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column. {sup}†GPT-5.6-SOL has fewer examples than the other models in this cell (300 Overall), so it is not bold. A dash marks a missing result.

Inference Cost and Response Time

Table 29 to Table 34 report, for each experiment, the number of answered examples, the output tokens, the response time, and the API cost of each model.

Model#ExamplesOutput tokensResponse time (s)Cost per example (¢)Total cost ($)
GPT-5.6-SOL∗^{\ast}3001,340.02±1,096.371{,}340.02 \pm1{,}096.3718.97±12.8618.97 \pm12.8644.12±6.3944.12 \pm6.39132.35
GPT-5.6-LUNA∗^{\ast}400715.69±529.55715.69 \pm529.5512.66±10.0212.66 \pm10.022.19±0.292.19 \pm0.298.77
GEMINI-3.1-PRO∗^{\ast}4004,642.54±4,621.154{,}642.54 \pm4{,}621.1592.16±99.7992.16 \pm99.7920.88±3.8120.88 \pm3.8183.54
GEMINI-3.8-FLASH40019,412.21±20,608.9019{,}412.21 \pm20{,}608.90104.96±104.96104.96 \pm104.9620.91±8.1920.91 \pm8.1983.62
GEMINI-3.6-FLASH∗^{\ast}40010,410.75±12,035.6210{,}410.75 \pm12{,}035.6261.35±59.8861.35 \pm59.887.79±3.687.79 \pm3.6831.16
QWEN3.5-397B-A17B40015,729.28±7,681.1315{,}729.28 \pm7{,}681.13439.19±371.56439.19 \pm371.5612.21±3.0812.21 \pm3.0848.85
QWEN3.5-122B-A10B40014,843.83±6,818.7014{,}843.83 \pm6{,}818.70210.63±79.53210.63 \pm79.537.87±1.567.87 \pm1.5631.47
QWEN3.5-35B-A3B40016,387.97±11,616.1916{,}387.97 \pm11{,}616.19162.29±99.04162.29 \pm99.045.10±1.545.10 \pm1.5420.41
All models440.17

Table 29. Cost and response time of each model in the question depth experiment (§4.2), with reasoning effort low. #Examples counts the examples that the model answered, and each cell shows mean±\pmstd over these examples. Output tokens are the completion tokens reported by the provider, including reasoning tokens. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total. ∗^{\ast}Sent at the flex service tier, which has lower prices and can respond more slowly.

ModelContext length#ExamplesOutput tokensResponse time (s)Cost per example (¢)Total cost ($)
GPT-5.6-LUNA*200k100823.21±539.75823.21 \pm539.7513.18±8.2713.18 \pm8.271.95±0.681.95 \pm0.681.95
500k50748.56±459.06748.56 \pm459.0614.51±6.1314.51 \pm6.1311.13±0.9311.13 \pm0.935.57
922k50844.76±544.15844.76 \pm544.1524.29±8.9124.29 \pm8.9120.60±1.7620.60 \pm1.7610.30
GEMINI-3.8-FLASH200k10023,086.40±23,049.8623{,}086.40 \pm23{,}049.86137.27±127.62137.27 \pm127.6222.46±8.8222.46 \pm8.8222.46
500k5031,639.22±25,951.4731{,}639.22 \pm25{,}951.47192.96±146.21192.96 \pm146.2147.85±9.8147.85 \pm9.8123.93
922k5032,091.20±26,945.3432{,}091.20 \pm26{,}945.34205.07±149.79205.07 \pm149.7978.99±10.1678.99 \pm10.1639.49
QWEN3.5-35B-A3B200k10015,219.30±5,487.9715{,}219.30 \pm5{,}487.97223.67±89.98223.67 \pm89.984.92±0.844.92 \pm0.844.92
All models108.62

Table 34. Cost and response time of each model in the target context length experiment (§ 9.2), with reasoning effort low. The 200k rows reuse the answers of the reasoning effort experiment. #Examples counts the examples that the model answered, and each cell shows mean±std_{\pm\mathrm{std}} over these examples. Output tokens are the completion tokens reported by the provider, including reasoning tokens. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total. *Sent at the flex service tier, which has lower prices and can respond more slowly.

Model100150200250Overall
GPT-5.6-SOL77.78{sup}† [59.24, 89.39]61.54{sup}† [42.53, 77.57]60.00 [50.20, 69.06]49.00 [39.42, 58.65]57.71{sup}† [51.55, 63.64]
GPT-5.6-LUNA44.00 [34.67, 53.77]38.00 [29.10, 47.79]42.00 [32.80, 51.79]37.00 [28.18, 46.78]40.25 [35.56, 45.13]
GEMINI-3.1-PRO74.00 [64.63, 81.60]57.00 [47.22, 66.27]55.00 [45.24, 64.39]46.00 [36.56, 55.74]58.00 [53.11, 62.74]
GEMINI-3.8-FLASH88.00 [80.19, 93.00]77.00 [67.85, 84.16]76.00 [66.77, 83.31]67.00 [57.31, 75.44]77.00 [72.63, 80.86]
GEMINI-3.6-FLASH69.00 [59.37, 77.22]52.00 [42.32, 61.54]54.00 [44.26, 63.44]43.00 [33.73, 52.78]54.50 [49.60, 59.31]
QWEN3.5-397B-A17B54.00 [44.26, 63.44]40.00 [30.94, 49.80]43.00 [33.73, 52.78]40.00 [30.94, 49.80]44.25 [39.46, 49.15]
QWEN3.5-122B-A10B50.00 [40.38, 59.62]40.00 [30.94, 49.80]41.00 [31.87, 50.80]36.00 [27.27, 45.76]41.75 [37.02, 46.64]
QWEN3.5-35B-A3B43.00 [33.73, 52.78]38.00 [29.10, 47.79]42.00 [32.80, 51.79]37.00 [28.18, 46.78]40.00 [35.32, 44.87]
Random guessing25.2225.2225.2225.2225.22

Table 24. Accuracy (%) of each model in the component size experiment (§4.2), with reasoning effort low. Each component size has 100 examples, and Overall pools all component sizes. Below each accuracy value is its 95% Wilson confidence interval. Bold marks the highest accuracy in each column. {sup}†GPT-5.6-SOL has fewer examples than the other models in these cells (27 at 100, 26 at 150, and 253 Overall), so they are not bold.

Model#ExamplesOutput tokensResponse time (s)Cost per example (¢)Total cost ($)
GPT-5.6-SOL∗^{\ast}2531,424.49±1,282.981{,}424.49 \pm1{,}282.9821.68±16.1021.68 \pm16.1041.83±10.6441.83 \pm10.64105.83
GPT-5.6-LUNA∗^{\ast}400785.28±541.69785.28 \pm541.6911.99±7.9511.99 \pm7.952.16±0.332.16 \pm0.338.66
GEMINI-3.1-PRO∗^{\ast}4004,744.15±5,510.224{,}744.15 \pm5{,}510.2242.75±45.8042.75 \pm45.8020.78±4.5020.78 \pm4.5083.13
GEMINI-3.8-FLASH40020,513.35±21,872.9920{,}513.35 \pm21{,}872.99120.58±122.39120.58 \pm122.3921.20±8.4421.20 \pm8.4484.79
GEMINI-3.6-FLASH∗^{\ast}40011,143.68±12,347.3411{,}143.68 \pm12{,}347.3458.21±60.7858.21 \pm60.786.89±3.566.89 \pm3.5627.56
QWEN3.5-397B-A17B40015,841.35±8,552.9415{,}841.35 \pm8{,}552.94350.65±214.14350.65 \pm214.1411.85±3.0511.85 \pm3.0547.39
QWEN3.5-122B-A10B40015,186.61±6,564.4315{,}186.61 \pm6{,}564.43213.11±85.54213.11 \pm85.547.90±1.527.90 \pm1.5231.61
QWEN3.5-35B-A3B40016,510.21±10,638.3416{,}510.21 \pm10{,}638.34152.97±81.26152.97 \pm81.265.08±1.455.08 \pm1.4520.33
All models409.29

Table 30. Cost and response time of each model in the component size experiment (§4.2), with reasoning effort low. #Examples counts the examples that the model answered, and each cell shows mean±\pmstd over these examples. Output tokens are the completion tokens reported by the provider, including reasoning tokens. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total. ∗^{\ast}Sent at the flex service tier, which has lower prices and can respond more slowly.

Model#ExamplesOutput tokensResponse time (s)Cost per example (¢)Total cost ($)
GPT-5.6-LUNA∗^{\ast}300738.38±529.67738.38 \pm529.6713.23±9.3313.23 \pm9.332.17±0.322.17 \pm0.326.51
GEMINI-3.8-FLASH30020,248.57±21,838.1420{,}248.57 \pm21{,}838.14114.54±114.74114.54 \pm114.7421.13±8.5421.13 \pm8.5463.39
QWEN3.5-35B-A3B30016,471.95±9,999.1116{,}471.95 \pm9{,}999.11155.31±76.50155.31 \pm76.505.08±1.365.08 \pm1.3615.25
All models85.15

Table 31. Cost and response time of each model in the task experiment (§ 9.1), with reasoning effort low. #Examples counts the examples that the model answered, and each cell shows mean±\pmstd over these examples. Output tokens are the completion tokens reported by the provider, including reasoning tokens. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total. ∗^{\ast}Sent at the flex service tier, which has lower prices and can respond more slowly.

Model#ExamplesOutput tokensResponse time (s)Cost per example (¢)Total cost ($)
GPT-5.6-LUNA*500759.95±519.65759.95 \pm519.6513.94±8.2613.94 \pm8.262.16±0.342.16 \pm0.3410.79
GEMINI-3.8-FLASH50021,187.22±21,680.0721{,}187.22 \pm21{,}680.07124.62±120.08124.62 \pm120.0821.43±8.5721.43 \pm8.57107.17
QWEN3.5-35B-A3B50017,419.60±11,881.6917{,}419.60 \pm11{,}881.69162.62±96.15162.62 \pm96.155.17±1.615.17 \pm1.6125.84
All models143.81

Table 32. Cost and response time of each model in the distractor type experiment (§ 9.3), with reasoning effort low. #Examples counts the examples that the model answered, and each cell shows mean±std_{\pm\mathrm{std}} over these examples. Output tokens are the completion tokens reported by the provider, including reasoning tokens. Response time runs from sending the successful request to receiving its response, so it excludes waits before sending and failed attempts. Cost is the LiteLLM estimate from the token counts of each response and the provider’s prices, in US cents per example and US dollars in total. *Sent at the flex service tier, which has lower prices and can respond more slowly.

Prompt Templates

We provide the system and user prompt templates for the three reachability tasks. The question templates and the APA-style template appear in Appendices § 12.§ 8 and § 12.§ 10, respectively. The context describes scientific papers and their citation relationships; author names serve only as publication metadata. Placeholders in braces denote example-specific values.

System Prompts

Figure 11 to Figure 13 show the system prompts for the YES/NO reachability, multiple-choice reachability, and reachability counting tasks, respectively.

System Prompt for YES/NO Reachability

Figure 11. System prompt for the YES/NO reachability task.

System Prompt for Reachability Counting. You are a careful research assistant answering questions about academic papers using only the provided context. You will receive a QUESTION and a CONTEXT containing academic paper documents. The CONTEXT is organized into structured blocks about academic papers and their references. A block beginning with Title describes one paper. Each item in References identifies a paper cited by the paper described in that block. The Authors field is publication metadata only; do not follow it as a relationship. Use only citation relationships stated in the CONTEXT. Whenever Paper A cites Paper B, there is a ‘cites’ relationship from A to B, and a ‘cited_by’ relationship from B to A. A paper is reachable from the SOURCE paper if there exists a sequence of relationships that leads from the SOURCE paper to that paper, where each relationship is either a ‘cites’ or ‘cited_by’ relationship. Read the QUESTION carefully to identify the SOURCE. Use only the information in the provided CONTEXT. The CONTEXT contains all information available for this task. Treat entities and relationships not stated in the CONTEXT as absent. Starting from the SOURCE paper, count every distinct paper reachable through one or more ‘cites’ or ‘cited_by’ relationships. A paper appearing only in a References list is still a paper node and must be counted when reachable. Do not count the SOURCE paper. Paper mentions with the same title and the same complete set of authors denote one paper node. Count that paper only once even if it appears in multiple blocks or reference lists. Paper mentions with the same title but different author sets denote different paper nodes. Ignore superficial differences in capitalization, punctuation, author order, and repeated author names when applying this identity rule. You must not include or exclude a paper based on how closely related it appears to be to the SOURCE paper, such as similarities in their topics, research fields, or titles. A paper on an unrelated topic must be counted if it is reachable, while a paper on a closely related topic must not be counted if it is unreachable. Your response may contain reasoning, but its final line must be exactly one answer block containing a non-negative base-10 integer, for example: \answer{17} Do not use commas or other text inside the answer block. Do not write more than one answer block or put anything after the final answer line.

Figure 13. System prompt for the reachability counting task.

User Prompt Template

All tasks share the user prompt template in Figure 14. The question is instantiated from the corresponding template in Appendix § 12.§ 8. The context contains shuffled natural language blocks and distractors, separated by a blank line, i.e., \n\n. The response-format reminder, which depends on the task, is shown in Figure 15. The ellipsis in the template stands for the remaining blocks and distractors and is not part of the generated message.

User Prompt Template Shared by All Tasks. # Question. {QUESTION}. # Context. {Natural language block 1}. {Natural language block 2}. ... # Response Format. {TASK_SPECIFIC_FORMAT_REMINDER}

Figure 14. User prompt template shared by all tasks.

Response-format reminders for YES/NO Reachability, Multiple-choice Reachability, and Reachability Counting

Figure 15. Response-format reminders for the three reachability tasks. Placeholders in braces denote example-specific values.

AI use statement

In this work, we used generative AI tools for supporting coding and writing. We have not used generative AI tools for literature review, forming research direction, finding research gap, ideation, or designing method/benchmark/experiments. We have reviewed all AI-assisted work. LLM-generated code was verified and tested for correctness by the first author. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

System Prompt for Multiple-choice Reachability
System Prompt for Multiple-choice Reachability

You are a careful research assistant answering questions about academic papers using only
the provided context.

You will receive a multiple-choice QUESTION and a CONTEXT containing academic paper documents.

The CONTEXT is organized into structured blocks about academic papers and their references.
A block beginning with Title describes one paper. Each item in References identifies a paper
cited by the paper described in that block. The Authors field is publication metadata only;
do not follow it as a relationship.

Use only citation relationships stated in the CONTEXT. Whenever Paper A cites Paper B, there
is a ‘cites’ relationship from A to B, and a ‘cited_by’ relationship from B to A. For
each option, starting from the SOURCE paper, determine whether there exists a sequence of
relationships that leads to the paper named by that option, where each relationship is either
a ‘cites’ or ‘cited_by’ relationship.

Read the QUESTION carefully to identify the SOURCE and every option. Use only the information
in the provided CONTEXT. The CONTEXT contains all information available for this task. Treat
entities and relationships not stated in the CONTEXT as absent. You must not guess the
answer based on how closely related the SOURCE paper appears to be to the papers listed in
the options, such as similarities in their topics, research fields, or titles. Two papers on
similar topics may have no connecting sequence of relationships, while two papers on very
different topics may still be connected through several intermediate papers.

Your response may contain reasoning, but its final line must be exactly one answer block
containing the selected uppercase option letter, for example:
\answer{C}

Do not write more than one answer block or put anything after the final answer line.

Figure 12. System prompt for the multiple-choice reachability task.

References

  1. [1]Adams, L., Busch, F., Han, T., Excoffier, J.-B., Ortala, M., Löser, A., Aerts, H. J., Kather, J. N., Truhn, D., & Bresssem, K. (2025). Longhealth: A question answering benchmark with long clinical doc-. *Journal of Healthcare Informatics Research*, 9(3), 280–296.
  2. [2]Aldous, D. J. (2006). Exchangeability and related topics. In École d’Été de Probabilités de Saint-Flour XIII—1983 (pp. 1–198).
  3. [3]Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. L-eval: Instituting standardized evaluation for long context language models. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 14388–14411, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2307.11088
  4. [4]Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., & Li, J. (2024). LongBench: A bilingual, multi-task benchmark for long context understanding. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3119–3137.
  5. [5]Béla Bollobás, Christian Borgs, Jennifer T Chayes, and Oliver Riordan. 2003. Directed scale-free graphs. In *SODA*, volume 3, pages 132–139. Baltimore, MD, United States.DOI
  6. [6]Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., & others, . 12 . (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901.
  7. [7]Chen, J., Wei, Z., Ren, Z., Li, Z., & Zhang, J. (n.d.). LR²Bench: Evaluating long-chain reflective reasoning capabilities of large language models via constraint satisfaction problems. Findings of the Association for Computational Linguistics: ACL 2025, 6006–6032.
  8. [8]Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. 2025b. Benchmarking large language models under data contamination: A survey from static to dynamic evaluation. In *Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 10080–10098, Suzhou, China. Association for Computational Linguistics.DOI
  9. [9]Simin Chen, Pranav Pusarla, and Baishakhi Ray. 2025c. Dycodeeval: Dynamic benchmarking of reasoning capabilities in code language models under data contamination. In *Forty-second International Conference on Machine Learning.arxiv.org/abs/2503.04149
  10. [10]Yuxing Cheng, Yi Chang, and Yuan Wu. 2025. A survey on data contamination for large language models. *Preprint*, arXiv:2502.14425.arXiv
  11. [11]Hyeong Kyu Choi, Maxim Khanov, Hongxin Wei, and Yixuan Li. 2025. How contaminated is your benchmark? measuring dataset leakage in large language models with kernel divergence. In *Forty-second International Conference on Machine Learning*.arxiv.org/abs/2502.00678
  12. [12]Juhwan Choi, Yeonghwa Kim, Seunguk Yu, JungMin Yun, and Youngbin Kim. 2024. UniGen: Universal domain generalization for sentiment classification via zero-shot dataset generation. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 1–14, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2405.01022
  13. [13]DeepSeek-AI. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence.arxiv.org/abs/2606.19348
  14. [14]Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiaheng Xu, Fan Yang, and Mao Yang. 2024. Longrope: extending llm context window beyond 2 million tokens. In *Proceedings of the 41st International Conference on Machine Learning*, pages 11091–11104.arxiv.org/abs/2402.13753
  15. [15]Mingxuan Du, Benfeng Xu, Chiwei Zhu, Licheng Zhang, Xiaorui Wang, and Zhengdong Mao. 2026. Deepresearch bench: A comprehensive benchmark for deep research agents. In *The Fourteenth International Conference on Learning Representations*.arxiv.org/abs/2506.11763
  16. [16]Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., MetropolitanSky, D., Ness, R. O., & Larson, J. (2025). From local to global: A graph rag approach to query-focused summarization. In arXiv preprint. arXiv:2404.16130
  17. [17]Lutfi Eren Erdogan, Hiroki Furuta, Sehoon Kim, Nicholas Lee, Suhong Moon, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. 2025. Plan-and-act: Improving planning of agents for long-horizon tasks. In *Forty-second International Conference on Machine Learning*.arxiv.org/abs/2503.09572
  18. [19]Yunfan Gao, Yun Xiong, Wenlong Wu, Bohan Li, Yijie Zhong, and Haofen Wang. 2026. U-niah: Unified rag and llm evaluation for long context needle-in-a-haystack. *ACM Transactions on Information Systems*, 44(3):1–30.arxiv.org/abs/2503.00353
  19. [20]Google. (n.d.). Gemini 3.1 Pro Preview. <a href="https://doi.org/https://deepmind.google/models/model-cards/gemini-3-1-pro/" rel="noopener noreferrer" target="_blank">https://doi.org/https://deepmind.google/models/model-cards/gemini-3-1-pro/</a>
  20. [21]Google. (2026). Gemini 3.6 flash. <a href="https://doi.org/https://deepmind.google/models/model-cards/gemini-3-6-flash/" rel="noopener noreferrer" target="_blank">https://doi.org/https://deepmind.google/models/model-cards/gemini-3-6-flash/</a>
  21. [22]Google. (2026). Gemini 3.8 flash. <a href="https://doi.org/https://deepmind.google/models/model-cards/gemini-3-8-flash/" rel="noopener noreferrer" target="_blank">https://doi.org/https://deepmind.google/models/model-cards/gemini-3-8-flash/</a>
  22. [23]Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., & al., . et . (2024). The llama 3 herd of models. In arXiv preprint. arXiv:2407.21783
  23. [24]Batu Guan, Xiao Wu, Yuanyuan Yuan, and Shaohua Li. 2025. Is your benchmark still useful? dynamic benchmarking for code language models. In NeurIPS 2025 Fourth Workshop on Deep Learning for Code.arxiv.org/abs/2503.06643
  24. [25]Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling.arxiv.org/abs/2404.06654
  25. [27]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2403.07974
  26. [28]Kamradt, G. (2023). Needle In A Haystack - pressure testing LLMs. GitHub.
  27. [29]Kinney, R., Anastasiades, C., Authur, R., Beltagy, I., Bragg, J., Buraczynski, A., Cachola, I., Candra, S., Chandrasekhar, Y., Cohan, A., & al., . et . (2023). The semantic scholar open data platform. In arXiv preprint. arXiv:2301.10140
  28. [30]Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022. An empirical survey on long document summarization: Datasets, models, and metrics. ACM computing surveys, 55(8):1–35.DOI
  29. [31]Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86.DOI
  30. [32]Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Igorevich Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.arxiv.org/abs/2406.10149
  31. [33]Kurtic, E., Moeini, A., & Alistarh, D. (2024). Mathador-LM: A dynamic benchmark for mathematical reasoning on large language models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 17020–17027.
  32. [34]Philippe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context LLMs and RAG systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9885–9903, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2407.01370
  33. [35]Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, and Kang Liu. 2024. S3Eval: A synthetic, scalable, systematic evaluation suite for large language model. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1259–1286, Mexico City, Mexico. Association for Computational Linguistics.DOI
  34. [36]Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024a. LooGLE: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16304–16333, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2311.04939
  35. [37]Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. 2024b. Infibench: Evaluating the question-answering capabilities of code large language models. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.arxiv.org/abs/2404.07940
  36. [39]Li, T., Zhang, G., Do, Q. D., Yue, X., & Chen, W. (2025). Long-context LLMs struggle with long in-context learning. Transactions on Machine Learning Research.
  37. [40]Yanyang Li, Wong Tin Long, Cheung To Hung, Jian-qiao Zhao, Duo Zheng, Liu Ka Wai, Michael R. Lyu, and Weizhi Wang. 2025. C2LEVA: Toward comprehensive and contamination-free language model evaluation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2283–2306, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2412.04947
  38. [41]Yucheng Li, Frank Guerin, and Chenghua Lin. 2024c. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18600–18607.arxiv.org/abs/2312.12343
  39. [42]Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356–2362.DOI
  40. [43]Liu, X., Dong, P., Hu, X., & Chu, X. (n.d.). LongGenBench: Long-context generation benchmark. Findings of the Association for Computational Linguistics: EMNLP 2024, 865–883.
  41. [44]Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. 2024b. Scaling laws of roPE-based extrapolation. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2310.05209
  42. [45]Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schuetze, H. (2025). Nolima: Long-context evaluation beyond literal matching. Forty-second International Conference on Machine Learning.
  43. [46]Hyeonseok Moon and Heuiseok Lim. 2025. Needlechain: Measuring intact long-context reasoning capability of large language models.DOI
  44. [47]OpenAI. (2026). Gpt-5.6 system card. https://deploymentsafety.openai.com/gpt-5-6deploymentsafety.openai.com/gpt-5-6
  45. [48]Anwesan Pal, Karen Hovsepian, Tinghao Guo, Mengnan Zhao, Somendra Tripathi, Nikos Kanakaris, George Mihaila, and Sumit Nigam. 2025. Tagging-augmented generation: Assisting language models in finding intricate knowledge in long contexts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2209–2220, Suzhou (China). Association for Computational Linguistics.arxiv.org/abs/2510.22956
  46. [49]Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang, Xuanming Zhang, Maximillian Chen, and Zhou Yu. 2024. VarBench: Robust language model benchmarking through dynamic variable perturbation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16131–16161, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2406.17681
  47. [50]Yifu Qiu, Varun R. Embar, Yizhe Zhang, Navdeep Jaitly, Shay B. Cohen, and Benjamin Han. 2025. Eliciting in-context retrieval and reasoning for long-context large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3176–3192, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2501.08248
  48. [51]Team, Q. (2026). Qwen3.5: Towards native multi-modal agents.
  49. [52]Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations.arxiv.org/abs/1911.05507
  50. [53]Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., & Aguirre, E. (2023). NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. Findings of the Association for Computational Linguistics: EMNLP 2023, 10776–10787.
  51. [54]Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics.arxiv.org/abs/2305.14196
  52. [55]Ali Shirali, Rediet Abebe, and Moritz Hardt. 2023. A theory of dynamic benchmarks. In The Eleventh International Conference on Learning Representations.arxiv.org/abs/2210.03165
  53. [56]Mingyang Song, Mao Zheng, and Xuan Luo. 2025. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3753–3763, Abu Dhabi, UAE. Association for Computational Linguistics.arxiv.org/abs/2403.11802
  54. [57]Team, G., El Abd, S., Aggarwal, V., Alvarez, R., Andreev, A., Bacher, O., Ballantyne, I., Brick, C., Cărbune, V., Casbon, M., & al., . et . (2026). Gemma 4 technical report. arXiv:2607.02770
  55. [58]Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sunil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. 2024. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640.arXiv
  56. [59]Ming-Yang Wang, Guang Yu, and Da-Ren Yu. 2010. The scale-free model for citation network. In 2010 IEEE International Conference on Intelligent Computing and Intelligent Systems, volume 1, pages 773–776. IEEE.DOI
  57. [60]Siyuan Wang, Zhuohan Long, Zhihao Fan, Xuanjing Huang, and Zhongyu Wei. 2025. Benchmark self-evolving: A multi-agent framework for dynamic LLM evaluation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3310–3328, Abu Dhabi, UAE. Association for Computational Linguistics.arxiv.org/abs/2402.11443
  58. [62]Yuxiang Wang, Xinnan Dai, Wenqi Fan, and Yao Ma. 2026b. Exploring graph learning tasks with pure LLMs: A comprehensive benchmark and investigation. In Findings of the Association for Computational Linguistics: ACL 2026, pages 7909–7942, San Diego, California, United States. Association for Computational Linguistics.arxiv.org/abs/2502.18771
  59. [63]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. Livebench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2406.19314
  60. [64]Edwin B Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212.DOI
  61. [65]Dustin Wright, Zain Muhammad Mujahid, Lu Wang, Isabelle Augenstein, and David Jurgens. 2025. Unstructured evidence attribution for long context query focused summarization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1839–1867, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2502.14409
  62. [66]Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, and William Yang Wang. 2025. AntiLeakBench: Preventing data contamination by automatically constructing benchmarks with updated real-world knowledge. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18403–18419, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2412.13670
  63. [67]Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824.arxiv.org/abs/2404.18824
  64. [68]Jiahao Ying, Yixin Cao, Yushi Bai, Qianru Sun, Bo Wang, Wei Tang, Zhaojun Ding, Yizhe Yang, Xuanjing Huang, and Shuicheng YAN. 2024. Automating dataset updates towards reliable and timely evaluation of large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.arxiv.org/abs/2402.11894
  65. [69]Yifei Yu, Qian-Wen Zhang, Lingfeng Qiao, Di Yin, Fang Li, Jie Wang, Chen Zeng Xi, Suncong Zheng, Xiaolong Liang, and Xing Sun. 2025. Sequential-NIAH: A needle-in-a-haystack benchmark for extracting sequential needles from long contexts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29450–29468, Suzhou, China. Association for Computational Linguistics.arxiv.org/abs/2504.04713
  66. [72]Haozhen Zhang, Tao Feng, Pengrui Han, and Jiaxuan You. 2025. Academiceval: Live long-context LLM benchmark. Transactions on Machine Learning Research.arxiv.org/abs/2510.17725
  67. [73]Zhehao Zhang, Jiaao Chen, and Diyi Yang. 2024. Darg: Dynamic evaluation of large language models via adaptive reasoning graph. Advances in Neural Information Processing Systems, 37:135904–135942.arxiv.org/abs/2406.17271
  68. [74]Xiaoshi Zhong and Huizhi Liang. 2024. On the scale-free property of citation networks: an empirical study. In Companion Proceedings of the ACM Web Conference 2024, pages 541–544.DOI
  69. [75]Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024a. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2309.17167
  70. [76]Qin Zhu, Qinyuan Cheng, Runyu Peng, Xiaonan Li, Ru Peng, Tengxiao Liu, Xue Peng Qiu, and Xuanjing Huang. 2024b. Inference-time decontamination: Reusing leaked benchmarks for large language model evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9113–9129, Miami, Florida, USA. Association for Computational Linguistics.arxiv.org/abs/2406.13990

Paper details

vlnc:W_tbj3peec
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

generative AI tools

View statement
Looking for related papers…
Contents