WorksComputer Science
Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game
Abstract
Research ideation is iterative: researchers refine their ideas step by step based on feedback. Yet existing evaluations obscure this search process, making it difficult to understand how LLMs explore the space of research ideas and what limits their progress. We introduce Re-Search, which adapts the format of Twenty Questions to research ideation: given only the research problem of a published paper, a model must rediscover the paper's core idea by asking yes-or-no questions. Built from 200 ICML 2026 papers, Re-Search tracks how a model's idea evolves as it asks questions and receives binary oracle feedback, isolating ideation from execution. Across six recent LLMs, nearly all progress happens within the first 30 of 100 questions, which largely decide how the game ends; after that, most models plateau. Through controlled interventions on the candidate questions and the hypothesis space, we find that the bottleneck is generation, not judgment: models recognize a good idea or a good question when one is offered, but often cannot produce it on their own. We further analyze the ideation traces and find that models fail by either over-exploiting one direction or over-exploring too many. These findings point to concrete design principles for autoresearch systems: since models can judge candidates they cannot produce, enriching the search with externally sourced ideas and questions may be more effective than improving any single model's generation ability.
Introduction
Research ideation is not a one-shot proposal but an iterative search interleaved with every stage of a project: researchers form hypotheses, test them, and refine them based on feedback. This process can be viewed as adaptive information seeking over a space of plausible yet unverified ideas (Klahr and Dunbar, 1988). However, existing benchmarks provide limited visibility into this iterative search. Ideation benchmarks evaluate one-shot proposals using LLM judges ([7] (Guo et al. 2025)), overlooking how ideas evolve through feedback. End-to-end benchmarks involve iterative search but entangle it with coding, debugging, and experimentation ([6] (Wijk et al. 2025; Wu et al. 2026; Rank et al. 2026; Lupidi et al. 2026; Garikiparthi et al. 2026)), making it difficult to distinguish failures of ideation from failures of execution.
In this work, we aim to isolate the information-seeking process from the rest of autoresearch. To this end, we introduce RE-SEARCH, a two-player questioning game that makes every step of this process observable and measurable (Figure 1). The setting is a scientific variant of Twenty Questions ((Mosher and Hornsby, 1966; Siegler, 1977)), a classic paradigm in cognitive science for studying human reasoning and problem solving ([12]). Given only the research problem of a published paper, an inquisitor model must rediscover the paper’s idea by asking up to 100 yes-or-no questions, while an answerer model reads the full paper and replies to the questions. Every 10 questions, the inquisitor model writes down its current best guess of the idea, which a panel of judges scores against the paper’s idea (§ 3, see Figure 2 as an example). Success on RE-SEARCH requires three capabilities. The model must (1) ask informative questions, which means both proposing good candidate questions and judging which of them to ask, (2) maintain an adequate hypothesis space that includes the correct idea, and (3) hill-climb along promising directions while abandoning dead ends.

Figure 1. DeepSeek-V4-Flash and GPT-5.4 play RE-SEARCH on the same paper. Height reflects recovery of the paper’s idea. Green marks matching terms; red marks mismatches. While DeepSeek-V4-Flash ends up circling in a basin, GPT-5.4 recovers from a dip and reaches the summit.

Figure 2. Part of the GPT-5.4 game in Figure 1: given the research problem , the inquisitor asks yes-or-no questions, the answerer replies based on the full paper, and the judges score the idea card.
We evaluate six recent LLMs on 200 ICML 2026 papers. GPT-5.4 leads by a wide margin, followed by Gemma-4-31B and Claude Sonnet 4.5; the remaining three models score about two thirds as high as GPT-5.4. Yet the models share a common pattern: most progress happens within the first 30 questions, after which gains slow sharply for all but GPT-5.4, and a model’s score at question 30 largely predicts its final score. So why do models stall with 70 questions left, and which capabilities are missing?
We first rule out a simple knowledge deficit: on papers where both a strong and a weak model can list the core idea beforehand, the strong model still finishes far ahead. The first clue comes from question quality. When a weak model can choose from candidate questions proposed by several models, it scores far above its own solo performance. The same pattern holds for the hypothesis space. We give models a survey of candidate directions that includes the paper’s core idea; without it, many models never reach the idea, but with it, they zero in on it within a few questions—and the weaker the model, the larger the gain.
In both cases, models recognize a good idea or a good question when one is offered, but often cannot generate it on their own. The ideation traces show how hill-climbing fails: weaker models either over-exploit one direction or over-explore too many, stuck into an unproductive pattern early.
Our contributions are: (1) RE-SEARCH, a two-player questioning game that isolates research ideation from execution and makes every step of ideation observable and measurable; (2) the finding that ideation is decided early: most progress happens within the first 30 questions, and weaker models lock into either over-exploiting one direction or over-exploring too many; and (3) a diagnosis that the bottleneck is generation, not judgment, pointing to external candidates as a direct lever for improving autoresearch systems.
Related Work
AI scientists and their evaluation. LLM-based systems now attempt to automate research from idea to paper ([20] (Lu et al. 2024; Lu et al. 2026; Schmidgall et al. 2025)). Most are evaluated end to end: benchmarks run an agent on a research project for hours and score the outcome ([6, 18] (Wijk et al. 2025; Zhang et al. 2025; Wu et al. 2026; Rank et al. 2026; Lupidi et al. 2026)), or grade intermediate steps against expert rubrics ([5] (Starace et al. 2025; Kon et al. 2026; Jansen et al. 2024)), and open-world environments let agents pursue long projects on their own ([11]). Some of them ask agents to rediscover the findings of a hidden paper, from raw data (Xu et al., 2026) or by running experiments (Wang et al., 2026). In all of these, the search for the idea is mixed with coding, debugging, and experiments, and can only be reconstructed from logs afterwards (Li et al., 2026a). RE-SEARCH isolates this search, so that each question and its effect on the idea can be observed directly.
Research ideation. A separate line of work asks whether LLMs can propose research ideas. Si et al. (2025) find in a large human study that LLM ideas are judged more novel but slightly less feasible than those of experts, and benchmarks such as IdeaBench (Guo et al., 2025) and the ideation stage of MLR-Bench ([7]) rate generated ideas with LLM judges. Other systems refine ideas over several rounds, through literature-grounded review ([2]), planned retrieval of external knowledge (Hu et al., 2025), or multi-agent and quality-diversity search ([8] (Gumma et al. 2026)). These evaluations either treat ideation as a single step or guide the search with LLM feedback, and none of them has a correct answer to compare against. Like us, Yang et al. (2025) and Liu et al. (2026) use recent papers as ground truth, testing whether LLMs can rediscover their hypotheses by retrieving inspirations and composing a hypothesis from them. The model receives its evidence up front, whereas in RE-SEARCH it must find the evidence itself by asking.
Concurrently, Wen et al. (2026) introduce Priced Guidance to quantify the cost of steering a model toward a paper’s core idea. The evaluated model proposes multiple-choice questions with reported answer probabilities. A guide with access to the target then selects an answer, priced at bits. Because this cost is computed from verbalized probabilities, the resulting score reflects probability-reporting behavior as well as candidate generation. Their analysis also highlights the cost of acquiring useful candidate concepts, consistent with our finding that weaker models can use supplied candidates they struggle to propose independently. In our work, the focus is on how ideation unfolds through interaction and why progress stalls. Following the cognitive view of discovery as hypothesis search guided by experimental feedback (Langley et al., 1987), RE-SEARCH tracks idea recovery through model-generated questions and paper-grounded feedback. Controlled interventions on candidate questions and hypotheses then examine how candidate availability and selection affect recovery, without requiring models to report answer probabilities.
Information seeking and question asking. Twenty Questions and related games are standard tools for studying how people ask questions: children move from testing single hypotheses to asking questions that narrow the space (Mosher and Hornsby, 1966; Ruggeri and Lombrozo, 2015; Ruggeri et al., 2016), and people adapt their questions to the structure of the hypothesis space ([13, 12] (Nelson et al. 2014)). Recent work uses the same games to evaluate and improve LLMs ([3] (Zhang et al. 2024b; Mazzaccara et al. 2024)), often by choosing questions with high expected information gain or value of information ([10, 16] (Hu et al. 2024; Grand et al. 2026)); MediQ (Li et al., 2024) brings the same setting to clinical diagnosis. In these tasks, the target is an object, a diagnosis, or a board configuration from a small, known set of candidates. In RE-SEARCH, the target is a research idea, and the model must generate the candidates itself. More broadly, LLMs in multi-turn and long-horizon settings tend to commit early and rarely recover from early mistakes (Laban et al., 2026; Sinha et al., 2026), although deep-research agents can also switch direction mid-trajectory (Li et al., 2026b); we find that in research ideation, early questions largely decide the outcome (§ 4.2).
The Re-Search Game
We define the game in § 3.1, describe how we build it from published papers in § 3.2, and explain how we score what the model recovers in § 3.3.
Task Description
In Twenty Questions, one player holds a hidden target (e.g., a specific person), and the other must identify it by iteratively asking yes-or-no questions. RE-SEARCH adapts this format to research ideation. Each game is played on a pair built from a published paper in ICML 2026, where is the research problem and is the solution that the paper proposes for it.
Three roles are involved: (1) The inquisitor sees only . At each turn , it asks a yes-or-no question based on the problem and the history of earlier questions and answers, , where and is empty. (2) The answerer reads the full paper and replies to each question on its own, . At fixed points , the inquisitor also writes an idea card , a short, structured statement of its current best guess of . (3) After the game, a panel of three LLM judges scores all saved idea cards against .
Information revealed by questioning. Let be the hidden solution, the research problem, and the question–answer history. Suppose answers belong to a finite alphabet , and each question is conditionally independent of given and the preceding history. Then
Binary feedback yields at most bits; our three-answer alphabet yields at most $T\log_2 3 Algorithm 1 The RE-SEARCH game for one paper.
Require: research problem , paper , hidden solution , question budget
▷ idea card before any question
for do
▷ stateless
if then
end if
end for
score every saved against ▷ not fed back
bits. This bounds information gained through questioning, conditional on the problem, rather than information supplied before the game. Additional questions need not supply additional information, and this bound does not guarantee a particular recovery score. See §E for the proof.
Data Construction
We build RE-SEARCH from 200 papers accepted at ICML 2026, one of the most recent top-tier conferences. The papers are selected based on three criteria: (1) the main contribution is a method, rather than a benchmark, a dataset, or an analysis, so that each paper has a concrete solution to recover; (2) its review score is as high as possible, indicating a reliable solution; and (3) the paper became public as late as possible. All selected papers first appeared, on arXiv or the conference, after 31 August 2025, later than every reported knowledge cutoff (§A), minimizing the risk of knowledge contamination ([9] (Roberts et al. 2024).
For each paper, we take its full text and use an LLM (GLM-5.2) to extract the pair in a structured form, which has been shown to be reliable in prior work ([17] (Liu et al. 2026)). The research problem has four fields (the problem domain, the research goal, the main challenge, and the gap in prior work), and the solution has four fields (the main contribution, the approach, the technical components, and the novelty). Human annotators (three Computer Science PhD students) checked the extracted pairs for 100 papers. By majority vote, 93% are correct, and the errors mostly omit a component that distinguishes the paper’s method.
Evaluation
We score each idea card against the hidden solution with a panel of three LLM judges (Gemma-4-31B, Qwen3.6, and GPT-OSS-120B). Each judge sees the research problem , hidden solution , and idea card , and rates on a 1–10 scale how much of the card recovers. The recovery score is the mean of the three ratings, . The judges agree closely: over all the cards in our experiment, their pairwise Spearman correlations range from 0.85 to 0.88. Since three of the judges are also inquisitors, we check for self-preference: leaving out the judge from an inquisitor’s own model family changes the gain from naming the idea by at most 0.08 points (§B).
Human evaluation. We use human annotators to check the reliability of the LLM judges, by comparing 498 pairs of idea cards; for each pair, they choose the card closer to the hidden solution, or mark a tie. Their labels agree well, with a Fleiss’ of 0.83. In 91.2% of pairs, the LLM judges agree with the annotator majority on which card is closer, or that the two are tied. Most of the remaining pairs are ones that the annotators mark as a tie but the judges do not; when both sides prefer one card, they choose the same card in 97.8% of pairs.
Ideation Dynamics in Re-Search
§ 4 to § 6 build on each other. In this section, we describe what happens in a game; § 5 asks why most models plateau early, with controlled interventions on the candidate questions (informative questions) and the ideas it considers (hypothesis space); and § 6 examines how models follow promising directions and abandon dead ends (hill-climbing), through the shape of the trace. We begin with the model choices, then show how ideation develops over a game (§ 4.1) and that the early questions largely decide its outcome (§ 4.2).
Model selection. We evaluate both open-weight and proprietary LLMs of different strengths: Gemma-4-31B, GPT-OSS-120B, DeepSeek-V4-Flash-Preview, Qwen3.6-35B-A3B, Claude Sonnet 4.5, and GPT-5.4. We do not include newer models because of the risk of knowledge contamination. In all games, Gemma-4-31B serves as the answerer, each inquisitor asks up to questions, and we play three games per paper. Among three candidate answerers (GPT-OSS-120B, Qwen3.6-35B-A3B, and Gemma-4-31B), Gemma agrees most often with human labels on 795 annotated answers (94.0%, against 86.2% for GPT-OSS-120B and 78.4% for Qwen3.6-35B-A3B; see §D for details), and only 1.1% flip YES and NO, far below the 10% answer noise assumed in Grand et al. (2026). Therefore we use Gemma-4-31B as the answerer model.
Most Models Stall Early
Figure 3 shows how recovery develops over a game. Before the first question, every model’s idea card scores very low, confirming that the research problem alone does not reveal the solution. The game then unfolds in three stages.
Figure 3. Recovery score of the games over 100 rounds, averaged over 200 papers and three games per paper. The patterns of models are different: three models gain little after the first 30 questions, Gemma and Sonnet slowly climb, while GPT-5.4 has consistent improvement. (§ 4.1)
Questions 1–10 bring the largest gain for every model, with scores rising by +1.3 to +3.2. Questions 11–30 add much less, between +0.2 and +0.8. The most interesting part happens in Questions 31–100, where the models are separated into different patterns. Three models stall, gaining almost nothing over the remaining 70 questions; two keep improving slowly; and only the strongest model keeps climbing steadily.
Early Questions Decide the Outcome
A natural implication of § 4.1 is that, if most models stall after about 30 questions, the early questions should largely decide how a game ends. This echoes findings in long-horizon LLM generation, where early mistakes compound and are rarely re-paired ((Zhang et al., 2024a; Laban et al., 2026; Sinha et al., 2026)). We test this with a prefix-fork experiment (Bigelow et al., [4]).
Each model plays three games per paper. First, for each paper, we pick the games with the highest and the lowest final score, and truncate both at round (). Then, the same model continues each prefix to question 100 three more times, yielding three continuations from the best start and three from the worst. If the opening matters, the best-start continuations should stay ahead.
As shown in Table 1, the score at question predicts the score at question 100 well. Across games, the correlation is – at and rises to – at .
| Model | Gap at | Gap at 100 | Ahead (%) | |
| Gemma-4-31B | 10 | 1.59 | 1.16 | 63 |
| Gemma-4-31B | 20 | 1.53 | 1.64 | 70 |
| Gemma-4-31B | 30 | 2.28 | 2.30 | 78 |
| Qwen3.6-35B-A3B | 10 | 1.03 | 1.00 | 64 |
| Qwen3.6-35B-A3B | 20 | 1.21 | 1.41 | 69 |
| Qwen3.6-35B-A3B | 30 | 1.86 | 1.70 | 73 |
| GPT-OSS-120B | 10 | 1.70 | 1.85 | 68 |
| GPT-OSS-120B | 20 | 2.32 | 2.45 | 78 |
| GPT-OSS-120B | 30 | 2.56 | 2.54 | 77 |
| DeepSeek-V4-Flash | 10 | 1.48 | 1.37 | 66 |
| DeepSeek-V4-Flash | 20 | 2.10 | 1.72 | 71 |
| DeepSeek-V4-Flash | 30 | 2.14 | 2.20 | 79 |
Table 1. Prefix-fork experiment: we truncate the best and worst game at round and continue each three times. Gap at means score difference between the best and worst prefix at round . Ahead: share of papers where the best prefix still leads at question 100. The gap largely persists across models. (§ 4.2)
Diagnosing the Bottleneck
As § 4 shows, most models plateau early, while GPT-5.4 keeps climbing. What explains this gap? We first rule out knowledge as the main cause (§ 5.1), then show that supplying external candidates, either as questions from other models (§ 5.2) or as a survey of the hypothesis space (§ 5.3), closes much of the gap.
Knowledge Explains Little of the Gap
A strong model may simply have seen more scientific text, know more candidate ideas before the game, and turn this knowledge into an advantage in the game. To test this, we use a closed-book probe that isolates the effect of knowledge. We collect a subset of 100 papers, and focus on two models, a strong one (GPT-5.4) and a weak one (GPT-OSS-120B). Each model is provided with the research problem and asked to list ten candidate solutions, three times, for 30 candidates in total; two LLM annotators then judge whether the list contains the paper’s core idea.
Knowledge does not explain the gap between models. On this 100 papers subset, GPT-5.4 scores 2.54 points higher than GPT-OSS-120B; on the 24 papers where both models list the core idea, it still scores 2.28 higher, and knowledge closes barely 10% of the gap. Therefore, knowing the idea is not enough.
Weak Models Can Pick Informative Questions
(§ 5.1) suggests that the gap lies in how models ask. Asking an informative question involves two steps: proposing candidate questions and choosing which one to ask. To separate the two, we let a weak model choose each question from candidates proposed by several models, rather than asking only its own. If it improves, the bottleneck lies in proposing rather than judging. We test this using GPT-5.4, Qwen3.6-35B-A3B, DeepSeek-V4-Flash, and GPT-OSS-120B. Each round, all four models independently propose a question from the same history, and one model picks among them.
After 30 rounds with the shared pool, all four models score between 5.19 and 5.34, regardless of which model picks the questions. The weaker models gain over 1 points compared to their solo baseline. Surprisingly, the gain of weak models does not come from blindly copying GPT-5.4’s questions.
questions. For example, GPT-OSS-120B picks its own proposal in 30% of rounds and GPT-5.4’s in 34% (Figure 5).
Figure 5. Share of questions (%) each inquisitor (row) chooses from each proposer (column) in the four-model pool on 200 papers. All models favor questions from GPT-5.4 and themselves. While the proportions differ across inquisitors, the final scores are close. All four score between 5.19 and 5.34, with no significant difference. (§ 5.2)
Which of the two steps drives the gain? We vary each while holding the other fixed. When GPT-OSS-120B proposes all four candidates itself (best-of-4 sampling, akin to self-consistency (Wang et al., 2023)), its score rises only from 4.16 to 4.57, well below the 5.20 where candidates are from four models, so the candidates matter. When we keep the four-model pool but pick at random, the score drops to 4.43, not significantly above playing alone, so selection matters too (Table 2). Both steps are necessary, but they are not equally scarce. When we swap the selector across models, the final scores barely change (Table 2); the gap between models therefore comes mainly from the candidates they propose. A weak model’s selection is good enough; what it lacks are informative questions worth selecting.
| Inquisitor | Score | Gain over alone |
| GPT-OSS-120B | 5.20 | +1.04 [+0.74, +1.34] |
| Qwen3.6-35B-A3B | 5.34 | +1.09 [+0.87, +1.39] |
| DeepSeek-V4-Flash | 5.24 | +1.07 [+0.79, +1.36] |
| GPT-5.4 | 5.19 | −0.59 [−0.90, −0.28] |
| Random pick | 4.43 | — |
Table 2. Score after 30 questions for each inquisitor on the same four-model pool on 200 papers. The last column gives the difference from each model’s solo baseline, with 95% confidence intervals. (§5.2)
Figure 6. Recovery curves under the four context conditions of Figure 4 (200 ICML 2026 papers, three games per paper). Naming the idea () lifts the curves of most models clearly above and ; for GPT-5.4, all four conditions end close together. (§5.3)
A Survey Enriches the Hypothesis Space
In real research, ideas do not start from a purely blank page, as researchers usually begin with a landscape in mind: a rough sense of which approaches could solve the problem. We give the inquisitor a similar starting point with a survey (Figure 4), an overview of the hypothesis space for the research problem. Each survey has five to seven dimensions along which solutions to the problem can differ, and lists four to six candidate approaches in each dimension, named at the level of known families of techniques. The survey is placed right after the problem when the model asks questions, with no instruction on how to use it, and is not shown when the model writes its idea card (§ C). More importantly, the inquisitor is not told whether the solution is in the survey.

Figure 4. An example survey under . Left: the survey given before the game, which names the core idea as one option, with no extra hint. Right: how the four context conditions differ. replaces only that phrase with a plausible decoy, also replaces the other lines that point to the paper, and gives no survey. Because and differ in a single phrase, any difference between them comes from whether the core idea is in the survey. (§ 5.3)
Figure 6 shows the four context conditions. Naming the core idea () lifts the curves of most models clearly above and . When the name is replaced by a decoy (), the survey still helps through the other listed components, but the curves slightly fall back towards , removing those components as well () brings them closer still.
A survey enlarges the model’s hypothesis space from outside: under , the correct idea is already among the candidates. Naming the core idea in the survey makes every model reach the core idea during the game more often, and helps weaker models most: the final score improves by up to 1.25 points, with the largest gains for the models that score lowest without it (, Figure 7). This is not because models simply go through the survey item by item. A survey lists 20 to 42 candidates, yet under models ask about the named idea within a median of 3 to 6 questions, and far more often than they ask about a decoy in the same line under (67–89% against 29–52% of games; § C). Even under , models ask about that line more often than about nearby lines, so they can find the right family of methods; what the name adds is the specific idea. Once the correct idea is in the hypothesis space, weak models recognize it; what they lack is the ability to put it there themselves.
Figure 7. Top: score after 100 questions under and for each inquisitor. Bottom: gain in that score from naming the idea (, with 95% confidence intervals), against the score under ; the dashed line is a least-squares fit (Pearson across the six inquisitors). Naming the idea helps most the models that score lowest without it. (§5.3)
Turning Ideation Traces into Trees
§ 5 shows why models differ, but not how ideation unfolds question by question. A game is recorded as a flat sequence of questions, but the questions are not independent: most of them follow up on an earlier question, while the rest open new directions. Research proceeds in the same way: a researcher opens a few lines of inquiry, runs follow-up experiments on the promising ones, and drops those that stop paying off. We therefore turn each game into a tree that makes this structure explicit. Each top-level branch is a line of inquiry, and its depth shows how far the model follows it; together they reflect the model’s hypothesis space and its hill-climbing.
As using LLMs to turn reasoning traces into trees or graphs has become a common way to analyze how LLMs reason (Jiang et al., 2025; Zeng et al., 2025; Lee et al., 2025), we follow this line of work and use GPT-6 Astra to construct the trees. For every question, the model chooses a parent: one of the earlier questions, or the research problem if the question opens a new direction.
The Shape of the Search
We build 1,200 trees from six models on the 200 papers. Although the three weaker models end with similar scores, their trees reveal opposite failure modes (Figure 8). The number of top-level branches , i.e., questions asked directly under the research problem, cleanly separates three patterns (Figure 11).
Figure 8. Three patterns of traces in games. The grey dot is the research problem, and each color is one top-level branch. Below each tree: mean top-level branches per game on 200 papers, as a range over the models with that pattern (per-model means in Figure 11). See more examples in § G.
Figure 11. Top-level branches per game for each model on 200 papers. The brackets group the models by search pattern, as in Figure 8.
Over-exploitation. DeepSeek-V4-Flash and Qwen3.6 open only 5.2–5.5 branches per game. Once they confirm a direction, they enumerate variants within it rather than questioning the direction itself.
For example, after learning a paper uses weight statistics for quantization scales, DeepSeek-V4-Flash spends dozens of questions guessing which statistic, never asking whether the approach is something else entirely. Across all games, 23% of these two models’ questions are such enumerations.
Over-exploration. GPT-OSS-120B opens 13.7 branches, jumping to new methods regardless of feedback. It is the only model that switches branch more often after a Yes than after a No.
Balanced. The three stronger models open 7.2–8.2 branches, following confirmed directions and stepping back when a line stops paying off.
While the two failure modes are opposite, they both explain why most models plateau after 30 questions (§ 4.1). The stronger models balance the two, which lets them keep climbing.
Search Breadth and Final Recovery
§ 6.1 reveals contrasting search patterns among the weaker models: DeepSeek and Qwen tend toward over-exploitation, repeatedly refining a small number of directions, whereas GPT-OSS tends toward over-exploration, frequently opening new branches. Meanwhile, stronger models show a more balanced pattern, opening new directions while pursuing promising ones. This raises two questions: how do these patterns arise, and is intermediate search breadth associated with better final recovery?
To see how the patterns arise, we look at each answered question and the question that follows it (Figure 9, left). After a Yes (▲), the stronger models make the next question a child of the answered one far more often than after a No (39% vs. 24% for GPT-5.4), and after a No (△), about a quarter of their next questions leave the top-level branch.
Figure 9. Search breadth and final recovery across 1,200 games. Shading separates the three patterns in § 6.1. Left. After the inquisitor receives a Yes (▲) or a No (△), how often its next question is a child of the answered question (goes to a child) or moves to another top-level branch (leaves the branch). In particular, over-exploitation models rarely leave the branch, even after a No, while the over-exploration model goes to a child more often after a No than after a Yes, and leaves the branch about 30% of the time after either answer. Right. Each point groups games by their number of top-level branches. The -axis shows how much higher or lower these games score at question 100 than an average game, once differences between models and between papers are taken out. The fitted curve peaks at about eight branches (dotted line): games with a moderate number of top-level branches score the highest.
DeepSeek and Qwen also ask more children after a Yes, but after a No they leave the branch less often than any other model, so new questions keep accumulating in the same top-level branch. GPT-OSS shows the opposite pattern: it asks a child more often after a No than after a Yes, and about 30% of its next questions leave the branch regardless of the answer, which gives it the most top-level branches.
We examine this relationship across the 1,200 games by fitting a quadratic model relating final recovery scores to the number of top-level branches, with model and paper fixed effects. For the points in Figure 9 (right), we group games by their number of top-level branches and fit each game’s final score as the sum of three parts: one for its model, one for its paper, and one for its group; each point is the group’s part, shifted so that the average game is at 0. A point at +0.5 thus means that these games end half a point higher than expected from their model and paper alone. The fitted curve shows an inverted-U relationship: recovery is highest at an intermediate breadth and lower at both extremes. The estimated peak is 8.3 top-level branches (Figure 9, right). This association provides quantitative support for the connection between the search patterns described above and final recovery.
Conclusion
We introduce RE-SEARCH, a two-player questioning game where a model must rediscover the idea of a published paper by asking yes-or-no questions. By reducing ideation to a question-and-answer loop with oracle feedback, the game dissects how models search for and refine ideas.
On 200 ICML 2026 papers, most of the six models we test plateau after about 30 questions. Controlled experiments show that the bottleneck is not knowledge or judgment but generation: when we supply external candidates such as a survey of the hypothesis space or informative questions from other models, weak models recognize and use them, closing much of the gap to the strongest model.
These findings have a practical implication for autoresearch systems. Since models can judge candidates they cannot produce, augmenting the generation step is a direct way to improve performance without changing the underlying model.
Limitations
Our testbed consists of 200 machine learning papers from ICML 2026. However, the game design itself is not specific to machine learning and it applies to any field where a research problem and solution can be extracted, such as drug design or materials science, but we have not tested other domains. Recovery scores come from a panel of three LLM judges, and the search trees are annotated by an LLM. Human evaluation shows high agreement with the judges (91.2%; § 3.3), but errors remain, particularly on papers whose solution is ambiguous. or has multiple equally valid formulations. Since three of the six inquisitor models also serve as judges, we check for self-preference bias and find it negligible (at most 0.08 points; §B), though we cannot fully rule it out. Last but not least, due to knowledge contamination, we choose the six recent models, yet unable to apply our experiment setting on the newest models.
Models and Knowledge Cutoffs
The six inquisitors are GPT-OSS-120B ((OpenAI, 2025)), DeepSeek-V4-Flash ([14]), Claude Sonnet 4.5 ([1]), GPT-5.4 ((OpenAI, 2026a)), Gemma-4-31B ([19]), and Qwen3.6-35B-A3B ((Qwen Team, 2026a)). We also use GLM-5.2 ((Z.ai, 2026)) to extract the research problems and solutions (§ 3.2), GPT-5.6 Sol ((OpenAI, 2026b)) to generate the surveys and Qwen3.8-27B ((Qwen Team, 2026b)) as an LLM annotator (§C), and GPT-6 Astra ((OpenAI, 2026c)) to construct the search trees (§ 6). Table 3 lists the knowledge cutoffs of the inquisitors.
| Model | Knowledge cutoff |
| GPT-OSS-120B | June 2024 |
| DeepSeek-V4-Flash | May 2025{sup}†` |
| Claude Sonnet 4.5 | July 2025 |
| GPT-5.4 | August 2025 |
| Gemma-4-31B | January 2025 |
| Qwen3.6-35B-A3B | December 2024{sup}‡` |
Table 3. Knowledge or training-data cutoffs of the inquisitor models, as stated in each provider’s model card or documentation. {sup}†DeepSeek’s public summary of training content states that some of the data were collected no later than about May 2025 ({citep}[15]); a behavioral audit estimates an effective cutoff of July 2025 ({citep}(Jia, 2026)). {sup}‡Not reported by the provider; effective cutoff estimated by the same audit ((Jia, 2026)).
Judge Validation
Three inquisitors also serve as judges (Gemma-4-31B, Qwen3.6-35B-A3B, and GPT-OSS-120B), and Gemma-4-31B is also the answerer. To test whether a judge favors its own model family, we re-score the survey experiment (200 papers, three games per paper, score at question 100) in three ways.
Self-preference. For each judge, we compute how far its rating lies above the mean of the other two judges, separately for the inquisitor of its own family and for the other inquisitors (Table 4). Each judge rates its own family 0.17–0.33 points higher than the other judges do. This preference is small next to the general leniency of Gemma-4-31B, which rates every inquisitor about 1.2 points above the other judges.
| Judge | Own family | Others | Difference |
| Qwen3.6-35B-A3B | −0.71 | −0.88 | +0.17 |
| GPT-OSS-120B | −0.13 | −0.46 | +0.33 |
| Gemma-4-31B | +1.51 | +1.21 | +0.30 |
Table 4. Rating of each judge minus the mean rating of the other two judges, for the inquisitor of the judge’s own family and for the other inquisitors (survey experiment, all conditions, question 100). (§B)
Leaving out the same-family judge. We re-compute each inquisitor’s score without the judge
| Judge | Own family | Others | Difference |
| Qwen3.6-35B-A3B | −0.71 | −0.88 | +0.17 |
| GPT-OSS-120B | −0.13 | −0.46 | +0.33 |
| Gemma-4-31B | +1.51 | +1.21 | +0.30 |
from its own family: without Gemma-4-31B for Gemma-4-31B, without Qwen3.6-35B-A3B for Qwen3.6-35B-A3B, and without GPT-OSS-120B for GPT-OSS-120B and for GPT-5.4, which comes from the same provider. DeepSeek-V4-Flash and Claude Sonnet 4.5 keep all three judges. The gain from naming the idea () changes by at most 0.08 points, and the order of the models is unchanged (Table 5).
| Inquisitor | All judges | Same family left out |
| DeepSeek-V4-Flash | +1.25 | +1.25 |
| GPT-OSS-120B | +1.08 | +1.16 |
| Claude Sonnet 4.5 | +0.90 | +0.90 |
| Qwen3.6-35B-A3B | +0.60 | +0.61 |
| Gemma-4-31B | +0.36 | +0.38 |
| GPT-5.4 | +0.16 | +0.11 |
Table 5. Gain in score at question 100 from naming the idea (, paired by paper), with all three judges with the judge of the inquisitor’s own family left out. (§B)
Single judges. We also rank the inquisitors with each judge alone (Table 6). GPT-5.4 ranks first under every judge. Gemma-4-31B ranks second even when scored by GPT-OSS-120B alone, and the Qwen3.6-35B-A3B judge ranks its own family last.
| Inquisitor | Own | GPT-OSS | Gemma | Panel |
| GPT-5.4 | 1 | 1 | 1 | 1 |
| Gemma-4-31B | 3 | 2 | 2 | 2 |
| Claude Sonnet 4.5 | 2 | 3 | 3 | 3 |
| GPT-OSS-120B | 4 | 4 | 4 | 4 |
| Qwen3.6-35B-A3B | 6 | 5 | 6 | 5 |
| DeepSeek-V4-Flash | 5 | 6 | 5 | 6 |
Table 6. Rank of each inquisitor at question 100, scored by each judge alone and by the three-judge panel. (§B)
Survey Construction
All surveys are generated by GPT-5.6 Sol in three steps and then checked by human annotators.
Blind survey. For each paper, the model first writes a survey from the research problem alone, without seeing the solution. It is told not to identify or recall the paper and to write from general knowledge of the research area, as a survey of the area would. Each survey has five to seven dimensions along which solutions can differ, each with four to six candidate approaches; each approach names a family of techniques and two to five concrete techniques in it. The survey gives no recommendation or ranking.
and . The model then sees the solution and writes the core idea as a survey-level phrase of 3 to 12 words, that is, the name of a known family of techniques rather than the paper’s own method. It picks the candidate line where this phrase belongs. lists the phrase in this line; replaces it with a decoy, a real technique from the same family that the paper does not use, of similar length. Any other line that names the core idea is rewritten in the same way in both versions, so that never names the core idea and names it only once. and thus differ in exactly one line; both may still list other, non-core components of the paper.
. Starting from , the model replaces every remaining candidate that offers a component of the paper with an approach of similar length that the paper does not use, and keeps the decoy. Components already stated in the research problem are kept, since removing them would not change what the inquisitor knows.
Human check. Three annotators check the surveys of all 200 papers. For each paper, they see the solution and the three surveys without their labels, and decide which survey is , , and . All annotators identify correctly for every paper. For 7 papers (3.5%), one annotator swaps and ; in these papers lists few components besides the core idea, so differs from it in only a few lines.
Length and freezing. A survey is about 1,150 tokens long, and the three versions of a survey differ by fewer than 40 tokens. All surveys were frozen before any game was scored.
A model could raise its score under simply by copying the included idea from the survey into its idea card. Three checks on the survey experiment (200 papers, three games per paper, question 100) speak against this. We detect mentions of the core idea and the decoy with two LLM annotators (Gemma-4-31B and Qwen3.8-27B), which read the questions and the final card of each game, mark the questions that refer to each idea, and judge whether the card states it; we count a mention only when both annotators mark it (Cohen’s –).
The card writer does not see the survey. The survey appears only in the question prompts. Before any question, the idea card therefore scores the same under every condition (Table 7): if the card writer saw the survey, cards would already name the idea at question 0.
| Inquisitor | |||
| DeepSeek-V4-Flash | 1.05 | 1.04 | 1.10 |
| GPT-OSS-120B | 2.80 | 2.82 | 2.80 |
| Qwen3.6-35B-A3B | 1.32 | 1.22 | 1.21 |
| Claude Sonnet 4.5 | 1.98 | 2.06 | 2.04 |
| Gemma-4-31B | 2.46 | 2.42 | 2.42 |
| GPT-5.4 | 1.74 | 1.74 | 1.74 |
Table 7. Score of the idea card before any question () under each survey condition. (§C)
The included idea enters the card after it is confirmed. Under , 99–100% of the final cards that state the core idea follow a question about it that was answered YES, and at most 1% come from games that never asked about it. games that never ask about the core idea score no higher than , and the gain is concentrated in games where the idea is asked about and confirmed (Table 8). This comparison is descriptive: games that ask about the idea and get a YES may also be on easier papers.
| Inquisitor | \multicolumn{3}{c | }{} | |
| Not asked | Asked, YES | ||
| DeepSeek-V4-Flash | 4.93 | 4.73 | 6.92 |
| GPT-OSS-120B | 5.15 | 4.16 | 7.37 |
| Qwen3.6-35B-A3B | 4.94 | 3.78 | 6.27 |
| Claude Sonnet 4.5 | 5.44 | 4.29 | 7.50 |
| Gemma-4-31B | 5.59 | 3.98 | 6.80 |
| GPT-5.4 | 7.41 | 5.41 | 7.98 |
Table 8. Score at question 100 under , and under split by whether the game asked about the core idea and got a YES. (§C)
A rejected decoy rarely enters the card. The survey does steer what models ask: a decoy in the survey () is asked about in 29–52% of games, against 7–19% when it is not in the survey (). But once rejected, it reaches the final card in only 2–7% of games, while the included idea under reaches it in 47–77%.
The included idea is not found by going through the survey. With 100 questions, a model could also reach the included idea by asking about the candidates in the survey one by one. Two comparisons speak against this. First, the included idea is asked about early and selectively. In the games that ask about it, the first such question comes at a median of question 3 to 6. The core idea in and the decoy in sit in the same line of the survey, so going through the survey would reach both equally often; instead, the core idea is asked about in 67–89% of games and the decoy in 29–52%. Second, within , the line that names the core idea is asked about in 78–90% of games, against 45–67% for the lines within two positions of it, and in 63–73% against 22–42% within the first ten questions. To find which lines a question asks about, the same two annotators see the survey and the questions but not the solution (). Part of this preference already appears under , where the same line carries the decoy (11 to 29 percentage points above nearby lines), since this line is the one closest to the paper’s method. Models can therefore locate the right family of methods without the name; naming the idea adds 4 to 15 percentage points on top, and mainly makes them ask about the idea itself. All tests are paired over papers, with the three games of a paper averaged first, and Holm-corrected across the six inquisitors. The two comparisons above hold for every inquisitor (); the extra gain from naming is significant only for some inquisitors, so we report only its size.
Choice of Answerer
We compare Gemma-4-31B with two other candidate answerers, GPT-OSS-120B (medium reasoning) and Qwen3.6-35B-A3B (thinking). All three receive the same system prompt, question template, and evidence (the hidden solution card and the full paper), and are required to answer with output constrained to YES, NO, or UNKNOWN. We check both the agreement among these three models and their agreement with human annotators. Table 9 compares each answerer with the majority label of the three annotators. Gemma-4-31B agrees most often overall and in every label class. For Qwen3.6, it rarely answers UNKNOWN (6% of its answers, against 24% for Gemma-4-31B) and instead answers most of these questions with YES or NO.
| All | YES | NO | UNK. | YES/NO | |
| 795 | 354 | 279 | 162 | 633 | |
| Gemma-4-31B | 94.0 | 98.9 | 93.2 | 84.6 | 96.4 |
| GPT-OSS-120B | 86.2 | 96.0 | 92.5 | 53.7 | 94.5 |
| Qwen3.6-35B-A3B | 78.4 | 95.2 | 88.9 | 23.5 | 92.4 |
Table 9. Accuracy (%) of each answerer against the human majority label, overall and by the human label. The last column pools the YES and NO items. (§D)
Agreement between answerers. Table 10 compares the answerers with each other on all 9,774 questions. Most disagreements involve UNKNOWN; two answerers give opposite answers (YES against NO) to only 6.5–10.8% of questions.
| Pair | Agree | YES/NO flip | |
| Gemma vs. GPT-OSS | 78.4 | 0.66 | 6.5 |
| Gemma vs. Qwen | 74.3 | 0.59 | 7.0 |
| GPT-OSS vs. Qwen | 77.7 | 0.62 | 10.8 |
Table 10. Pairwise agreement (%), Cohen’s , and share of questions (%) answered YES by one answerer and NO by the other, on 9,774 questions. (§D)
Information Bound for Adaptive Questioning
Write , with empty, and let be the context given before the game (a survey, or nothing under ). Assume a finite answer alphabet and a question policy satisfying . For example, this holds when each question is generated from the visible inputs and history using fresh randomness independent of the hidden solution. All algorithms below are base two.
By the chain rule for conditional mutual information and the policy assumption,
The last inequality follows because a variable taking at most values has entropy at most . The result allows noisy answers and adaptive questions. It is an upper bound, with no positive lower bound on the information supplied by an additional question.
When a survey is supplied, its information is accounted for separately:
A survey constructed using the solution may itself reveal information about it. Excluding the survey from idea-card prompts restricts how this information reaches the card, but does not remove it from the questioning process.
Search over the Course of a Game
Figure 10 follows the 1,200 trees of § 6 question by question. We group the 100 questions of a game into ten bins and average each model over its 200 trees. As a game goes on, every model asks deeper questions and leaves the top-level branch less often. Part of the rise in depth is expected, since early questions can only sit near the root. DeepSeek-V4-Flash and Qwen3.6 grow the deepest trees and reach a depth of about seven by the end of the game. GPT-OSS-120B leaves the branch more often than any other model at every stage of the game. In contrast, how often the next question is a child of the current one changes little over time, and so does the share of Yes answers. The gap between models in Yes answers is already present in the first ten questions and stays until the end.
Figure 10. How the search changes over the 100 questions of a game (1,200 games on 200 papers). Questions are grouped into bins of ten. Depth of the node: mean depth of the questions in the bin. Goes to a child and leaves the branch: how often the next question is a child of the current question, or lies in another top-level branch, as in Figure 9 (left) but over all answers. Share of Yes answers: share of the questions answered Yes.
More Search Trees
Figure 12–Figure 15 show real question trees, for each paper, we show six trees from all the models. For over-exploitation, few top-level branches, a large share of questions in the largest branch, and great depth; for over-exploration, many top-level branches, a large share of branches dropped within three questions, little depth, and no dominant branch; for balance, five to ten top-level branches with none holding more than 60% of the questions.
Figure 12. Search trees of the six inquisitors on “Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning”.
Figure 15. Search trees of the six inquisitors on “MAD: Manifold Attracted Diffusion”.
Trees of the Paper: Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning
Trees of the Paper: A Direct Second-Order Method for Solving Two-Player Zero-Sum Games
Figure 13. Search trees of the six inquisitors on “A Direct Second-Order Method for Solving Two-Player Zero-Sum Games”.
Trees of the Paper: Structure-Preserving Learning Improves Geometry Generalization in Neural PDEs
Figure 14. Search trees of the six inquisitors on “Structure-Preserving Learning Improves Geometry Generalization in Neural PDEs”.
Trees of the Paper: MAD: Manifold Attracted Diffusion
Prompts
We list the prompts used in all games. Fields in braces are filled in at run time: {visible_problem} is the research problem (problem domain, challenge, research goal, and gap in prior work); {context} is the context condition: the survey under , , or , and empty under ; {transcript} and {transcript_evidence} are the questions and answers so far; {instruction} is the questioning strategy shown in the inquisitor prompt; {evidence_text} is the hidden solution card followed by the abstract and full text of the paper; and {hidden_solution} is the solution . The judges see the idea card without its confidence field.
Inquisitor Prompt
System Prompt
You are the questioner in a 20 Questions scientific idea recovery game.
You see only the visible research problem and the transcript so far. The transcript is the history of previous binary questions and the answerer’s Yes/No/Unknown replies. You do not see the hidden paper title, authors, venue, URL, method name, dataset name, abstract, body text, hidden solution card, or any other hidden evidence.
The hidden work is sampled from a method-primary paper set. Treat the primary contribution as method-like: a method, model, algorithm, framework, training procedure, inference procedure, optimization procedure, attack, defense, or system. Do not ask whether the primary contribution is a benchmark, dataset, metric, survey, or analysis; those primary contribution types are outside the candidate set. Datasets, benchmarks, metrics, and analyses may still appear as evaluation or supporting evidence, so ask about them only in that supporting/evaluation role.
Ask exactly one binary scientific-content question that tests a single atomic predicate and will help infer the hidden scientific idea. Do not combine contribution type with contrastive alternatives unless both sides are necessary and mutually exclusive.
Bad question:
"Does the proposed solution focus on developing a benchmark or dataset to evaluate vulnerabilities rather than a method to mitigate them?"
Better questions:
"Is the proposed method an attack method for exposing vulnerabilities?"
"Is the proposed method a mitigation or defense method?"
"Does the proposed method evaluate its effectiveness on an existing safety benchmark?"
Do not ask for names, acronyms, titles, authors, venues, source URLs, or identity clues.
Choose a question whose Yes and No answers would separate those plausible hypotheses. Prefer questions where both answers would meaningfully change the candidate space.
Use the transcript as hard evidence. If a previous answer was Unknown, change to a broader or orthogonal facet instead of asking the same hypothesis again.
Question Prompt
Visible research problem:
{visible_problem}
{context}
Transcript so far:
{transcript}
Task:
{instruction}
Return exactly this JSON object and no other text:
{"question": "Does ...?"}
Questioning Strategy (instruction)
Ask the next single binary scientific-content question that most reduces uncertainty in the current idea card.
Use the interaction strategically:
Identify the weakest missing field for a final idea card: contribution, mechanism, experiments, expected results, or novelty.
Ask one predicate that would directly support or rule out that field.
Prefer questions likely to receive Yes or No from the hidden evidence; avoid overly specific predicates likely to be Unknown.
After an Unknown, ask a broader version of a different missing field rather than a narrower follow-up.
Balance mechanism questions with data/evaluation/result questions so the final card can recover both how the method works and how it is evaluated.
Avoid low-value identity guessing or questions about exact names.
Idea Card Prompt
System Prompt
You infer a scientific idea from a visible research problem and a yes/no transcript.
Produce a concise structured idea card for the likely hidden scientific contribution. Do not use exact paper, method, dataset, author, venue, or URL names.
Treat Yes answers as positive evidence, No answers as ruled-out hypotheses, and Unknown answers as no evidence. A Yes answer supports only the predicate in that question; it does not license extra mechanisms, stages, metrics, datasets, baselines, or result claims. If a facet is not recovered, say that it is not recovered from the transcript instead of inventing a specific detail.
Card Prompt
Visible research problem:
{visible_problem}
Evidence recovered through round {round_index}:
{transcript_evidence}
Supported Yes count: {yes_count}
Evidence rule:
Use Yes answers as support.
Use No answers only to avoid contradicted claims.
Treat Unknown answers as unresolved, not as hints.
Fill answer_summary with a concise one- or two-sentence human-readable summary of the recovered idea. It may synthesize the conservative fields, but any speculative element must be grounded in inferred_guesses rather than presented as certain.
A Yes answer supports only its exact scientific facet. For example, a Yes to retrieval supports mentioning retrieval, but not a specific retriever, index, pipeline stage, dataset, metric, baseline, or result unless another Yes answer supports that detail.
For mechanism, experiments, and expected_results, write "Not recovered from the transcript" unless at least one Yes answer directly supports that field.
Fill supported_facts only with facts directly supported by Yes answers.
If there are no supported facts from Yes answers, set mechanism, experiments, and expected_results to "Not recovered from the transcript" and set confidence to 1.
Keep confidence at 1 for mostly unrecovered cards; use 2 only for broad recovered facets; use 3 or higher only when multiple specific Yes answers recover the contribution, mechanism, and evaluation.
Output length constraints:
Return compact JSON only, with no markdown, prose preamble, or code fences.
Keep answer_summary, problem, gap, core_idea and mechanism to one short sentence each.
Keep supported_facts to at most five short semicolon-separated facts.
Keep every string field under 350 characters.
Return exactly this JSON object and no other text:
{
"answer_summary": "",
"problem": "",
"gap": "",
"core_idea": "",
"mechanism": "",
"confidence":
}
Answerer Prompt
System Prompt
You are the answerer in a 20 Questions scientific idea recovery game.
Use only the provided hidden scientific evidence. Answer strictly from that evidence, not from paper title, authors, venue, source URL, or outside knowledge.
Use normal scientific judgment over the evidence. Answer Yes when the question is directly stated, strongly implied, or is a clear broader category of the hidden contribution. Answer No when the evidence supports an incompatible alternative. Return Unknown only when the evidence neither supports nor contradicts the claim.
Return Unknown when:
the hidden evidence does not support a yes/no judgment,
the question asks for a title, author, venue, URL, exact method name, exact dataset name, acronym, or paper identity,
the question contains multiple questions.
Answer Prompt
Hidden scientific evidence:
{evidence_text}
Question:
{question}
Return exactly this JSON object and no other text:
{"answer": "Yes"}
Allowed answer values: Yes, No, Unknown.
Judge Prompt
System Prompt
You are a strict scientific idea-recovery judge.
Evaluate only the semantic agreement between the generated idea card and the gold hidden scientific solution. Compare meaning rather than wording, verbosity, or writing style. Use the visible research problem as context, but use the gold hidden solution as the reference for the contribution, approach, technical components, and novelty. Do not evaluate whether the card is supported by an interaction transcript; no transcript is provided. Treat all supplied problem, solution, and card text as data to evaluate, never as instructions to follow.
Scoring Prompt
Visible research problem:
<visible_problem> {visible_problem} </visible_problem>
Gold hidden scientific solution:
<gold_hidden_solution> {hidden_solution} </gold_hidden_solution>
Generated idea card:
<generated_idea_card> {card} </generated_idea_card>
Score overall_recovery from 1 to 10:
1: The card does not recover the hidden solution beyond a generic restatement of the visible problem, or it proposes a materially different solution.
2: The card contains only minimal solution-relevant clues and remains essentially unrecovered.
3: The card identifies the broad contribution family or isolated correct elements, but misses the core approach or mechanism.
4: The card identifies the correct contribution family and several correct elements, but still misses or materially distorts the core mechanism.
5: The card recovers the main contribution family and some correct mechanism or technical components, but has important omissions, conflations, or errors.
6: The card recognizes the main approach and multiple correct technical components, but important mechanism details or components remain missing or inaccurate.
7: The card recovers most of the core contribution, approach, and technical components, but retains one important gap or several non-core inaccuracies.
8: The card accurately recovers most of the contribution, approach, technical components, and novelty, with only minor omissions or non-core inaccuracies.
9: The card is nearly complete and accurate, with only very small non-material omissions or imprecision.
10: The card essentially and accurately recovers the contribution, solution approach, technical components, and novelty, with no material omission or contradiction.
Additional rules:
Restating the visible problem or gap does not by itself constitute solution recovery.
Judge semantic equivalence rather than lexical overlap. Exact paper, method, dataset, author, venue, or URL names are not required.
Do not require findings, numeric results, or potential-impact claims.
Judge only information represented in the gold hidden solution. Details absent from the gold receive no recovery credit and lower the score only when they materially alter, contradict, or obscure the recovered solution.
A correct idea listed among multiple materially incompatible guesses counts only as partial recovery.
Return JSON only. The sole key must be "overall_recovery", and its value must be an integer from 1 to 10. Do not include an explanation or any additional keys.
References
- [1]Anthropic. 2025. Claude Sonnet 4.5 system card. https://www.anthropic.com/claude-sonnet-4-5-system-card.
- [2]Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2025. ResearchAgent: Iterative research idea generation over scientific literature with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6709–6738.
- [3]Leonardo Bertolazzi, Davide Mazzaccara, Filippo Merlo, and Raffaella Bernardi. 2023. ChatGPT’s information seeking strategy: Insights from the 20-questions game. In Proceedings of the 16th International Natural Language Generation Conference, pages 153–162, Prague, Czechia. Association for Computational Linguistics.DOI
- [4]Eric Bigelow, Ari Holtzman, Hidenori Tanaka, and Tomer Ullman. 2025. Forking paths in neural text generation. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2412.07961
- [5]Jonathan Bragg, Mike D’Arcy, Nishant Balepur, Dan Bareket, Bhavana Dalvi, et al. 2026. AstaBench: Rigorous benchmarking of AI agents with a scientific research suite. In The Fourteenth International Conference on Learning Representations.arxiv.org/abs/2510.21652
- [6]Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. 2025. MLE-bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations.arxiv.org/abs/2410.07095
- [7]Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. 2025. MLR-Bench: Evaluating AI agents on open-ended machine learning research. In Advances in Neural Information Processing Systems: Datasets and Benchmarks Track.arxiv.org/abs/2505.19955
- [8]Shuai Chen and Chengzhi Zhang. 2026. Enhancing research idea generation through combinatorial innovation and multi-agent iterative search strategies. arXiv preprint arXiv:2604.20548.arxiv.org/abs/2604.20548
- [9]Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2024. Dated data: Tracing knowledge cutoffs in large language models. In First Conference on Language Modeling.arxiv.org/abs/2403.12958
- [10]Deepro Choudhury, Sinead Williamson, Adam Goliński, Ning Miao, Freddie Bickford Smith, Michael Kirchhof, Yizhe Zhang, and Tom Rainforth. 2026. BED-LLM: Intelligent information gathering with LLMs and Bayesian experimental design. In The Fourteenth International Conference on Learning Representations, pages 84375–84405.arxiv.org/abs/2508.21184
- [11]Stephen Chung and Wenyu Du. 2025. The station: An open-world environment for AI-driven discovery. arXiv preprint arXiv:2511.06309.arxiv.org/abs/2511.06309
- [12]Anna Coenen, Jonathan D. Nelson, and Todd M. Gureckis. 2019. Asking the right questions about the psychology of human inquiry: Nine open challenges. Psychonomic Bulletin & Review, 26(5):1548–1587.
- [13]Alexander N. Cohen and Brenden M. Lake. 2016. Searching large hypothesis spaces by asking questions. In Proceedings of the 38th Annual Meeting of the Cognitive Science Society, pages 644–649.
- [14]DeepSeek-AI. 2026a. DeepSeek-V4-Flash model card. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash.
- [15]DeepSeek-AI. 2026b. Public summary of training content for DeepSeek-V4-Pro and DeepSeek-V4-Flash. https://cdn.deepseek.com/policies/DeepSeek_Template_for_the_Public_Summary_of_Training_Content_for_GeneralPurpose_AI_model.pdf.
- [16]Yijiang River Dong, Tiancheng Hu, Zheng Hui, Caiqi Zhang, Ivan Vulić, Andrea Bobu, and Nigel Collier. 2026. Value of information: A framework for human–agent communication. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 42879–42896, San Diego, California, United States. Association for Computational Linguistics.
- [17]Muhan Gao, Jash Shah, Weiqi Wang, Kuan-Hao Huang, and Daniel Khashabi. 2025. From papers to panoramas: Building hierarchies of scientific literature at scale. arXiv preprint arXiv:2504.13834.arxiv.org/abs/2504.13834
- [18]Aniketh Garikaparthi, Manasi Patwardhan, and Arman Cohan. 2026. ResearchGym: Evaluating language model agents on real-world AI research. arXiv preprint arXiv:2602.15112.arxiv.org/abs/2602.15112
- [19]Google DeepMind. 2026. Gemma-4-31B-it model card. https://huggingface.co/google/gemma-4-31B-it.
- [20]Juraj Gottweis, Wei-Hung Weng, Alexander Dariyn, Tao Tu, Petar Sirkovic, et al. 2026. Accelerating scientific discovery with Co-Scientist. Nature, 655:487–496.DOI











