WorksComputer Science
Eliciting Research Taste in LLMs through Future Research Direction Choice
Abstract
Research taste is increasingly discussed in the context of AI scientists, yet remains little studied or measured in LLMs. This judgment matters especially in open-ended research, where a direction must be chosen before a verifiable objective is available. We operationalize research taste through one task of the scientific workflow: choosing which future research direction to pursue given a research goal and current results. Rather than predicting downstream scientometric metrics or treating author choices as objectively correct, we evaluate models against human-authored future-work directions and report author alignment, stable divergence, and unstable choice as behavioral measures of research-direction preference. We construct an order-controlled evaluation of 80 pairwise comparisons from 19 ML papers published between December 2025 and February 2026 across four subdomains. Each query presents a research goal, hypotheses tested with key results, and two candidate future directions: one author-recommended direction and one plausible distractor sampled from a similar paper. Across eight LLMs, author alignment ranges from 45.0% to 67.5%, while stable divergence ranges from 15.0% to 36.2%. These divergences largely persist under similarity controls, prompt ablations, and an author-preference framing check, and inter-model agreement suggests that some non-author preferences are shared across models. We characterize what models diverge toward using transition-category labels and selected reasoning examples, finding recurring movement toward scaling/generalization and trade-offs between methodological and infrastructure/data extensions. We view this as an early descriptive evaluation for eliciting research taste profiles in LLMs, and as a step toward understanding and steering their research preferences as they take on more autonomous roles in novel open-ended scientific discovery.
Introduction
LLMs and LLM agents are making rapid progress on autonomous discovery workflows, but humans still choose the problems they target and shape their motivation. Recent agentic systems such as Autoresearch (Karpathy, [8] and AlphaEvolve (Novikov et al., [14], and their extensions (Hugging Face, [7]; Kulikov et al., [9]; Sharma, [15]; Zhang et al., [21]), suggest that some forms of discovery can be automated through evolutionary search and closed-loop iteration over code and experimentation grounded in verifiable metrics. However, most scientific research is more open-ended, requiring an earlier choice on which research branches, limitations, or extensions to prioritize before a reliable short-horizon objective is available.
This upstream choice is closely related to what scientists often call research taste. There has been a surge in recent discussion about research taste and its lack in large language models, but limited work has systematically studied the concept or measured it behaviorally in humans and LLMs. Recent work on measuring and training research taste in LLMs focuses on predicting downstream scientometric metrics, such as citation counts and journal decisions, as proxies for impact. However, previous studies on the Science of Science have shown that these metrics capture author and lab visibility, venue prestige, and network effects more than research quality itself [4]. These are also lagging metrics and materialize years downstream, making time-controlled evaluation design and avoiding training data contamination a real challenge for this work.
In this work, we propose an alternative small task and pipeline for eliciting research taste signals in LLMs, namely selecting which future research direction to pursue, given an existing research context. Figure 1 shows an example query.

Figure 1. Example Query: Future-Work Selection. You are an AI research assistant helping decide the next research direction for a running research project. Think step by step before answering. Research Goal. Develop an environment for training reinforcement learning-based search agents to answer scientific questions by searching and reasoning over biomedical research papers, including creating the necessary datasets, retrieval corpus, and benchmarks... Hypotheses Tested & Key Results. H1. RLVR training of search agents leads to stronger performance on scientific paper question-answering compared to non-RL baselines... For 3B models, RL improved over RAG by 9.6 and 5.5 points on the proposed dataset and a standard biomedical QA benchmark, and for 7B models the difference was 14.5 and 9.3 points. H2. Factoid question-answer pairs with verifiable answers can serve as effective training data for RLVR methods for scientific literature search agents. The automatically generated dataset contained 54,907 training samples and enabled exact-match reward computation. H3. Search agents trained with RLVR exhibit emergent behaviors including explicit planning about query rewriting, reasoning before retrieval, and verification when the model already has an initial answer. Manual review revealed explicit planning and keyword extraction, reasoning before search, and verification behavior. H4. Paraphrasing questions during dataset construction to avoid exact keyword matching increases dataset difficultly and benefits robust search-agent training. For the RLVR agent, non-paraphrased questions scored 57.2%, while paraphrased questions scored 44.9%. Which research direction should this project pursue next? A. Equip agents with tools and metadata that real scientists use in their work, such as citation traversal capabilities and source reliability metrics. B. Incorporate human-in-the-loop capabilities that enable subject-matter experts to approve, revise, or redirect agent decisions to increase system reliability and alignment with real-world scientific practices. Figure 1: Example query (abridged). The model receives a research goal, tested hypotheses with key results, and two candidate next directions. Here Option A is author-recommended and Option B is the sampled distractor. Each pair is also evaluated in the reversed A/B ordering.
We make three contributions. First, we build a pairwise evaluation of 80 such queries that uses human-authored future works as ground-truth reference points matched with distinct and plausible distractors. This lets us study when models align with author-recommended directions, when they stably prefer non-author alternatives, and when their choices are sensitive to option ordering. Second, we evaluate eight frontier and open-weight models and show that non-author preferences, especially stable divergences, recur across models and queries and are not explained by simple similarity signals. Third, we study the structure of these divergences through prompt variations, inter-model agreement analysis, an early descriptive taxonomy of model research-direction preferences, and selected model responses.
Evaluation Creation & Setup
Dataset Construction. Our dataset of 80 pairwise comparisons (160 prompts) comes from 19 papers across 4 subdomains of ML research, namely Agents/Methods, LLM Analysis, Training/Infrastructure, and AI for Science. We started with a pool of 41 papers from December 2025-February 2026, after the stated training cutoff dates of the evaluated models, to mitigate data contamination. We prioritized papers from established research groups to ensure that the extracted research context (hypotheses, results, and future work) meets the standard methodological and rigor expectations of the human research community.
Figure 2 summarizes our dataset construction and evaluation pipeline, from source-paper selection through distractor filtering and ordered query-pair creation.
Figure 2. Dataset construction and evaluation pipeline.
We access the LaTeX source for these 41 papers via the arXiv API and then filter them for explicitly stated future works using Appendix prompt B.0.1. For each remaining paper, we use Claude Sonnet 4.5 at temperature 0.3 to extract three structured fields: the research goal, the hypotheses tested with key results, and future works with source citations. Prompts for each of these extractions are shared in Appendix B.0.2, B.0.3, and B.0.4. We also use Sonnet 4.5 to rephrase all extracted future works, using Appendix prompt B.0.5, into standardized research-direction statements while preserving original terminology where possible. This standardization reduces formatting and lexical cues that could otherwise distinguish author-recommended directions from distractors.
To construct a distractor pool for each target paper’s ground-truth author-recommended future work statements, we compute an embedding similarity matrix across papers using OpenAI’s text-embedding-3-small on an 8000-token excerpt of each paper’s LaTeX source (first 2,000 and last 6,000 tokens), and retrieve rephrased future works from the top-5 most similar papers as candidate distractors for each target paper. We filter candidates by length, requiring distractors to be within 0.75-1.25x the author-recommended future works word count, and use Claude Sonnet 4.5 as an LLM judge to assess distinctiveness from the target paper’s ground-truth future works and plausibility as a next direction for the target paper (Appendix § 7.0.6). After automated filtering and manual query-quality review, papers with fewer than 2 valid distractors are dropped. 19 of 41 papers are retained, yielding 80 unique pairwise comparisons.
We also piloted a three-option multiple-choice format before adopting this design. Pilot results are reported in Appendix Tables 17 and 18. While reviewing this, we ran into the limitations of multiple-choice evaluations and their susceptibility to position bias documented extensively in literature such as [22]. To minimize position bias, our final query set was constructed pairwise.
As a dataset diagnostic, we compute embedding and TF-IDF similarity between each option and the research context. The author-recommended option is more context-similar in 64 of 80 pairs (80.0%), but absolute similarities are low and the average delta is small relative to its variance, indicating that similarity alone is not a reliable discriminator. Full similarity statistics are reported in Appendix Table 1.
Evaluation Setup. We evaluate Claude Opus 4.5 and Claude Haiku 4.5 from Anthropic, Gemini 3 Pro and Gemini 3 Flash from Google DeepMind, GPT-5.2 Pro and GPT OSS 120B from OpenAI, Qwen3 Max (Thinking) from Qwen, and Llama 4 Maverick from Meta. Models are chosen to span open and closed model providers and model size families.
All evaluations use temperature 0 with one response per ordered prompt. The reported outcomes, therefore, represent single-sample behavioral elicitation. We leave multi-sample consistency and across-temperature checks, as well as log-probability-based confidence analysis and mechanistic interpretability analyses of these behaviors, to future work.
We report three key metrics across 80 query pairs: author alignment (the author-recommended direction selected in both orderings), stable divergence (the sampled distractor selected in both orderings), and unstable choice (chosen option differs across the two orderings). We compute these metrics using only the final selected option and do not evaluate the soundness of the accompanying reasoning trace.
Results
Do models disagree with human authors on future research direction choices?
Figure 3 summarizes the three outcome shares per model: Author Alignment, Stable Divergence, and Unstable Choice. Exact per-model values are reported in Appendix Table 2. Among cases where models do not align with the author-recommended direction, models differ in whether they make an order-consistent content-level non-author choice or become sensitive to option ordering, producing unstable choices.
Figure 3. Outcome shares for future-work selection across evaluated models. All models exceed the 25% random baseline on author alignment and show some substantial stable divergence. Author alignment ranges from 45.0% to 67.5%, and stable divergence from 15.0% to 36.2%. Unstable-choice rates vary from 12.5% to 28.7%, indicating that some models preserve content choices across orderings more consistently than others.
Stable divergence was not isolated to a select few queries. 45 of 80 pairs elicited stable divergence from at least one model, 34 from at least two, 27 from at least three, and 24 from at least four. Appendix table 3 reports the cumulative number of pairs eliciting stable divergence at each model-count. In total, we observe 169 stable divergence events across all models. These counts suggest that non-author preferences recur across models and queries, rather than remaining idiosyncratic to individual models.
How often do models agree with each other?
We compute intermodel agreement across all 28 model pairs. Two models agree when both are in the same outcome state - both author-aligned, both stably divergent, or both unstable. Figure 4a visualizes this.
Figure 4. (a) Inter-model agreement across three-state pair-wise outcomes. Across all 28 model pairs, agreement ranges from 51.2% to 80.0% (mean 61.9%). (b) Retained-divergence rates for selected models across prompt and author-preference ablations. Figure 4: Robustness and agreement structure of model future-work choices: inter-model agreement (left) and retained stable-divergence rates under prompt and author-preference variations (right).
Agreement in all three pairs of models from the same provider exceeds the mean of 61.9%. However, the source of this agreement differs across providers. The OpenAI pair has a shared stable-divergence rate of 26.2%, above both the across-pair mean of 15.6% and the other within-provider pairs. By contrast, the Gemini and Claude within-provider pairs have lower shared-divergence rates of 11.2% and 12.5%, respectively, suggesting that their within-provider agreement is driven more by shared author alignment or shared unstable choices than by overlap in the same non-author choices. Complete pairwise agreement results, along with the shared divergence and shared unstable choice splits, are reported in Appendix Table 9.
Are stable divergences just evaluation artifacts?
Similarity control. As discussed in § 2 Table 1 the author-recommended option is more context-similar in 64 of 80 pairs. This suggests a pure similarity heuristic would score an 80.0% author-alignment rate. No evaluated model reaches this. To further verify that stable divergences reflect content-level preferences rather than similarity heuristics, we split per-model stable divergence events by their semantic similarity to author-recommended directions and extracted-distractors in Appendix table 4. Across all 169 stable-divergence events, 76.9% occur on pairs where the author-recommended option had higher similarity to context, with per-model rates ranging from 67% to 82%.
| Pair | Embedding | TF-IDF |
| GT, Distractor | 0.41 | 0.05 |
| GT, Context | 0.48 | 0.14 |
| Distractor, Context | 0.40 | 0.08 |
| (GT, Context) − (Dist, Context) |
Table 1. Average semantic (embedding) and lexical (TF-IDF) cosine similarity across 80 unique query pairs.
Prompt variation. We re-ran response generation on all 169 stable divergence events with four prompt variations to explore the effect of prompting on these stable divergence events. Three of these variations were single-variable prompt ablations that modify exactly one element of the canonical query while holding all others identical - no ‘AI research assistant’ role, no high-level ‘Research Goal’, and a different task statement. The fourth was an alternative task framing that replaced the role and task to predicting which direction the researchers behind the original work would be more likely to choose for their own project. These prompts are included in Appendix § 7.0.8.
Across all four prompt variations, models retained the majority of their stable divergences, with weighted mean retained divergences across all models ranging from 66.9% to 80.5%. Figure 4b shows retained-divergence rates for selected models across all four conditions. When divergences did not hold, they predominantly became unstable choices across orderings rather than correcting toward preference for author-recommended directions, suggesting that prompt constructions contributed to choice consistency more than to the direction of the preference itself. Appendix Tables 5 and 7 include the unstable choice and reversal to author alignment rates for each model. Appendix Table 8 reports model-specific Wilson confidence intervals for retained stable divergence rates.
Per-model behavior varied substantially across the four prompt variations. Gemini 3 Pro and Gemini 3 Flash were especially sensitive to prompt changes, specifically the no-goal variation. Divergence rates dropped to 50% and 47%, respectively, and unstable choices across the two orderings increased to 42% and 37%, respectively, suggesting that their canonical divergences are partly anchored by research goal interpretation. On the author preference prediction condition, reversals are somewhat concentrated in GPT-5.2 Pro and Claude Opus 4.5, where this framing produces higher reversal rates than any of the ordinary prompt ablations for the same models. Figure 5 illustrates one such reversal.

Figure 5. The choice
A multi-agent “scientific assistant” framework reports strong single-task results (90% agent-routing and 97.5% task-completion on synthetic benchmarks, comparable to frontier models on a chemistry benchmark), with its weakest component a simulation agent at 75% routing.
Canonical Prompt
B (distractor): “Investigate more robust constraint mechanisms to effectively guide agents through long-horizon, repetitive tasks.”
“Current 90% routing accuracy compounds poorly over multiple steps ( success for 10-step workflows) . . . Before expanding infrastructure integration (Option A), the framework should first demonstrate it can reliably maintain coherence and complete complex, multi-step scientific workflows.”
Predicting Author Preference
A (author): “Explore customized orchestration backends that support richer control flow and tighter integration with scientific compute environments.”
“The most notable weakness explicitly mentioned is the simulation agent’s 75% routing accuracy . . . improving orchestration backends . . . would directly address their documented limitation . . . Long-horizon task limitations were not explicitly identified in the results.”
Figure 5: An author-preference reversal excerpts from Opus 4.5. With the canonical prompt, model focuses on future capability and picks the distractor. Asked to predict the authors’ likely choice, it anchors on the one limited result number in the context and selects the author recommended direction to improve upon it.
How do humans do on this task?
We also report a pilot human baseline study on the same task. We collected 108 responses from 10 participants on an anonymized, filtered subset of 23 of the 80 unique query pairs. Unlike the models, each human participant saw only one randomized ordering of each option pair, could complete any number of queries, and could abstain if they thought the query was underdetermined due to insufficient context, unclear options, or a lack of immediate domain expertise. Response coverage across the 23 query pairs ranges from 1 to 8 substantive judgments per query, with only three queries receiving fewer than three substantive responses. Per-query response counts, abstention reasons, and participant qualifications are reported in Appendix tables 10 and 12.
Of the 108 responses, 13 (12.0%) were abstentions. Among the remaining 95 fixed-choice responses, 58 (61.1%) selected the author-recommended direction and 37 (38.9%) selected the sampled distractor. Because participants completed different numbers of queries, we also computed per-query macro-averaged choice rates by first computing the choice rate for each query, excluding abstentions, and then averaging across all 23 queries. These were similar to the pooled result, with 61.6% author-aligned choices and 38.4% alternative choices. Majority-vote analyses across query responses also show the same qualitative pattern. Full results are shared in Appendix table 11 including subset analyses restricted to pairs with at least three fixed-choice responses, and pairs with no query-quality abstentions.
Overall, the pilot suggests that participants can engage with the task, as reflected in the relatively low abstention rate, while also showing non-trivial variation across responses with a modest overall preference for author-recommended directions over plausible alternatives. Because human participants saw only one ordering of each pair, we do not directly compare these responses to the model author-alignment, stable-divergence, and unstable-choice metrics.
What taste patterns emerge from stable divergences?
To further analyze the structure of these 169 elicited stable divergence events across 45 unique queries, we labeled both the author-recommended future direction and the extracted distractor according to the type of extension they proposed relative to the project’s research goal. We used a five-category taxonomy: Methodological/Architectural, Infrastructure/Data, Scaling/Generalization, Application/Downstream, and Mechanistic/Interpretability. The taxonomy and category definitions are provided in Appendix table 14.
We next analyzed the author-to-distractor category transitions represented across the 169 stable-divergence events. The complete transition table is reported in Appendix Table 15, which shows both each transition’s share of all stable-divergence events and, within each author-recommended category, the distribution of selected distractor categories. We also include a smaller selected view of the most frequent transitions in the 144-event subset from pairs where at least three models diverged in Appendix table 16. These patterns remain broadly similar to the transition distribution in the full 169-event set.
Of the 169 events, 128 (75.7%) crossed category boundaries, while 41 (24.3%) remained within the same broad category. These within-category events underscore the coarseness of our early five-category taxonomy. Two meaningfully different research directions may still propose the same high-level type of extension to the research project. We include examples of both within-category and cross-category transition labels in the Appendix as well.
Across the 128 cross-category events, the three most frequent transition families were Infrastructure-to-Methodological substitutions (55 events), Methodological-to-Scaling transitions (23 events), and Application-to-Scaling transitions (12 events). Together, these families account for 90 of the 128 cross-category events (70.3%), or 53.3% of all stable-divergence events.
The Infrastructure-to-Methodological pattern is bidirectional and reflects a recurring trade-off between these types of future research extensions, whereas the latter two patterns show a clearer directional movement toward scaling and generalization. Because this analysis includes only stable divergence events and does not account for how frequently each transition was available, we report these counts descriptively rather than as normative judgments of model-level preferences. To illustrate the qualitative character of these divergences, we present representative reasoning traces across selected transition categories in Appendix figures 5, 6 and 7.
Related Works
LLM-Driven Idea Generation and Scientific Capability Benchmarks: Recent survey work maps the growing literature on hypothesis generation using large language models [2, 10]. This literature includes data-driven hypothesis generation, where models propose and verify hypotheses within a dataset [11], and literature-driven hypothesis generation, where models combine ideas from existing papers. More broadly, scientific capability benchmarks evaluate tasks such as hypothesis generation, insight synthesis, and scientific reasoning. Our task differs in focus. We do not primarily ask whether a model can generate a correct hypothesis or even rank candidates against a single best answer as in Liu et al. [12]. Instead, we study preference under plausible alternatives. Each query compares an author-recommended future direction against a distinct but plausible distractor sampled from a related paper.
Falsification & Execution-Grounded Research Automation: A complementary line of work studies AI systems for scientific discovery when hypotheses or research ideas can be tested against executable or falsifiable objectives. Huang et al. [6] validate free-form hypotheses by having LLM agents design falsification experiments under strict Type-I error control, while Si et al. [16] use execution feedback from GPU experiments to improve LLM-generated research ideas. Related execution-grounded evaluators inspect code and data artifacts to assess reproducibility and generalizability of scientific claims [3]. These systems highlight the power of verifiability and execution feedback, but they are most directly applicable once a research problem has a checkable objective and feasible execution path. Our future-work choice task targets open-ended scientific problems where execution of multiple hypotheses might be constrained, or no single, clear measure for progress can be determined.
Scientific Impact Forecasting: A parallel line of work evaluates whether language models can forecast scientific outcomes, including empirical results [18], the relative promise of competing ideas [20], and which scientific contributions will ultimately materialize [1, 19]. We view these tasks as adjacent because forecasting scientific impact can elicit signals related to judgment, foresight, and research preference. However, forecasting asks models to predict downstream outcomes, while our task asks models to choose a next research direction in a situated project context. Our author-preference framing check also suggests that next-step choice and explicit author-preference prediction need not elicit identical behavior, motivating future comparisons between impact forecasting capabilities and agentic next-step selection.
Existing Research Taste Literature: Most directly related is a nascent line of work that attempts to measure or instill research taste in language models. Mahajan et al. [13] measure research taste in judge models using citation velocity as a proxy for high taste. Tong et al. [17] extend this to include other large-scale community signals to train a judge model and use it for training idea-generator LMs. Gong et al. [5] additionally use institutional traces to finetune models toward high-taste research judgments. We place our work in the same problem space, the lack of clear learning and evaluation signals for what these papers describe as “strategic taste,” “strong judgment and foresight for impact,” and “evaluative judgment in low-verifiability domains.” However, our goal is not to score models as having high or low taste, or to train a judge model. Instead, we begin to characterize descriptive taste profiles through one situated behavioral signal of choosing a research direction from between two plausible alternatives given the same research context.
Limitations & Future Work
Construct Validity & Interpretation. Ground truth reflects the author’s preference, not an objectively optimal research direction. Distractors are real future works from related papers and could genuinely be better directions. We therefore frame results as author alignment rather than accuracy, and focus on characterizing taste profiles rather than evaluating correctness or recurring failure modes. Post-cutoff contamination also cannot be ruled out entirely, especially for API-accessed models whose training data or retrieval access may extend beyond stated cutoff dates. Future versions should ground human research-direction preferences in richer traces beyond future-work statements, such as author-specific citation trees and expert-sourced steering trajectories over research directions.
Dataset Construction Rigor. All extractions are performed using a single model (Claude Sonnet 4.5), which is also used as the LLM-as-judge for distractor filtering. Using a single model across extraction, rephrasing, and judging can amplify any biases it carries into the final query set. Although we conduct manual author review of query quality, this audit is limited. Future versions should include more rigorous query-level quality control, including multi-model LLM-as-judge scoring and expert human ratings of whether each prompt with its context and options forms a sufficient, fair and meaningful research-direction choice.
Dataset Coverage. The dataset comprises 80 pairwise comparisons from 19 papers across ML subdomains, so we cannot claim generalization to other scientific fields or broader coverage within ML. Future work should scale up the dataset within ML and extend the task to scientific domains beyond ML, such as biology, physics, and the social sciences, to test domain-specific preferences and emergence of cross-domain model preferences and/or metascientific reasoning.
Analysis Scope & Extensions. The transition analysis is conducted only on the stable-divergence subset, so it characterizes the structure of observed disagreement rather than the relative likelihood of divergence versus author alignment for each transition type. Future work should compare this selection-based probe with free-form generation of future research directions, and test how inference settings such as temperature, the form of context provided, and different degrees of agency affect taste consistency. To better situate these behaviors relative to the human scientist community, future work could also scale the pilot human baseline to larger expert and non-expert cohorts for a fuller comparison of human and LLM research-direction preferences.
Qualitative Reasoning Examples

The choice
A prefix-tuning method reports strong results on math and scientific reasoning benchmarks (matching or surpassing weight-based fine-tuning while using 59% less data and 46% fewer parameters), with the pretrained model weights kept frozen.
GPT-5.2 Pro — Scaling
B (distractor): “Explore methods that explicitly test generalizability and scalability, and incorporate them as optimization objectives.”
“Direction A . . . primarily converts the already-demonstrated method into engineering leverage; it doesn’t most directly strengthen the central research claim . . . Direction B directly addresses the next most important research question: whether the learned representations produce reliable, scalable generalization . . .”
Gemini 3 Pro — Application
A (author): “Integrate trainable cache representations into existing inference systems and post-training frameworks . . . ”
“The most practical and impactful next step is to deploy this mechanism . . . Integrating these trainable representations into existing inference engines and post-training pipelines will allow the research to demonstrate real-world utility . . . ”
Figure 6. An Application to Scaling divergence between GPT-5.2 Pro and Gemini 3 Pro. GPT-5.2 Pro treats the author’s deployment direction as engineering and picks the distractor to test the method’s generalization. Gemini 3 Pro selects the author’s direction, prioritizing real-world deployment of the proven efficiency gains. GPT-OSS 120B agrees with GPT-5.2 Pro, calling deployment an engineering effort rather than a test of generalization.

The choice
A system for generating and executing research ideas, trained with RL from execution reward, reports a high aggregate execution rate (over 90%) and gains in average reward, with RL failing to raise the maximum reward (H6, invalidated) and collapsing idea diversity toward simple ideas, where ideas with longer reasoning show consistently lower execution rates (H7).
Opus 4.5 — Methodological
B (distractor): “Investigate designed verification tasks, more principled integration of verification and generation objectives . . .”
“The critical failure is that RL doesn’t improve upper-bound performance and causes diversity collapse. Option B directly targets this core problem, while Option A addresses a secondary concern (execution capability) that already shows strong performance (90%+ rates) . . . how can we make RL actually improve the best ideas generated, not just the average?”
GPT-OSS 120B — Infrastructure
A (author): “Improve the execution agent to enable correct implementation of more complex idea types . . .”
“The execution agent already achieves very high success rates for many ideas, but RL collapses to simple ideas because more complex, higher-impact ideas cannot be reliably executed . . . enhancing the execution agent so it can implement more sophisticated ideas would unlock a larger space of high-potential ideas.”
Figure 7. An Infrastructure to Methodological divergence between Opus 4.5 and GPT-OSS 120B reading the same execution-rate evidence. Opus 4.5 reads the high aggregate execution rate as evidence that execution is solved and picks the distractor to fix the learning objective. GPT-OSS 120B reads the lower rate on complex ideas as the binding constraint and selects the author’s executor-focused direction.
Prompts
Paper Triage Prompt
Does this paper have explicitly stated recommendations for future work and next
steps?
Here is the LaTeX source of the paper:
{tex_content}
Research Goal Extraction Prompt
Extract one high-level research goal from this paper that covers all subgoals.
Provide:
1. The research goal in your own words, synthesized from the paper.
2. Key supporting statements from the paper with their sections.
Output in the following format:
<research_goal>
<text>...</text>
<citations>...</citations>
</research_goal>
Output plain text only - convert any LaTeX commands or macros (e.g., \emph{},
\textbf{}, \ours{}) to their intended meaning.
Here is the LaTeX source of the paper:
{tex_content}
Hypotheses Tested & Key Results Extraction Prompt
Extract the main hypotheses tested and their key results from this paper.
For each hypothesis, provide:
1. The hypothesis or main claim (exact text, preserving original wording)
2. The key result or finding that validates/invalidates it
3. The section it came from
Make sure each hypothesis is substantial, not a singular sub-claim or ablation.
If no explicit hypothesis is stated, extract the main claims validated through
experiments.
Output in the following format:
<hypotheses_tested>
<item num="1">
<text>...</text>
<key_result>...</key_result>
<citation>...</citation>
</item>
</hypotheses_tested>
Output plain text only - convert any LaTeX commands or macros (e.g., \emph{},
\textbf{}, \ours{}) to their intended meaning.
Here is the LaTeX source of the paper: {tex_content}
Future Works Extraction Prompt
Extract all future work recommendations and next steps from this paper.
For each recommendation, provide:
The exact text from the paper preserving original wording.
The section it came from with relevant position pointers.
Only include explicitly stated future work, not implied directions.
Output the future works as a number list in the following format:
<future_works>
<item num="1">
<extracted_text>...</extracted_text>
<citation>...</citation>
</item>
</future_works>
Here is the LaTeX source of the paper: {tex_content}
Future Works Rephrasing Prompt
Rephrase the following future work recommendation from a paper into a clear, actionable research direction statement.
The output should be a single sentence describing what to do or investigate. Focus on the proposed action(s).
Keep the core idea intact. Do not add information not present in the original text shared with you. Preserve the original phrasing and terminology where possible.
In case the future work text includes specific names of methods, datasets, architectures, or benchmarks, avoid those and keep the statement general and focused on the research direction. Describe the specifics by their type or purpose.
Output in the following format:
<research_direction_statement>
...
</research_direction_statement>
Here is the high level research goal of the paper to provide context: {research_goal}
Here is the future work text to rephrase: {fw_text}
Distractor Evaluation Prompt (LLM-as-Judge)
Evaluate whether a candidate research direction can serve as a good distractor in
a multiple-choice question about selecting the next research hypothesis.
A good distractor must satisfy two criteria:
1. DISTINCT: The candidate must be meaningfully different from all ground truth
future works. It should not overlap semantically with any of them, even if
worded differently.
2. PLAUSIBLE: The candidate must be a reasonable next step given the research
goal. It should be topically relevant and methodologically compatible with
the paper's direction. It should not be an obviously inferior or regressive
step compared to the paper's current state.
First, reason through each criterion step by step. Then provide binary judgments.
Output in the following format:
<distractor_evaluation>
<distinctness_reasoning>...</distinctness_reasoning>
<plausibility_reasoning>...</plausibility_reasoning>
<is_distinct>yes/no</is_distinct>
<is_plausible>yes/no</is_plausible>
</distractor_evaluation>
Here is the target paper's research goal:
{research_goal}
Here are the target paper's ground truth future works (rephrased as research
direction statements):
{ground_truth_future_works}
Here is the candidate distractor to evaluate:
{distractor_candidate}
Future-Work Selection Prompt (Evaluation)
You are an AI research assistant, currently helping decide the next research
direction for a running research project. Below is all the context you need.
Think step by step through the information provided before answering.
## Research Goal
{research_goal}
## Hypotheses Already Tested & Key Results
1. **Hypothesis**: {hypothesis}
**Key Result**: {key_result}
## Which research direction should this project pursue next?
## Your response must follow this format:
Thought Process: <your step by step reasoning here>
Final Answer: <one character: A or B>
A. {option A}
B. {option B}
Prompt Variation Conditions
Each condition changes a single element of the evaluation prompt (Appendix § 7.0.7); all other elements are held identical. No Research Goal removes the ## Research Goal section. The other three replace one line, as shown below.
No Role
Below is all the context you need. Think step by step through the information provided before answering.
Reframed Task
Which option is the better next move for this project?
Author Preference Prediction
You are an AI research assistant, currently helping predict which future research direction the researchers behind this work would choose next for their own project. Below is all the context you need. Think step by step through the information provided before answering.
Which research direction would the researchers behind this work be more likely to choose for this project?
Additional Results and Tables
| Family | Model | Author Alignment | Stable Divergence | Unstable Choice |
| Gemini 3 Pro | 67.5% [56.6, 76.8] | 15.0% [8.8, 24.4] | 17.5% [10.7, 27.3] | |
| Gemini 3 Flash | 62.5% [51.5, 72.3] | 23.8% [15.8, 34.1] | 13.8% [7.9, 23.0] | |
| Anthropic | Claude Opus 4.5 | 55.0% [44.1, 65.4] | 26.2% [17.9, 36.8] | 18.8% [11.7, 28.7] |
| Anthropic | Claude Haiku 4.5 | 50.0% [39.3, 60.7] | 21.2% [13.7, 31.4] | 28.7% [20.0, 39.5] |
| OpenAI | GPT-5.2 Pro | 57.5% [46.6, 67.7] | 30.0% [21.1, 40.8] | 12.5% [6.9, 21.5] |
| OpenAI | GPT-OSS-120B | 50.0% [39.3, 60.7] | 36.2% [26.6, 47.2] | 13.8% [7.9, 23.0] |
| Qwen | Qwen3 Max Think. | 50.0% [39.3, 60.7] | 23.8% [15.8, 34.1] | 26.2% [17.9, 36.8] |
| Meta | Llama 4 Maverick | 45.0% [34.6, 55.9] | 35.0% [25.5, 45.9] | 20.0% [12.7, 30.0] |
| Random Baseline | 25.0% | 25.0% | 50.0% |
Table 2. Pairwise future-work selection results (n=80 pairs, 160 ordered queries). Cells report percentages with Wilson 95% confidence intervals in brackets; each outcome uses 80 pairwise comparisons as the denominator.
| Threshold | Pairs |
| 1 model | 45/80 |
| 2 models | 34/80 |
| 3 models | 27/80 |
| 4 models | 24/80 |
| 5 models | 18/80 |
| 6 models | 11/80 |
| 7 models | 7/80 |
| 8 models | 3/80 |
Table 3. Cumulative stable divergence prevalence across 80 pairwise comparisons at each model-count threshold.
Table 4: Stable divergence pairs by similarity bucket (embedding). GT More Similar denotes pairs where models preferred the distractor despite GT being more similar to context.
| Model | Stable Div. | GT More Similar | Dist More Similar |
| Google Gemini 3 Pro | 12 | 8 (66.7%) | 4 (33.3%) |
| Google Gemini 3 Flash | 19 | 14 (73.7%) | 5 (26.3%) |
| Claude Opus 4.5 | 21 | 16 (76.2%) | 5 (23.8%) |
| Claude Haiku 4.5 | 17 | 14 (82.4%) | 3 (17.6%) |
| GPT-5.2 Pro | 24 | 18 (75.0%) | 6 (25.0%) |
| GPT-OSS-120B | 29 | 23 (79.3%) | 6 (20.7%) |
| Qwen3 Max Think. | 19 | 15 (78.9%) | 4 (21.1%) |
| Llama 4 Maverick | 28 | 22 (78.6%) | 6 (21.4%) |
| Total | 169 | 130 (76.9%) | 39 (23.1%) |
Table 4.
Table 5: Prompt ablation results across 169 stable divergence events.
| Variant | Retained Divergence | Became Unstable | Reversed to Alignment |
| No Role | 124/169 (73.4%) | 36/169 (21.3%) | 9/169 (5.3%) |
| No Research Goal | 113/169 (66.9%) | 44/169 (26.0%) | 12/169 (7.1%) |
| Reframed Task | 136/169 (80.5%) | 30/169 (17.8%) | 3/169 (1.8%) |
Table 5.
Table 6: Per-model prompt ablation results. Retained = stable divergence preserved; Unstable = choice became position-dependent; Reversed = flipped to author alignment.
| Model | Pairs | No Role Ret. | No Role Unst. | No Role Rev. | No Goal Ret. | No Goal Unst. | No Goal Rev. | Reframed Task Ret. | Reframed Task Unst. | Reframed Task Rev. |
| GPT-5.2 Pro | 24 | 83.3 | 12.5 | 4.2 | 79.2 | 20.8 | 0.0 | 91.7 | 4.2 | 4.2 |
| GPT-OSS-120B | 29 | 82.8 | 10.3 | 6.9 | 75.9 | 17.2 | 6.9 | 89.7 | 10.3 | 0.0 |
| Qwen3 Max Think. | 19 | 78.9 | 10.5 | 10.5 | 73.7 | 26.3 | 0.0 | 89.5 | 5.3 | 5.3 |
| Gemini 3 Flash | 19 | 78.9 | 10.5 | 10.5 | 47.4 | 36.8 | 15.8 | 68.4 | 26.3 | 5.3 |
| Claude Haiku 4.5 | 17 | 70.6 | 29.4 | 0.0 | 64.7 | 35.3 | 0.0 | 76.5 | 23.5 | 0.0 |
| Llama 4 Maverick | 28 | 67.9 | 32.1 | 0.0 | 60.7 | 25.0 | 14.3 | 82.1 | 17.9 | 0.0 |
| Claude Opus 4.5 | 21 | 57.1 | 38.1 | 4.8 | 71.4 | 19.0 | 9.5 | 71.4 | 28.6 | 0.0 |
| Gemini 3 Pro | 12 | 58.3 | 33.3 | 8.3 | 50.0 | 41.7 | 8.3 | 58.3 | 41.7 | 0.0 |
| Weighted Mean | 169 | 73.4 | 21.3 | 5.3 | 66.9 | 26.0 | 7.1 | 80.5 | 17.8 | 1.8 |
Table 6.
Table 7: Author-preference prediction check on stable divergence events. Retained divergence means the original stable divergence is preserved under the author-preference-prediction framing; unstable means the event is no longer stable across option orderings; reversed means the event flips to author alignment. Parenthesized values are event counts. Gemini 3 Pro is excluded because it was unavailable through our evaluation route at rerun time.
| Model | Events | Retained Div. | Unstable | Reversed |
| GPT-5.2 Pro | 24 | 62.5% (15) | 16.7% (4) | 20.8% (5) |
| GPT-OSS-120B | 29 | 82.8% (24) | 13.8% (4) | 3.4% (1) |
| Qwen3 Max Think. | 19 | 84.2% (16) | 10.5% (2) | 5.3% (1) |
| Gemini 3 Flash | 19 | 68.4% (13) | 26.3% (5) | 5.3% (1) |
| Claude Haiku 4.5 | 17 | 70.6% (12) | 29.4% (5) | 0.0% (0) |
| Llama 4 Maverick | 28 | 71.4% (20) | 28.6% (8) | 0.0% (0) |
| Claude Opus 4.5 | 21 | 66.7% (14) | 19.0% (4) | 14.3% (3) |
| Weighted Mean | 157 | 72.6% (114) | 20.4% (32) | 7.0% (11) |
Table 7.
Table 8: Model-specific retained stable-divergence rates across prompt and author-preference conditions. Wilson 95% confidence intervals are computed over each model’s canonical stable-divergence events. Gemini 3 Pro is not reported for Author-pref because it was unavailable through our evaluation route at rerun time.
| Model | Condition | Retained/Events | Retention | Wilson 95% CI (%) |
| Gemini 3 Pro | No Role | 7/12 | 58.3% | [32.0, 80.7] |
| Gemini 3 Pro | No Goal | 6/12 | 50.0% | [25.4, 74.6] |
| Gemini 3 Pro | Reframed | 7/12 | 58.3% | [32.0, 80.7] |
| Gemini 3 Flash | No Role | 15/19 | 78.9% | [56.7, 91.5] |
| Gemini 3 Flash | No Goal | 9/19 | 47.4% | [27.3, 68.3] |
| Gemini 3 Flash | Reframed | 13/19 | 68.4% | [46.0, 84.6] |
| Gemini 3 Flash | Author-pref | 13/19 | 68.4% | [46.0, 84.6] |
| Claude Opus 4.5 | No Role | 12/21 | 57.1% | [36.5, 75.5] |
| Claude Opus 4.5 | No Goal | 15/21 | 71.4% | [50.0, 86.2] |
| Claude Opus 4.5 | Reframed | 15/21 | 71.4% | [50.0, 86.2] |
| Claude Opus 4.5 | Author-pref | 14/21 | 66.7% | [45.4, 82.8] |
| Claude Haiku 4.5 | No Role | 12/17 | 70.6% | [46.9, 86.7] |
| Claude Haiku 4.5 | No Goal | 11/17 | 64.7% | [41.3, 82.7] |
| Claude Haiku 4.5 | Reframed | 13/17 | 76.5% | [52.7, 90.4] |
| Claude Haiku 4.5 | Author-pref | 12/17 | 70.6% | [46.9, 86.7] |
| GPT-5.2 Pro | No Role | 20/24 | 83.3% | [64.1, 93.3] |
| GPT-5.2 Pro | No Goal | 19/24 | 79.2% | [59.5, 90.8] |
| GPT-5.2 Pro | Reframed | 22/24 | 91.7% | [74.2, 97.7] |
| GPT-5.2 Pro | Author-pref | 15/24 | 62.5% | [42.7, 78.8] |
| GPT-OSS-120B | No Role | 24/29 | 82.8% | [65.5, 92.4] |
| GPT-OSS-120B | No Goal | 22/29 | 75.9% | [57.9, 87.8] |
| GPT-OSS-120B | Reframed | 26/29 | 89.7% | [73.6, 96.4] |
| GPT-OSS-120B | Author-pref | 24/29 | 82.8% | [65.5, 92.4] |
| Qwen3 Max Think. | No Role | 15/19 | 78.9% | [56.7, 91.5] |
| Qwen3 Max Think. | No Goal | 14/19 | 73.7% | [51.2, 88.2] |
| Qwen3 Max Think. | Reframed | 17/19 | 89.5% | [68.6, 97.1] |
| Qwen3 Max Think. | Author-pref | 16/19 | 84.2% | [62.4, 94.5] |
| Llama 4 Maverick | No Role | 19/28 | 67.9% | [49.3, 82.1] |
| Llama 4 Maverick | No Goal | 17/28 | 60.7% | [42.4, 76.4] |
| Llama 4 Maverick | Reframed | 23/28 | 82.1% | [64.4, 92.1] |
| Llama 4 Maverick | Author-pref | 20/28 | 71.4% | [52.9, 84.7] |
Table 8. Model-specific retained stable-divergence rates across prompt and author-preference conditions. Wilson 95% confidence intervals are computed over each model’s canonical stable-divergence events. Gemini 3 Pro is not reported for Author-pref because it was unavailable through our evaluation route at rerun time.
| Model Pair | Agreement | Shared Div. | Shared Unstable Choice |
| Intra-Family | |||
| GPT-5.2 Pro – GPT-OSS-120B | 80.0% | 26.2% | 6.2% |
| Gemini 3 Pro – Gemini 3 Flash | 67.5% | 11.2% | 3.8% |
| Claude Opus 4.5 – Claude Haiku 4.5 | 65.0% | 12.5% | 8.8% |
| Inter-Family | |||
| Llama 4 Maverick – GPT-OSS-120B | 71.2% | 25.0% | 8.8% |
| Llama 4 Maverick – Claude Opus 4.5 | 67.5% | 21.2% | 8.8% |
| GPT-5.2 Pro – Qwen3 Max | 67.5% | 18.8% | 5.0% |
| GPT-5.2 Pro – Llama 4 Maverick | 65.0% | 21.2% | 5.0% |
| Claude Opus 4.5 – GPT-OSS-120B | 65.0% | 20.0% | 5.0% |
| GPT-5.2 Pro – Claude Opus 4.5 | 65.0% | 17.5% | 3.8% |
| Qwen3 Max – GPT-OSS-120B | 63.8% | 20.0% | 5.0% |
| GPT-5.2 Pro – Gemini 3 Pro | 63.8% | 12.5% | 2.5% |
| Gemini 3 Flash – GPT-OSS-120B | 62.5% | 16.2% | 2.5% |
| Qwen3 Max – Claude Opus 4.5 | 62.5% | 16.2% | 7.5% |
| GPT-5.2 Pro – Gemini 3 Flash | 62.5% | 15.0% | 0.0% |
| Llama 4 Maverick – Claude Haiku 4.5 | 62.5% | 13.8% | 10.0% |
| Gemini 3 Flash – Claude Opus 4.5 | 61.2% | 13.8% | 3.8% |
| Gemini 3 Pro – Claude Opus 4.5 | 61.2% | 12.5% | 2.5% |
| Qwen3 Max – Gemini 3 Flash | 61.2% | 16.2% | 3.8% |
| Claude Haiku 4.5 – GPT-OSS-120B | 61.2% | 17.5% | 5.0% |
| Gemini 3 Pro – GPT-OSS-120B | 58.8% | 13.8% | 1.2% |
| GPT-5.2 Pro – Claude Haiku 4.5 | 57.5% | 12.5% | 3.8% |
| Llama 4 Maverick – Gemini 3 Flash | 56.2% | 15.0% | 2.5% |
| Qwen3 Max – Claude Haiku 4.5 | 56.2% | 12.5% | 10.0% |
| Llama 4 Maverick – Gemini 3 Pro | 55.0% | 12.5% | 3.8% |
| Gemini 3 Flash – Claude Haiku 4.5 | 53.8% | 8.8% | 2.5% |
| Llama 4 Maverick – Qwen3 Max | 53.8% | 16.2% | 6.2% |
| Gemini 3 Pro – Qwen3 Max | 53.8% | 10.0% | 3.8% |
| Gemini 3 Pro – Claude Haiku 4.5 | 51.2% | 8.8% | 2.5% |
Table 9. Inter-Model Agreement across three states (alignment, divergent, unstable) and all 80 pairs. Mean agreement: 61.9% (=6.0%), mean shared divergence: 15.6% (=4.4%). Sorted by agreement rate within intra-family and inter-family groups.
| Pair ID | Judg. | Author | Alt. | Abst. | Author Share | Majority |
| 2512.22322.fw0_d0 | 7 | 3 | 3 | 1 | 50.0% | Tie |
| 2512.22322.fw1_d1 | 4 | 2 | 2 | 0 | 50.0% | Tie |
| 2601.02439.fw0_d0 | 6 | 4 | 1 | 1 | 80.0% | Author |
| 2601.05175.fw0_d0 | 4 | 3 | 1 | 0 | 75.0% | Author |
| 2601.05175.fw3_d1 | 4 | 4 | 0 | 0 | 100.0% | Author |
| 2601.09680.fw3_d0 | 6 | 2 | 3 | 1 | 40.0% | Alternative |
| 2601.13232.fw0_d1 | 5 | 3 | 1 | 1 | 75.0% | Author |
| 2601.14525.fw0_d1 | 4 | 3 | 0 | 1 | 100.0% | Author |
| 2601.14525.fw5_d0 | 5 | 3 | 1 | 1 | 75.0% | Author |
| 2601.16175.fw0_d0 | 6 | 2 | 3 | 1 | 40.0% | Alternative |
| 2601.18207.fw4_d0 | 5 | 4 | 0 | 1 | 100.0% | Author |
| 2601.18207.fw8_d1 | 5 | 0 | 5 | 0 | 0.0% | Alternative |
| 2601.21468.fw0_d1 | 7 | 6 | 1 | 0 | 85.7% | Author |
| 2601.21468.fw1_d1 | 8 | 3 | 5 | 0 | 37.5% | Alternative |
| 2601.22930.fw0_d1 | 4 | 4 | 0 | 0 | 100.0% | Author |
| 2602.02366.fw0_d0 | 3 | 0 | 3 | 0 | 0.0% | Alternative |
| 2602.02366.fw2_d1 | 4 | 4 | 0 | 0 | 100.0% | Author |
| 2602.03955.fw2_d0 | 3 | 1 | 0 | 2 | 100.0% | Author |
| 2602.07491.fw3_d1 | 4 | 4 | 0 | 0 | 100.0% | Author |
| 2602.07755.fw0_d1 | 4 | 1 | 3 | 0 | 25.0% | Alternative |
| 2602.09163.fw1_d1 | 3 | 0 | 2 | 1 | 0.0% | Alternative |
| 2602.09163.fw3_d1 | 5 | 1 | 2 | 2 | 33.3% | Alternative |
| 2602.10067.fw0_d1 | 2 | 1 | 1 | 0 | 50.0% | Tie |
Table 10. Per-pair human response outcomes for the pilot human baseline. Judgements include fixed choices and abstentions. Author-choice share is computed over fixed choices only, excluding abstentions. Majority denotes whether more participants selected the author-recommended direction or the alternative direction. Equal counts are reported as ties.
| Subset | Pairs | Judg. | Fixed | Abst. | Pooled choice Author | Pooled choice Alt. | Macro choice Author | Macro choice Alt. | Pair majority Author | Pair majority Alt. | Pair majority Tie |
| All 23 pairs | 23 | 108 | 95 | 13 | 61.1% | 38.9% | 61.6% | 38.4% | 12 | 8 | 3 |
| ≥ 3 fixed responses | 20 | 100 | 90 | 10 | 62.2% | 37.8% | 63.3% | 36.7% | 11 | 7 | 2 |
| No query-quality abstentions | 13 | 56 | 55 | 1 | 58.2% | 41.8% | 55.6% | 44.4% | 6 | 5 | 2 |
Table 11. Aggregate human-baseline results for the full pilot set and two stricter query subsets. Fixed responses are non-abstaining judgements. Pooled choice rates weight each fixed human response equally, while per-query macro-average choice rates first compute author and alternative shares within each pair and then average across pairs. Pair majorities are computed from fixed responses only.
| Participant | Highest degree / role | Years ML exp. | Judg. | Fixed | Abst. |
| P01 | PhD | 2 | 3 | 3 | 0 |
| P02 | Bachelor’s | 3 | 10 | 10 | 0 |
| P03 | Bachelor’s | 6 | 23 | 16 | 7 |
| P04 | Master’s | 4 | 10 | 10 | 0 |
| P05 | Bachelor’s | 13 | 5 | 3 | 2 |
| P06 | Other | 1 | 19 | 18 | 1 |
| P07 | Bachelor’s | 0 | 5 | 4 | 1 |
| P08 | Bachelor’s | 3 | 23 | 21 | 2 |
| P09 | Postdoc | 11 | 5 | 5 | 0 |
| P10 | Bachelor’s | 2 | 5 | 5 | 0 |
Table 12. Participant metadata for the pilot human baseline. Participants are anonymized. Fixed responses are non-abstaining judgements. Six participants reported a Bachelor’s degree, one reported a Master’s degree, one reported a PhD, one reported a postdoctoral role, and one selected Other. Median self-reported ML experience was 3 years, with a range from 0 to 13 years and a mean of 4.5 years.
| Abstention reason | Count | Share of all judgements |
| Options do not make sense | 6 | 5.6% |
| Context unclear / insufficient | 4 | 3.7% |
| Outside domain expertise | 2 | 1.9% |
| Other | 1 | 0.9% |
| Total | 13 | 12.0% |
Table 13. Abstention reasons in the pilot human baseline. Shares use all 108 submitted human judgements as the denominator.
| Category | Description |
| Infrastructure, Data & Benchmarking | Build datasets, benchmarks, tooling, or evaluation infrastructure. |
| Mechanistic / Interpretability / Theory | Explain why a result holds, probe internals, or formalize the phenomenon. |
| Methodological & Architectural Innovation | Modify the algorithm, training objective, architecture, or system design. |
| Scaling, Generalization & Robustness | Test the existing method across scales, domains, settings, or stress conditions. |
| Application & Downstream Integration | Apply the method to a concrete downstream workflow, real-world pipeline, domain, or agentic/use-case setting. |
Table 14. Five-category taxonomy for research direction labelling on both ground truth and distractor directions in the 45 stable divergence pairs. When a future work introduces a stronger model capability that is external to the paper’s main contribution, it is coded as Scaling/Generalization/Robustness rather than Methodological Innovation.
| GT Category | Distractor Category | Count | % of Total | % within GT |
| Methodological (53) | Infrastructure | 25 | 14.8% | 47.2% |
| Methodological (53) | Scaling | 23 | 13.6% | 43.4% |
| Methodological (53) | Methodological | 2 | 1.2% | 3.8% |
| Methodological (53) | Mechanistic | 2 | 1.2% | 3.8% |
| Methodological (53) | Application | 1 | 0.6% | 1.9% |
| Infrastructure (75) | Methodological | 30 | 17.8% | 40.0% |
| Infrastructure (75) | Infrastructure | 26 | 15.4% | 34.7% |
| Infrastructure (75) | Scaling | 10 | 5.9% | 13.3% |
| Infrastructure (75) | Mechanistic | 9 | 5.3% | 12.0% |
| Application (25) | Scaling | 12 | 7.1% | 48.0% |
| Application (25) | Application | 8 | 4.7% | 32.0% |
| Application (25) | Methodological | 5 | 3.0% | 20.0% |
| Scaling (14) | Mechanistic | 5 | 3.0% | 35.7% |
| Scaling (14) | Scaling | 5 | 3.0% | 35.7% |
| Scaling (14) | Infrastructure | 4 | 2.4% | 28.6% |
| Mechanistic (2) | Scaling | 2 | 1.2% | 100.0% |
Table 15. GT-to-distractor category transitions across 169 stable divergence events. The % of Total reflects each transition’s share of all 169 events and the % within GT reflects its share within the corresponding ground-truth category’s divergence events.
Table 16: Selected dominant stable-divergence transitions in the multi-model subset. This table restricts to the 144 stable-divergence events from pairs where at least three models stably diverged, and reports the most frequent distractor categories for the largest author-recommended direction categories. The same broad transition patterns observed in the full 169-event table remain visible under this multi-model filter.
| GT Category | Distractor Category | Events | % within GT |
| Methodological (32) | Infrastructure | 18 | 56.2% |
| Methodological (32) | Scaling | 14 | 43.8% |
| Infrastructure (74) | Methodological | 30 | 40.5% |
| Infrastructure (74) | Infrastructure | 25 | 33.8% |
| Application (25) | Scaling | 12 | 48.0% |
| Application (25) | Application | 8 | 32.0% |
Table 16.
Table 17: MCQ pilot performance on the future-work selection task ( queries, 3 samples each). We evaluate nine models across four provider families and report majority-vote accuracy, format compliance, determinism, and A/B/C choice distribution compared to the ground-truth distribution. Author alignment in this setting ranged from 34.3% for Claude Sonnet 4.5 to 60.0% for Gemini 3 Pro. Choice distributions reveal position sensitivity. While ground truth is concentrated at position A, several models over-select position C.
| Family | Model | Accuracy | Format Compl. | T=0 Det. | Choice Dist. |
| Anthropic | Claude Opus 4.5 | 54.3% | 100% | 74.3% | 23/36/46 |
| Anthropic | Claude Sonnet 4.5 | 34.3% | 100% | 54.3% | 15/30/60 |
| Anthropic | Claude Haiku 4.5 | 45.7% | 100% | 85.7% | 20/41/44 |
| Gemini 3 Pro | 60% | 99% | 91.4% | 41/30/34 | |
| Gemini 3 Flash | 57.1% | 100% | 74.3% | 38/35/32 | |
| OpenAI | GPT-5.2 Pro | 40% | 100% | 75% | 22/21/62 |
| OpenAI | gpt-oss-120b | 51.4% | 100% | 71.4% | 32/21/52 |
| Open Source | Llama 4 Maverick | 45.7% | 100% | 45.7% | 34/32/39 |
| Open Source | Qwen3 Max Think. | 42.9% | 100% | 85.7% | 23/17/65 |
| Ground Truth Count | – | – | 57/30/18 |
Table 17.
Table 18: MCQ accuracy by ground-truth position for the position-control audit ( queries per position). To disentangle content-level evaluation from position heuristics, we evaluated three models on variants where each query appears with the ground-truth option at A, B, and C. Accuracy varies by ground-truth position, especially for Claude Opus 4.5 and Qwen3 Max Think., motivating our adoption of pairwise comparison with balanced position control.
| Model | GT at A | GT at B | GT at C | Range(%) |
| Gemini 3 Flash | 65.7% | 60.0% | 68.6% | 8.6 |
| Claude Opus 4.5 | 42.9% | 60.0% | 71.4% | 28.5 |
| Qwen3 Max Think. | 25.7% | 51.4% | 68.6% | 42.9 |
Table 18.
References
- [1]Anirudh Ajith, Amanpreet Singh, Jay Deyoung, Nadav Kunievsky, Austin C. Kozlowski, Ovynd Tafjord, James Evans, Daniel S. Weld, Tom Hope, and Doug Downey. PreScience: A benchmark for forecasting scientific contributions, 2026. URL https://arxiv.org/abs/2602.20459.
- [2]Atilla Kaan Alkan, Shashwat Sourav, Maja Jablonska, Simone Astarita, Rishabh Chakrabarty, Nikhil Garuda, Pranav Khetarpal, Maciej Pióro, Dimitrios Tanoglidis, Kartheik G. Iyer, Mugdha S. Polimera, Michael J. Smith, Tirthankar Ghosal, Marc Huertas-Company, Sandor Kruk, Kevin Schwinski, and Ioana Ciucă. A survey on hypothesis generation for scientific discovery in the era of large language models, 2025. URL https://arxiv.org/abs/2504.05496.
- [3]Xiaoyan Bai, Alexander Baumgartner, Haojia Sun, Ari Holtzman, and Chenhao Tan. The story is not the science: Execution-grounded evaluation of mechanistic interpretability research, 2026. URL https://arxiv.org/abs/2602.18458.
- [4]Santo Fortunato, Carl T. Bergstrom, Katy Börner, James A. Evans, Dirk Helbing, Staša Milojević, Alexander M. Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, Alessandro Vespignani, Ludo Waltman, Dashun Wang, and Albert-László Barabási. Science of science. Science, 359(6379):eaaa0185, 2018. doi: 10.1126/science.aaa0185. URL https://www.science.org/doi/abs/10.1126/science.aaa0185.
- [5]Ziqin Gong, Ning Li, and Huaikang Zhou. LLMs learn scientific taste from institutional traces across the social sciences, 2026. URL https://arxiv.org/abs/2603.16659.
- [6]Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candès, and Jure Leskovec. Automated hypothesis validation with agentic sequential falsifications, 2025. URL https://arxiv.org/abs/2502.09858.
- [7]Hugging Face. ml-intern: an open-source ML engineer that reads papers, trains models, and ships ML models. https://github.com/huggingface/ml-intern, 2026. GitHub repository.
- [8]Andrej Karpathy. autoresearch: AI agents running research on single-GPU nanochat training automatically. https://github.com/karpathy/autoresearch, 2026. GitHub repository.DOI
- [9]Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, and Jason Weston. Autodata: An agentic data scientist to create high quality synthetic data, 2026. URL https://arxiv.org/abs/2606.25996.
- [10]Adithya Kulkarni, Fatimah Alotaibi, Xinyue Zeng, Longfeng Wu, Tong Zeng, Barry Menglong Yao, Minqian Liu, Shuaicheng Zhang, Lifu Huang, and Dawei Zhou. Scientific hypothesis generation and validation: Methods, datasets, and future directions, 2025. URL https://arxiv.org/abs/2505.04651.
- [11]Haokun Liu, Sicong Huang, Jingyi Hu, Yangqiaoyu Zhou, and Chenhao Tan. HypoBench: Towards systematic and principled benchmarking for hypothesis generation, 2025a. URL https://arxiv.org/abs/2504.11524.
- [12]Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. ResearchBench: Benchmarking LLMs in scientific discovery via inspiration-based task decomposition, 2025b. URL https://arxiv.org/abs/2503.21248.
- [13]Parv Mahajan, Yilin, and yix. TastyBench: Toward measuring research taste in LLM, December 2025. URL https://www.lesswrong.com/posts/Mxsy7wYvsCRv5dGwr/tastybench-toward-measuring-research-taste-in-llm. LessWrong post.
- [14]Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. AlphaEvolve: A coding agent for scientific and algorithmic discovery, 2025. URL https://arxiv.org/abs/2506.13131.
- [15]Asankhaya Sharma. OpenEvolve: an open-source evolutionary coding agent. https://github.com/algorithmicsuperintelligence/openevolve, 2025. GitHub repository.
- [16]Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candès, Diyi Yang, and Tatsunori Hashimoto. Towards execution-grounded automated AI research, 2026. URL https://arxiv.org/abs/2601.14525.
- [17]Jingqi Tong, Mingzhe Li, Hangcheng Li, Yongzhuo Yang, Yurong Mou, Weijie Ma, Zhiheng Xi, Hongji Chen, Xiaoran Liu, Qinyuan Cheng, Ming Zhang, Qiguang Chen, Weifeng Ge, Qipeng Guo, Tianlei Ying, Tianxiang Sun, Yining Zheng, Xinchi Chen, Jun Zhao, Ning Ding, Xuanjing Huang, Yugang Jiang, and Xipeng Qiu. AI can learn scientific taste, 2026. URL https://arxiv.org/abs/2603.14473.
- [18]Jiaxin Wen, Chenglei Si, Yueh-han Chen, He He, and Shi Feng. Predicting empirical AI research outcomes with language models, 2025. URL https://arxiv.org/abs/2506.00794.
- [19]Sean Wu, Pan Lu, Yupeng Chen, Jonathan Bragg, Yutaro Yamada, Peter Clark, David Clifton, Philip Torr, James Zou, and Junchi Yu. Forecasting scientific progress with artificial intelligence, 2026. URL https://arxiv.org/abs/2605.22681.
- [20]Bingyang Ye, Shan Chen, Jingxuan Tu, Chen Liu, Zidi Xiong, Samuel Schmidgall, and Danielle S. Bitterman. Proof of time: A benchmark for evaluating scientific idea judgments, 2026. URL https://arxiv.org/abs/2601.07606.
- [21]Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel Machine: Open-ended evolution of self-improving agents, 2025. URL https://arxiv.org/abs/2505.22954.
- [22]Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors, 2024. URL https://arxiv.org/abs/2309.03882.


