WorksComputer Science
PRIME: Predictive Routing of Interventions by Modeling Effects for Scientific Ideation
Abstract
Feedback-driven iteration is widely used to improve LLM-generated scientific ideas, but existing systems either leave revision decisions implicit in generation, align revisions with feedback-localized weaknesses, or restrict action selection to system-specific operations. This is problematic because problems, hypotheses, and solutions are interdependent: the location where a weakness manifests does not itself determine the revision scope that will produce a better successor. We term this phenomenon diagnosis-intervention mismatch. We introduce Predictive Routing of Interventions by Modeling Effects (PRIME), a plug-and-play feedback-to-revision decision layer for existing scientific ideators. PRIME operates in three stages: (1) Structural Attribution interprets feedback over the problem-hypothesis-solution structure without fixing a revision target; (2) Counterfactual Routing constructs alternative scope-conditioned interventions and compares their projected successor utility and risks through lightweight rollouts; and (3) Intervention Execution compiles the selected intervention into an ordered generator-facing plan with semantic actions, preservation constraints, and dependency requirements for successor generation. We validate diagnosis intervention mismatch on 112 human-annotated idea-feedback pairs. Across four benchmarks, two ideators, and fixed- and open-problem settings, PRIME improves successor quality and routing reliability over strong baselines; removing counterfactual routing reduces quality from 3.463 to 1.960 and increases route regret from 0.426 to 1.929, confirming cross-scope effect modeling as the primary source of its gains.
Introduction
With the widespread adoption of large language models in scientific research automation, LLM-based scientific idea generation has emerged as an important research direction ([33, 26, 12, 34, 32, 5, 9, 44, 28, 18, 23] (Shahhosseini et al. 2025)). Within this area, feedback-conditioned scientific idea revision supports the generation of successor ideas based on evaluations of a current idea. For example, in idea refinement, feedback is used to address the limitations of the current idea and improve its hypotheses, methodology, and validation plan ([6, 27, 1, 23, 17]); in idea evolution, feedback can guide the system in adjusting its research trajectory and exploring alternative hypotheses, solutions, or problem formulations ([49, 50, 19, 29]). Despite their different settings, these scenarios share the same core objective: translating feedback into concrete updates to successor ideas. Reliable feedback also depends on judge quality: recent evidence shows that LLM judges can favor presentation style over scientific substance when evaluating research ideas (Ji et al. 2026). At a later stage of scientific automation, LabGuard grounds natural-language laboratory rules into runtime guards for embodied laboratory agents ([47]); PRIME addresses the earlier step of revising ideas in response to feedback.
Based on how feedback is translated into subsequent revision actions, existing systems can be broadly categorized into three types. The first is holistic rewriting, exemplified by Self-Refine ([27]), in which the current idea, together with an overall score, natural-language critique, or reviewer feedback, is provided to a generative model to produce a complete successor version. In this approach, both what should be revised and the appropriate scope of revision remain implicit in the rewriting process. The second is localized corrective revision, exemplified by ResearchAgent ([2]) and Evolveator ([36]), in which feedback is first grounded in specific text spans, evaluation dimensions, or idea components, after which localized corrections are generated for the diagnosed weaknesses. The third is workflow-coupled revision, exemplified by Deep Ideation (Zhao et al. [51]), VirSci (Su et al. 2025b), and EIG (Dong, Li, and Lin [7]), in which the next action is selected from a predefined set of revision operations according to the current idea’s evaluation scores, defect categories, or structured state, using either predefined rules or a learned controller. These three categories respectively leave revision decisions implicit in holistic rewriting, bind them directly to the weaknesses localized by feedback, or constrain them to an operation space predefined by a particular system.
Table 1 situates these revision mechanisms within the broader landscape of scientific ideation methods. The first category leaves the revision decision implicit in generation, while the second commonly treats the diagnosed weakness as the direct target of revision ([27, 2, 36]). The third makes action selection explicit but couples the decision to a system-specific state representation and a predefined operation space (Dong, Li, and Lin [7]). None of these mechanisms provides a general decision layer for translating feedback into revision scopes that can be executed by existing idea generators. This limitation is particularly important in scientific ideation, where the problem, hypothesis, and solution are interdependent ([45, 7] (Pu et al. 2025)). The same observable weakness may arise from causes at different levels, while interventions at different levels may propagate through these dependencies in different ways. Taken together, these methods leave a fundamental feedback-to-revision gap unresolved: "where a weakness is diagnosed does not by itself determine where revision should be applied".
| Method | Feedback update | Explicit action | Editable state | Output search | Scope-effect routing | Plug-in layer | Mismatch audit |
| Self-Refine / Reflection | ✓ | × | × | × | × | × | × |
| ResearchAgent / Evolveator | ✓ | ✓ | × | × | × | × | × |
| IdeaSynth | ✓ | ✓ | × | × | × | × | × |
| Deep Ideation / VirSci | ✓ | ✓ | × | ✓ | × | × | × |
| AI Scientist / Co-Scientist | ✓ | × | × | ✓ | × | × | × |
| Nova / Chain of Ideas | ✓ | ✓ | × | × | × | × | × |
| EIG | ✓ | ✓ | ✓ | × | × | × | × |
| PRIME | ✓ | ✓ | ✓ | × | ✓ | ✓ | × |
Table 1. Positioning PRIME relative to output-search and feedback-driven revision methods. Output-level search selects among generated ideas or trajectories, whereas PRIME selects among alternative revisions for the same idea–feedback pair before final generation.
Bridging this gap presents three key challenges. First, the locus where feedback manifests may not be the component that actually causes the problem. The same feedback may correspond to multiple plausible intervention loci, making the revision scope difficult to determine uniquely (Xiong et al. [45]; Pu et al. (2025); Dong, Li, and Lin [7]). Second, the appropriate intervention scope cannot be directly observed before revision. For a given idea, only the successor produced by the selected revision is typically observed, while exhaustively executing all alternatives is prohibitively expensive ([42, 36] (Hu et al. 2024; Li et al. 2026)). Third, translating the intervention scope into actionable instructions that guide the generator to revise target components, update dependencies, and preserve valid content ([2, 7] (Pu et al. 2025)).
We annotate 112 human-annotated idea–feedback pairs, and evaluate six baselines, revealing a diagnosis–intervention mismatch in scientific ideation: feedback may target a different scope from the one that produces a higher-quality successor idea. To bridge this gap, as shown in Figure 1, we introduce Predictive Routing of Interventions by Modeling Effects, PRIME, a plug-and-play feedback-to-revision decision layer that can be inserted between an existing feedback provider and idea generator. PRIME consists of three components: (i) Structural Attribution, which interprets feedback over the research problem, scientific hypothesis, solution, and their dependencies to distinguish where a weakness manifests from its potential structural sources; (ii) Counterfactual Routing, which constructs candidate interventions covering the feedback-localized component, its potential upstream causes, cross-component dependencies, and problem-reframing directions, and uses lightweight successor rollouts to evaluate their feedback resolution, prospective utility, and structural risks before selecting an intervention scope; and (iii) Intervention Execution, which translates the selected intervention into an ordered, generator-facing revision plan specifying the target content, concrete semantic operations, preservation constraints, and required dependency updates. Through these three components, PRIME invokes the original idea generator to produce a complete successor idea. We evaluate PRIME on four benchmarks. Compared with direct feedback updating, PRIME-Refinement and PRIME-Evolution improve 14/16 and 15/16 metrics, respectively, and jointly achieve the best results on 10/16 metrics, showing that effective revision requires deciding not only how to address feedback, but also where to intervene.
Figure 1. Position–Execute Iterative Refinement. PRIME structures each idea into problem (P), hypothesis (H), and solution (S), positions feedback across their components and relations, selects an action, remove, revise, or replace; rewrites the prompt with explicit reasoning to iteratively generate improved ideas.
Our main contributions and findings are: (i) We formulate the diagnosis–intervention mismatch in scientific ideation, where a feedback-localized weakness does not uniquely determine the revision scope, and validate it through a controlled audit of 114 human-annotated idea–feedback pairs; (ii) We propose PRIME, a plug-and-play feedback-to-revision decision layer that performs revision planning through Structural Attribution, Counterfactual Routing, and Intervention Execution without replacing the underlying ideator; (iii) Across four benchmarks, two ideators, and both refinement and evolution settings, PRIME jointly achieves the best results on 20/32 metric ideator combinations; in the intervention audit, PRIME-Evolution further achieves the highest successor utility of 64.16 and the lowest route regret of 3.93.
Related Works
Iterative Scientific Ideation and Direct Revision. As shown in Figure 2(a,Figure 2b), Iterative scientific ideation systems improve candidate ideas through repeated critique, review, refinement, or evolution. General LLM self-improvement methods, such as Self-Refine and Reflexion, show that verbal feedback can drive iterative revision (Shinn et al. [38]). In scientific ideation, ResearchAgent, IdeaSynth, AI Scientist, Co-Scientist, Chain of Ideas, Many Heads, and SciDeator employ reviewing agents, literature-grounded critiques, rubric-based judgments, or multi-agent interactions to improve generated proposals ([2, 10] (Lu et al. 2024; Li et al. 2025; Su et al. 2025a)). These systems demonstrate the value of evaluation-driven iteration, but they use evaluation as a direct revision signal, for example, by rewriting the entire proposal, revising its weakest aspect, or continuing along the current trajectory. PRIME explicitly models the diagnosis-to-intervention decision instead of treating the diagnosed locus as the revision target.
Figure 2. Feedback-guided revision through structured reasoning. (a) Holistic and localized revision: decisions are implicit in generation or tied to feedback-localized weaknesses. (b) Workflow-coupled revision: actions are selected from a system-specific operation space. (c) Feedback-reasoned revision: diagnosis revision mismatch is explicitly modeled to guide revision.
Structured State and Graph-Based Edit Control. Structured ideation methods reduce the opacity of text-only revision by exposing intermediate idea states. SciPIP, KG-CoI, Graph2Idea, and IdeaSynth ground ideation in paper graphs, knowledge graphs, graph-structured contexts, or decomposed idea facets ([43, 45] (Li, Tu, and Han 2026; Pu et al. 2025)). Most closely related to our setting, EIG represents an evolving proposal as an idea graph and learns edit-and-commit operations over localized graph structures (Dong, Li, and Lin [7]). This line of work shows that structured weakness localization and editable state representations are useful. EIG controls localized edits within its idea-graph workflow, whereas PRIME selects an intervention scope without equating the diagnosed locus with the revision target.
Search and Evolution over Idea Spaces. Search- and evolution-based methods broaden scientific ideation by exploring multiple candidate ideas, branches, or trajectories. Nova, Chain of Ideas, AI Scientist, Co-Scientist, FlowPIE, Evoldeator, and EvoGens ([16, 15, 46, 4, 24, 48] (Du et al. 2026)) improve idea generation through planning, search, population evolution, checklist-guided optimization, or test-time exploration ([10, 42, 36] (Hu et al. 2024; Lu et al. 2024; Li et al. 2026)). Their selection targets are typically generated ideas, branches, population members, or trajectories. These methods do not explicitly ask whether, given the same feedback-localized weakness, the next update should take the form of a local refinement, an upstream rationale revision, or a broader problem-level evolution. PRIME instead selects a semantic intervention scope before successor generation, turning feedback into an explicit revision decision for the existing ideator.
Problem Formulation
We denote the research context, scientific idea generator, and current idea by , , and , respectively. Given evaluation criteria , an evaluator produces feedback . Rather than using directly as an edit instruction, we study how to select a revision by jointly considering the feedback and the structural context of the current idea.
The feasible intervention set is . Each specifies a revision scope and a corresponding revision, inducing a potential successor . The feedback-to-revision decision is
where measures feedback resolution and scientific quality, while captures revision risks such as loss of valid content, dependency inconsistency, and task drift. Conditioning the intervention set on both and captures that the same feedback may admit different revisions under different idea structures, while the expectation reflects that their successor effects are unobserved before execution.
The feedback-localized locus is , and a diagnosis-aligned intervention satisfies . Diagnosis–intervention mismatch occurs when an intervention with achieves a higher objective value than . Thus, the diagnosed locus informs intervention selection but does not by itself determine the revision scope.
PRIME
Overview
PRIME approximates the feedback-to-revision decision in Equation (1) through three stages. Structural Attribution interprets feedback over the problem, hypothesis, solution, and their dependencies without fixing a revision target; Counterfactual Routing compares alternative intervention scopes using their projected successor effects; and Intervention Execution translates the selected intervention into an ordered revision plan for the original generator. PRIME therefore operates as a plug-in decision layer without modifying the feedback provider or idea generator.
Structural Attribution
Feedback identifies an observable weakness but does not uniquely reveal its structural source. Structural Attribution therefore interprets over the structure of without selecting an intervention scope. PRIME normalizes into , which contains an improvement goal, supporting evidence, preservation requirements, and revision constraints. The current idea is then represented as
where , , and denote the research problem, scientific hypothesis, and solution, respectively. The dependencies require the hypothesis to address the problem and the solution to operationalize or test the hypothesis.
PRIME interprets through six structural views . The first three views examine individual components, and examine adjacent dependencies, and examines the complete reasoning chain. For each , PRIME produces one attribution record containing supporting evidence, the observed failure, a plausible structural explanation, the corresponding repair requirement, downstream consequences, severity, and confidence. The module output is
The records in provide diagnostic evidence rather than revision actions: they describe which structural conditions are deficient without prescribing which components should be modified. Consequently, one attribution may support multiple intervention scopes in Counterfactual Routing.
Counterfactual Routing
Structural Attribution provides evidence for revision but does not determine which intervention will produce a better successor. Counterfactual Routing therefore constructs candidate interventions and compares their projected effects before selecting one for execution.
Candidate Intervention Construction
An intervention separates where to revise from how to revise. Its scope satisfies , where each scope is an edit mask. The mask covers the problem framing, scope, bottleneck, and objective; covers the hypothesis and its mechanism; and covers the solution, its mechanism, and validation. Composite masks permit joint revisions to the corresponding components, while fields outside the selected mask are preserved. Dependencies are represented through explicit requirements and risks rather than automatic dependency closure.
The revision within a scope is specified by an ordered action plan composed of Add, Remove, Revise, and Replace operations. Each step identifies its target component and field, concrete instruction, success condition, and dependencies on preceding steps. Add introduces missing content, Remove deletes unsupported content, Revise modifies an existing commitment while retaining its role, and Replace substitutes it with a new commitment.
For interventions containing , the problem operation is Update or Switch. Update changes the problem framing while preserving its identity, whereas Switch moves to another context-grounded problem and is available only in evolution settings. Interventions not containing use None.
Problem-Reference Conditioning
Interventions involving are conditioned on a persistent Problem Pool , which maintains the active problem identity and grounded alternative references. A P-Update uses the active reference as an identity anchor, whereas a P-Switch uses an alternative reference as its target. The pool only supplies reference conditions for candidate construction and does not rank interventions. PRIME-Refinement uses a fixed reference policy and disables P-Switch; PRIME-Evolution uses an open policy and additionally constructs P-Switch candidates from grounded alternatives in . Both variants otherwise use the same attribution, routing, and execution procedure. The open policy permits but does not force evolution: only a selected P-Switch realizes an evolution step.
Using the structural representation , normalized feedback , and attribution , PRIME constructs
Each candidate additionally cites supporting attributions and carries preservation requirements and dependency requirements . The supporting attribution need not have the same label as : evidence from one structural view may support a local, upstream, or joint intervention.
Counterfactual Evaluation and Selection. For each candidate , PRIME generates a lightweight structured successor
The rollout applies the declared actions while copying fields outside their targets unchanged. It approximates the potential successor in Equation (1), but is not the final idea produced by .
A routing evaluator estimates the prospective utility and revision risk of each rollout. The former captures feedback resolution and scientific quality, while the latter captures unsupported changes, infeasibility, dependency breakage, task drift, overcomplexity, and uncertainty. Candidates violating predefined scope, preservation, risk, or problem-reference constraints are removed. PRIME then selects
If no candidate passes the strict validity constraints, PRIME selects the highest-ranked executable candidate. Counterfactual Routing outputs the selected intervention and its projected successor for Intervention Execution.
Intervention Execution
The selected intervention must be translated into instructions that the original generator can execute without reconsidering its scope. PRIME therefore compiles and its projected successor into a generator-facing specification and invokes once:
The execution specification contains the selected scope , ordered action plan , preservation requirements , and dependency requirements . The projected successor provides a semantic target for these actions rather than a draft to be copied.
The generator is instructed to execute the selected actions in order, preserve content outside , satisfy the specified dependencies, and return a complete successor idea in the same format as . The compiled specification instructs not to introduce additional revision targets from the raw feedback or to modify content outside the selected scope. Because PRIME communicates with through without modifying its parameters, it can be combined with different instruction-conditioned idea generators.
Experiments
Experimental Setup
Baselines. We compare PRIME with six feedback-to-revision strategies. (1) Flat Feedback directly injects a single natural-language improvement direction into the subsequent generation task. (2) Problem-First H/S fixes the initial problem and revises only the hypothesis and solution. (3) Problem Regeneration updates only the problem used to guide the next iteration. (4) Direct Feedback Decomposition generates an ordered, free-form list of revisions and directly executes the highest-ranked item. (5) Feedback Reasoning separates feedback generation from revision planning and serves as a simplified PRIME variant without counterfactual cross-scope comparison. (6) Graph-Based grounds feedback and localized edits in a structured problem–hypothesis–solution graph, while its reasoning variant explicitly selects a graph edit before generation. For each baseline, we evaluate two models: Direct Prompting and SciMon ([41]).
Benchmark. We evaluate PRIME on four scientific idea benchmarks: AI Idea Bench ([31]), IdeaBench ([11]), LiveIdeaBench ([35]), and MLR-Bench ([3]). These benchmarks follow two evaluation protocols. The first assesses idea reconstruction by comparing generated ideas with target ideas from real papers. AI Idea Bench uses I2I, I2T, and MCQ to evaluate idea matching, topic alignment, and identification of key research content. IdeaBench further uses the novelty score NIS and feasibility score FIS to assess innovation relative to existing work and practical implementability. The second category follows a paper-review-style protocol, with multiple LLM judges directly assessing overall idea quality. LiveIdeaBench evaluates originality, feasibility, and clarity, focusing on whether an idea forms a complete, coherent, and worthwhile research plan. MLR-Bench evaluates clarity, novelty, feasibility, and significance to determine whether an idea approaches the quality expected of a machine learning research paper. For fair comparison, all methods use the same scientific context, reference materials, feedback provider, Qwen3.5-4B generator, 20 iterations, and a generation budget of 2,048 tokens on each benchmark. For all experiments, we compute the score for each idea generated across all iterations and then report the average score.
Main Results
The main experiment examines whether explicitly modeling the feedback-to-revision decision improves the quality of iteratively generated successor ideas across ideators and revision settings. Table 2 compares five general update policies with PRIME-Refinement and PRIME-Evolution under Direct Prompting and SciMon. The Graph-Based block further tests whether PRIME can be integrated with a structured ideator. Using random seed 101, we sample 50 tasks from each benchmark and perform 20 feedback-to-revision iterations for every task under each policy–ideator configuration. All methods share the same Qwen3.5-4B base model, research contexts, feedback provider, generation budget, and benchmark-specific evaluation protocol. Across the five general policies and two PRIME variants, PRIME achieves the best results on 10/16 metrics under both Direct Prompting and SciMON. Compared with Flat Feedback, PRIME-Refinement and PRIME-Evolution improve 14/16 and 15/16 metrics under Direct Prompting, and 11/16 and 13/16 under SciMON, respectively. PRIME-Refinement leads all three AI Idea Bench metrics under Direct Prompting, while PRIME-Evolution achieves the highest MLR-Bench overall scores under both ideators (7.260 and 7.120). Although results on LiveIdeaBench are mixed, adding PRIME-Refinement to the Graph-Based ideator improves all LiveIdeaBench and MLR-Bench metrics, increasing the MLR-Bench overall score from 6.589 to 7.060. These results support PRIME as an effective plug-in layer for feedback-to-revision routing.
| Update Policy | Ideator | I2I | I2T | IMCQ | SemSim | Overlap | NIS | FIS | Orig. | Feas. | Clar. | Cons. | Clar. | Nov. | Feas. | Sig. | Overall |
| Flat Feedback | Direct Prompting | 2.656 | 4.690 | 0.744 | 0.647 | 1.774 | 0.885 | 0.165 | 5.506 | 4.810 | 5.112 | 7.192 | 6.338 | 6.657 | 5.689 | 7.061 | 6.411 |
| SciMON | 2.648 | 4.560 | 0.734 | 0.599 | 1.540 | 0.996 | 0.029 | 6.50 | 2.528 | 3.776 | 7.839 | 6.533 | 7.676 | 5.113 | 7.673 | 6.561 | |
| Problem-First H/S | Direct Prompting | 2.685 | 4.716 | 0.759 | 0.647 | 2.011 | 0.937 | 0.164 | 6.366 | 4.683 | 5.582 | 7.734 | 6.657 | 6.905 | 6.237 | 7.567 | 6.896 |
| SciMON | 2.621 | 4.469 | 0.707 | 0.642 | 1.660 | 0.997 | 0.017 | 6.938 | 2.426 | 3.637 | 7.240 | 6.144 | 7.774 | 4.506 | 7.508 | 6.273 | |
| Problem Regeneration | Direct Prompting | 2.519 | 4.556 | 0.766 | 0.647 | 1.699 | 0.864 | 0.186 | 6.180 | 4.171 | 4.916 | 7.577 | 6.660 | 7.052 | 5.703 | 7.595 | 6.750 |
| SciMON | 2.163 | 3.799 | 0.684 | 0.641 | 1.433 | 0.994 | 0.003 | 6.581 | 2.282 | 3.287 | 6.989 | 6.139 | 7.865 | 4.502 | 7.405 | 6.208 | |
| Feedback Decomp. | Direct Prompting | 2.609 | 4.668 | 0.736 | 0.649 | 2.029 | 0.799 | 0.238 | 5.616 | 4.261 | 4.626 | 7.001 | 6.170 | 6.615 | 5.684 | 7.092 | 6.357 |
| SciMON | 2.550 | 4.257 | 0.717 | 0.620 | 1.858 | 0.994 | 0.015 | 6.798 | 2.170 | 3.501 | 7.475 | 6.373 | 8.077 | 4.396 | 7.882 | 6.504 | |
| Feedback Reasoning | Direct Prompting | 2.687 | 4.744 | 0.767 | 0.648 | 1.873 | 0.922 | 0.528 | 6.403 | 4.622 | 5.072 | 7.827 | 6.614 | 7.333 | 5.879 | 7.899 | 7.055 |
| SciMON | 2.480 | 4.700 | 0.720 | 0.646 | 1.700 | 0.940 | 0.600 | 6.400 | 4.800 | 5.160 | 7.640 | 6.400 | 7.400 | 5.800 | 7.80 | 6.900 | |
| PRIME-Refinement | Direct Prompting | 3.490 | 4.857 | 0.939 | 0.655 | 2.500 | 0.958 | 0.271 | 6.060 | 3.900 | 4.800 | 8.180 | 6.540 | 7.040 | 5.880 | 8.060 | 7.100 |
| SciMON | 2.735 | 4.735 | 0.755 | 0.650 | 2.082 | 0.980 | 0.163 | 5.449 | 2.388 | 3.327 | 7.898 | 6.918 | 7.980 | 4.878 | 8.224 | 6.939 | |
| PRIME-Evolution | Direct Prompting | 3.460 | 4.840 | 0.920 | 0.651 | 2.400 | 0.920 | 0.200 | 6.220 | 4.660 | 5.340 | 8.320 | 6.820 | 7.020 | 6.140 | 7.960 | 7.260 |
| SciMON | 2.667 | 4.688 | 0.792 | 0.657 | 1.918 | 1.000 | 0.143 | 5.469 | 2.347 | 3.510 | 8.020 | 7.080 | 7.820 | 5.180 | 8.240 | 7.120 | |
| Graph-Based | Direct Prompting | 3.184 | 4.953 | 0.884 | 0.665 | 2.518 | 0.911 | 0.051 | 5.193 | 3.683 | 3.306 | 8.015 | 5.707 | 6.966 | 5.228 | 7.898 | 6.589 |
| w Reasoning | 3.377 | 4.868 | 0.866 | 0.660 | 2.337 | 0.852 | 0.264 | 5.515 | 4.139 | 4.043 | 8.044 | 6.170 | 7.088 | 5.607 | 7.928 | 6.870 | |
| w PRIME-Refinement | 3.440 | 4.940 | 0.840 | 0.670 | 2.340 | 0.920 | 0.220 | 5.940 | 3.940 | 4.720 | 8.240 | 6.460 | 7.040 | 5.640 | 8.020 | 7.060 | |
| w PRIME-Evolution | 3.460 | 4.900 | 0.920 | 0.648 | 2.300 | 0.840 | 0.320 | 6.100 | 4.240 | 5.040 | 8.100 | 6.480 | 6.880 | 5.780 | 7.900 | 7.020 |
Table 2. Main results across four scientific idea-generation benchmarks. Rows compare feedback-to-revision policies across two ideators, with the Graph-Based block additionally reporting reasoning and PRIME-enhanced variants. PRIME-Refinement fixes the problem reference, whereas PRIME-Evolution permits context-grounded problem switching. All methods use the same experimental settings within each benchmark. All metrics are higher-is-better.
Statistical Significance and Multi-Judge Consistency. To ensure robust evaluation and reduce judge-specific bias, we randomly sample 50 generated ideas from each of the eight result groups, yielding 400 sampled ideas in total. Each idea is independently scored by three LLM judges, resulting in 1,200 evaluation records. We report exact agreement, ; adjacent agreement, ; and the Pearson correlation between judges. The resulting agreement scores, EA = 68.7%, AA = 78.9%, and Pearson , indicate strong inter-judge consistency.
Diagnostic Intervention Mismatch
We test whether the locus identified by feedback reliably determines the intervention scope that produces a better successor idea. From 114 human-annotated idea–feedback pairs, we retain 112 complete instances from IdeaBench, LiveIdeaBench, and MLR-Bench, comprising 42, 42, and 28 instances, respectively. For each instance, we hold the idea, feedback, context, generator, and generation budget fixed, and vary only the intervention scope over , yielding 672 scope-conditioned outcomes. Human annotators identify the diagnostic locus using only the original idea and feedback, without observing successors or policy decisions, while blind evaluators independently assess the resulting successors. A selection split defines the -near-optimal intervention set, and a separate evaluation split measures held-out mismatch and regret. Panel (a) audits whether diagnosis identifies a near-optimal intervention, while Panel (b) applies seven update policies to the same outcome bank to evaluate feedback-to-intervention routing.
Table 3 shows a substantial diagnosis–intervention mismatch. At the primary threshold , the human-ground-truth diagnosis falls outside the near-optimal intervention set in 65.2% of instances; the mismatch remains 40.2% when all admissible diagnostic loci are considered and replicates on the held-out split in 43.8% of instances. Directly following the diagnosed locus incurs an average held-out regret of 3.79 points (95% CI: [2.74, 4.93], ), with similar mismatch rates across model and individual-human diagnoses (61.6%–71.2%). For routing, PRIME-Evolution achieves the highest Target Hit@2 (43.8%), Benefit@2 (65.2%), and successor utility (64.16%), while obtaining the lowest Repr@2 (16.1%), Harm@2 (2.7%), and evaluation regret (3.93). PRIME-Refinement does not exhibit the same advantage, suggesting that restricting problem-level alternatives can be limiting when broader intervention is beneficial. Overall, the results show that feedback localization alone is insufficient and that comparing the projected effects of alternative intervention scopes improves revision routing.
| Source or Setting | [95% CI] | [95% CI] | Rep. [95% CI] | 95% CI | ||||||
| Human Groundtruth | 112 | 73.2 | 72.3 | 65.2 [56.2, 74.1] | 41.1 | 40.2 [31.2, 49.1] | 43.8 [34.8, 52.7] | 3.79 | [2.74, 4.93] | .0001 |
| Model Attribution | 112 | 73.2 | 73.2 | 61.6 [52.7, 70.5] | 38.4 | 42.9 [33.9, 51.8] | 25.0 [17.0, 33.0] | 1.13 | [−0.06, 2.42] | .0386 |
| Human 1 | 112 | 73.4 | 73.4 | 68.8 [59.6, 77.1] | 38.5 | 65.1 [56.0, 74.3] | 28.4 [20.2, 37.6] | 1.15 | [0.21, 2.15] | .0137 |
| Human 2 | 112 | 76.6 | 76.6 | 71.2 [62.2, 79.3] | 38.7 | 68.5 [59.5, 77.5] | 29.7 [21.6, 38.7] | 1.03 | [0.05, 2.02] | .0212 |
| Human–Human Reliability | 77 | 67.5 | 66.2 | 62.3 [51.9, 72.7] | 29.9 | 62.3 [51.9, 72.8] | 20.8 [11.7, 29.9] | 0.05 | [−1.01, 1.14] | .4356 |
| Update Policy | Diag. Match | Target Hit@ | RepErr@ | Benefit@ | Harm@ | |||||
| Flat Feedback | 112 | 23.2 | 37.5 | 19.6 | 57.1 | 8.9 | 63.83 | 3.61 | 4.25 | |
| Problem-First H/S | 112 | 16.1 | 37.5 | 22.3 | 56.2 | 10.7 | 63.81 | 3.58 | 4.27 | |
| Problem Regeneration | 112 | 22.3 | 24.1 | 43.8 | 28.6 | 28.6 | 60.69 | 0.46 | 7.40 | |
| Feedback Decomp. | 112 | 11.6 | 34.8 | 25.0 | 53.6 | 13.4 | 63.47 | 3.24 | 4.62 | |
| Feedback Reasoning | 112 | 16.1 | 38.4 | 24.1 | 55.4 | 12.5 | 63.60 | 3.37 | 4.49 | |
| PRIME-Refinement | 112 | 45.5 | 36.6 | 25.0 | 50.0 | 12.5 | 62.52 | 2.29 | 5.56 | |
| PRIME-Evolution | 112 | 77.7 | 43.8 | 16.1 | 65.2 | 2.7 | 64.16 | 3.93 | 3.93 |
Table 3. Diagnosis intervention location mismatch and feedback-to-intervention routing. Panel (a) establishes whether the diagnostic location identifies the most effective intervention location. Panel (b) evaluates how six update policies convert the same feedback into intervention actions on the 112 complete annotated instances.
Base-Model and Ideators Generalization
We evaluate whether PRIME generalizes across both base models and ideation workflows through two controlled substitutions on AI Idea Bench. First, we replace the primary base model with Llama while retaining Direct Prompting as the ideator. Second, we retain the primary base model but replace Direct Prompting with CoI-Agent. All other settings, including the update policies, feedback provider, generation budget, and evaluation protocol, remain unchanged. Table 4 reports the results. Under the Llama substitution, PRIME obtains the best I2I, I2T, and IMCQ scores across its refinement and evolution variants. Under the CoI-Agent substitution, PRIME-Evolution achieves the best IMCQ and a near-best I2T score, although Flat Feedback remains stronger on I2I. These results show that PRIME’s gains transfer strongly across base models, while its benefits under ideator substitution are metric-dependent.
| Update Policy | I2I | I2T | IMCQ |
| (a) Base-model transfer: Llama with Direct Prompting | |||
| Flat Feedback | 3.114 | 4.374 | 0.787 |
| Problem-First H/S | 3.094 | 4.231 | 0.821 |
| Problem Regeneration | 2.876 | 4.238 | 0.816 |
| Direct Feedback Decomp. | 3.122 | 4.415 | 0.821 |
| Feedback Reasoning | 3.573 | 4.568 | 0.883 |
| PRIME-Refinement | 3.740 | 4.700 | 0.900 |
| PRIME-Evolution | 3.740 | 4.720 | 0.860 |
| (b) Ideator transfer: CoI-Agent with the original base model | |||
| Flat Feedback | 3.402 | 4.964 | 0.840 |
| Problem-First H/S | 3.254 | 4.897 | 0.811 |
| Problem Regeneration | 2.792 | 4.684 | 0.790 |
| Direct Feedback Decomp. | 3.386 | 4.809 | 0.861 |
| Feedback Reasoning | 3.118 | 4.984 | 0.862 |
| PRIME-Refinement | 3.250 | 4.917 | 0.854 |
| PRIME-Evolution | 3.286 | 4.980 | 0.898 |
Table 4. Generalization results on AI Idea Bench. Panel (a) replaces the base model with Llama under Direct Prompting, while Panel (b) replaces Direct Prompting with CoI-Agent using the original base model.
Ablation
We conduct ablation studies to examine the key design choices of PRIME by removing counterfactual routing, upstream intervention candidates, problem-level candidates,
and the risk penalty from the full model. As reported in Table 5, removing counterfactual routing replaces successor-effect comparison with diagnosis-based intervention selection, removing upstream or problem-level candidates restricts the intervention space, and removing the risk penalty selects interventions solely according to their predicted utility. Counterfactual routing is the most influential component: removing it decreases Quality from 3.463 to 1.960, increases Regret from 0.426 to 1.929, and decreases Consistency from 3.173 to 1.773. Excluding upstream and problem-level candidates also consistently reduces successor quality and increases routing regret. In contrast, removing the risk penalty results in only a minor performance degradation, suggesting that projected successor utility dominates routing decisions on this benchmark, while risk modeling primarily provides a modest regularization effect.
| Variant | Quality | Regret | Cons. |
| PRIME | 3.463 | 0.426 | 3.173 |
| w/o Counterfactual Routing | 1.960 | 1.929 | 1.773 |
| w/o Dependency Closure | 3.490 | 0.399 | 3.211 |
| w/o Upstream Candidates | 3.177 | 0.712 | 2.909 |
| w/o Problem Candidates | 3.110 | 0.779 | 2.831 |
| w/o Risk Penalty | 3.459 | 0.430 | 3.158 |
Table 5. Ablation results for PRIME. Quality, Regret, and Cons. denote successor utility, regret relative to the post-hoc best intervention, and structural consistency, respectively. Higher Quality and Cons. and lower Regret indicate better performance.
Conclusion
In this paper, we identify reference-frame ambiguity as a key challenge in context-driven scientific idea generation, where ideas may target different problems and flat comparison may conflate problem selection with solution quality. We propose PRIME, a plug-and-play framework that recovers latent problem frames, organizes ideas in a dynamic problem tree, and provides problem-conditioned evaluation and feedback. Our results show that feedback localization alone is insufficient for selecting revision scope; by making both problem positioning and intervention routing explicit, PRIME enables more reliable comparison, actionable feedback, and controllable iterative improvement for scientific ideation.
References
- [1]Alkan, A. K.; Sourav, S.; Jablonska, M.; Astarita, S.; Chakrabarty, R.; Garuda, N.; Khetarpal, P.; Pióro, M.; Tanoglidis, D.; Iyer, K. G.; Polimera, M. S.; Smith, M. J.; Ghosal, T.; Huertas-Company, M.; Kruk, S.; Schawinski, K.; and Ciucă, I. 2025. A Survey on Hypothesis Generation for Scientific Discovery in the Era of Large Language Models. arXiv:2504.05496.arxiv.org/abs/2504.05496
- [2]Baek, J.; Jauhar, S. K.; Cucerzan, S.; and Hwang, S.-w. J. 2025. ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models. In Proceedings of NAACL 2025.
- [3]Chen, H.; Xiong, M.; Lu, Y.; Han, W.; Deng, A.; He, Y.; Wu, J.; Li, Y.; Liu, Y.; and Hooi, B. 2026a. Mlr-bench: Evaluating ai agents on open-ended machine learning research. Advances in Neural Information Processing Systems, 38.arxiv.org/abs/2505.19955
- [4]Chen, J.; Mishra, B. D.; Nam, J.; Meng, R.; Pfister, T.; and Yoon, J. 2026b. MARS: Modular Agent with Reflective Search for Automated AI Research. arXiv preprint arXiv:2602.02660.arxiv.org/abs/2602.02660
- [5]Chen, W. 2025. Automated, Interpretable, and Scalable Scientific Machine Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 39(27): 28708–28708.
- [6]Deshwal, A. 2025. Adaptive Experimental Design to Accelerate Scientific Discovery and Engineering Design. Proceedings of the AAAI Conference on Artificial Intelligence, 39(27): 28709–28709.
- [7]Dong, J.; Li, B.; and Lin, W. 2026. Evolving Idea Graphs with Learnable Edits-and-Commits for Multi-Agent Scientific Ideation. arXiv:2605.04922.arxiv.org/abs/2605.04922
- [9]Gelles, R.; Kinoshita, V.; Musser, M.; and Dunham, J. 2024. Resource Democratization: Is Compute the Binding Constraint on AI Research? Proceedings of the AAAI Conference on Artificial Intelligence, 38(18): 19840–19848.
- [10]Gottweis, J.; Weng, W.-H.; Daryin, A.; Tu, T.; Sirkovic, P.; Myasovskyy, A.; Glowaty, G.; Weissenberger, F.; Orlandi, A.; Popovici, D.; Palepu, A.; Rong, K.; Tanno, R.; Saab, K.; Zhang, F.; Blum, J.; Carroll, A.; Kulkarni, K.; Tomašev, N.; Zverinski, D.; Rendulic, I.; Vedadi, E.; Hasler, F.; Rimanic, L.; Boia, M.; Budiselić, I.; Feinstein, B.; Bealiche, M.; Sheffer, T.; Freyberg, J.; Ratcliff, J.; Bertolli, O.; Chou, K.; Hassidim, A.; Gokturk, B.; Vahdat, A.; Guan, Y.; Dhillon, V.; Vaishnav, E. D.; Lee, B.; Costa, T. R. D.; Penadés, J. R.; Peltz, G.; Matias, Y.; Manyika, J.; Hassabis, D.; Xu, Y.; Kohli, P.; Pawlosky, A.; Karthikesalingam, A.; and Natarajan, V. 2026. Accelerating scientific discovery with Co-Scientist. Nature.DOI
- [11]Guo, S.; Shariatmadari, A. H.; Xiong, G.; Huang, A.; Kim, M.; Williams, C. M.; Bekiranov, S.; and Zhang, A. 2025. IdeaBench: Benchmarking large language models for research idea generation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, 5888–5899.arxiv.org/abs/2411.02429
- [12]Hasib, M. M.; Jo, S.; Sinha, H.; Song, J.; Das, A.; Liu, Z.; Galloway, H.; Huang, H.; Zhang, K.; Gao, S.-J.; et al. 2025. A Process-Centric Survey of AI for Scientific Discovery Through the EXHYTE Framework.DOI
- [15]Jiang, Z.; Schmidt, D.; Srikanth, D.; Xu, D.; Kaplan, I.; Jacenko, D.; and Wu, Y. 2025. Aide: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138.arxiv.org/abs/2502.13138
- [16]Jin, J.; Hu, Y.; Qiu, K.; Dai, Q.; Luo, C.; Dong, G.; Li, X.; Zhao, T.; Ma, X.; Zhang, G.; et al. 2026. Toward Generalist Autonomous Research via Hypothesis-Tree Refinement. arXiv preprint arXiv:2606.11926.arxiv.org/abs/2606.11926
- [17]Keya, F.; Rabby, G.; Mitra, P.; Vahdati, S.; Auer, S.; and Jaradeh, Y. 2025. SCI-IDEA: Context-Aware Scientific Ideation Using Token and Sentence Embeddings. arXiv:2503.19257.arxiv.org/abs/2503.19257
- [18]Kon, P. T. J.; Liu, J.; Zhu, X.; Ding, Q.; Peng, J.; Xing, J.; Huang, Y.; Qiu, Y.; Srinivasa, J.; Lee, M.; et al. 2025. Exp-bench: Can ai conduct a research experiments? arXiv preprint arXiv:2505.24785.arxiv.org/abs/2505.24785
- [19]Kulkarni, A.; Alotaibi, F.; Zeng, X.; Wu, L.; Zeng, T.; Yao, B. M.; Liu, M.; Zhang, S.; Huang, L.; and Zhou, D. 2025. Scientific hypothesis generation and validation: Methods, datasets, and future directions. arXiv preprint arXiv:2505.04651.arxiv.org/abs/2505.04651
- [23]Liu, H.; Huang, S.; Hu, J.; Zhou, Y.; and Tan, C. 2025a. Hy-ponbench: Towards systematic and principled benchmarking for hypothesis generation. arXiv preprint arXiv:2504.11524. Liu, Y.; Sharma, P.; Oswal, M.; Xia, H.; and Huang, Y. 2025b. PersonaFlow: Designing LLM-Simulated Expert Perspectives for Enhanced Research Ideation. In Proceedings of the ACM Conference.
- [24]Liu, Z.; Shi, Z.; Sang, Y.; He, B.; Lin, M.; Wei, T.; Wang, D.; Dumoulin, B.; Jin, W.; and Lu, H. 2026. Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams. arXiv preprint arXiv:2606.01770.arxiv.org/abs/2606.01770
- [26]Luo, Z.; Yang, Z.; Xu, Z.; Yang, W.; and Du, X. 2025. LLM4SR: A Survey on Large Language Models for Scientific Research. arXiv:2501.04306.arxiv.org/abs/2501.04306
- [27]Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in neural information processing systems, 36: 46534–46594.arxiv.org/abs/2303.17651
- [28]Mishra, S.; Sahu, G.; Pedersoli, M.; Charlin, L.; Dolz, J.; and Pal, C. 2026. AInstein: Can LLMs Solve Research Problems From Parametric Memory Alone? arXiv:2510.05432.arxiv.org/abs/2510.05432
- [29]Nigam, H.; Patwardhan, M.; Vig, L.; and Shroff, G. 2024. Acceleron: A Tool to Accelerate Research Ideation. arXiv (Cornell University).arxiv.org/abs/2403.04382
- [31]Qiu, Y.; Zhang, H.; Xu, Z.; Li, M.; Song, D.; Wang, Z.; and Zhang, K. 2025. AI idea bench 2025: A research idea generation benchmark. arXiv preprint arXiv:2504.14191.arxiv.org/abs/2504.14191
- [32]Radensky, M.; Shahid, S.; Fok, R.; Siangliulue, P.; Hope, T.; and Weld, D. S. 2024. Scideator: Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation. arXiv preprint arXiv:2409.14634.arxiv.org/abs/2409.14634
- [33]Radensky, M.; Shahid, S.; Fok, R.; Siangliulue, P.; Hope, T.; and Weld, D. S. 2026. Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation. arXiv:2409.14634.arxiv.org/abs/2409.14634
- [34]Reddy, C. K.; and Shojaee, P. 2025. Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges. Proceedings of the AAAI Conference on Artificial Intelligence, 39(27): 28601–28609.arxiv.org/abs/2412.11427
- [35]Ruan, K.; Wang, X.; Hong, J.; Wang, P.; Liu, Y.; and Sun, H. 2024. LiveIdeaBench: Evaluating LLMs’ Divergent Thinking for Scientific Idea Generation with Minimal Context. arXiv preprint arXiv:2412.17596.arxiv.org/abs/2412.17596
- [36]Sauter, A.; Zhao, Y.; Urbani, J.; Hu, W.; Meng, Z.; Zhou, L.; Yan, X.; and Lyu, Y. 2026. EvolIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning. arXiv preprint arXiv:2603.21728.arxiv.org/abs/2603.21728
- [38]Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36: 8634–8652.arxiv.org/abs/2303.11366
- [41]Wang, Q.; Downey, D.; Ji, H.; and Hope, T. 2024a. SciMON: Scientific Inspiration Machines Optimized for Novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 279–299. Bangkok, Thailand: Association for Computational Linguistics.
- [42]Wang, Q.; Wang, H.; Chen, L.; Yang, Z.; Chen, G.; Alinejad-Rokny, H.; Li, H.; Lin, Y.; and Yang, M. 2026. FlowPIE: Test-Time Scientific Idea Evolution with Flow-Guided Literature Exploration. arXiv preprint arXiv:2603.29557.arxiv.org/abs/2603.29557
- [43]Wang, W.; Gu, L.; Zhang, L.; Luo, Y.; Dai, Y.; Shen, C.; Xie, L.; Lin, B.; He, X.; and Ye, J. 2024b. SciPIP: An LLM-based Scientific Paper Idea Proposer. arXiv preprint arXiv:2410.23166.arxiv.org/abs/2410.23166
- [44]Xiong, G.; Xie, E.; Shariatmadari, A. H.; Guo, S.; Bekiranov, S.; and Zhang, A. 2024a. Improving Scientific Hypothesis Generation with Knowledge Grounded Large Language Models. arXiv:2411.02382.arxiv.org/abs/2411.02382
- [45]Xiong, G.; Xie, E.; Shariatmadari, A. H.; Guo, S.; Bekiranov, S.; and Zhang, A. 2024b. Improving Scientific Hypothesis Generation with Knowledge Grounded Large Language Models. arXiv preprint arXiv:2411.02382.arxiv.org/abs/2411.02382
- [46]Yamada, Y.; Lange, R. T.; Lu, C.; Hu, S.; Lu, C.; Foerster, J.; Clune, J.; and Ha, D. 2025. The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066.arxiv.org/abs/2504.08066
- [47]Yang, J.; Ji, F.; Lai, Z.; Cui, Z.; Ouyang, G.; Jiang, Q.; Zhang, F.; Peng, M.; Xie, Q.; Nakov, P.; and Xie, Z. 2026. LabGuard: Grounding Natural-Language Laboratory Rules into Runtime Guards for Embodied Laboratory Agents. arXiv:2606.31045.arxiv.org/abs/2606.31045
- [48]Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 11809–11822.arxiv.org/abs/2305.10601
- [49]Yuksekgonul, M.; Koceja, D.; Li, X.; Bianchi, F.; McCaleb, J.; Wang, X.; Kautz, J.; Choi, Y.; Zou, J.; Guestrin, C.; et al. 2026. Learning to discover at test time. arXiv preprint arXiv:2601.16175.
- [50]Zhang, Q. 2025. Knowledge-driven Scientific Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 39(27): 28736–28736.DOI
- [51]Zhao, K.; Lin, W.; Zheng, Q.; Xu, F.; and Li, Y. 2025. Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network. arXiv preprint arXiv:2511.02238.arxiv.org/abs/2511.02238

