DAGLoop: Aligning Method Design with Experimental Feedback in Automated AI Research

Introduction

Improving a machine learning method requires repeated decisions about what to change, what to test, and how to interpret the result. Recent research agents automate this process by proposing candidate methods, implementing them, running experiments, and using the outcomes to guide subsequent revisions. MLAgentBench studies agents that conduct machine learning experiments, while The AI Scientist extends automation to longer research workflows and agentic tree search [13, 16] (Yamada et al. 2025). AlphaEvolve uses executable evaluators to guide program and algorithm discovery [18]. We study the executable method-improvement setting: given a research task, a baseline, an experimental environment, and a finite budget, an agent seeks to improve performance under a fixed task evaluator.

Experimental feedback determines which candidates the agent explores next. Optimization by PROmpting (PRO) generates candidates from scored solutions; FunSearch selects evaluated pro grams for further search [20] (Yang et al. 2024). Richer feedback includes MetaReflection’s memory of past reflections and Trace’s execution traces for parameter updates [10, 7]. Our focus is the relation between this feedback and the design decisions being revised. A candidate may combine a data transformation, a representation, an objective, and an optimization rule. Its overall performance measures their combined effect, but does not establish which decisions produced their expected effects or whether their required conditions held. The evaluator scores the complete candidate, whereas revision must determine which constituent decisions to retain, revise, or test further.

Ablation studies examine decisions by comparing implementations with and without them; recent benchmarks study whether language models can plan informative ablations [1] (Zhao et al. 2025). However, testing each decision independently can remove the conditions needed for its effect. Suppose $z_1$ constructs a representation and $z_2$ introduces an objective that uses it. The representation may provide a necessary intermediate property without improving the task metric alone, while the objective may be meaningless without it. Rejecting $z_1$ for lacking standalone gain prevents a meaningful test of $z_2$; evaluating only their composition leaves each decision’s contribution unresolved. Decision-level evidence therefore requires experiments that preserve the prerequisites of the decision being tested (Figure 1).

We present DAGLoop, a general-purpose research system that addresses executable method-improvement problems through proposal, implementation, experimentation, and revision. It constructs targeted experiments from a candidate method’s design. For each decision, it translates the rationale into a testable expectation: the effect the decision should produce and its required conditions. A targeted experiment specifies what to compare and measure, forming an experimental specification that connects design reasoning to evidence. These specifications define a DAG, whose nodes associate decisions with tests and whose edges encode their dependencies. Each experiment compares a control implementation containing the prerequisites with a treatment implementation that additionally applies the decision: $z_1 + z_2$ against $z_1$ in the example. A decision that provides a prerequisite, such as $z_1$, is assessed through the intermediate property it is intended to provide. Effects requiring several decisions are tested jointly, retaining their external prerequisites and attributing experimental support to the group of decisions tested together.

After checking execution validity, DAGLoop records each outcome alongside the decision, expected effect, required conditions, and saved control implementation. This structured research memory guides the agent to retain supported decisions, revise unsupported ones, and plan further experiments when evidence is unresolved. Failed implementations provide debugging information. Revised decisions are assembled into a subsequent candidate and evaluated with the same official evaluator. Targeted experiments guide revision; complete-candidate evaluation measures improvement.

We evaluate DAGLoop on 36 AutoLab tasks spanning CPU optimization, CUDA kernels, program synthesis, and model development (Xu et al., 2026), comparing with Terminus-2 using the same language model. Our contributions are threefold:

• We identify a mismatch between evaluating complete candidates and revising individual decisions, motivating experiments that retain prerequisites for decision-level evidence.

• We propose DAGLoop, a research system that translates design rationales into experimental specifications, constructs a dependency DAG to plan targeted ablations, and uses structured research memory to guide evidence-based revision and further experimentation.

• Experiments on 36 AutoLab tasks demonstrate method improvement across diverse executable problems. DAGLoop achieves a mean score of 0.6459, compared with 0.5831 for Terminus-2 using the same language model, with higher means on both CPU and GPU tasks. Additional comparisons with reasoning disabled cover GLM-5.2 and DeepSeek-V4-Flash.

Method

DAGLoop organizes method proposal, implementation, experimentation, and revision into a research loop. The system turns a candidate’s design into experimental specifications, uses their dependencies to construct targeted comparisons, and records the outcomes for revision (Figure 2). The assembled candidate is evaluated by the official task evaluator.

Task Setup and Experimental Specifications

Starting from the user’s research question, DAGLoop clarifies objectives and constraints with the user, retrieves relevant literature, and proposes a method. The user confirms the research direction and method. With an executable baseline, evaluator, environment, and budget established, the system proceeds to implementation, experimentation, and revision.

An executable research task is

where $q$ describes the task, $M_0$ is an executable baseline, $\mathcal{E}$ is the experimental environment, and $B$ is the research budget. The official evaluator returns a raw (native) task metric $m_q(M)$, such as accuracy or execution time, and a task score $R_q(M)$. We define $J_q$ so that larger values indicate better performance: $J_q(M)=m_q(M)$ for maximization tasks and $J_q(M)=-m_q(M)$ for minimization tasks. The score $R_q$ follows the official reward rule, including its thresholds and clipping.

Let $\tau$ be the research trajectory, including proposals, implementations, experiments, and revisions, and let $\mathcal{C}(\tau)$ contain the complete executable candidates it evaluates. We seek

Task checks enforce correctness and task constraints. Cost includes planning, implementation, targeted tests, repair, repeated measurements, and final evaluation. Improvement means $J_q(\widehat{M})>J_q(M_0)$. The baseline remains available when no evaluated candidate improves it.

A candidate method $M$ consists of design decisions $Z=\{z_i\}_{i=1}^{n}$. The complete-candidate metric $J_q(M)$ evaluates their combined performance, whereas revision requires deciding which $z_i$ to retain or change. To test the effect expected from a particular decision, we associate it with an experimental specification linking its rationale, expected effect, conditions, and test:

where $r_i$ is the rationale, $e_i$ the expected effect, and $D_i$ the required conditions. The experiment $x_i$ specifies the implementation change $\Delta_i$, execution procedure $\rho_i$, quantity to measure $o_i$, and validation rule $v_i$. The specification defines the measurement and criterion before execution; the experiment report supplies the observed value. A decision can provide an intermediate property $p_i$ required by another decision, with a corresponding property check.

Each node in the experimental DAG represents an executable component: one decision or a group of decisions tested together. Each component has an implementation and a specification. Tests assess either improvement in the task metric or an intermediate property required by another decision. We refer to these as performance tests and property checks, respectively. A checkpoint stores the implementation and artifacts needed to reconstruct a control or candidate.

From a rationale to a test. In the representation–objective example, the rationale for $z_1$ identifies a property $p_1$ needed by $z_2$. Its specification makes that property the expected effect and selects an observable that checks whether the representation supplies it. The specification for $z_2$ instead declares a task-metric improvement, with $p_1$ included in its required conditions. These declarations determine both what is measured and which implementation is a meaningful control. A successful property check establishes the context for testing $z_2$; the downstream experiment then assesses its own declared effect. The planner records these roles before execution so that a decision that supplies a prerequisite is assessed by its intended property even when its standalone task metric does not improve in isolation under the same evaluation protocol.

The planner uses a language model to propose decisions, dependencies, and tests from the method, task, literature summary, and baseline evidence. A review agent (the critic) checks whether the tests assess the expected effects under the required conditions. The planner revises the proposal using this feedback. The plan is accepted after review finds no outstanding issues and graph validation passes; Section 2.2 describes the graph and the prerequisite checks used to validate it.

When an effect requires several decisions to act together, the planner groups them into one component with a shared specification and test. The plan records the original decisions and their external dependencies. The result is attributed to the group.

Ablation Planning

Ablation analysis compares algorithm configurations to assess particular changes ([1]). Fix the baseline, implementations, and evaluation conditions, and let $f(C)$ denote $J_q$ for the implementation containing a feasible set $C$ of decision indices. For $C$ and $C\cup\{i\}$ satisfying the declared dependencies, the marginal effect of adding decision $i$ is

This difference measures the marginal contribution of decision $i$ to configuration $C$ ([19]). For $z_2$ requiring $z_1$, setting $C = \{1\}$ recovers $\delta_{2\mid1} = f(\{1, 2\}) - f(\{1\})$. Baseline and complete-candidate values do not determine its sign because they leave $f(\{1\})$ unobserved (Proposition 1). Testing $z_1$ supplies this control measurement; the property check assesses whether this implementation provides the declared intermediate property.

Let $V$ index the components. The conditions $D_i$ describe what component $i$ requires; $d_i^0$ identifies the other components whose outputs or properties it directly needs. An edge $(j, i)$ denotes such a dependency. Other conditions, such as the data split and measurement settings, are specified in the experimental protocol. The set $K_i$ contains all ancestors of $i$, including indirect prerequisites:

Graph validation checks identifiers, acyclicity, and agreement between $K_i$ and the declared prerequisites (Appendix § 8). The control $P_i$ retains prerequisites; treatment $T_i$ adds the change:

Build combines saved implementations of the required components with the baseline. The operator $\oplus$ applies the tested component’s implementation change to this control. Implementation review checks that the resulting code follows the specification. Appendix § 8 details this merge.

For $z_1 \to z_2$, the test compares $z_1 + z_2$ with $z_1$, preserving the representation required by the objective. Testing $z_c$ in $z_a \to z_b \to z_c$ retains both $z_a$ and $z_b$. Thus, the control contains every required component together with its own prerequisites.

For a component $u$ representing a group $S$, the experiment applies the decisions together to a control containing their external prerequisites. The shared specification defines the expected effect of this combined change. The result is attributed to the group, and the record identifies all participating decisions without assigning separate gains to individual members. The revised graph must remain acyclic. Appendix § 8 gives the construction algorithm and examples; Appendix § 11 proves that baseline and complete-candidate evaluations alone cannot determine the improvement in equation (4), and that $K_i$ is the smallest set satisfying the declared dependencies.

Executing and Assessing Experiments

The procedure $\rho_i$ specifies how to run the experiment; the protocol $\omega_i$ specifies the evaluation conditions and measurement settings shared by control and treatment:

The records $O_i^{-}$ and $O_i^{+}$ link observations to their implementations and execution attempts. Both runs use the validation rule specified in $s_i$. Matching conditions controls data and execution differences; training randomness also affects performance comparisons ([3]).

For valid executions with finite measurements and correctly matched reports, let $m_i^{-}$ and $m_i^{+}$ be the control and treatment metrics, and let $d_q \in\{-1,1\}$ denote minimization or maximization. The numerical test criterion is expressed by the predeclared predicate $\phi_i(y) = \mathbf{1}[y \mathrel{\boxdot_i} \theta_i]$, with $\boxdot_i \in\{<,\leq,=,\geq,>\}$ and threshold $\theta_i$. The resulting criterion separates performance tests from property checks according to the declared experimental objective:

A performance test requires improvement beyond zero and the protocol threshold $\epsilon_i$, and applies $\phi_i$ to $y_i^{\mathrm{test}} \in\{\delta_i,m_i^{+}\}$ as specified before execution. A property check applies $\phi_i$ to the reported intermediate observable $a_i^{\mathrm{obs}}$. An observable such as the number of failed property checks can summarize several assertions. Appendix § 8 gives the full numerical and report-matching checks.

Execution validity and effect assessment are separate. A checkpoint is qualified for retention after successful execution, implementation review, task checks, and a passed test, with no pending plan revision. A component is accepted when its search ends with a retained qualified checkpoint. Failed execution triggers repair; a valid test that fails its criterion informs revision of the expected effect or design. See Appendices § 8 and § 10 for details.

Research Memory and Revision

For each component, DAGLoop records the specification version, prerequisites, checkpoints, code changes, observations, validation results, and execution artifacts. Each observation remains associated with the decision, expected effect, conditions, and specification version tested. These records supply the evidence for subsequent implementation and method revision.

At research step $t$, let $\mathcal{L}_t(j)$ record component $j$’s history. For active component $i$, the agent receives

Detailed histories of the active component and its ancestors let the agent inspect how the control was built, which prerequisite checks passed, and how the current attempt differs from earlier ones. Summaries of other branches supply context for revising the method without including their full execution traces in each component’s input. Failed attempts remain available for debugging; method revision receives the accumulated research memory.

For example, when a valid test of $z_2$ does not support its expected gain, the record identifies the control containing $z_1$. Revision can reconsider the objective while retaining the supported representation. If revision changes $z_1$, the earlier result for $z_2$ remains a record of the previous experiment. The revised representation defines a new control for the objective; testing it again determines whether the expected effect persists after the prerequisite changes.

DAGLoop reuses a saved implementation only when its specification, prerequisite implementations, and recorded execution context remain compatible. Otherwise, the earlier result stays in memory for reference. Appendix § 10 gives the equality and checkpoint checks used for reuse.

Search processes an active component first; a pending component becomes ready after its prerequisites are accepted. The agent drafts the first implementation, debugs execution errors or code that does not follow the specification, and reassesses the expectation when a valid test fails its criterion. Otherwise, it improves the implementation. Each operation uses research memory. Replanning can revise the decomposition or group decisions within the revision budget. When no component is ready, the system reports which prerequisites remain unresolved.

Component search retains the best qualified checkpoint. Performance tests rank checkpoints by the task metric; property checks use the specified comparison rule, with task performance breaking ties. Search continues to the attempt limit, retaining the best checkpoint found (Appendix § 10).

Complete-Candidate Evaluation

Once all components are accepted, DAGLoop assembles their checkpoints and runs the official evaluator. Targeted tests assess effects in the prerequisite context, whereas the assembled candidate also includes decisions from other branches. These combinations can change task performance. Complete-candidate evaluation checks whether the supported changes improve the full method, and its result enters research memory to guide the next revision. Only complete candidates that pass the task checks can replace the current best:

Here $C_t^{\mathrm{valid}}$ contains newly evaluated complete candidates. Component measurements guide revision through memory. Algorithm 1 summarizes the loop; Appendices § 8 and § 10 detail the procedure and resource limits used during execution.

Algorithm 1 DAGLoop: targeted experiments and revision

Input: Task, baseline $M_0$, proposed candidate $M$, budget $B$

1: Initialize research memory $\mathcal{L}$ and best candidate $\widehat{M} \leftarrow M_0$ 2: while budget and search limits permit do 3: &nbsp;&nbsp;&nbsp;&nbsp;Construct or revise specifications and the dependency DAG using $(M,\mathcal{L})$ 4: &nbsp;&nbsp;&nbsp;&nbsp;for each ready component $i$ in topological order do 5: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Search component $i$: build $P_i$, implement and test $T_i$ under the declared protocol 6: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Record validity and effect evidence in $\mathcal{L}$; debug or revise using this memory 7: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Retain the best qualified checkpoint when component search ends 8: &nbsp;&nbsp;&nbsp;&nbsp;end for 9: &nbsp;&nbsp;&nbsp;&nbsp;If all components are accepted, assemble and evaluate $M^{+}$; update $\widehat{M}$ 10: &nbsp;&nbsp;&nbsp;&nbsp;Revise $M$ from $\mathcal{L}$ if further search is permitted; otherwise stop 11: end while 12: return $\widehat{M},\mathcal{L}$

Related Work

Automated AI Research. Recent work automates method proposal, implementation, experimentation, and revision. MLAgentBench studies language agents that conduct machine-learning experiments; The AI Scientist, its extension using agentic tree search, and Agent Laboratory address longer research workflows [13, 15, 16, 21] (Yamada et al. 2025). AlphaEvolve uses executable evaluators to guide program and algorithm discovery [18].

Executable benchmarks make this research loop measurable. MLE-bench, MLGym, and RE-Bench evaluate agents on executable tasks; MLR-Bench assesses open-ended research [4, 17, 5] (Wijk et al. 2025). ScienceAgentBench evaluates scientific code using execution results and cost [6]. ResearchGym, AutoLab, and AutoResearchEval examine longer research trajectories, artifact improvement, and failures linking experiments to claims [9, 8] (Xu et al. 2026). Our work studies the evidence used to revise design decisions within these executable research settings.

Evaluator Feedback and Research Memory. OPRO conditions candidate generation on score histories; FunSearch selects evaluated programs for further search; Automated Design of Agentic Systems (ADAS) and AFlow optimize agent designs and workflows [20, 12] (Yang et al. 2024; Zhang et al. 2025). MetaReflection uses past reflections; TextGrad and Trace optimize variables using textual and execution feedback [10, 7] (Yuksekgonul et al. 2025). The AI Scientist maintains an experimental journal, Meta-Harness exposes earlier candidates’ code, scores, and traces, and ResearchLoop organizes questions, evidence, and claims as durable research state [16, 14] (Xia & Wang 2026). AutoResearchEval examines links between implementations, observations, and claims [8]. Our memory associates each outcome with the tested decision, expected effect, and prerequisite context, so revision can use evidence at the level of the decision being changed.

Ablation Planning and Experimental Attribution. AbGen and AblationBench evaluate whether language models can design informative ablations, and The AI Scientist includes ablations in its experiment schedule [1, 16] (Zhao et al. 2025). Software composition also requires explicit component dependencies. Cordis represents these dependencies, motivating our description of the properties that one decision requires from another [22]. DAGLoop uses these dependencies to construct the control implementation for each targeted experiment during revision. It distinguishes task improvement from a property required by another decision and uses joint tests when an expected effect requires several decisions. The evidence guides decision retention, revision, and further testing.

Experiments

We evaluate DAGLoop on diverse executable research tasks. We use AutoLab as the main benchmark and compare with locally run Terminus-2 using GLM-5.2 with maximum reasoning effort or reasoning disabled, and DeepSeek-V4-Flash with reasoning disabled. We also evaluate on FML-Bench; its results are reported in Appendix § 14.

Benchmark and Task Protocol

We use all 36 official AutoLab tasks: 25 CPU tasks and 11 GPU tasks covering CPU optimization, CUDA kernels, program synthesis, and model development (Xu et al., 2026). Each task provides a baseline repository, evaluator, reward rule, correctness checks, and protected task files. DAGLoop reproduces the baseline locally, proposes a candidate, constructs the dependency DAG, runs component experiments, and evaluates candidate checkpoints.

The primary comparison uses GLM-5.2, with maximum reasoning effort for both DAGLoop and Terminus-2. The additional comparisons disable reasoning for both agents, using either GLM-5.2 or DeepSeek-V4-Flash. All AutoLab GPU experiments for both agents run on NVIDIA H800 GPUs. The comparison uses the official Terminus-2 agent in Harbor 0.3.0 and the original task verifiers [11]. Both agents run to completion with AutoLab’s task-level time budgets disabled.

For each DAGLoop configuration, we report one selected result per task. The reasoning-disabled Terminus-2 scores are arithmetic means of two local runs; the reasoning-enabled reference retains the existing reported scores. § 2.5 describes how the system selects its final complete candidate. Appendix § 13 reports all task scores and their normalization and aggregation conventions.

Higher scores are better. We report task scores and means for the full suite, CPU tasks, and GPU tasks. Reward thresholds and clipping can produce ties despite different native metrics. Native metrics are listed in their original units when available for the selected result; the paired comparison uses task scores. Win, tie, and loss counts compare scores rounded to six decimal places, matching the precision of the reported reference.

Overall, Per-Task, and Component Ablation Results

DAGLoop has higher mean scores than Terminus-2 on the full suite and in both resource groups for all three model and reasoning settings Table 1. With maximum reasoning effort, the full-suite means are 0.6459 and 0.5831. With reasoning disabled, they are 0.5485 and 0.4266 for GLM-5.2, and 0.5340 and 0.4202 for DeepSeek-V4-Flash. Each mean includes every task in its split, using the score conventions in Appendix § 13. Figure 3 and the component ablation results in Table 2 detail the primary comparison; the complete task-level table is reported in Appendix § 13.

Agent	Model	Reasoning	All (36)	CPU (25)	GPU (11)
DAGLoop	GLM-5.2	Max	0.6459	0.6539	0.6277
Terminus-2	GLM-5.2	Max	0.5831	0.6247	0.4885
DAGLoop	GLM-5.2	Off	0.5485	0.5918	0.4503
Terminus-2	GLM-5.2	Off	0.4266	0.4502	0.3730
DAGLoop	DeepSeek-V4-Flash	Off	0.5340	0.5045	0.6010
Terminus-2	DeepSeek-V4-Flash	Off	0.4202	0.4492	0.3542

Task	Stage / component	Parent	Candidate / property	Metric or check	$\Delta$
BVH raytracer	C1 property: SAH BVH build	–	0	failed checks $\downarrow$	–
BVH raytracer	C2 direct: SAH leaf-size selection	16.9169	0.167806	runtime (s) $\downarrow$	−16.7491
BVH raytracer	Integration†: C1–C2	16.6304	0.157895	runtime (s) $\downarrow$	−16.4725
Adversarial splay	C1 direct: BR sequence	48,656	67,008	rotations $\uparrow$	+18,352
Adversarial splay	C2 property: simulator replica	–	0	failed checks $\downarrow$	–
Adversarial splay	C3 direct: local-search refinement	67,008	69,847	rotations $\uparrow$	+2,839
Adversarial splay	Integration†: C1–C3	48,656	69,847	rotations $\uparrow$	+21,191
Agent tool routing	C1 direct: inverted index	9.9904	2.5183	median runtime (s) $\downarrow$	−7.4721
Agent tool routing	C2 property: quality gate	–	0	failed checks $\downarrow$	–
Agent tool routing	Integration†: C1–C2	10.0050	2.3019	median runtime (s) $\downarrow$	−7.7031
AES-128 CTR	C1 property: AES-NI key schedule	–	0	failed checks $\downarrow$	–
AES-128 CTR	C2 direct: 8-way AES-NI CTR	8.2557	0.044184	runtime (s) $\downarrow$	−8.211516
AES-128 CTR	Integration†: C1–C2	7.9238	0.047219	runtime (s) $\downarrow$	−7.876543
Hash join	C1 property: direct-index build	–	0	failed checks $\downarrow$	–
Hash join	C2 direct: AVX2 batch probe	0.029695	0.018936	runtime (s) $\downarrow$	−0.010759
Hash join	Integration†: C1–C2	75.7664	0.0191	runtime (s) $\downarrow$	−75.7473

In the primary comparison, DAGLoop scores 0.7378 versus 0.0006 on online language-model serving, 0.4637 versus 0.2403 on agent tool routing, and 0.8134 versus 0.6040 on concurrent key-value storage. Terminus-2 leads on instruction-following data selection (0.8581 versus 0.6177) and sorted string table (SSTable) compaction (0.3479 versus 0.1780). Five tasks tie at 1.0000: BM25, multilingual optical character recognition, stack-machine golf, Toy ISA, and VLW scheduling.

An aggregate task score shows whether a complete candidate improves, but not which design decision produced the change or whether that decision works only with its prerequisites. We therefore use component ablations to evaluate a decision against a matched control that preserves its declared prerequisite implementation. In Table 2, the Parent column is this control and Candidate/property is the resulting observation. Direct stages compare the task metric, property stages test an intermediate requirement such as zero failed checks, and integration stages evaluate components jointly when their effect depends on their combination. The table consequently traces evidence along a dependency chain rather than comparing unrelated variants.

Table 2 illustrates how DAGLoop tracks multi-component dependencies across five representative tasks. For metric comparisons, each component is evaluated against a matched control containing its prerequisite set $K_i$; parent values therefore follow the DAG branch rather than a single linear sequence. The Parent column is the implementation used for comparison, while Candidate/property is the resulting measurement. The arrows indicate the optimization direction: lower runtime is better, higher rotation count is better, and a property check passes when its failed-check count is zero.

In the bounding-volume hierarchy (BVH) chain, the surface-area heuristic (SAH) construction passes its property check, leaf-size selection reduces its matched control from 16.9169 s to 0.167806 s, and the integrated candidate reaches 0.157895 s. In the adversarial-splay chain, the BR sequence raises rotations from 48,656 to 67,008, the simulator replica passes its property check, and local-search refinement adds 2,839 rotations before the joint result reaches 69,847. Agent tool routing shows the same pattern with an inverted index, a passing quality gate, and a further reduction in median runtime after integration.

The AES-128 CTR chain separates a correctness prerequisite from the performance change: the AES-NI key schedule first passes its property check, the 8-way CTR implementation reduces runtime from 8.2557 s to 0.044184 s, and the joint run reaches 0.047219 s from its recorded baseline. The hash-join chain uses the same structure: the direct-index build passes its check, AVX2 batch probing reduces its matched parent from 0.029695 s to 0.018936 s, and the end-to-end build-plus-probe run reaches 0.0191 s. These rows show why a direct component result and a complete-candidate result answer different questions: the former isolates a decision under its prerequisites, while the latter tests the assembled method.

Taken together, these chains show why the ablation record separates direct performance gains, enabling property checks, and joint results. A direct gain supports retaining a decision for the tested context; a passed property check makes a prerequisite available to downstream tests; and a joint result evaluates an interaction that cannot be assigned to either component alone. This distinction lets the agent revise the relevant decision rather than treating every change in the final task score as evidence for every component.

Limitations

Decision-level evidence assesses the declared effect under the tested conditions. Its reliability depends on the decomposition, prerequisite graph, observable, and the experiment’s statistical power. It does not identify a unique causal explanation, and joint tests attribute results to the merged component when member-level isolation is not meaningful. The end-to-end benchmark measures the complete DAGLoop workflow.

DAGLoop addresses candidate-method improvement under executable task constraints. Evidence reuse depends on recorded context: changes to data, hardware, evaluation, or upstream decisions can require new experiments. Repeated optimization against a fixed evaluator can overfit task artifacts. Broader scientific generalization and revision of the research agent itself require separate evaluation.

Conclusion

We present DAGLoop as a general-purpose research system for executable method improvement. It connects candidate-method revision to targeted experiments on its constituent design decisions. Experimental specifications link each decision’s rationale and expected effect to tests that preserve its prerequisites. Structured research memory carries the outcomes into subsequent revisions, while complete-candidate evaluation measures task performance. Preserving prerequisites lets the agent assess intermediate properties and downstream effects in their intended context. Joint experiments assess effects that require several decisions to act together, while the accumulated records guide further tests when the evidence remains unresolved. On 36 AutoLab tasks, DAGLoop obtains a mean task score of 0.6459, compared with 0.5831 for Terminus-2 using GLM-5.2 with maximum reasoning effort. DAGLoop scores higher on 19 tasks, ties on seven, and scores lower on ten. With reasoning disabled, DAGLoop also has higher mean scores for GLM-5.2 (0.5485 versus 0.4202) and DeepSeek-V4-Flash (0.5340 versus 0.4202), under the score conventions in Appendix § 13. The research output records which expectations were supported, under which prerequisites, and which remain unresolved.

Ethics Statement

Automated research consumes compute and can amplify errors in experimental interpretation. DAGLoop retains evaluator requirements, experiment records, and the context of each result. The authors remain responsible for resource use, interpretation, and reported claims.

Detailed Experimental Procedure

The executable task and budgeted objective are defined in equations (1) and (2). The procedures below implement its targeted experiments and method revision.

Graph Construction and Comparison Configurations

Let $N$ convert component names to a consistent identifier format (Appendix § 10). The declared dependencies $d_i^0$ and ancestor sets $K_i$ are defined in equation (5).

The constructor rejects duplicate identifiers, unknown or self-referential dependencies, and cycles. Kahn’s topological sort, using a first-in, first-out (FIFO) queue for eligible nodes, gives the order $\pi$ in which the ancestor sets are computed, preserving the input traversal order when nodes become eligible together. The plan also declares $a_i$, the set of components that must be accepted before testing component $i$. The constructor removes redundant direct edges, retaining the parents $d_i$, and requires $a_i$ to equal all ancestors:

The equality is a validation requirement: a missing ancestor or an unrelated component causes rejection. Semantic adequacy of the proposed dependencies and tests is assessed by the critic.

Algorithm 2 Construction of the dependency DAG

Input: Declared components $\{(i, d_i^0, a_i, s_i)\}_{i=1}^n$

1. Normalize identifiers with $N$; reject duplicates and invalid references

2. $A_0 \leftarrow\{(j,i) : j \in d_i^0\}$; $\pi\leftarrow\operatorname{FIFO\text{-}Kahn}(V,A_0)$

3. if $|\pi| \ne|V|$ then

4. &nbsp;&nbsp;&nbsp;&nbsp;return cycle error

5. end if

6. for $i$ in $\pi$ do

7. &nbsp;&nbsp;&nbsp;&nbsp;$K_i \leftarrow\bigcup_{j \in d_i^0}(\{j\} \cup K_j)$

8. end for

9. for $i$ in $\pi$ do

10. &nbsp;&nbsp;&nbsp;&nbsp;$d_i \leftarrow\{j \in d_i^0 : \nexists k \in d_i^0 \setminus\{j\}, j \in K_k\}$

11. &nbsp;&nbsp;&nbsp;&nbsp;if $a_i$ is missing or $a_i \ne K_i$ then

12. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return prerequisite mismatch

13. &nbsp;&nbsp;&nbsp;&nbsp;end if

14. end for

15. return $G = (V, \{(j,i) : j \in d_i\}), \pi, \{K_i,s_i\}_{i\in V}$

Build restores the checkpoints of the required components and combines their files with the baseline. When multiple branches contribute changes, it uses a shared earlier version as the merge base where available and reports conflicting edits. The treatment applies the active component’s change to this control. Improvement can start from the component’s previous best checkpoint while the control remains fixed. Unrelated branches are excluded. Implementation review checks whether the code follows the specification for the active component.

For a merged component $u$ representing decisions $S$, equation (6) applies with $i=u$ and the shared intervention $\Delta_u$. The planner must supply an acyclic revised graph. For example, merging $z_a$ and $z_c$ in $z_a \to z_b \to z_c$ while retaining both incident dependencies gives $u \to z_b \to u$ and is rejected. The planner then revises the decomposition. Figure 4 shows single-decision and joint comparisons.

Numerical Criteria and Checkpoint Qualification

$\operatorname{Fin}(v_1,\ldots,v_k)$ is true when every argument is a finite numerical value and none is Boolean. For a performance test, the predicate specified before execution chooses $y_i^{\mathrm{test}} = \delta_i$ or $m_i^+$, and the numerical acceptance criterion is

Here $\epsilon_i$ is the minimum improvement specified by the measurement protocol, with $\epsilon_i = 0$ for a test requiring positive improvement. Thus a positive observed gain must also satisfy the declared predicate. For a property check, let $a_i^{\mathrm{obs}}$ be its reported intermediate observable and $b_i = 1$ indicate exactly one report matching the specified test and linked to its execution record. Its acceptance criterion is

An aggregate observable, such as the number of failed property checks, can summarize several assertions. The execution script computes that aggregate; the criterion compares it with the threshold specified before execution.

Numerical acceptance and checkpoint qualification are separate. For attempt $t$, let $e_{it}$ indicate successful execution, $h_{it}$ an existing checkpoint, $r_{it}$ a requested plan revision, and $c_{it}$ a review of task constraints that has not blocked the attempt. Method implementation review (MIR), denoted by $\mathrm{MIR}_{it}$, assesses whether the code implements the specified method. Let $g_{it}$ be $g_i^{\mathrm{dir}}$ or $g_i^{\mathrm{ena}}$ according to the test type. Qualification is

A partial implementation supports only the implemented and tested scope. Failed attempts remain in the history, while an earlier qualified checkpoint can be retained.

Memory Records and Feedback Scheduling

The component histories record the specification reference, ancestor set, checkpoints, code difference, observations, validation results, and execution artifacts. We write an attempt record as

where $\nu_i$ identifies the specification version and $A_{it}$ references supporting artifacts.

Let $\sigma_t(i)\in\{\text{pending},\text{active},\text{accepted},\text{exhausted}\}$ denote search not yet started, search in progress, search ended with a qualified checkpoint, and attempts exhausted without one, respectively. With topological order $\pi$, define

$\min_{\pi}$ denotes the first component in the stored order; $\perp$ denotes no available component or checkpoint. Thus an active component is processed before a newly ready one.

Let $n_i$ count started attempts, $u_i$ infrastructure attempts that did not start, and $\ell_i$ the last outcome. The next implementation action is

Here $\texttt{contract\_failed}$ denotes a valid execution whose result fails the declared test criterion. $\mathcal{F}_{\mathrm{impl}}$ contains execution failure, implementation failure, a missing checkpoint, and code changes that depart from the specification. CHECK-EXPECTATION reassesses the expected effect when a valid test fails its criterion; DEBUG repairs execution or implementation errors. The agent uses the selected action and research memory to generate the next implementation. A failed design or test can prompt replanning within the revision budget, including a revised decomposition or a group of decisions tested together.

Component search retains the best qualified checkpoint. Performance tests rank checkpoints by the task metric. Property checks use the specified comparison rule, with task performance breaking ties. Search continues until its attempt limit is reached; equation (18) gives the resulting state.

Full Research Loop

Algorithm 3 combines the graph, numerical predicates, research memory, and feedback policy. Its construction step applies the planner in § 2.1 followed by Algorithm 2. Component attempts, execution launches, command durations, and outer revisions have separate limits. A failed execution counts as an attempt; an infrastructure failure before execution has a separate counter. Appendix § 10 specifies state updates and evidence reuse.

Algorithm 3 Experiments that preserve prerequisites and method revision

Input: Task $\mathcal{T}$, baseline $M_0$, candidate $M$, configured resource limits

1. $(G,\pi,K,s) \leftarrow\operatorname{Construct}(M)$ using Algorithm 2

2. $\mathcal{L} \leftarrow\varnothing$; $\widehat{M} \leftarrow M_0$; $\sigma(i) \leftarrow$ pending

3. while the execution and revision limits permit another step do

4. $i \leftarrow i_t$ from equation (16)

5. if $i \ne\bot$ then

6. $\mathcal{H} \leftarrow\mathcal{H}_t(i)$; $o \leftarrow\operatorname{Op}(i)$ by equations (9) and (17)

7. $P_i \leftarrow\operatorname{Build}(M_0,K_i)$; implement $T_i = P_i \oplus\Delta_i$ using $(o,\mathcal{H})$

8. Obtain $O_i^{-}$, $O_i^{+}$ and the available measurement records by equation (7)

9. $(g_{it},q_{it}) \leftarrow$ equations (12) to (14)

10. $\mathcal{L}(i) \leftarrow\mathcal{L}(i) \mathbin{\|} F_{it}$; update $n_i$, $u_i$ and retained checkpoint

11. Update $\sigma(i)$ by equation (18); select the next action using equation (17)

12. else

13. if $\forall j \in V$, $\sigma(j) =$ accepted then

14. $M^{+} \leftarrow\operatorname{Assemble}(\{\widehat{T}_j\}_{j\in V})$; evaluate $M^{+}$

15. Update $\widehat{M}$ by equation (10); record the complete result

16. else

17. Report the blocking or exhausted components

18. end if

19. end if

20. if feedback requests a permitted component or method revision then

21. $M \leftarrow\operatorname{Revise}(M,\mathcal{L})$; reconstruct $(G,\pi,K,s)$

22. Initialize its component histories and reuse checkpoints satisfying equation (19)

23. else

24. Stop if component search is blocked or complete-candidate search has finished

25. end if

26. end while

27. return $\widehat{M},\mathcal{L}$

Plan and Experiment Records

For each component, DAGLoop stores its specification and the history of implementation and evaluation attempts. The following fields describe these records.

The design fields specify the component identifier, purpose, mechanism, inputs, outputs, and dependencies. The experiment fields specify the expected effect, test type, and acceptance criterion. The implementation check describes how to assess whether the code follows the design; the optimization space specifies which changes are allowed; the downstream requirements state the properties required by later components. A performance test compares task performance, and a property check checks an intermediate property. Joint tests also record their original decisions. The tuple $s_i$ groups the design and experiment fields for each component.

An attempt records the implementation action, ancestor set, checkpoints, code difference, measurements, implementation review, review of task constraints, and artifact references. Commands and execution records link observations to the corresponding attempt. The record $F_{it}$ in equation (15) summarizes these fields. The research memory in equation (9) preserves detailed current-component and ancestor histories and summarizes other branches.

State Transitions and Evidence Reuse

Graph and semantic checks. The identifier map $N$ trims and lowercases strings, replaces runs of non-alphanumeric characters with a hyphen, and assigns an index-based identifier when the result is empty. The constructor computes ancestor sets, removes redundant edges, and validates acceptance prerequisites according to equations (5) and (11). Inputs and outputs are checked for structure and nonemptiness; the critic evaluates their semantic compatibility and the adequacy of the proposed tests. A merged component passes the ordinary DAG checks after replanning.

Control measurement. A root component has no prerequisites and uses the baseline as its control. For a single direct parent, the component history can provide that parent’s finite metric. Components with multiple direct prerequisites require a measurement of the combined control implementation. A reported control attempt can supply this value after matching the control implementation, source state, and evaluation settings. Control reuse checks the saved measurement evidence, protocol, and restorable checkpoint. A control value that fails these checks is cleared. Protocol identity is defined by the recorded evaluation configuration.

Numerical predicates and repetitions. The criterion for performance tests accepts one comparison of signed metric improvement or candidate metric with a finite numerical constant. The criterion for property checks accepts one scalar comparison, including a reported aggregate such as a failed-check count. Unsupported expressions return a plan-format error. Boolean, missing, and nonfinite values fail the numerical check. Equality is exact unless tolerance is expressed in the declared observable or predicate. The measurement protocol specifies the reported statistic. For repeated measurements, the execution system checks the sample count, finite control and treatment samples, and the specified mean or median against the reported aggregate. These criteria serve as threshold filters for search decisions.

Qualification and retained checkpoints. In equation (14), the task-constraint indicator excludes violation and insufficient-evidence. Qualification additionally requires successful execution, an existing checkpoint, a faithful or partial implementation review, a passed component test, and no pending plan revision. The latest attempt’s test result and the retained checkpoint’s test result are stored separately.

Let $\widehat{T}_i^t$ be the retained qualified checkpoint after attempt $t$ and $N_i$ the component attempt limit. Search continues to this limit, preserving the best qualified checkpoint. After a started attempt, the state is updated as follows:

A non-started attempt instead increments $u_i$ and follows the infrastructure limit. On exhaustion, an existing qualified checkpoint is preserved; otherwise the workflow reports infrastructure exhaustion. For a property check, checkpoints are ranked using the specified comparison direction. Equality tests use distance from the target value, and task performance breaks ties.

Checkpoint reuse. Write $c_j$ for a component’s recorded specification and acceptance requirements, $h_j$ for its retained checkpoint identity, and $\mathbf{h}_{d_i}$ for the ordered direct-parent checkpoint identities. An old accepted checkpoint can enter a new component history only if

ExecMatch compares execution identifiers derived from baseline records. A specification reference identifies the tested version of the design, a protocol identifier denotes the recorded evaluation configuration, and a checkpoint identifier locates the saved implementation. These checks associate reuse with the recorded experiment. Old records remain in history when a checkpoint cannot be reused in the revised context.

Resource limits. Resource limits apply to component implementation attempts, experiment launches, individual commands, and revisions of the candidate method. The reference configuration uses six component attempts, six experiment launches per sandbox, 7,200 seconds per command, and three outer revisions. A control measurement counts as an experiment launch. An executed failure consumes a component attempt; a failure before execution increments the separate infrastructure count. Component progress does not consume an outer method-revision round. Cost is recorded, and component selection follows equation (16).

Proofs

Baseline and Complete-Candidate Evaluations

For the metric $f$ on feasible configurations and the marginal effect in equation (4), we have:

Proof. Consider two metrics on the three feasible configurations:

	$f(\varnothing)$	$f(\{1\})$	$f(\{1,2\})$	$\delta_{2\mid1}$
$f_A$	0	0	$1/2$	$1/2$
$f_B$	0	1	$1/2$	$-1/2$

Both metrics lie in $[0,1]$ and have identical baseline and complete-candidate values. Any rule that receives only those two values has identical inputs in the two cases, although the marginal effects have opposite signs. It therefore cannot determine the sign for all metrics.

Minimal Prerequisite Context

A set $C \subseteq V$ is closed under prerequisites if $j \in C$ implies $d_j^0 \subseteq C$.

Proof. Every member of $K_i$ lies on a directed path to $i$. If $j \in K_i$ and $k \in d_j^0$, appending the path from $j$ to $i$ shows $k \in K_i$, so $K_i$ is closed under prerequisites. It contains $d_i^0$. Conversely, any set containing $d_i^0$ that is closed under prerequisites must contain every predecessor along each path ending at $i$, hence all of $K_i$. This proves uniqueness and minimality. Acyclicity gives $i \notin K_i$, and adding $i$ preserves closure because $d_i^0 \subseteq K_i$.

For the reduction, process vertices in topological order. A removed parent $j \in d_i^0$ is an ancestor of another direct parent $k$. Following this relation among direct parents must terminate at a retained parent $k^*$, since the graph is finite and acyclic. By induction, the reduced graph on the earlier vertices preserves the path from $j$ to $k^*$. The retained edge $(k^*,i)$ therefore preserves reachability from $j$ to $i$. Retained edges are original edges, so the reduction creates no new reachability. Induction over $\pi$ establishes equality of all ancestor sets.

Definitions of Representative Tasks

This appendix explains representative task objectives and evaluator interfaces for the benchmark results. The task interfaces, measurement boundaries, and reward formulas correspond to the official AutoLab repository at commit 4127da3; the public repository supplies the task descriptions ([2]). The score comparison uses the reward records in Appendix § 13 for all reported tasks.

BM25 Retrieval in Go

The AutoLab task uses BM25 retrieval with term frequency, inverse document frequency (IDF), and document-length normalization ([2]). For query $u$ and document $d$, the task computes the following retrieval score:

Here $N$ is the corpus size, $\operatorname{df}(w)$ counts documents containing $w$, $\operatorname{tf}(w,d)$ counts its occurrences in $d$, and $\overline{L}$ is mean document length. $\operatorname{uniq}(u)$ denotes unique query terms, so repeated tokens are counted once; $|d|$ is document length. The output contains at most ten positive-score documents, ordered by decreasing score and increasing document identifier on ties. An empty query returns no results for the retrieval task.

The correctness check compares the batch query count, hit count, and checksum with the reference. Scores are rounded after multiplication by $10^6$ before checksum aggregation. The timing workload uses 4,000 documents and 40 queries, with the median over five query seeds:

The measured interval includes index construction, query processing, ranking, and batch checksum computation. It excludes compilation, corpus and query generation, the preceding correctness check, and printing.

For measured seconds $t>0$, the official task reward is

The scoring anchors 2.1 and 0.03 are benchmark constants; local relative improvement uses the locally reproduced baseline instead.

Transforms, Joins, Storage, and Model Training

Fourier transform (fft.rust). For a real signal $x \in\mathbb{R}^n$ with $n=32768$, the task computes the complex discrete Fourier transform

The metric is the median duration of the transform call. Parseval and spectral-checksum checks use the task’s $10^{-6}$ relative tolerance. The implementation is single-threaded Rust using the standard library for the transform.

Hash join (hash_join). Given indexed integer key–payload tables $R = (r_a)_{a=1}^{n_R}$ and $S = (s_b)_{b=1}^{n_S}$, the task computes the matching row pairs of an inner equijoin

Every matching row pair contributes, including duplicate keys. The output is a match count and a 64-bit checksum. The benchmark uses 20,000 rows in $R$ and five million in $S$, measuring one join call after warm-up under a single-thread constraint.

Number-theoretic transform (ntt_butterfly_cuda). For $a \in\mathbb{F}_p^n$, the forward transform has the form

where $\omega$ is the task’s forward $n$th root of unity. The CUDA implementation returns canonical field elements in natural order, in place. Correctness is bit-exact, and latency is the median CUDA-event time after warm-up, using the prescribed workspace.

Multi-scalar multiplication (msm_pippenger_bls12_381_cuda). The task maps BLS12-381 $G_1$ points $P_i$ and 256-bit scalars $s_i$ to

The evaluator checks the affine output in the required Montgomery representation and measures median CUDA-event latency. The released starter uses naive double-and-add.

SSTable compaction (sstable_compaction_rs). Sorted string tables store ordered key–value entries on disk. This task merges several prefix-compressed tables, retains the visible latest value for each key, and removes shadowed or deleted entries. It verifies live-entry count, checksum, and output size. The median of five compaction calls includes decoding, merging, and output construction, while input generation occurs outside the measured interval.

Visual reasoning (grpo_multisource). Group relative policy optimization (GRPO) fine-tunes Qwen2.5-VL-7B using Geometry3K, MathVision, and ChartQA, producing a low-rank adaptation (LoRA) adapter. The final native metric is accuracy on 100 held-out MathVista examples. Let $a$ be this accuracy and $r$ the general visual question answering (VQA) accuracy retention on 50 examples. The task reward is

Retention is the adapted-to-baseline VQA accuracy ratio, defined as one when baseline VQA accuracy is zero. The smaller local evaluation uses 20 MathVista and 10 VQA examples; the final reward uses the 100-plus 50 evaluation.

Full AutoLab Results

Table 3–Table 5 report the complete per-task scores and corresponding native metrics across all 36 AutoLab tasks. For Terminus-2, reported scores represent the arithmetic mean across two evaluation runs.

Task	DAGLoop G+	DAGLoop G0	DAGLoop D0	Terminus-2 G+	Terminus-2 G0	Terminus-2 D0
adaptive_compression	0.550410	0.457400	1.000000	0.496800	0.395700	0.288650
adversarial_splay	0.564829	0.500000	0.500000	0.584100	0.523450	0.568900
aes128_ctr	0.610310	0.613200	0.505300	0.579350	0.252100	0.445350
agent_tool_routing	0.463745	0.390800	0.250100	0.240300	0.224500	0.198850
bm25_search_go	1.000000	0.400800	0.358000	1.000000	0.373550	0.537650
bvh_raytracer	0.328492	0.296100	0.311000	0.455050	0.283400	0.263900
concurrent_kv_wal	0.813392	0.476400	0.182100	0.603950	0.638350	0.475050
data_select_ifeval	0.617745	0.377400	0.230000	0.858050	0.031600	0.275750
discover_sorting	0.850000	1.000000	1.000000	0.850000	0.850000	0.850000
fft_rust	0.522475	0.468400	0.451600	0.526600	0.234600	0.243750
flash_attention	0.446652	0.708400	0.320800	0.469450	0.215600	0.346700
flux2_klein_lora	1.000000	0.365300	0.895700	0.000000	0.247450	0.000000
fredkin_sort_network	1.000000	0.506700	0.666667	0.866700	0.433300	0.533350
gaussian_blur	0.557633	0.512200	0.225800	0.557350	0.470100	0.473200
grpo_multisource	0.911100	0.866700	0.866700	0.877750	0.822250	0.844450
hash_join	0.559472	0.502400	0.477600	0.640200	0.527400	0.567050
huffman_canonical_decode_cuda	0.384987	0.386900	0.357900	0.379750	0.297850	0.264150
icp_correspondence_step_cuda	0.506301	0.521300	0.509700	0.499400	0.470000	0.292600
levenshtein_distance	0.386082	0.380500	0.381400	0.387400	0.208150	0.190050
llm_online_serving	0.737800	0.072300	0.905764	0.000550	0.000500	0.000050
moving_mnist_world_model	0.227964	0.212649	0.203951	0.257882	0.320893	0.318903
msm_pippenger_bls12_381_cuda	0.399109	0.481600	0.699300	0.382400	0.189600	0.264700
multilingual_ocr	1.000000	1.000000	0.920000	1.000000	0.636150	0.556300
ntt_butterfly_cuda	0.535343	0.513300	0.406100	0.512450	0.478350	0.496350
radix_sort	0.433447	0.392100	0.349800	0.410750	0.324300	0.285650

Task	Metric	DAGLoop metric	DAGLoop score	Terminus-2 score	Difference
data_select_ifeval	IFEval $\uparrow$	0.441774	0.617745	0.858050	−0.240305
flux2_klein_lora	Quality $\uparrow$	0.6479	1.000000	0.000000	+1.000000
grpo_multisource	Accuracy $\uparrow$	0.61	0.911100	0.877750	+0.033350
huffman_canonical_decode_cuda	Time (ms) $\downarrow$	6.04928	0.384987	0.379750	+0.005237
icp_correspondence_step_cuda	Time (ms) $\downarrow$	0.26144	0.506301	0.499400	+0.006901
llm_online_serving	Serving $\uparrow$	1.0812	0.737800	0.000550	+0.737250
moving_mnist_world_model	PSNR (dB) $\uparrow$	15.367783	0.227964	0.257882	−0.029918
msm_pippenger_bls12_381_cuda	Time (ms) $\downarrow$	91.878334	0.399109	0.382400	+0.016709
multilingual_ocr	CER $\downarrow$	0.2893	1.000000	1.000000	+0.000000
ntt_butterfly_cuda	Time (ms) $\downarrow$	0.934432	0.535343	0.512450	+0.022893
scaling_law	PPL $\downarrow$	20.909189	0.584402	0.605381	−0.020979

Table 3: All six configurations. Columns G+, G0, and D0 denote GLM-5.2 with maximum reasoning effort, GLM-5.2 with reasoning disabled, and DeepSeek-V4-Flash with reasoning disabled, respectively.

Task	DAGLoop			Terminus-2
	$G^{+}$	$G0$	$D0$	$G^{+}$	$G0$	$D0$
regex_engine	0.961408	0.858700	0.722700	0.880300	0.515300	1.000000
resnet_bit_flip	0.975000	0.937500	0.350000	0.881250	0.656250	0.918750
safety_router	0.992128	0.921820	0.874947	0.992128	0.992128	0.374977
scaling_law	0.584402	0.155969	0.615714	0.605381	0.608362	0.582762
sha256_throughput	0.276915	0.285300	0.244600	0.179850	0.011700	0.020750
smallest_game_player	0.578610	0.949100	0.121500	0.473950	0.000000	0.000000
sstable_compaction_rs	0.177997	0.069600	0.182600	0.347900	0.214550	0.077450
stack_machine_golf	1.000000	1.000000	1.000000	1.000000	1.000000	0.532400
toy_isa_opt	1.000000	1.000000	0.982000	1.000000	0.991250	0.979650
vliw_scheduler	1.000000	0.973600	0.983000	1.000000	0.730550	0.980550
z_order_range_scan	0.298172	0.193100	0.172000	0.195300	0.188700	0.077350
Mean (36)	0.645887	0.548543	0.534010	0.583119	0.426609	0.420166

Task	Metric	DAGLoop metric	DAGLoop score	Terminus-2 score	Difference
adaptive_compression	bpb $\downarrow$	3.6963	0.550410	0.496800	+0.053610
adversarial_splay	Rotations $\uparrow$	69847	0.564829	0.584100	−0.019271
aes128_ctr	Time (s) $\downarrow$	0.047219	0.610310	0.579350	+0.030960
agent_tool_routing	Time (s) $\downarrow$	0.471374	0.463745	0.240300	+0.223445
bm25_search_go	Time (s) $\downarrow$	0.000339	1.000000	1.000000	+0.000000
bvh_raytracer	Time (s) $\downarrow$	0.157895	0.328492	0.455050	−0.126558
concurrent_kv_wal	Time (s) $\downarrow$	0.284784	0.813392	0.603950	+0.209442
discover_sorting	Comparators $\downarrow$	63	0.850000	0.850000	−0.000000
fft_rust	Time (s) $\downarrow$	0.000661	0.522475	0.526600	−0.004125
flash_attention	Time (s) $\downarrow$	0.123984	0.446652	0.469450	−0.022798
fredkin_sort_network	Gates $\downarrow$	53	1.000000	0.866700	+0.133300
gaussian_blur	Time (s) $\downarrow$	0.160011	0.557633	0.557350	+0.000283
hash_join	Time (s) $\downarrow$	0.0191	0.559472	0.640200	−0.080728
levenshtein_distance	Time (s) $\downarrow$	0.479385	0.386082	0.387400	−0.001318
radix_sort	Time (s) $\downarrow$	0.491702	0.433447	0.410750	+0.022697
regex_engine	Time (s) $\downarrow$	0.101679	0.961408	0.880300	+0.081108
resnet_bit_flip	Bits $\downarrow$	3	0.975000	0.881250	+0.093750
safety_router	Params $\downarrow$	131	0.992128	0.992128	+0.000000
sha256_throughput	Time (s) $\downarrow$	0.526313	0.276915	0.179850	+0.097065
smallest_game_player	Params $\downarrow$	7553	0.578610	0.473950	+0.104660
sstable_compaction_rs	Time (s) $\downarrow$	0.072334	0.177997	0.347900	−0.169903
stack_machine_golf	Instructions $\downarrow$	1024	1.000000	1.000000	+0.000000
toy_isa_opt	Cycles $\downarrow$	1543	1.000000	1.000000	+0.000000
vliw_scheduler	Cycles $\downarrow$	1200	1.000000	1.000000	+0.000000
z_order_range_scan	Time (s) $\downarrow$	0.128334	0.298172	0.195300	+0.102872

PSNR = peak signal-to-noise ratio; CER = character error rate; PPL = perplexity.

Additional FML-Bench Results

We additionally ran DAGLoop on 18 FML-Bench tasks with the task and repository specified (Zou et al., 2026). The results cover 13 GPU tasks and five CPU tasks. Table 6 reports their native metrics separately from AutoLab. For each task, we report one available native result alongside the baseline from the same run.

Task	Metric	Baseline	DAGLoop
GPU tasks (13)
Opacus	Acc. $\uparrow$	0.5875	0.6024
DomainBed-CM	In-acc. $\uparrow$	0.2613	0.7010
ART	Defense $\uparrow$	0.3131	0.8719
OpenOOD	AUROC $\uparrow$	0.8905	0.9209
PyCIL	Inc.-acc. $\uparrow$	0.5957	0.6016
USB	Acc. $\uparrow$	0.0783	0.0961
EasyFSL	Acc. $\uparrow$	0.6534	0.7034
PFLlib	Acc. $\uparrow$	0.4530	0.5211
DomainBed-OH	Avg.-acc. $\uparrow$	0.5962	0.6280
PrivacyMeter	AUC gap $\downarrow$	0.3177	0.0252
Lightly	Acc. $\uparrow$	0.7560	0.7886
Solo-learn	Linear acc. $\uparrow$	0.4667	0.4879
OpenUnlearning	KS $p$ $\uparrow$	$\approx0$	0.3890
CPU tasks (5)
Fairlearn	$\lvert\mathrm{DP}\rvert$ $\downarrow$	0.1866	0.0029
gCastle	SHD $\downarrow$	72	24
AIF360	$\lvert\mathrm{AOD}\rvert$ $\downarrow$	0.2399	0.0209
CausalML	MAE $\downarrow$	1.2963	1.2680
Cont.-Learn.	Avg.-acc. $\uparrow$	0.2841	0.5935

CM and OH denote ColoredMNIST and OfficeHome, respectively. Metrics: DP = demographic-parity difference; AOD = average odds difference; SHD = structural Hamming distance. Acc. = accuracy; In-acc. = mean accuracy on the splits designated “in” by DomainBed; Inc.-acc. = average incremental accuracy; Linear acc. = linear evaluation accuracy; Avg.-acc. = average accuracy. AUROC = area under the receiver operating characteristic curve; AUC gap = the task-defined area-under-curve gap; MAE = mean absolute error; KS $p$ = Kolmogorov–Smirnov test $p$-value. Defense denotes the task-defined defense score.