Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

Introduction

Advances in LLMs and research harnesses have enabled increasingly capable autonomous research systems ((Tang et al., 2025; Liu et al., 2026)). These systems iteratively propose ideas, conduct experiments, and use the results to guide further research. Auto-research benchmarks provide a common basis for comparing these systems, but their role extends beyond evaluation. By defining clear objectives and constraints, a well-designed benchmark helps researchers identify, validate, and build on useful improvements. However, existing benchmarks remain limited both in evaluating research systems and in supporting broad research ideas [7, 31, 26] (Wijk et al. 2025; Nathani et al. 2025).

To address this gap, we introduce Auto-Research Arena, a comprehensive benchmark for evaluating auto-research systems and advancing idea discovery in language modeling. Departing from prior benchmarks that define progress primarily through a particular objective within each research problem, Arena defines multiple research objectives within a common experimental environment. These objectives address complementary bottlenecks across the model lifecycle, including model size, data consumption, training computation and memory, and inference memory and latency. The shared environment provides a common experimental basis for evaluating whether a harness remains effective across different research objectives. Meanwhile, different objectives reward different types of improvements. For example, reducing model size or inference memory can count as progress even when prediction loss remains unchanged. By recognizing these complementary forms of progress, Arena supports exploration of a broader range of model designs and systems optimizations.

To support this broad range of research objectives within a unified framework, Arena specifies both what to improve and which requirements a candidate must satisfy. That is, Arena separates the standard for admissibility from the direction of improvement. Each objective is formulated as a constrained optimization problem. Constraints on model quality and relevant non-target resources determine whether a candidate is admissible. Among admissible candidates, a reduction in the selected resource metric constitutes improvement. Changing the target metric defines a different research objective within the same framework. This formulation supports consistent evaluation across diverse objectives, while excluding gains that violate the specified quality or resource constraints. Beyond evaluating objectives independently, Arena supports research across objectives. Separate searches can pursue different objectives in parallel, each exploring improvements relevant to its target. Compatible improvements discovered in one search can then be incorporated into the starting program for a subsequent search targeting a different objective. This enables both parallel exploration of distinct research directions and cumulative progress through the reuse of earlier discoveries. By explicitly specifying objectives, constraints, budgets, and starting programs, Arena makes it possible to measure how prior discoveries affect subsequent research.

We evaluate six LLM-based auto-research methods and a Tree-structured Parzen Estimator baseline [18] across Arena’s nine objectives. Arena provides a discriminative and meaningful evaluation of research harnesses: methods exhibit clear differences across objectives, with no single harness dominating the benchmark. Two results further support the evaluation. TPE, specialized for the original validation loss-oriented setting, performs worst in Arena’s broader multi-target evaluation. Moreover, our simple Sequential Search baseline ranks second overall with both foundation models, outperforming several substantially more sophisticated harnesses and suggesting large headroom in current auto-research system design. The recorded search trajectories help explain how these final outcomes arise. Higher-performing harnesses tend to sustain progress throughout the search budget, whereas lower-performing methods often plateau earlier or improve less consistently.

Multi-target research also exposes a substantially broader idea frontier. We empirically find that different objectives lead to qualitatively different discoveries, yielding 32 findings across nine idea families spanning model architecture, optimization, memory, and inference systems. Many of these discoveries align with active directions in LLM research. For example, when optimizing parameter efficiency, agents discover looped Transformers through weight sharing. They further learn depth-specific residual gains to adapt the residual updates across repeated applications of the shared block. This finding closely parallels recent work identifying residual adaptation as a key ingredient of looped Transformers (Wang et al., 2026). Similar connections appear across other discoveries, although alignment with existing research does not imply identical solutions. Several discoveries follow design principles similar to those of prior work but implement them through distinct mechanisms, offering alternative approaches to the same research problems. Together, these findings show how Arena’s diverse objectives expand the range of ideas explored by autonomous research systems.

Finally, we examine whether ideas discovered through Arena’s lightweight searches remain useful beyond the search setting. The original searches train each candidate for only 10 minutes. We provide the collected discoveries to a new search focused on model quality, which can select, combine, and modify these ideas. We then train the resulting model for one hour from random initialization. Unlike the original searches, which optimize individual efficiency targets, this experiment evaluates improvements in model quality directly. Relative to the reference run, training loss decreases from 2.675 to 2.544 at matched training data and from 2.617 to 2.544 at matched cumulative FLOPs. Held-out BPB also decreases from 0.9295 to 0.9112 at essentially the same model size. These results show that discoveries from Arena’s small-scale, multi-target searches can improve model quality through combination and adaptation. Arena thus supports both the evaluation of autonomous research systems and the discovery of reusable research ideas.

Related Work

Autonomous research benchmarks. PaperBench (Starace et al. (2025)) evaluates replication of published research, while EXP-Bench (Kon et al. (2026)) assesses the design and execution of paper-derived experiments. These benchmarks emphasize implementing and validating specified research content. More open-ended settings include MLE-bench (Chan et al. [7]) and RE-Bench (Wijk et al. (2025)), which evaluate ML engineering and research, alongside MLGym (Nathani et al. (2025)), MLRC-Bench (Zhang et al. (2025c)), and AstaBench (Bragg et al. [5]), which support standardized experimentation and evaluation across ML and scientific tasks. Within language modeling, Karpathy’s autoresearch (Karpathy [31]) minimizes validation loss under a fixed training-time budget, whereas NanoGPT Bench (Intology [26]) targets faster training at a required model quality. Related benchmarks evaluate reproduction of NanoGPT improvements (Zhao et al. (2025)) and correctness-constrained computational speedups ((Ouyang et al., 2025; Press et al., 2025; Shetty et al., 2025)). However, existing benchmarks provide limited evaluation of auto-research systems across complementary objectives. Arena instead unifies diverse objectives within a shared experimental setting, revealing how the research objective shapes both harness performance and the resulting idea frontier.

Autonomous research systems. Complementing these benchmarks, research harnesses determine how agents organize experiments and use accumulated evidence. End-to-end systems connect ideation, experimentation, and reporting ((Lu et al., 2026; Tang et al., 2025)), while AIDE (Jiang et al. [28]) and AIRA (Toledo et al. (2025)) organize iterative code improvement through search. AutoScientists (Gao et al. [19]) coordinates decentralized teams, Arbor (Jin et al. [30]) maintains a persistent hypothesis tree, and GEAR (Jeddi et al. [27]) evolves populations of research states. Heuresis (Antoniades et al. [3]) further compares search strategies across quality, diversity, and novelty. Arena evaluates how the relative strengths of research harnesses vary across objectives.

Auto-Research Arena

Auto-Research Arena defines a family of research problems that optimize different resource targets within a shared language-modeling environment. Figure 2 illustrates the research loop, from selecting a target and evaluating candidate programs to identifying each harness’s best admissible program and comparing performance across targets. In the following, we present the general formulation in § 3.1 and its language-modeling instantiation in § 3.2.

Auto-Research as Constrained Optimization

Consider a testbed with a fixed data pool, evaluator, and execution environment. An auto-research harness searches over executable programs $p \in\mathcal{P}$, where candidate programs may modify the model architecture, optimizer, training schedule, and inference implementation.

For each objective $j$, Arena minimizes a resource metric $\operatorname{Target}_{j}(p)$ subject to a quality threshold $\tau$ shared by all targets $j$, target-specific resource limits $\mathbf{b}_{j}$, and correctness checks:

Here, $\mathcal{L}(p)$ denotes validation loss, for which lower values are better. The vector $\operatorname{Resources}_{j}(p)$ contains the constrained resource measurements, and $\mathbf{b}_{j}$ specifies their limits. Resource inequalities apply componentwise. This formulation separates admissibility from improvement: candidates qualify by satisfying the quality, resource, and correctness requirements, while lower target values indicate better outcomes among qualifying candidates.

Efficiency targets. Arena instantiates this formulation with nine objectives spanning model, training, and inference efficiency. These objectives address complementary bottlenecks in model development and deployment. Table 1 summarizes their measurements and practical motivations.

Target	Measured quantity	Why it matters
Model
Total parameters	Number of model parameters	Model storage and weight-memory footprint (Lan et al., 2019).
Training
Data consumption	Tokens used to reach target quality	Learning efficiency under a token budget [23].
Optimizer updates	Updates to reach the quality threshold	Convergence in fewer updates [22].
Training FLOPs	FLOPs per training token	Compute demand per token [23].
Peak memory	Peak allocated GPU memory	Training feasibility on memory limits [9].
Training time	Time to reach the quality threshold	Training turnaround (Mattson et al., 2020).
Inference
Live-state memory	Memory per active request	Concurrent serving capacity (Kwon et al., 2023).
Decode latency	Sequential decoding latency	Autoregressive generation latency (Shazeer, 2019).
Long-prompt latency	Prefill + decoding latency	Long-context response latency (Zhong et al., 2024).

Each metric captures a distinct dimension of efficiency. Parameter efficiency measures the model size needed to achieve the required quality. Training efficiency measures the resources needed to reach that quality, while inference efficiency measures the resources needed to use the trained model. Optimizing these metrics separately defines distinct research problems within the same experimental environment.

Language-Modeling Instantiation

We instantiate Arena using Karpathy’s autoresearch language-modeling testbed [31], in which candidate solutions are compact LLM pretraining programs. We retain its fixed training-data pool and lightweight experimental setup, replacing loss minimization with the constrained formulation in Eq. (1). Each track optimizes one of nine efficiency metrics subject to requirements on model quality and relevant non-target resources.

Model quality. Following the original testbed [31], we measure language-modeling quality using validation bits per byte (BPB). A candidate is admissible only if its validation BPB is below 1.05. This threshold is slightly above the reference program’s BPB, providing flexibility for structural changes while limiting quality degradation.

Resource constraints. Each track limits relevant non-target resources to control the trade-offs permitted during optimization. These constraints cover quantities such as model size, training FLOPs, peak training memory, and available training data. Inference tracks also constrain other serving costs. For example, the live-state-memory track imposes a latency bound, while the decode-latency track imposes a live-state-memory bound. Each track therefore measures improvement in its selected metric while requiring other resource costs to remain within specified limits.

Measurement. The Arena evaluator measures all target metrics and constrained resources directly. Depending on the target, measurements are taken after a fixed 600-second training period or when the candidate first meets the quality requirement. Training metrics include consumed tokens, FLOPs per token, peak allocated GPU memory, elapsed training time, and optimizer updates. Inference metrics are measured after training using standardized prompt and generation workloads. Appendix A provides complete metric definitions and resource constraints for all nine tracks.

Example: parameter efficiency. The parameter-efficiency track seeks the smallest model that reaches the required quality within the specified training-compute and memory budgets. The remaining eight tracks follow the same formulation, each with its own target metric and corresponding resource constraints.

The BPB threshold of 1.05 permits a modest quality decrease relative to the reference program to accommodate structural changes. The compute and memory limits match the reference program’s measured costs, ensuring that parameter reductions do not exceed either resource budget.

Experiments

We evaluate Arena along its two central roles: benchmarking autonomous research systems and supporting broad idea discovery. We then use its controlled multi-target setting to study how research evolves under different objectives, constraints, budgets, and research histories.

Benchmark–harness interface. Arena provides the environment, task specification, reference program, evaluator, and experiment budget, while each harness determines how to conduct the search. Starting from the reference program, a harness proposes code modifications, submits candidates for evaluation, observes the measured outcomes, and decides what to explore next. Arena records every experiment and independently reports the best recorded candidate satisfying the quality, resource, and correctness requirements, regardless of which program the harness ultimately retains. Each search receives 100 charged experiments; an experiment is charged once GPU execution begins, including runs that subsequently fail or are abandoned. Appendix § 7 provides the full interface and protocol.

Reference baselines. We introduce two simple reference harnesses that differ primarily in how many research directions they retain. Sequential Search maintains one program and evaluates one modification at a time, adopting an admissible improvement before continuing. Beam Search retains up to four programs and evaluates two modifications of each before selecting the next frontier. Both condition subsequent proposals on the accumulated experimental history, including unsuccessful attempts. Appendix § 7 provides their detailed proposal, selection, and stopping policies.

Tasks and resources. We evaluate all nine Arena objectives defined in § 3, each with target-specific resource constraints and the shared quality requirement $\mathrm{BPB} < 1.05$. Each candidate runs on one NVIDIA A100 80GB GPU with seed 42 and up to 600 seconds of training, excluding compilation, validation, and the first ten warmup updates. All searches use the same approximately 631M-token ClimbMix [15] training pool and held-out BPB evaluator; training-token consumption includes repeated exposure to the data.

Models and harnesses. We evaluate six LLM-based research harnesses with Claude Opus 5 [2] and GPT-6 Astra (OpenAI, 2026), both using medium reasoning effort. In addition to Sequential Search and Beam Search, we evaluate AutoScientists [19], Arbor [30], GEAR [27], and Heuresis [3], which respectively use multi-agent coordination, hypothesis trees, evolutionary search, and diversity-preserving archives. We additionally include TPE [18], a non-agentic baseline restricted to a predefined hyperparameter space, in the Opus 5 experiments.

Scoring. Each objective is scored by the best recorded admissible target value, with lower values better. For training-token consumption, optimizer updates, and training time, we use the first 30-second validation checkpoint satisfying $\mathrm{BPB} < 1.05$. We summarize performance across objectives with different units and scales using Elo ratings from pairwise harness comparisons within each objective. Appendix § 9 gives the scoring procedure.

Benchmarking Auto-Research Harnesses

Table 2 reports the best results across Arena’s nine objectives, while Figure 3 shows representative search trajectories.

Agent model	Harness	Params. (M)	Tokens (M)	Updates	Time (s)	FLOPs (M/token)	Memory (GiB)	KV cache (KiB)	Decode (ms)	Request (ms)	Charged experiments (mean)	Elo ↑
Reference starting point		50.33	398.98	761	600	239.08	43.957	33,790	465.28	641.60	–	–
Opus 5	AutoScientists (Gao et al., [19])	13.91	184.55	267	240	52.15	0.521	58.00	49.70	29.46	84.7	1729.4
Opus 5	Sequential Search	15.15	152.46	295	210	56.66	0.567	1.000	98.56	61.30	87.7	1687.6
Opus 5	Arbor (Jin et al., [30])	14.55	159.78	337	270	58.30	0.373	94.50	57.09	57.88	54.0	1632.4
Opus 5	GEAR (Jeddi et al., [27])	13.20	164.43	306	270	62.23	0.434	121.5	56.01	97.84	100.0	1622.4
Opus 5	Beam Search	14.22	197.66	335	270	69.85	0.951	68.00	65.15	180.56	96.9	1438.6
Opus 5	Heuresis (Antoniades et al., [3])	17.83	156.17	377	270	84.15	1.582	488.5	97.82	129.03	100.0	1350.4
Opus 5	TPE (Ferreira et al., [18])	26.35	387.71	753	510	100.27	5.949	17,790	230.74	295.39	100.0	1039.2
GPT-6	AutoScientists (Gao et al., [19])	19.84	311.16	508	420	103.81	6.006	423.5	177.78	472.56	30.2	1298.7
GPT-6	Sequential Search	15.28	223.61	350	330	74.54	0.646	362.0	72.39	116.05	100.0	1702.4
GPT-6	Arbor (Jin et al., [30])	18.87	263.19	405	480	77.71	1.317	3177	137.75	163.25	14.7	1459.6
GPT-6	GEAR (Jeddi et al., [27])	18.61	159.91	302	240	67.25	0.709	51.51	62.74	47.34	100.0	1835.3
GPT-6	Beam Search	27.26	274.46	339	480	81.00	3.914	272.5	142.69	405.06	95.8	1423.3
GPT-6	Heuresis (Antoniades et al., [3])	33.03	264.37	355	420	125.24	2.318	8193	317.78	503.57	86.8	1280.6

Arena provides a discriminative evaluation. Research harnesses exhibit substantial differences across Arena’s objectives, with no single method dominating the benchmark. With Opus 5, AutoScientists achieves the highest aggregate Elo, followed by Sequential Search and Arbor; with GPT-6, GEAR ranks first, followed by Sequential Search. The variation across objectives shows that Arena distinguishes complementary strengths of research systems rather than reducing their performance to a single research setting.

Simple search is surprisingly competitive. Despite its minimal design, Sequential Search ranks second overall with both foundation LLMs, outperforming several substantially more sophisticated harnesses. Its strong performance suggests substantial headroom in current auto-research system design: more elaborate orchestration does not yet consistently translate into stronger performance.

Specialization does not necessarily transfer across research problems. TPE provides a complementary sanity check. Designed for Karpathy’s autoresearch setting (Ferreira et al., [18]), TPE performs substantially worse than all LLM-based harnesses across Arena’s broader set of objectives. Strong performance in a specialized research setting therefore need not translate into broad research capability, illustrating the value of evaluating research systems across complementary objectives.

Research trajectories support the benchmark scores. The endpoint differences in Tab. 2 are also reflected in the search trajectories in Figure 3. Stronger harnesses typically make sustained progress over the experiment budget, whereas weaker methods plateau earlier or improve less consistently. For example, TPE stops improving relatively early in the parameter search, while GEAR and AutoScientists continue to find improvements near the end of their recorded searches. These trajectories provide complementary evidence that the benchmark scores reflect systematic differences in research progress rather than isolated final outcomes.

Similar outcomes can hide different research efficiency. Endpoint performance does not fully characterize how a result is reached. On the Opus 5 training-time objective, GEAR, Arbor, Beam Search, and Heuresis all reach 270 seconds, but first attain this value after 12, 39, 50, and 99 charged experiments, respectively. Thus, harnesses with identical final scores can differ substantially in how efficiently they discover the same level of improvement.

A Broader Idea Frontier

Multi-target research substantially broadens the idea frontier. Beyond routine hyperparameter tuning, Karpathy’s reported loss-driven search identifies two architectural changes, smear and back-out [31]. Across Arena’s efficiency objectives, our search catalogue contains 32 distinct findings, including 20 that were accepted as improvements during search, which we consolidate into nine idea families spanning model architecture, training, and inference. Different objectives expose qualitatively different research directions, from architectural changes under model-efficiency targets to memory and systems optimizations under training and inference targets.

The discovered idea frontier mirrors active LLM research. The breadth of Arena’s discoveries is not merely a larger collection of benchmark-specific optimizations: many of them reach questions central to current LLM research. A representative example emerges from parameter efficiency (Table 3). Agents first discover recurrent weight sharing, effectively turning the model into a looped Transformer, and further identify the need to adapt residual updates across repeated depth, learning separate attention and MLP residual gains at each depth. This closely mirrors recent work studying residual adaptation as a key component of looped Transformers (Wang et al., 2026). Similar correspondences appear across attention-state reduction through cross-layer K/V sharing [6], MQA/MLA (Shazeer, 2019; DeepSeek-AI, 2024), conditional memory [12], optimizer-state reduction (Zhang et al., 2025d), activation-memory efficiency (Korthikanti et al., 2022), and inference optimization [1]. Importantly, these discoveries are not simply reproductions of existing methods: they often share the same underlying principles while arriving at different technical mechanisms, offering alternative perspectives on active research questions. The resulting idea frontier therefore demonstrates a broader value of Arena: its diverse objectives expand not only what research systems are evaluated on, but also the space of meaningful research directions they explore.

Discovered idea:	Loop Transformer with depth-specific residual gain
Optimization target:	Parameter efficiency
BEFORE: INDEPENDENT BLOCKS	for $\ell= 0, \ldots, 7$: $u \leftarrow h + \operatorname{Attn}_{\ell}(h)$; $h \leftarrow u + \operatorname{MLP}_{\ell}(u)$
AFTER: SHARED BLOCKS, DEPTH-AWARE GAINS	for $\ell= 0, \ldots, 7$: $u \leftarrow h + \alpha_{\ell}\operatorname{Attn}_{\ell\bmod B}(h)$; $h \leftarrow u + \beta_{\ell}\operatorname{MLP}_{\ell\bmod B}(u)$
	$B = 1$: one stored block, eight applications. The learned $\alpha_{\ell}, \beta_{\ell}$ control attention and MLP residual updates separately at each depth.
RELATION TO PRIOR WORK	Recent work scales residual updates with loop count (Wang et al., 2026); here the gains are learned separately for each depth.
Notation.	$h$: hidden state; $\ell$: depth; $B$: stored blocks. Schematic: normalization and auxiliary paths omitted.

The broader idea frontier accompanies substantial measured progress. The searches producing these discoveries also achieve large admissible gains across distinct resource dimensions. In the Opus 5 group, the best programs reduce model size from 50.33M to 13.20M parameters, training computation from 239.08M to 52.15M FLOPs per token, and peak training memory from 43.96 to 0.373 GiB, while satisfying BPB $< 1.05$ and the corresponding resource constraints. However, these endpoint programs combine multiple changes accumulated during search, so their gains do not establish whether the discovered ideas themselves provide reusable knowledge beyond the searches that found them. We test this directly in § 4.3.

Validating Discoveries Beyond the Search Setting

Arena discovers mechanisms under specific objectives and short experimental budgets. To assess their usefulness beyond these conditions, we first test whether a library of discovered ideas can guide a new search for better model quality. We then ask a stricter question: can winning programs from the original Arena searches retain their advantages when retrained for longer without further search?

Combining discovered ideas improves model quality. We collect discoveries across Arena’s objectives and harnesses into an idea library to guide a quality-driven search. The agent can select, modify, and combine mechanisms from different research trajectories rather than continue from a single winning program. We extend the training time budget to 1 hour in this search.

The resulting design improves model quality under both matched-budget evaluations (Figure 1). At matched training data, it reduces training loss from 2.675 to 2.544 (4.9%); at matched training FLOPs, it reduces training loss from 2.617 to 2.544 (2.8%). Final held-out BPB also decreases from 0.9295 to 0.9112 at the same model size (50.33M parameters). The resulting design selectively adapts and simplifies ideas rather than merely combining their original implementations. These results demonstrate that efficiency-targeted discoveries can serve as reusable model-design knowledge for improving quality. Appendix § 8.§ 3 details how these ideas are selected and transformed.

Discovered programs can transfer far beyond the search horizon. We next ask a stricter question: can a program found under a short training budget be scaled directly, without further search? Arena normally evaluates candidate programs after 10 minutes of training. We select two programs from each of the training-time, token, and update objectives and retrain them from random initialization for 60 minutes alongside the reference program.

Several discoveries retain their advantages far beyond the horizon at which they were found. A token-optimized GEAR program reaches 0.9593 BPB after 30 minutes using 1.004B tokens, while the reference requires approximately 47.6 minutes and 1.771B tokens to reach the same quality, corresponding to 43.3% fewer tokens. Its quality advantage persists until approximately 51 minutes, extending to five times the original search horizon. Similarly, a training-time program discovered by AutoScientists outperforms the reference over the 15–46-minute interval and reaches 0.9647 BPB after 30 minutes, a quality level that the reference reaches only after approximately 45.8 minutes, corresponding to 34.5% less training time (Figure 4; Table 4). Reference costs at matched quality are estimated by linear interpolation.

Objective	Harness	Resource saving	60-min quality advantage to BPB
Reference	–	–	0.9295
Training time	Seq. Search	28.5%	∼44 min, 0.9662
Training time	AutoScientists	34.5%	∼46 min, 0.9647
Training tokens	GEAR	43.3%	∼51 min, 0.9478
Training tokens	Seq. Search	34.4%	∼42 min, 0.9765
Training updates	AutoScientists	30.1%	– 0.9409
Training updates	Seq. Search	39.6%	– 0.9365

The transfer is substantial but not indefinite. The GEAR program’s learning rate reaches its 2% floor around 27 minutes, while the AutoScientists schedule reaches zero around 24 minutes. Their early advantages consequently diminish as training continues; by 60 minutes, for example, the reference reaches 0.9295 BPB compared with 0.9478 for the GEAR program. Thus, short-horizon search can discover programs whose advantages persist for four to five times the original search horizon, while horizon-dependent training policies can ultimately limit direct extrapolation to longer training.

The Science of Auto-Research

Arena also enables controlled study of what shapes the auto-research process. We examine quality constraints, budget allocation between evaluation depth and search breadth, and the role of research history in cross-target search.

Constraints shape the research space. We vary the BPB threshold across 1.02, 1.05, and 2.0 for parameter- and memory-efficiency searches with Sequential Search and Beam Search (Table 5). Relaxing the threshold from 1.05 to 2.0 allows Sequential Search to reduce parameters from 18.94M to 0.69M, while its separate peak-memory search reduces memory from 0.83 GiB to 0.035 GiB. The additional quality slack permits substantially more aggressive efficiency improvements. Conversely, tighter thresholds restrict which structural changes remain admissible. Quality constraints thus shape not only the final results, but also which regions of the design space can be developed.

Harness	BPB Limit	Params. (M)	Parameters BPB	Memory (GiB)	Peak GPU Memory BPB
Beam Search	< 1.02	23.5934	1.0185	3.9753	1.0175
Beam Search	< 1.05	19.6610	1.0448	1.6707	1.0371
Beam Search	< 2.0	0.4649	1.5030	0.1625	1.2312
Sequential Search	< 1.02	26.4767	1.0195	6.8889	1.0162
Sequential Search	< 1.05	18.9399	1.0497	0.8341	1.0402
Sequential Search	< 2.0	0.6924	1.9963	0.0348	1.7200

Budget allocation creates a depth–breadth trade-off. For parameter minimization under BPB < 2.0, we vary candidate training time from 1,200 to 6,000 seconds under a shared 20 GPU-hour training budget, trading fewer, longer evaluations for more, shorter ones (Table 8). The shortest setting evaluates 60 candidates and finds the smallest model; longer settings yield better BPB but more parameters. These outcomes illustrate the trade-off between evaluation depth and search breadth, with different allocations favoring different quality–efficiency balances.

Training time/candidate (s)	Completed candidates	Parameters	BPB
1,200	60	123,272	1.986241
2,400	30	3,194,960	1.201839
4,800	15	11,542,672	1.060628
6,000	12	7,864,464	1.151816

Cross-target research builds on prior discoveries. We initialize searches under new objectives from programs discovered under earlier objectives, transferring code while reinitializing model weights (Table 6). Subsequent searches can accumulate compatible gains: parameter savings are fully preserved while training FLOPs decrease by a further 33.7%, and a FLOP-optimized program preserves its FLOP count while uncached decode latency decreases by 87.5%. Cross-target continuation can also introduce trade-offs: optimizing training time from the parameter-optimized program reduces time to quality by 21.1% but increases parameters by 24.6%. Prior discoveries therefore provide foundations for both cumulative gains and new trade-offs across objectives.

Research sequence	Metric	Reference	After first search	After second search	$\Delta_2$
Parameters → FLOPs	Parameters (M)	50.33	14.09	14.09	0.0%
Parameters → FLOPs	Training FLOPs (M/token)	239.08	113.80	75.46	−33.7%
FLOPs → Decode latency	Training FLOPs (M/token)	239.08	66.55	66.55	0.0%
FLOPs → Decode latency	Uncached decode latency (ms)	552.09	520.36	65.17	−87.5%
Parameters → Training time	Parameters (M)	50.33	14.09	17.56	+24.6%
Parameters → Training time	Time to BPB < 1.05 (s)	600	570	450	−21.1%

Together, these studies show how constraints define admissible designs, budgets balance evaluation depth against search breadth, and research history supports cumulative gains or trade-offs across targets. Therefore, Arena enables systematic study of how autonomous research develops over time.

Conclusion

We introduced Auto-Research Arena, a multi-target benchmark for autonomous research across nine quality-constrained language-model efficiency objectives. Arena reveals complementary harness strengths and a broad frontier of architectural and systems ideas. Discoveries can seed searches under new objectives, while a program combining selected ideas can improve model quality. Controlled studies further show how constraints, budgets, and research history shape progress. Together, these results support evaluating autonomous research across complementary objectives and studying how useful discoveries accumulate.

Limitations. Our scope is explicitly limited to auto-research in language modeling, rather than the full end-to-end research process or scientific novelty. Idea-family groupings are interpretive, and noisy system measurements require care when interpreting small differences.

AI Use Statement

LLM-based research agents are the subjects of evaluation in this work; their roles in proposing ideas, implementing programs, and conducting experiments are described in the paper.

Separately, we used AI tools to assist with literature search, compilation of experimental records, text polishing for readability and figure preparation.

We have not used generative AI tools for helping develop theoretical models or conceptual frameworks, proposing or refining hypotheses, designing or providing feedback on research methodology or experiments, implementing methods, assisting with translation, supporting qualitative and thematic data analysis, and interpreting results.

Generating synthetic data sets, formulating mathematical claims, providing critical ingredients for proving mathematical claims, assisting in the writing of proofs, cleaning and reformating dataset, are not applicable to this work.

We have reviewed AI-assisted content against the relevant literature, source code, and experimental records. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Ethics Statement

This work studies autonomous research systems in a controlled language-modeling environment. The experiments use existing language-model training data and do not involve human subjects, personal data collection, or deployment to users.

The evaluated agents can autonomously propose code changes and execute experiments, which may raise broader concerns about increasingly capable autonomous research systems. In our setting, their actions are restricted to a sandboxed experimental environment, a fixed program interface, predefined compute budgets, and standardized evaluation. We do not evaluate or enable autonomous interaction with external systems beyond the research environment.

We report both successful and unsuccessful research outcomes and distinguish measured improvements from interpretations of scientific novelty. We hope that the benchmark can support more systematic study of the capabilities and limitations of autonomous research systems.

Efficiency Target Definitions

The following specifications use the objective names in Table 1. Displayed resource limits are rounded; each data-pool constraint fixes the available pool rather than the number of tokens consumed.

Total parameters

Why it matters. Fewer parameters shrink checkpoints and reduce model storage and transfer costs.

How we measure it. The model’s total parameter count after the fixed 600-second training schedule.

Optimize Minimize total parameters.

Maintain quality Validation BPB $< 1.05$ (fixed quality threshold)

Stay within limits Training FLOPs $\le$ 239.1 M/token

Peak training memory $\le$ 44.0 GiB

Available training-data pool Fixed 631.2 M-token pool

Training tokens to quality

Why it matters. Consuming fewer tokens reduces the data processing needed to train a model of the required quality.

How we measure it. Total training tokens consumed by the first scheduled validation check with BPB below 1.05, including warmup updates, gradient accumulation, and repeated use of data. This is token consumption, not the number of unique dataset positions visited.

Optimize Minimize consumed training tokens.

Maintain quality Validation BPB $< 1.05$ (fixed quality threshold)

Stay within limits Training FLOPs $\le$ 239.1 M/token

Total parameters $\le$ 50.3 M

Peak training memory $\le$ 44.0 GiB

Available training-data pool Fixed 631.2 M-token pool

Training FLOPs

Why it matters. Fewer FLOPs reduce training computation independently of hardware and implementation speed.

How we measure it. Forward-and-backward FLOPs per token, measured on the uncompiled model using executed operations and registered counting formulas. The count includes matrix, convolution, and attention work, but excludes optimizer updates and unregistered operations such as elementwise operations and normalization.

Optimize Minimize FLOPs per token.

Maintain quality Validation BPB $< 1.05$ (fixed quality threshold)

Stay within limits Total parameters $\le$ 50.3 M

Peak training memory $\le$ 44.0 GiB

Available training-data pool Fixed 631.2 M-token pool

Peak training memory

Why it matters. Lower memory lets training fit on devices with less capacity or use larger batches.

How we measure it. Peak PyTorch-allocated GPU memory from process setup through the reporting read, including FLOPs probing, training, and validation under the fixed 600-second schedule. Validation uses a fixed batch of 128 sequences of 2,048 tokens and can determine the peak. This is allocated memory, not allocator-reserved memory or total device usage.

Optimize Minimize peak allocated memory.

Maintain quality Validation BPB $< 1.05$ (fixed quality threshold)

Stay within limits Training FLOPs $\le$ 239.1 M/token

Total parameters $\le$ 50.3 M

Available training-data pool Fixed 631.2 M-token pool

Training time

Why it matters. Shorter training time reaches the required quality sooner and captures hardware and implementation effects that FLOPs miss.

How we measure it. Measured training time at the first scheduled validation check with BPB below 1.05. Startup, compilation, evaluation, and the first ten warmup updates are excluded; each subsequent update includes all accumulation microbatches and the optimizer step, with GPU synchronization at its boundaries.

Optimize Minimize measured training time.

Maintain quality Validation BPB $< 1.05$ (fixed quality threshold)

Stay within limits Training FLOPs $\le239.1$ M/token Total parameters $\le50.3$ M Peak training memory $\le44.0$ GiB Available training-data pool Fixed 631.2 M-token pool

Training updates

Why it matters. Fewer updates indicate faster optimization progress, separate from the duration or arithmetic cost of each update.

How we measure it. Total logical optimizer updates completed by the first scheduled validation check with BPB below 1.05, including warmup updates. Accumulation microbatches within one update are not counted as separate updates.

Optimize Minimize optimizer updates.

Maintain quality Validation BPB $< 1.05$ (fixed quality threshold)

Stay within limits Training FLOPs $\le239.1$ M/token Total parameters $\le50.3$ M Peak training memory $\le44.0$ GiB Available training-data pool Fixed 631.2 M-token pool

Live-state memory (KV cache)

Why it matters. Retained state grows with the live context and limits either context length or concurrent requests during generation.

How we measure it. The increase in PyTorch-allocated GPU memory while retaining one request state after a 1,536-token prefill and 512 decoding steps, with capacity 2,048. Measurement follows a discarded warmup request and constructs the state with CUDA graph capture disabled. It measures retained allocation rather than transient peak memory or a logical tensor-size sum, and includes allocator rounding (approximately 1 MiB in the reference measurement).

Optimize Minimize long-prompt retained GPU allocation.

Maintain quality Validation BPB $< 1.05$ (fixed quality threshold)

Stay within limits Request latency $\le750$ ms Training FLOPs $\le239.1$ M/token Total parameters $\le50.3$ M Peak training memory $\le44.0$ GiB Available training-data pool Fixed 631.2 M-token pool

Decode latency

Why it matters. Lower decode latency produces generated tokens faster after the prompt has been processed.

How we measure it. Median latency for a one-token prefill followed by 512 decoding steps, measured over 30 timed passes after three warmups. The timer includes prefill and synchronized GPU completion, but excludes state allocation and reset.

Optimize Minimize latency.

Maintain quality Validation BPB $< 1.05$ (fixed quality threshold)

Stay within limits Retained GPU state allocation at 513 tokens $\le10$ MiB Training FLOPs $\le239.1$ M/token Total parameters $\le50.3$ M Peak training memory $\le44.0$ GiB Available training-data pool Fixed 631.2 M-token pool

Long-prompt latency (request latency)

Why it matters. Lower request latency reduces the full delay of processing a long context and generating a response.

How we measure it. Median latency for a 1,536-token prefill followed by 512 decoding steps, measured over 30 timed passes after three warmups. The timer includes prefill and synchronized GPU completion, but excludes state allocation and reset.

Optimize                 Minimize latency. Maintain quality         Validation BPB $<$ 1.05 (fixed quality threshold) Stay within limits        Retained GPU state allocation at 2,048 tokens $\le40$ MiB Training FLOPs $< 239.1$ M/token Total parameters $\le50.3$ M Peak training memory $\le44.0$ GiB Available training-data pool Fixed 631.2 M-token pool

Shared inference correctness requirements. The three inference tasks supply ground-truth tokens at each decoding step and compare the resulting next-token distributions with the same model’s full forward pass. For both the one-token and 1,536-token prefill settings, the maximum total-variation distance across all 513 scored positions must not exceed 0.05. The returned request-state object must own all retained KV data, including compressed representations or replacement token history, and these data must remain on the GPU throughout the request. Memory constraints use the same retained-allocation procedure as the KV-cache objective, with capacities of 513 and 2,048 for the short- and long-prompt settings, respectively.

Harness Integration and Reference Baselines

This section specifies the integration interface and the two reference search policies used in § 4. Arena provides execution, measurement, and scoring; the baselines determine proposals and retention. Their selection rules are harness choices, not requirements imposed on other integrations.

Interface and Recorded Artifacts

Minimal interface. A harness must inspect the provided workspace and task definition, modify only the permitted program, submit experiments through Arena, read standardized results, and choose subsequent candidates. Arena defines the research problem and evaluation environment while leaving the research strategy open: an integration may retain an incumbent, beam, population, hypothesis tree, or another search state.

Recorded artifacts. Each run records the reference program, proposed ideas, candidate implementations, execution outcomes, measured metrics, admissibility decisions, and search lineage. The terminal score is computed from measured admissible outcomes; the accompanying record supports audits and post-hoc trace analysis. Charged-experiment accounting follows Section B.4, independently of harness-side bookkeeping.

Initialization

Arena prepares the environment and evaluates the reference program. The harness receives its source and measurements, the task’s objective, BPB threshold, resource constraints, measurement procedures, experiment budget, and a workspace for candidate programs and research records.

Sequential Search starts from one incumbent and proposes one candidate per round. Beam Search retains up to four programs and proposes two modifications per retained program in each generation. A full generation therefore contains eight candidate experiments; selection occurs after all outcomes are available.

A persistent controller coordinates an ideator, executor, and reviewer. The implementation calls the reviewer an evaluator, distinct from Arena’s evaluator. Agent sessions are fresh for each round or generation. Instructions, source code, task rules, and saved research history are supplied to each session, so continuation does not rely on conversational memory.

Proposal Preparation

The ideator inspects the current code and prior experimental history, then records a proposal, rationale, timestamp, and position in that history. Edits may change architecture, attention, optimization, data processing, or training within the permitted boundary. A proposal can coordinate several edits implementing one mechanism; Sequential Search also requests follow-up experiments that could separate their contributions.

The executor implements each proposal in an isolated copy of its parent program and checks the edits before Arena’s benchmark evaluation.

Execution and Budget Accounting

Arena checks the submitted source, protected files, available launch budget, and hardware, then runs training and task-specific evaluation. It records source, measurements, execution status, and logs. Successful execution alone does not establish admissibility.

Sequential Search uses one worker; Beam Search uses four. Parallel submission does not alter accounting: each candidate is one experiment. A full eight-candidate generation needs eight available slots and consumes eight launches if all reach chargeable GPU work, although at most four run simultaneously. Local CPU checks do not replace candidate training through Arena.

The budget is 100 charged launches, including the reference evaluation. A recorded GPU-work-start marker determines charging. Once GPU work begins, crashes, out-of-memory errors, and task failures still consume budget. A resolved launch without the marker is not charged; an unresolved launch reserves a slot. A normally charged reference therefore leaves at most 99 candidate experiments. The harness reads Arena’s remaining budget rather than counting its own rounds or generations. Failed experiments remain recorded.

Review and Selection

The reviewer compares the proposal, parent and candidate code, measurements, and logs, then records feedback. Both baselines exclude failed or constraint-violating candidates from further development. Other harnesses may continue refining them, but only admissible results contribute to Arena’s score.

Unchanged-target attempts. An efficiency improvement may increase BPB while remaining below the quality threshold. Both baselines also permit limited attempts to improve quality without changing the target. Target equality means exact equality of recorded values, regardless of the proposal’s description. Such attempts consume the allowance even when rejected.

Sequential Search. An admissible candidate replaces the incumbent when it strictly improves the target. At exactly equal target value, it may replace the incumbent while the allowance remains if it improves either the task-defined tie-break metric or BPB. Otherwise the incumbent is retained. The allowance starts at four unchanged-target attempts and resets only after an accepted strict target improvement; accepting a quality improvement alone does not reset it.

Beam Search. Parents remain fixed throughout a generation. After every outcome is resolved, the reviewer pools parents and new candidates, removes exact program duplicates, and excludes failures, constraint violations, and missing required ranking measurements. Parents need not be replaced when their modifications fail.

Up to two unchanged-target attempts are permitted per generation only if the best retained target strictly improved during the preceding two completed generations. This allowance is determined before proposal generation. When permitted, a parent and modification with equal target are reduced to the admissible member with lower BPB. The remaining programs are ranked by target, ordered task tie-break metrics, and finally BPB, retaining up to four. Remaining ties favor lower complexity, lower resource use, and then an older program where evidence is available. A read-only helper computes identities, admissibility checks, and ranking values; the reviewer applies the selection policy.

Persistence, Interruption, and Stopping

Saved state retains programs, measurements, decisions, feedback, and the remaining unchanged-target allowance, together with Beam generation membership and outcomes. Rejected and failed experiments remain in the history. Continuation reconciles outstanding requests with Arena and completes pending review before new proposals; recovery itself is not a new experiment.

Sequential Search normally stops when the budget is exhausted. Beam Search also stops when the remaining budget cannot cover two candidates per retained program; it does not shrink the generation or select from a partially evaluated one. Both record the stopping reason and unused budget.

Arena independently ranks all admissible recorded results and reports the best program, which need not be the final harness-selected endpoint. Computing this result requires no additional training.

Discovery Evidence and Supplementary Analysis

Representative Discovery Catalogue

This catalogue presents nine representative findings, detailing their mechanisms, objectives, and relationships to prior work.

Table 7 gives mechanisms, objectives, and comparisons with prior work. These examples use the BPB thresholds specified in Table 7. Correspondences identify related research questions and technical differences, not necessarily exact rediscoveries. Endpoints combine multiple changes and do not isolate individual mechanisms’ contributions.

RESEARCH FINDINGS Model and optimizer design (a–f)
(a) Looped Transformers<br>Depth-specific residual gains<br>Parameters<br>BEFORE<br>for $\ell= 0,\ldots,7$:<br>$u \leftarrow h + \operatorname{Attn}_{\ell}(h)$<br>$h \leftarrow u + \operatorname{MLP}_{\ell}(u)$<br>AFTER<br>for $\ell= 0,\ldots,7$:<br>$u \leftarrow h + \alpha_{\ell}\operatorname{Attn}_{\ell\bmod B}(h)$<br>$h \leftarrow u + \beta_{\ell}\operatorname{MLP}_{\ell\bmod B}(u)$<br>RELATION TO PRIOR WORK<br>One block ($B = 1$), eight uses; each depth learns residual gains, unlike loop-count scaling (Wang et al., 2026).	(b) Lookup-augmented FFNs<br>Token lookup before activation<br>Training FLOPs $\operatorname{MLP}_{\ell}(h)$<br>BEFORE<br>$z \leftarrow W_{\ell}^{\mathrm{up}}h$<br>return $W_{\ell}^{\mathrm{down}}\phi(z)$<br>AFTER<br>$e \leftarrow\operatorname{Gate}_{\ell}(h)\odot E_{\ell}[\tau]$<br>$z \leftarrow G_{\ell}^{h+e}$<br>return $G_{\ell}^{\mathrm{down}}\phi(z)$<br>RELATION TO PRIOR WORK<br>Grouped FFNs receive context-gated token-ID features before activation; Engram [12] instead retrieves hashed $N$-grams.	(c) Cross-layer K/V sharing<br>Layer-specific queries<br>Live-state memory<br>BEFORE<br>$K,V \leftarrow\operatorname{KV}_{\ell}(h)$<br>$y \leftarrow\operatorname{Attn}_{\ell}(h;K,V)$<br>AFTER<br>if $\ell\in\mathcal{P}$:<br>$K,V \leftarrow\operatorname{KV}_{\ell}(h)$<br>$y \leftarrow\operatorname{Attn}_{\ell}(h;K,V)$<br>RELATION TO PRIOR WORK<br>Unequal CLA-style groups [6] share K/V with private queries; LISA (Mu et al., 2024) instead shares and corrects attention weights.
(d) Low-memory Adam<br>Token-row second moments<br>Training memory<br>BEFORE<br>$m_{ij} \leftarrow\operatorname{EMA}_{1}(g_{ij})$<br>$v_{ij} \leftarrow\operatorname{EMA}_{2}(g_{ij}^{2})$<br>AFTER<br>$m_{ij} \leftarrow\operatorname{EMA}_{1}(g_{ij})$<br>$v_i \leftarrow\operatorname{EMA}_{2}(d^{-1}\sum_j g_{ij}^{2})$<br>RELATION TO PRIOR WORK<br>Adam-mini-style (Zhang et al., 2025d) token-row second moments apply to input/value embeddings; first moments and output-head moments remain full.	(e) Shared low-rank bases<br>Shared reads; private projections<br>Parameters<br>BEFORE<br>for $p \in\{q,k,v\}$:<br>$y_p \leftarrow L_pR_ph$<br>AFTER<br>for $p \in\{q,k,v\}$:<br>$y_p \leftarrow L_pRh$<br>RELATION TO PRIOR WORK<br>Learned Q/K/V read basis, private output factors; ASVD (Yuan et al., 2023) instead factorizes pretrained weights using activation statistics.	(f) Unified vocabulary<br>Input / value / output<br>Parameters<br>BEFORE<br>$h_0 \leftarrow E^{\mathrm{in}}[\tau]$<br>$v_{\ell}^{\mathrm{tok}} \leftarrow E_{\ell}^{\mathrm{val}}[\tau]$<br>$o \leftarrow E^{\mathrm{out}}h$<br>AFTER<br>$h_0 \leftarrow E[\tau]$<br>$v_{\ell}^{\mathrm{tok}} \leftarrow E[\tau]$<br>$o \leftarrow Eh$<br>RELATION TO PRIOR WORK<br>Input–output tying (Press & Wolf, 2016) extends to token-indexed value residuals: one table for all three roles.
INFRASTRUCTURE ENGINEERING Training and inference execution (g–i)
(g) Stochastic rounding<br>BF16 parameter write-back<br>Update precision<br>BEFORE<br>$w \leftarrow\theta+ \Delta\theta$<br>$\theta\leftarrow\operatorname{RN}_{\mathrm{BF16}}(w)$<br>AFTER<br>$w \leftarrow\theta+ \Delta\theta$<br>$\theta\leftarrow\operatorname{SR}_{\mathrm{BF16}}(w)$<br>RELATION TO PRIOR WORK<br>BF16 storage is retained; stochastic write-back preserves small updates in expectation (Zamirai et al., 2020).	(h) Activation reconstruction<br>Backward from retained outputs<br>Activation memory<br>BEFORE<br>$r \leftarrow\operatorname{ReLU}(z)$<br>$s \leftarrow r^{2}$; save$(r)$<br>$g_z \leftarrow2r \odot g_s$<br>AFTER<br>$s \leftarrow\phi(z)$; save$(s)$<br>$g_z \leftarrow2\sqrt{s}\odot g_s$<br>RELATION TO PRIOR WORK<br>Reuse retained outputs for backward; selective recomputation (Korthikanti et al., 2022) instead reruns operations to save memory.	(i) Decode compilation regions<br>Gathers inside Q/K/V preparation<br>Decode latency<br>BEFORE<br>$e \leftarrow E[\tau]$<br>$c,s \leftarrow T[\tau]$<br>$q,k,v \leftarrow\operatorname{In}(h,e,c,s)$<br>AFTER<br>$F \leftarrow\operatorname{Compile}(\operatorname{Prep})$<br>$q,k,v \leftarrow F(h,\tau,t)$<br>RELATION TO PRIOR WORK<br>Execution-overhead optimization, as in DeepSpeed Inference [1]; the baseline already has CUDA graphs.

Scope. (a,e) Beam Search and (f) Sequential Search, BPB $< 2.0$; (b–d,g,h) Sequential Search and (i) Beam Search, BPB $< 1.02$.

Notation. $h$: hidden state; $\ell$: depth; $B$: stored blocks; $\tau$: token ID; $\phi(z) = \operatorname{ReLU}(z)^2$; $\mathcal{P}$: K/V producer layers. $\operatorname{Attn}_{\ell}$ and $\operatorname{MLP}_{\ell}$ are full modules; supplied K/V retain layer-specific query/output projections. $G$: grouped projections; $E_{\ell}$: table assigned to depth $\ell$; $\operatorname{Gate}_{\ell}$: broadcast gate. In (d), $g_{ij}$ is an embedding gradient with row width $d$; $\operatorname{EMA}_{1,2}$ use Adam decay rates. In (f), $E$ stores token vectors by row; $o$: logits. In (g), $\Delta\theta$: computed update; RN/SR: nearest/stochastic rounding. In (h), $g_x = \partial L/\partial x$. In (i), $T[\tau]$: rotary factors at position $t$; Prep includes both gathers and Q/K/V preparation. Normalization, other auxiliary paths, and decode-state reconstruction are omitted.

Supplementary Protocols for Research-Process Studies

Budget Allocation

Four parameter-minimization searches use BPB $< 2.0$ and candidate training caps of 1,200, 2,400, 4,800, and 6,000 seconds. Obtaining 60, 30, 15, and 12 completed candidates, respectively, gives 20 nominal GPU-hours per setting (Table 8). The searches share starting code, a 379,260,514-token data pool, seed 42, A100 80GB hardware, tokenizer, and held-out evaluator. Beam width four, two proposals per parent, two retuning slots, and four GPU workers are fixed.

These comparisons vary duration, candidate count, explored programs, and achieved quality together; they establish neither an optimal allocation nor a causal or matched-quality effect.

Cross-Target Transfer Protocol

Table 6 reports parameter-to-FLOP, FLOP-to-uncached-decode-latency, and parameter-to-training-time continuations. Program code is transferred; weights are freshly initialized. Each receiving evaluator remeasures the transferred program under its own objective and constraints. Earlier gains are not imposed as hard constraints.

There is one search per receiving objective. These selected examples demonstrate compatible improvements and permitted trade-offs; without matched restarts from the reference program, they establish neither a causal benefit of transfer nor a transfer-success rate.

How Discovered Ideas Are Reused

§ 4.3 shows that ideas discovered in Arena’s short, target-specific searches can be reused to improve model quality at a larger training scale. Each candidate trains on a shared 10B-token pool for at most one hour or 2.217B tokens, whichever comes first. Here we examine how this reuse occurs. Rather than transferring a single winning program or mechanically combining all previous discoveries, the quality-driven search receives an idea library and can select, combine, modify, or discard its contents. We analyze the provenance of the selected design, representative interactions encountered during search, and how the candidate distribution evolves across search rounds.

The search draws on discoveries across objectives and harnesses. The recorded sources represented in the selected design span four research harnesses and four classes of Arena objectives: parameter efficiency, training tokens, training time, and training updates. They include mechanisms for sharing representations, allocating MLP capacity, changing optimization schedules, gating attention and value residuals, restricting context, and changing where computations are applied across depth. The resulting search is therefore not a continuation of one original objective or one harness trajectory. Instead, discoveries from heterogeneous searches form a common library of candidate research ingredients.

Reuse progressively transforms prior mechanisms. Figure 5 summarizes the main architectural progression leading to the selected design. An early improvement shares the value-embedding table and reallocates MLP capacity, reducing BPB from the reference value of 0.9295 to 0.9192. Subsequent search changes how capacity is distributed across depth and where value embeddings are injected, eventually reaching 0.9122. The final step changes a different aspect of the architecture: instead of one MLP update per block, the model applies two independent half-width MLP sublayers, reaching 0.9112 BPB. Thus, the selected design is not a direct union of library entries. The search progressively reinterprets how previously discovered mechanisms are implemented and combined.

Idea provenance spans multiple research histories. Table 9 lists the ten recorded idea sources represented in the selected search lineage. They originate from Sequential Search, GEAR, Auto-Scientists, and Heurisis, and from searches targeting parameters, training tokens, training time, and training updates. This diversity supports the intended role of the idea library: mechanisms discovered under one research objective can become hypotheses for subsequent research under another. At the same time, provenance only records where an idea entered the search; it does not establish that every listed mechanism independently contributes to the final improvement.

Recorded idea	Source harness	Objective	Use in the quality-driven search
Shared value-embedding table	Sequential Search	Parameters	Shares token-dependent value representations rather than maintaining separate lookup tables.
MLP width as a search variable	GEAR	Parameters	Reallocates model capacity through MLP width rather than uniformly shrinking the architecture.
Reduced weight decay	GEAR	Training tokens	Changes optimization while retaining the architectural backbone.
Square-root cooldown	Heuresis	Training time	Changes the learning-rate schedule during later training.
Per-head attention output gate	Sequential Search	Training time	Introduces learned control over individual attention-head outputs.
Sparse value-residual gate channels	AutoScientists	Parameters	Restricts value-residual gating to a subset of channels.
Shorter attention window	GEAR	Training updates	Changes the amount of context processed by attention.
Layer-specific MLP widths	AutoScientists	Parameters	Redistributes MLP capacity across depth while approximately preserving total width.
Value embeddings at selected stack depths	Sequential Search	Training time	Moves value-residual injection layers at depth $\{1,3,5,7\}$ to $\{0,1,6,7\}$.
Two independent half-width MLP sublayers	Heuresis	Training tokens	Motivates a final design with two independent MLP updates within each block.

Useful ideas do not combine monotonically. Representative outcomes in Table 10 reveal both positive and negative interactions. For example, simply reapplying an existing MLP adds another transformation without introducing independent parameters and performs worse than the reference, whereas introducing a genuinely independent second MLP appears in the strongest final design. Other search outcomes show that individually strong modifications can lose their advantage when combined and that different allocations of approximately the same total MLP width can yield different quality. These observations are not controlled ablations, but they show why the idea library should not be viewed as a collection of independently additive improvements. Its value instead lies in providing structured hypotheses that subsequent research can test, reject, and reinterpret.

Representative design	BPB $\downarrow$
Fixed reference	0.929497
Shared value embedding + redistributed MLP capacity	0.919165
Reapply the same MLP without introducing independent capacity	0.934888
Redistribute approximately the same total MLP width across depth	0.913908
Inject value embeddings only at selected stack depths	0.912195
Further tune the selective-injection design	0.911221
Two independent half-width MLP sublayers	0.911213

Search progressively moves into a stronger design region. Figure 6 shows the distribution of candidate quality across the six search rounds. None of the 56 valid candidates in the first round outperforms the fixed reference, compared with 44/56, 29/56, 50/56, 55/56, and 25/25 in subsequent rounds. The temporary broadening in the third round shows that the search does not simply accumulate accepted improvements, while later rounds increasingly concentrate around substantially lower BPB. Across all rounds, 203 of 305 valid candidates outperform the reference, indicating that the improvement reflects a broad region of stronger designs rather than a single isolated candidate.

The exact top-ranked combination should not be overinterpreted. Although the search moves toward stronger designs overall, its best candidates are extremely close. The two best recorded candidates achieve BPB values of 0.911213 and 0.911221, differing by only $8 \times10^{-6}$ in their one-hour evaluations. With one measurement per candidate, differences at this scale do not establish that either design is uniquely superior to nearby alternatives. We therefore interpret the experiment as evidence that the discovered idea library guides subsequent research toward a productive region of model designs, rather than that one particular combination is uniquely optimal.

Elo Scores

Table 2 reports the best recorded admissible target values across nine objectives for Opus 5 and GPT-6. We aggregate these values using the same margin-aware Elo procedure, with equal weight assigned to each objective.

Training times are compared after rounding to the nearest 30-second checkpoint; matching checkpoints are ties. Ratings are calculated from these stored target values before display rounding or unit conversion. The two agent-model groups use separate method pools, each with a mean rating of 1500, so Elo comparisons are made within each group.

All harnesses begin each calculation with the initial rating $R = 1500$. Lower values are better for every included objective.

Turn best admissible results into pairwise outcomes

$x_{a,j}$ is harness $a$’s best recorded admissible target value on objective $j$. $y_{a,b,j}$ is the outcome when harness $a$ is compared with harness $b$: 1 for a win, $\frac{1}{2}$ for a tie, and 0 for a loss.

Estimate the expected outcome from the current ratings

$R_a$ and $R_b$ are the harnesses’ current ratings. $E_{a,b}$ is the outcome expected for harness $a$, between 0 and 1. A higher-rated harness is expected to win more often.

Measure how large the win is

$d_{a,b,j}$ is the logarithm of the ratio between the two target values. Using a ratio makes proportional improvements comparable within an objective. $s_j$ is the median nonzero absolute log margin between all harness pairs on objective $j$, bounded below by $10^{-9}$; it defaults to 1 when every pair is tied. It defines a typical margin for that objective. $z_{a,b,j}$ expresses the observed margin in units of that typical margin. The margin multiplier $m_{a,b,j}$ grows gradually with the margin and is capped at 3 so that one unusually large win cannot dominate the ranking. Exact ties have a zero multiplier and cause no rating update.

Weight objectives and update both ratings

$K$ is 32, which controls the size of each update. $w_j$ is objective $j$’s weight, and $\overline{w}$ is the average objective weight. Dividing by the average keeps an ordinary comparison near the standard Elo update size. The term $y-E$ rewards results that exceed the current expectation and penalizes results that fall short.

Reduce sensitivity to comparison order

Because Elo updates happen one comparison at a time, their order can slightly affect the result. The calculator therefore generates 4,000 permutations using NumPy’s default random generator with seed 20260902. Each pass starts all harnesses at 1500 and processes every pair once per objective; the reported score is the mean final rating across passes. The zero-sum updates preserve a cohort mean of 1500 for both agent-model groups, without a final shift or fixed maximum. The published score is a relative rating of the best recorded admissible results, not a percentage or a measurement in any resource unit.

Extended Related Work

Autonomous-Research Benchmarks

Implementation, reproduction, and research tasks. Benchmarks cover repository execution, method implementation, reproduction, and expert-specified extensions [4, 25, 16] (Siegel et al. 2025; Yan et al. 2025; Starace et al. 2025; Kon et al. 2026). Related capability suites assess software and command-line tasks [29] (Merrill et al. 2026), retrieval (Wei et al., 2025; Xiong et al., 2026), scientific discovery and programming [11] (Majumder et al. 2025; Tian et al. 2024), and theorem proving (Tsoukalas et al., 2024). Arena instead leaves the improvement mechanism open and evaluates a measured optimization target under quality and resource constraints.

Open-ended research and research artifacts. Machine-learning engineering, research competitions, and paper-derived research environments support open-ended experimentation under task-specific evaluation protocols [7, 20] (Miller et al. 2025; Wijk et al. 2025; Nathani et al. 2025; Lupidi et al. 2026; Wu et al. 2026; Zhang et al. 2025c). Other evaluations examine proposals, experiments, reports, alignment, and artifact support across research stages [8, 13, 5] (Tang et al. 2025; Liu et al. 2026; Xu et al. 2026; Zhang et al. 2026). Arena focuses on how harness performance changes across objectives in one language-modeling environment, ranking measured endpoints and using traces as explanatory evidence.

Language-model and computational optimization. Language-model benchmarks evaluate validation loss, training speed, reproduction of historical improvements, or post-training under compute constraints [31, 26] (Zhao et al. 2025; Rank et al. 2026); systems benchmarks target kernels, numerical routines, and software performance under correctness requirements (Ouyang et al., 2025; Press et al., 2025; Shetty et al., 2025). Arena spans nine model, training, and inference objectives with a common quality requirement and objective-specific resource limits.

Evaluation validity and reliability. Studies examine efficiency and scaffold contributions after benchmark saturation (Nadgir et al., 2026), specification and evaluator inconsistencies (Tu et al., 2026), and hardware sensitivity and score aggregation [10]. These concerns motivate distinguishing valid resource improvements from apparent gains caused by evaluation artifacts.

Autonomous-Research Harnesses

End-to-end workflows. Research systems connect literature review, idea generation, implementation, experiments, analysis, and reporting (Lu et al., 2026; Tang et al., 2025; Trirat et al., 2025; Schmidgall et al., 2025). The research harness orchestrates these activities around a foundation model.

Tree, archive, and evolutionary search. Harnesses allocate experiments through code trees, hypothesis trees, archives, and alternative selection policies [28, 30, 3] (Toledo et al. 2025). Population-based methods retain competing programs or heuristics and generate candidates using executable feedback, mutation, or crossover [27] (Romera-Paredes et al. 2024; Liu et al. 2024; Novikov et al. 2025). These search procedures organize exploration; performance gains alone do not establish scientific novelty.

Multi-agent coordination and persistent memory. Teams, proposal critique, execution recovery, and shared successful or failed experiments support sustained research [19] (Liu et al. 2026). Semantic, episodic, and structured scientific memory provide complementary mechanisms for retaining knowledge across sessions [17, 21] (Mitchener et al. 2025).

Harness design and self-improvement. Other methods optimize agent architectures, workflows, or implementations through meta-search and recursive improvement (Hu et al., 2025; Zhang et al., 2025b;a; Srikanth et al., 2026). Their search object is the research procedure itself, rather than only the candidate experimental program.

Arena evaluates harness–model combinations across multiple optimization objectives in a shared language-modeling environment, leaving the search strategy open and retaining standardized experimental traces.