WorksComputer Science
IRAH: Hierarchical Agentic Tree Search for Retrieval Pipeline Optimization
Abstract
Optimizing Information Retrieval (IR) pipelines for Retrieval-Augmented Generation (RAG) requires jointly selecting structural components and numerical hyperparameters under limited evaluation budgets. We introduce the Information Retrieval Agentic Harness (IRAH), which combines Language Agent Tree Search for IR (LATS-IR) with offline meta-learned initialization. LATS-IR performs UCT-guided hierarchical search, first exploring macro-structural pipeline choices and then refining numerical parameters, with natural-language reflections to guide subsequent mutations and provide inspectable search traces. We evaluate IRAH on three in-distribution benchmarks and one out of-distribution benchmark to assess generalization under distribution shift. Under the same meta-learned initialization, LATS-IR outperforms TPE on two of three in-distribution benchmarks, with gains of up to 6.4%, and remains competitive on the out-of-distribution benchmark. Across all four benchmarks, warm-started IRAH achieves the highest average held-out nDCG@10 (0.4816) and improves over its cold-start counterpart. These results establish IRAH as a competitive approach to retrieval pipeline optimization under constrained evaluation budgets.
†Equal contribution; work conducted while both authors were interns at Capital One.
Introduction
Modern retrieval-augmented generation (RAG) systems rely on retrieval pipelines whose effectiveness is shaped by interdependent design choices (Lewis et al., 2020; Hsia et al., 2025). These include how documents are chunked, which dense encoder is used, whether sparse and dense scores are fused, which reranker is applied, and how numerical parameters such as fusion weight and retrieval depth are set. A chunking strategy changes the distribution of candidate passages seen by the embedder; reranking can compensate for some retrieval errors while amplifying others; and the best retrieval depth depends on both the underlying corpus and the evaluation objective (Hsia et al., 2025). As a result, RAG pipeline optimization is better viewed as a structured configuration-search problem than as isolated tuning of individual hyperparameters. Existing automated optimization methods reduce manual tuning burden [2], but general-purpose black-box optimization does not explicitly exploit the macro–micro structure of retrieval pipeline decisions. Recent work on RAG hyperparameter optimization demonstrates the value of structured search in large configuration spaces [11]. However, existing approaches do not explicitly organize optimization as a hierarchical search trajectory in which structural and numerical decisions are explored at different stages and prior outcomes inform subsequent proposals. Moreover, many hyperparameter optimization algorithms provide limited visibility into the optimization process (Moosbauer et al., 2022). For IR pipelines, inefficient exploration can also incur substantial systems costs when structural changes require constructing or loading new indices. Starting optimization from scratch for each new corpus further discards potentially useful information from prior retrieval tasks. These limitations motivate an approach that can exploit the structure of the configuration space, adapt its search based on previous evaluations, transfer useful experience across datasets, and retain an inspectable optimization trace.
We introduce the Information Retrieval Agentic Harness (IRAH) (Figure 1), which combines LATS-IR, a reflection-guided hierarchical search process, with offline meta-learned initialization. LATS-IR first performs macro-structural exploration, evaluating structural choices together with coarse numerical settings, before locally refining numerical parameters within a promising pipeline architecture. Upper Confidence Bound applied to Trees (UCT) guides tree selection, while natural-language reflections from evaluated configurations condition subsequent mutations and provide an inspectable record of the search process. To mitigate cold-start overhead, an offline meta-learner uses dataset-level characteristics to identify promising configurations from prior retrieval tasks for warm-start initialization.
Figure 1. IRAH: Given a target dataset and task metric, an offline meta-learner proposes promising seed configurations. LATS-IR then performs two-phase hierarchical search over structural and numerical pipeline decisions and returns the best configuration identified under the evaluation budget.
We evaluate IRAH on three in-distribution retrieval benchmarks and one out-of-distribution benchmarks under a fixed evaluation budget. Under the same meta-learned initialization, LATS-IR improves held-out over TPE by up to 6.4% on the in-distribution benchmarks and remains competitive on the out-of-distribution target. Across all four benchmarks, warm-started IRAH achieves the highest average held-out and outperforms its cold-start counterpart. These results demonstrate the effectiveness of combining hierarchical agentic search with meta-learned initialization for retrieval pipeline optimization under constrained evaluation budgets.
Our main contributions are as follows:
We introduce LATS-IR, a reflection-guided hierarchical search method for IR pipeline optimization that separates macro-structural exploration from local numerical refinement and uses prior evaluations to guide subsequent search.
We integrate LATS-IR with offline meta-learning in IRAH, allowing promising configurations from prior retrieval tasks to warm-start hierarchical search.
We evaluate IRAH across three in-distribution and one out-of-distribution retrieval benchmarks. Under shared initialization, LATS-IR outperforms TPE on two of three in-distribution benchmarks. Warm-starting improves IRAH on all four benchmarks, and performance remains competitive on the out-of-distribution target.
Related Work
Retrieval-Augmented Generation and Pipeline Design. Retrieval-augmented generation (RAG) grounds generation in externally retrieved evidence (Lewis et al., 2020), with performance depending on interacting choices in preprocessing, chunking, retrieval, reranking, and generation [12]. Systems such as DSPy treat language-model applications as compositional pipelines whose components can be optimized (Khattab et al., 2024). We focus on retrieval configuration, where structural and numerical choices jointly affect effectiveness and systems costs such as index construction and reuse.
Hyperparameter and Structured RAG Optimization. General-purpose methods such as Tree-structured Parzen Estimator (TPE) [3, 2] and AutoML under the CASH formulation (Thornton et al., 2013; Feurer et al., 2015) automate configuration search from observed evaluations, but do not explicitly encode the retrieval-specific hierarchy considered here.
Recent RAG optimization methods increasingly exploit search-space structure. Fu et al. [11] use a two-level hierarchical bandit; Afzal et al. [1] compare global and component-wise Bayesian optimization; and Chen and Chen [7] use cyclic sequential optimization of retriever and generator parameters. Language models have also been used as iterative optimizers that condition proposals on prior outcomes (Yang et al. (2024)). In contrast, LATS-IR organizes structural and coarse numerical exploration separately from local numerical refinement while maintaining an explicit search tree over retrieval configurations.
Tree Search and Language-Agent Reflection. Monte Carlo Tree Search and UCT balance exploitation and exploration ([5] (Kocsis and Szepesvári 2006)). Language Agent Tree Search (LATS) extends tree search to language-model agents using generation and self-reflection (Zhou et al. (2023)). LATS-IR adapts these ideas to retrieval configuration optimization: nodes are evaluated pipeline configurations, edges are phase-constrained mutations, UCT guides selection, and reflections condition subsequent candidate generation.
Meta-Learning for Warm Starts. Meta-learning transfers information across tasks to improve learning or optimization (Vanschoren, 2019). AutoML systems use dataset meta-features and prior evaluations to identify promising initial configurations (Feurer et al., [10]). IRAH applies this idea to retrieval optimization by using an offline meta-learner to rank shared anchor configurations and supply the highest-ranked candidates as warm starts for LATS-IR.
IRAH: Information Retrieval Agentic Harness
In this section, we introduce IRAH, a framework for retrieval configuration optimization. IRAH centers on LATS-IR, a reflection-guided hierarchical search procedure, and uses offline meta-learning to provide warm-start initialization. We formalize the hierarchical configuration problem (§ 3.1), then describe LATS-IR (§ 3.2), and finally introduce the meta-learned warm-start mechanism (§ 3.3).
Preliminaries
We formulate information retrieval (IR) configuration optimization as a hierarchical configuration-search problem, extending the Combined Algorithm Selection and Hyperparameter Optimization (CASH) paradigm ([10] (Thornton et al. 2013)) to modular retrieval pipelines used in RAG systems ([12] (Khattab et al. 2024)). Rather than treating all pipeline decisions as an undifferentiated set of parameters, we distinguish between macro-structural decisions and micro-level numerical parameters. A candidate configuration is , where denotes categorical structural choices, such as the chunking strategy or embedding model, and denotes numerical parameters associated with that structure, such as hybrid fusion weight .
This hierarchy reflects both the dependencies and computational costs of retrieval pipeline optimization. Numerical settings are tuned within the context of a particular pipeline structure, while changing structural choices such as the chunking strategy or embedding model can require constructing a new physical index. In contrast, parameters such as retrieval depth or fusion weight can often be varied while reusing an existing index. Prior RAG optimization work similarly demonstrates benefits from structured or component-wise search ([11, 1, 7]). We therefore define the configuration space as
Given a target dataset and an optimization-query set , the objective is to identify the configuration that maximizes retrieval effectiveness:
where denotes the retrieval utility of configuration for query , measured using a task-specific metric such as nDCG@ or Recall@.
Because evaluating configurations can be computationally expensive, we consider a budgeted setting in which the optimizer may evaluate at most configurations. The goal is therefore to identify a high-performing approximation to within a limited evaluation budget. This budgeted setting motivates allocating evaluations strategically
Algorithm 1 LATS-IR Hierarchical Retrieval Configuration Search
Require: optional seeds $\mathcal{C}_{\mathrm{seed}}$, evaluation budget $N_{\max}$, Phase 1 budget $N_1$, candidates per expansion $K$
1: $\mathcal{T}, \mathcal{H}, n \leftarrow \operatorname{INITIALIZE}(\mathcal{C}_{\mathrm{seed}}, Q_{\mathrm{opt}})$
2: while $n < N_{\max}$ do
3: if $n \ge N_1$ and Phase 2 not initialized then
4: $\mathbf{c}^{(1)*} \leftarrow \operatorname{BESTPHASE1}(\mathcal{T})$
5: end if
6: $\mathcal{S} \leftarrow \begin{cases} \mathcal{S}_{\mathrm{P1}}, & n < N_1, \\ \Theta_{\mathrm{local}}(\mathbf{c}^{(1)*}), & n \ge N_1 \end{cases}$
7: $w \leftarrow \operatorname{ADAPTIVEUCT}(\mathcal{T}, n)$
8: $c_p \leftarrow \operatorname{SELECT}(\mathcal{T}, w)$
9: $\mathcal{C} \leftarrow \operatorname{MUTATE}(c_p, \mathcal{H}, \mathcal{S}, K)$
10: $\mathcal{C} \leftarrow \operatorname{VALIDATE}(\mathcal{C}, \mathcal{S})$
11: for all $c \in \mathcal{C}$ do
12: $r \leftarrow \operatorname{EVALUATE}(c, Q_{\mathrm{opt}})$
13: $h \leftarrow \operatorname{REFLECT}(c, c_p, r)$
14: $\mathcal{H} \leftarrow \mathcal{H} \cup \{h\}$
15: $\operatorname{ADDCHILD}(\mathcal{T}, c_p, c, r, h)$
16: $\operatorname{BACKPROPAGATE}(\mathcal{T}, c, r)$
17: $n \leftarrow n + 1$
18: end for
19: end while
20: return $\operatorname{BESTEVALUATED}(\mathcal{T})$
between structural exploration and numerical refinement.
LATS-IR: Language Agent Tree Search for Information Retrieval
LATS-IR is our optimization engine. It adapts Language Agent Tree Search (LATS) (Zhou et al., 2023) to the hierarchical IR configuration problem defined in § 3.1. The hierarchy is imposed through the search procedure: tree nodes represent fully specified retrieval configurations, while phase-constrained edges first permit macro-structural changes and later restrict search to local numerical refinement. LATS-IR can operate from an uninformed cold start with agent-generated initial configurations, or from configurations supplied by the meta-learned warm-start mechanism described in § 3.3. For a warm start, the supplied seeds are evaluated and the highest-reward seed initializes the tree root.
At each tree expansion, LATS-IR (1) selects a previously evaluated node, (2) generates valid candidate mutations conditioned on the active search phase, and (3) evaluates the candidates on . The candidates are added as sibling children of the selected parent, and each reward is backpropagated through its tree path to update visit counts and empirical value estimates. This process repeats until the evaluation budget is exhausted. Algorithm 1 summarizes LATS-IR.
UCT-Guided Node Selection. Each evaluated configuration produces a retrieval reward , which is backpropagated along the selected tree path. For each node on this path, we increment its visit count and update its empirical value as the mean reward backpropagated through that node. Consequently, ancestor nodes summarize the observed performance of configurations explored within their descendant branches.
Nodes are selected for expansion using Upper Confidence Bounds for Trees (UCT) (Kocsis and Szepesvári, 2006):
where and are the visit counts of the node and its parent, respectively, and controls the exploration-exploitation trade-off. Rather than using a fixed exploration coefficient, LATS-IR adapts to the current search phase and observed reward history. Phase 1 favors broader exploration, whereas Phase 2 places greater emphasis on local exploitation. The complete adaptive schedule is provided in Appendix § 7.2.2.
Two-Phase Hierarchical Search. LATS-IR divides optimization into a macro phase centered on structural exploration followed by micro-level numerical refinement, exploiting the structure of the configuration space under a limited evaluation budget. In Phase 1, candidate mutations may modify structural components , including the embedding model, chunking strategy, and reranker, together with coarse numerical settings. The goal is to explore alternative retrieval architectures and identify a promising structural region of the search space. At the phase transition, we select the Phase 1 configuration with the highest optimization reward, , and fix its categorical structural choices . Phase 2 then refines the numerical parameters within a local neighborhood centered on the Phase 1 configuration. This separation prevents the search from revisiting alternative macro-architectures during refinement and concentrates the remaining budget on fine-grained optimization of a promising structural configuration.
Reflection-Guided Candidate Generation. Candidate mutations are generated by an LLM conditioned on the selected parent configuration, its observed reward, relevant prior reflections, the active search phase, and the valid parameter choices or numerical bounds for that phase. For each expansion, the LLM proposes structured child configurations, which are validated against the allowable search space before evaluation. After each evaluation, the agent produces a short reflection comparing the evaluated configuration with its parent and, when available, their observed reward difference. The reflection is retained in search memory and supplied as context for subsequent candidate generation, allowing later mutations to incorporate information from earlier successes and failures. Complete prompting, validation, and output-schema details are provided in Appendix § 7.2.3.
Meta-Learned Warm-Start Initialization
To mitigate cold-start overhead, we construct an offline meta-learner from configuration evaluations collected across multiple source retrieval datasets, following meta-learning approaches in AutoML [10] (Vanschoren 2019). For an IR dataset , comprising corpus and query set , we extract a dataset-level representation capturing corpus, query, and query-corpus characteristics. Each candidate retrieval configuration is represented by configuration features .
Given a dataset-configuration pair, the meta-learner produces a score indicating the predicted utility of that configuration for the dataset:
Let denote the set of anchor configurations evaluated during offline meta-training. For an unseen target dataset , we score each anchor configuration and select the top- ranked candidates:
where returns the anchor configurations with the highest predicted scores. The resulting seed set provides the starting configurations for warm-started LATS-IR, transferring promising configurations from prior retrieval tasks. When , the agent instead generates cold-start configurations; otherwise, the meta-learned seed configurations initialize the search. Full details of the source benchmarks, anchor configuration set, meta-feature construction, and meta-learner training and selection are provided in Appendix § 7.§ 1.
Experimental Setup
We evaluate IRAH on four retrieval benchmarks spanning diverse domains. We compare warm- and cold-started variants of IRAH and Tree-structured Parzen Estimator TPE, together with Grid Search, using a shared retrieval engine and evaluation protocol. We optimize all methods using nDCG@10 on an optimization-query split and report performance on a held-out query set. Additional implementation details are in Appendix § 7.
Datasets
We evaluate IRAH on three diverse, in-distribution natural-language retrieval benchmarks: SciFact (Wadden et al., 2020), FiQA-2018 (Maia et al., 2018), and CQADupStack-STATs [13] (Thakur et al. 2021). These benchmarks span scientific fact verification, financial question answering, and duplicate-question retrieval, respectively. We additionally evaluate on CoSQA (Huang et al., 2021; Li et al., 2025), a code-retrieval benchmark that pairs natural-language queries with source code, as an out-of-distribution (OOD) target. This setup allows us to evaluate both performance on targets similar to the meta-training distribution and transfer under a different retrieval distribution. Table 1 summarizes the benchmark characteristics.
| Dataset | Domain | Corpus Size | Opt. Queries | Held-out Queries | |
| SciFact | Scientific fact verification | 5,183 | 240 | 60 | 0.679 |
| FiQA-2018 | Financial QA | 57,638 | 519 | 129 | 0.354 |
| CQADupStack-STATS | Statistics duplicate questions | 42,269 | 522 | 130 | 0.396 |
| CoSQA | Natural-language code search | 6,267 | 400 | 100 | 2.092 |
Table 1. Characteristics of the target retrieval benchmarks. We report dataset domain, corpus size, optimization and held-out query counts, and meta-feature distance .
To quantify benchmark-to-meta-training distribution shift, we use the same dataset representation introduced in § 3.3. Each feature is normalized using statistics computed exclusively from the meta-training datasets. Let and denote the normalized meta-feature vectors of the target dataset and meta-training dataset , respectively. We define the meta-feature distance as the Euclidean distance to the nearest meta-training dataset:
The OOD benchmark, CoSQA, is substantially farther from the meta-training distribution () than the three in-distribution targets: SciFact (0.679), FiQA-2018 (0.354), and CQADupStack-STATS (0.396). Complete meta-feature definitions and normalization details are provided in Appendix A.1.
For each dataset, we partition the evaluation queries into an approximately 80/20 optimization and held-out evaluation split. All configuration search and selection decisions are made exclusively using the optimization set , while the held-out queries are used only for final evaluation of the selected configuration.
IRAH Implementation
We instantiate IRAH as follows. For warm-start initialization, we use an XGBRanker meta-learner trained on 11 source retrieval benchmarks using performance measurements from a 36-configuration anchor set. Each dataset–configuration pair is represented by 13 dataset-level meta-features and 14 configuration features. At inference, XGBRanker scores the 36 anchor configurations for the target dataset and returns the top configurations for warm-start initialization. Complete meta-training details, including the source benchmarks, feature definitions, anchor configurations, and model-selection procedure, are provided in Appendix A.1.
The downstream search space spans four chunking strategies (none, token, sentence, and recursive_character); three dense embedding models (multi-qa-mpnet-base-cos-v1, bge-large-en-v1.5, and bge-m3); three cross-encoder rerankers plus a no-reranking option; and numerical parameters controlling chunk size, retrieval depth , and hybrid lexical–dense fusion weight . Phase 1 searches over pipeline structure together with coarse numerical values and . Phase 2 fixes the selected categorical architecture and performs localized refinement of chunk size, , and . Complete search-space specifications and Phase 2 refinement bounds are provided in Appendix A.2.1.
The fixed search budget () is divided between macro-structural exploration and micro-level refinement using a phase-allocation fraction . Warm-started runs on the in-distribution benchmarks use , allocating 12 of 50 evaluations to Phase 1 and 38 to Phase 2, whereas cold-started runs use (30 and 20 evaluations, respectively). To reduce reliance on transferred configurations under substantial distribution shift, warm-started targets with use the cold-start allocation. We treat this threshold as a heuristic experimental decision rule rather than a calibrated OOD boundary. Thus, warm-started in-distribution search devotes more of its budget to local refinement, while cold-started and OOD search retain greater structural exploration.
Node selection follows the adaptive UCT procedure described in § 3.2. Reflection and candidate mutation are performed using openai/gpt-oss-120b ((OpenAI, 2025)), and generated candidates are validated against the phase-specific search-space constraints before evaluation. Additional UCT, prompting, and search-memory details are provided in Appendices A.2.2 and A.2.3.
Baselines and Comparison Protocol
We benchmark IRAH against two standard configuration-optimization baselines: a fixed 108-configuration Grid Search (Appendix Table 16) and Tree-structured Parzen Estimator (TPE) ([3]), implemented with Optuna ([2]). All methods evaluate configurations using the same retrieval engine, query splits, and retrieval objective.
Table 16: The 108 pipeline configurations spanning 18 physical indexes used for Grid Search. The first 36 configurations are also used for meta-learner training.
| Index | Dense Embedder | Chunking Strategy | Config ID | Reranker Model | ||
| PART 1: Meta-Learner Training Anchors & Grid Search Baseline (cfg_01–cfg_36) | ||||||
| 1 | bge-m3 | token (512, 50) | cfg_01 | 1.00 | 10 | none |
| 1 | bge-m3 | token (512, 50) | cfg_02 | 0.67 | 30 | none |
| 1 | bge-m3 | token (512, 50) | cfg_03 | 0.33 | 20 | bge-reranker-v2-m3 |
| 1 | bge-m3 | token (512, 50) | cfg_04 | 0.00 | 50 | ms-marco-MiniLM-L-6-v2 |
| 1 | bge-m3 | token (512, 50) | cfg_05 | 1.00 | 50 | bge-reranker-large |
| 1 | bge-m3 | token (512, 50) | cfg_06 | 0.33 | 10 | none |
| 2 | bge-m3 | none (full doc) | cfg_07 | 1.00 | 10 | none |
| 2 | bge-m3 | none (full doc) | cfg_08 | 0.67 | 30 | ms-marco-MiniLM-L-6-v2 |
| 2 | bge-m3 | none (full doc) | cfg_09 | 0.33 | 20 | none |
| 2 | bge-m3 | none (full doc) | cfg_10 | 0.00 | 50 | none |
| 2 | bge-m3 | none (full doc) | cfg_11 | 0.67 | 10 | bge-reranker-large |
| 2 | bge-m3 | none (full doc) | cfg_12 | 0.00 | 30 | bge-reranker-v2-m3 |
| 3 | bge-large-v1.5 | recursive (3000, 300) | cfg_13 | 1.00 | 10 | ms-marco-MiniLM-L-6-v2 |
| 3 | bge-large-v1.5 | recursive (3000, 300) | cfg_14 | 0.67 | 30 | bge-reranker-v2-m3 |
| 3 | bge-large-v1.5 | recursive (3000, 300) | cfg_15 | 0.33 | 20 | none |
| 3 | bge-large-v1.5 | recursive (3000, 300) | cfg_16 | 0.00 | 50 | bge-reranker-large |
| 3 | bge-large-v1.5 | recursive (3000, 300) | cfg_17 | 1.00 | 50 | none |
| 3 | bge-large-v1.5 | recursive (3000, 300) | cfg_18 | 0.33 | 10 | bge-reranker-v2-m3 |
| 4 | bge-large-v1.5 | none (full doc) | cfg_19 | 1.00 | 20 | bge-reranker-large |
| 4 | bge-large-v1.5 | none (full doc) | cfg_20 | 0.67 | 30 | none |
| 4 | bge-large-v1.5 | none (full doc) | cfg_21 | 0.33 | 50 | bge-reranker-v2-m3 |
| 4 | bge-large-v1.5 | none (full doc) | cfg_22 | 0.00 | 10 | none |
| 4 | bge-large-v1.5 | none (full doc) | cfg_23 | 0.67 | 10 | ms-marco-MiniLM-L-6-v2 |
| 4 | bge-large-v1.5 | none (full doc) | cfg_24 | 0.00 | 50 | bge-reranker-large |
| 5 | mpnet-base | recursive (3000, 300) | cfg_25 | 1.00 | 10 | ms-marco-MiniLM-L-6-v2 |
| 5 | mpnet-base | recursive (3000, 300) | cfg_26 | 0.67 | 30 | none |
| 5 | mpnet-base | recursive (3000, 300) | cfg_27 | 0.33 | 20 | bge-reranker-v2-m3 |
| 5 | mpnet-base | recursive (3000, 300) | cfg_28 | 0.00 | 50 | bge-reranker-large |
| 5 | mpnet-base | recursive (3000, 300) | cfg_29 | 1.00 | 30 | none |
| 5 | mpnet-base | recursive (3000, 300) | cfg_30 | 0.33 | 50 | ms-marco-MiniLM-L-6-v2 |
| 6 | mpnet-base | sentence (512, 50) | cfg_31 | 1.00 | 50 | none |
| 6 | mpnet-base | sentence (512, 50) | cfg_32 | 0.67 | 10 | bge-reranker-large |
| 6 | mpnet-base | sentence (512, 50) | cfg_33 | 0.33 | 20 | none |
| 6 | mpnet-base | sentence (512, 50) | cfg_34 | 0.00 | 30 | bge-reranker-v2-m3 |
| 6 | mpnet-base | sentence (512, 50) | cfg_35 | 0.67 | 50 | ms-marco-MiniLM-L-6-v2 |
| 6 | mpnet-base | sentence (512, 50) | cfg_36 | 0.00 | 10 | none |
| PART 2: Expanded Grid Search Baseline Only (cfg_37–cfg_108) | ||||||
| 7 | bge-m3 | token (256, 25) | cfg_37 | 1.00 | 10 | none |
| 7 | bge-m3 | token (256, 25) | cfg_38 | 0.67 | 20 | ms-marco-MiniLM-L-6-v2 |
| 7 | bge-m3 | token (256, 25) | cfg_39 | 0.33 | 30 | none |
| 7 | bge-m3 | token (256, 25) | cfg_40 | 0.00 | 50 | bge-reranker-v2-m3 |
| 7 | bge-m3 | token (256, 25) | cfg_41 | 0.67 | 50 | bge-reranker-large |
| 7 | bge-m3 | token (256, 25) | cfg_42 | 1.00 | 30 | none |
| 8 | bge-m3 | token (512, 100) | cfg_43 | 1.00 | 20 | bge-reranker-v2-m3 |
| 8 | bge-m3 | token (512, 100) | cfg_44 | 0.67 | 10 | none |
| 8 | bge-m3 | token (512, 100) | cfg_45 | 0.33 | 50 | ms-marco-MiniLM-L-6-v2 |
| 8 | bge-m3 | token (512, 100) | cfg_46 | 0.00 | 30 | bge-reranker-large |
| 8 | bge-m3 | token (512, 100) | cfg_47 | 1.00 | 50 | none |
| 8 | bge-m3 | token (512, 100) | cfg_48 | 0.33 | 20 | none |
| 9 | bge-large-v1.5 | token (512, 50) | cfg_49 | 1.00 | 10 | none |
| 9 | bge-large-v1.5 | token (512, 50) | cfg_50 | 0.67 | 30 | bge-reranker-large |
| 9 | bge-large-v1.5 | token (512, 50) | cfg_51 | 0.33 | 20 | ms-marco-MiniLM-L-6-v2 |
| 9 | bge-large-v1.5 | token (512, 50) | cfg_52 | 0.00 | 50 | none |
| 9 | bge-large-v1.5 | token (512, 50) | cfg_53 | 1.00 | 50 | bge-reranker-v2-m3 |
| 9 | bge-large-v1.5 | token (512, 50) | cfg_54 | 0.67 | 10 | none |
| 10 | bge-large-v1.5 | sentence (256, 25) | cfg_55 | 1.00 | 20 | none |
| 10 | bge-large-v1.5 | sentence (256, 25) | cfg_56 | 0.67 | 10 | ms-marco-MiniLM-L-6-v2 |
| Index | Dense Embedder | Chunking Strategy | Config ID | Reranker Model | ||
| cfg_57 | 0.33 | 50 | bge-reranker-v2-m3 | |||
| cfg_58 | 0.00 | 30 | bge-reranker-large | |||
| cfg_59 | 1.00 | 50 | none | |||
| cfg_60 | 0.33 | 10 | none | |||
| 11 | bge-large-v1.5 | sentence (512, 50) | cfg_61 | 1.00 | 10 | bge-reranker-v2-m3 |
| 11 | bge-large-v1.5 | sentence (512, 50) | cfg_62 | 0.67 | 30 | none |
| 11 | bge-large-v1.5 | sentence (512, 50) | cfg_63 | 0.33 | 20 | bge-reranker-large |
| 11 | bge-large-v1.5 | sentence (512, 50) | cfg_64 | 0.00 | 50 | ms-marco-MiniLM-L-6-v2 |
| 11 | bge-large-v1.5 | sentence (512, 50) | cfg_65 | 1.00 | 30 | none |
| 11 | bge-large-v1.5 | sentence (512, 50) | cfg_66 | 0.67 | 50 | none |
| 12 | bge-large-v1.5 | sentence (512, 100) | cfg_67 | 1.00 | 20 | none |
| 12 | bge-large-v1.5 | sentence (512, 100) | cfg_68 | 0.67 | 50 | ms-marco-MiniLM-L-6-v2 |
| 12 | bge-large-v1.5 | sentence (512, 100) | cfg_69 | 0.33 | 10 | bge-reranker-large |
| 12 | bge-large-v1.5 | sentence (512, 100) | cfg_70 | 0.00 | 30 | bge-reranker-v2-m3 |
| 12 | bge-large-v1.5 | sentence (512, 100) | cfg_71 | 0.67 | 20 | none |
| 12 | bge-large-v1.5 | sentence (512, 100) | cfg_72 | 1.00 | 50 | none |
| 13 | mpnet-base | token (256, 25) | cfg_73 | 1.00 | 10 | none |
| 13 | mpnet-base | token (256, 25) | cfg_74 | 0.67 | 30 | bge-reranker-large |
| 13 | mpnet-base | token (256, 25) | cfg_75 | 0.33 | 20 | ms-marco-MiniLM-L-6-v2 |
| 13 | mpnet-base | token (256, 25) | cfg_76 | 0.00 | 50 | none |
| 13 | mpnet-base | token (256, 25) | cfg_77 | 1.00 | 50 | bge-reranker-v2-m3 |
| 13 | mpnet-base | token (256, 25) | cfg_78 | 0.33 | 10 | none |
| 14 | mpnet-base | token (512, 50) | cfg_79 | 1.00 | 20 | bge-reranker-v2-m3 |
| 14 | mpnet-base | token (512, 50) | cfg_80 | 0.67 | 10 | none |
| 14 | mpnet-base | token (512, 50) | cfg_81 | 0.33 | 50 | ms-marco-MiniLM-L-6-v2 |
| 14 | mpnet-base | token (512, 50) | cfg_82 | 0.00 | 30 | bge-reranker-large |
| 14 | mpnet-base | token (512, 50) | cfg_83 | 0.67 | 50 | none |
| 14 | mpnet-base | token (512, 50) | cfg_84 | 1.00 | 30 | none |
| 15 | mpnet-base | sentence (256, 25) | cfg_85 | 1.00 | 10 | none |
| 15 | mpnet-base | sentence (256, 25) | cfg_86 | 0.67 | 20 | ms-marco-MiniLM-L-6-v2 |
| 15 | mpnet-base | sentence (256, 25) | cfg_87 | 0.33 | 30 | bge-reranker-v2-m3 |
| 15 | mpnet-base | sentence (256, 25) | cfg_88 | 0.00 | 50 | bge-reranker-large |
| 15 | mpnet-base | sentence (256, 25) | cfg_89 | 1.00 | 50 | none |
| 15 | mpnet-base | sentence (256, 25) | cfg_90 | 0.67 | 10 | none |
| 16 | mpnet-base | recursive (1500, 150) | cfg_91 | 1.00 | 20 | bge-reranker-large |
| 16 | mpnet-base | recursive (1500, 150) | cfg_92 | 0.67 | 30 | none |
| 16 | mpnet-base | recursive (1500, 150) | cfg_93 | 0.33 | 10 | ms-marco-MiniLM-L-6-v2 |
| 16 | mpnet-base | recursive (1500, 150) | cfg_94 | 0.00 | 50 | bge-reranker-v2-m3 |
| 16 | mpnet-base | recursive (1500, 150) | cfg_95 | 1.00 | 50 | none |
| 16 | mpnet-base | recursive (1500, 150) | cfg_96 | 0.33 | 20 | none |
| 17 | bge-m3 | recursive (1500, 150) | cfg_97 | 1.00 | 10 | none |
| 17 | bge-m3 | recursive (1500, 150) | cfg_98 | 0.67 | 20 | bge-reranker-v2-m3 |
| 17 | bge-m3 | recursive (1500, 150) | cfg_99 | 0.33 | 50 | ms-marco-MiniLM-L-6-v2 |
| 17 | bge-m3 | recursive (1500, 150) | cfg_100 | 0.00 | 30 | bge-reranker-large |
| 17 | bge-m3 | recursive (1500, 150) | cfg_101 | 0.67 | 50 | none |
| 17 | bge-m3 | recursive (1500, 150) | cfg_102 | 1.00 | 30 | none |
| 18 | bge-m3 | recursive (3000, 600) | cfg_103 | 1.00 | 20 | bge-reranker-large |
| 18 | bge-m3 | recursive (3000, 600) | cfg_104 | 0.67 | 10 | none |
| 18 | bge-m3 | recursive (3000, 600) | cfg_105 | 0.33 | 50 | bge-reranker-v2-m3 |
| 18 | bge-m3 | recursive (3000, 600) | cfg_106 | 0.00 | 30 | ms-marco-MiniLM-L-6-v2 |
| 18 | bge-m3 | recursive (3000, 600) | cfg_107 | 1.00 | 50 | none |
| 18 | bge-m3 | recursive (3000, 600) | cfg_108 | 0.33 | 20 | none |
Table 17.
We evaluate both IRAH and TPE under cold and warm initialization. Cold variants begin without meta-learned configuration seeds, whereas warm variants receive the same top- configurations proposed by the offline meta-learner. In particular, the shared warm-start initialization enables LATS-IR and TPE to be compared while holding the initialization strategy fixed.
Both IRAH and TPE receive a fixed optimization budget of configuration evaluations. Grid Search deterministically evaluates its predefined configuration set and is therefore reported as a single run. The stochastic methods are evaluated using three independent random seeds, and results are reported as mean ± standard deviation.
Evaluation Metrics
The optimization objective for all methods is nDCG@10 computed over . After optimization, the configuration selected by each run is evaluated on the corresponding held-out query set, and we report held-out nDCG@10 as the retrieval performance metric.
We additionally report optimization-loop runtime, measured as the wall-clock time attributed to each method’s search and evaluation procedure. Because structural configuration changes can require distinct physical retrieval indexes, we also analyze the number of unique index collections constructed during optimization. For IRAH, we further report LLM token usage as a measure of agentic optimization overhead.
Results and Analysis
We first analyze retrieval performance, including out-of-distribution generalization (§ 5.1), followed by runtime efficiency (§ 5.2) and qualitative search behavior (§ 5.3). Additional analyses, including meta-initialization performance, search trajectories, and runtime and index diagnostics, are provided in Appendix § 8.
Retrieval Performance
Table 2 compares held-out retrieval quality across Grid Search, TPE, and IRAH. Because IRAH (Warm) and TPE (Warm) receive the same meta-learned initialization, their results provide a direct comparison of the downstream optimization procedures under a shared warm start. Across all four benchmarks, IRAH (Warm) achieves the highest unweighted average held-out nDCG@10 of 0.4816.
| Method | SciFact | FiQA | CQADupStack-STATS | CoSQA (OOD) | Average |
| Grid Search | 0.7321 | 0.4407 | 0.3602 | 0.3350 | 0.4670 |
| TPE (Cold) | 0.7158 ± .0035 | 0.4843 ± .0152 | 0.3630 ± .0053 | 0.3181 ± .0090 | 0.4703 |
| TPE (Warm) | 0.7176 ± .0113 | 0.4662 ± .0198 | 0.3675 ± .0036 | 0.3420 ± .0050 | 0.4733 |
| IRAH (Cold) | 0.7267 ± .0210 | 0.4835 ± .0186 | 0.3542 ± .0111 | 0.3088 ± .0222 | 0.4683 |
| IRAH (Warm) | 0.7368 ± .0075 | 0.4959 ± .0021 | 0.3601 ± .0013 | 0.3336 ± .0131 | 0.4816 |
Table 2. Held-out retrieval performance across target benchmarks. Results are mean ± standard deviation over three seeds. Grid Search is reported from a single run. Warm variants use meta-learned initialization, whereas Cold variants do not. CoSQA is the out-of-distribution (OOD) target. Average is the unweighted mean held-out nDCG@10 across all four benchmarks. Best result per dataset is shown in bold.
Across the three in-distribution benchmarks, IRAH (Warm) achieves the highest held-out nDCG@10 on SciFact and FiQA-2018, improving over TPE (Warm) by 2.7% (+0.0192) and 6.4% (+0.0297), respectively. On CQADupStack-STATS, IRAH (Warm) remains competitive, trailing TPE (Warm) by 2.0% (−0.0074). The weaker result on CQADupStack-STATS coincides with narrower structural exploration: IRAH (Warm) evaluates 15.7 ± 3.2 unique physical collections, compared with 20.7 ± 1.5 for TPE (Warm), and all three IRAH runs favor similar embedding and chunking choices (Appendix § 8.§ 4, Table 12). This pattern is consistent with greater exploitation of the meta-learned initialization, although these diagnostics do not establish a causal explanation for the performance gap.
| Dataset | Method | Opt. nDCG@10 | Held-out nDCG@10 | Embedding | Chunker | Chunk Size | Reranker | ||
| SciFact | Grid Search | 0.7629 | 0.7321 | bge-large-en-v1.5 | sentence | 512 | bge-reranker-large | 0.33 | 20 |
| SciFact | TPE (Cold) | 0.7759 | 0.7160 | bge-large-en-v1.5 | sentence | 256 | bge-reranker-large | 0.70 | 16 |
| SciFact | TPE (Warm) | 0.7710 | 0.7182 | bge-large-en-v1.5 | token | 512 | bge-reranker-large | 0.65 | 10 |
| SciFact | IRAH (Cold) | 0.7757 | 0.7122 | bge-large-en-v1.5 | sentence | 256 | bge-reranker-large | 0.62 | 20 |
| SciFact | IRAH (Warm) | 0.7645 | 0.7281 | bge-large-en-v1.5 | sentence | 512 | bge-reranker-large | 0.80 | 30 |
| FiQA-2018 | Grid Search | 0.4784 | 0.4407 | multi-qa-mpnet-base-cos-v1 | token | 256 | bge-reranker-v2-m3 | 1.00 | 50 |
| FiQA-2018 | TPE (Cold) | 0.4863 | 0.4667 | multi-qa-mpnet-base-cos-v1 | recursive_character | 3000 | bge-reranker-v2-m3 | 1.00 | 14 |
| FiQA-2018 | TPE (Warm) | 0.4883 | 0.4523 | multi-qa-mpnet-base-cos-v1 | sentence | 512 | bge-reranker-v2-m3 | 0.90 | 16 |
| FiQA-2018 | IRAH (Cold) | 0.4793 | 0.4980 | multi-qa-mpnet-base-cos-v1 | none | 0 | none | 0.75 | 30 |
| FiQA-2018 | IRAH (Warm) | 0.4797 | 0.4938 | multi-qa-mpnet-base-cos-v1 | recursive_character | 2760 | none | 0.79 | 50 |
| CQADupStack-STATS | Grid Search | 0.3803 | 0.3602 | bge-large-en-v1.5 | none | 0 | none | 0.67 | 30 |
| CQADupStack-STATS | TPE (Cold) | 0.3845 | 0.3655 | multi-qa-mpnet-base-cos-v1 | sentence | 512 | ms-marco-MiniLM-L-6-v2 | 0.80 | 10 |
| CQADupStack-STATS | TPE (Warm) | 0.3830 | 0.3702 | multi-qa-mpnet-base-cos-v1 | recursive_character | 1500 | none | 0.60 | 38 |
| CQADupStack-STATS | IRAH (Cold) | 0.3818 | 0.3632 | multi-qa-mpnet-base-cos-v1 | sentence | 512 | none | 0.67 | 50 |
| CQADupStack-STATS | IRAH (Warm) | 0.3837 | 0.3616 | bge-large-en-v1.5 | sentence | 832 | none | 0.69 | 50 |
| CoSQA | Grid Search | 0.3406 | 0.3350 | bge-large-en-v1.5 | sentence | 256 | none | 1.00 | 20 |
| CoSQA | TPE (Cold) | 0.3391 | 0.3148 | bge-large-en-v1.5 | token | 256 | none | 1.00 | 35 |
| CoSQA | TPE (Warm) | 0.3414 | 0.3414 | bge-large-en-v1.5 | token | 512 | none | 0.80 | 27 |
| CoSQA | IRAH (Cold) | 0.3388 | 0.3100 | bge-large-en-v1.5 | token | 256 | none | 1.00 | 30 |
| CoSQA | IRAH (Warm) | 0.3427 | 0.3411 | bge-large-en-v1.5 | token | 512 | none | 1.00 | 20 |
Table 12. Selected configurations for each optimization method, together with their optimization-set and held-out nDCG@10 and pipeline parameters. For stochastic methods, the configuration is taken from the run with the highest optimization-set score.
The warm-start setting also yields consistently higher performance than the cold-start setting for IRAH on all three in-distribution benchmarks. Held-out nDCG@10 increases by 0.0101, 0.0124, and 0.0059 on SciFact, FiQA-2018, and CQADupStack-STATS, respectively. Per-evaluation analyses further show that the meta-learned initialization does not account for the gains of warm-started IRAH alone. LATS-IR improves upon the best warm-start seed on every benchmark, with the largest gain on the OOD CoSQA target. The source of these gains is dataset dependent. Local Phase 2 refinement contributes most on FiQA-2018 and CQADupStack-STATS, whereas broader Phase 1 exploration drives the improvement on CoSQA (Appendix § 8.§ 1).
Out-of-Distribution Generalization We further evaluate transfer to CoSQA, our OOD benchmark. Although all four evaluation benchmarks are held out from meta-learner training, CoSQA lies substantially farther from the meta-training distribution, with (Table 1). Consistent with this larger distribution shift, the meta-learner’s top-ranked anchor configuration achieves only 0.2593 held-out nDCG@10 on CoSQA (Appendix § 8.§ 2), substantially below its direct transfer performance on the in-distribution targets.
Despite this weaker transfer signal, meta-learned initialization benefits downstream search. IRAH (Warm) improves from 0.3088 to 0.3336 held-out nDCG@10 over IRAH (Cold), a gain of 0.0248 (8.0%). Although TPE (Warm) achieves the highest score on this benchmark (0.3420), IRAH (Warm) remains competitive and substantially improves over its cold-start counterpart. This suggests that LATS-IR can recover strong performance from weaker meta-learned initialization, and that transferred configuration information can remain useful under distribution shift even when direct meta-learner recommendations are less reliable.
Runtime Efficiency
Method runtime varies across datasets. As shown in Table 3, IRAH (Warm) is the fastest method on SciFact and CQADupStack-STATS, while TPE is faster on CoSQA. On SciFact, IRAH (Warm) reduces method runtime by approximately 50% relative to Grid Search. FiQA-2018 incurs substantially greater runtime for IRAH (Warm). Appendix § 8.§ 4 further analyzes this behavior using physical index usage and pure search/evaluation time, showing that the increase is not explained by collection count alone.
| Method | SciFact | FiQA | CQADupStack-STATS | CoSQA (OOD) |
| Grid Search | 6.75 | 56.92 | 42.83 | 18.02 |
| TPE (Cold) | 5.73 ± .20 | 50.89 ± 3.19 | 37.11 ± 4.14 | 2.06 ± .47 |
| TPE (Warm) | 5.62 ± .41 | 44.71 ± 4.72 | 38.73 ± 2.65 | 1.77 ± .03 |
| IRAH (Cold) | 5.99 ± 1.15 | 47.41 ± 6.72 | 59.92 ± 2.75 | 5.69 ± 2.93 |
| IRAH (Warm) | 3.37 ± .11 | 74.49 ± 20.06 | 33.28 ± 2.53 | 2.89 ± .95 |
Table 3. Method runtime (hours). Results are mean ± standard deviation over three seeds. Grid Search is a single run. Lower is better. Best per dataset is shown in bold.
Qualitative Search Behavior
Beyond retrieval performance, IRAH retains natural-language reflections that provide an inspectable record of the search process. Figure 2 shows a representative three-trial sequence on SciFact. After a strong configuration at Trial 13 (nDCG@10 = 0.763), Trial 14 jointly changes chunk size and embedding model, reducing performance to 0.731. The subsequent reflection identifies the embedding-model change as a likely source of degradation and motivates a more targeted mutation. At Trial 15, the search restores bge-large while retaining finer chunk granularity, increasing nDCG@10 to 0.770 and establishing a new trajectory peak.

Figure 2. Representative IRAH reflection trace on SciFact. An unsuccessful mutation is followed by a reflection-guided corrective mutation that reaches a new trajectory peak.
Although multiple configuration choices still change across the sequence, the trace illustrates how stored reflections can redirect search after an unsuccessful mutation while exposing the rationale behind subsequent decisions. Such process-level traceability is particularly valuable in real-world deployments, where inspectable records of system decisions can support human oversight, debugging, and internal auditing ((Raji et al., 2020)).
Conclusion
We introduced IRAH, an agentic framework for IR configuration optimization that combines LATS-IR, a reflection-guided hierarchical search procedure, with meta-learned warm-start initialization. A key aspect is that LATS-IR organizes search into macro-structural exploration followed by local numerical refinement, with natural-language reflections to inform subsequent mutations and provide an inspectable trace. Across all four benchmarks, warm-started IRAH achieves the highest average held-out nDCG@10 (0.4816) and outperforms its cold-start counterpart. These results show that hierarchical agentic search with meta-learned warm starts is a competitive approach. to retrieval pipeline optimization under constrained evaluation budgets.
Limitations
Our evaluation covers four target benchmarks and one retrieval backend. Although these benchmarks span scientific, financial, community-question, and code-retrieval domains, broader evaluation across retrieval systems, larger collections, and additional domains is needed to establish how consistently IRAH’s search behavior generalizes.
LATS-IR relies on an LLM for reflection and candidate mutation throughout optimization. Our experiments use only gpt-oss-120b and we do not evaluate sensitivity to the choice of language model. This incurs inference overhead, consuming roughly 0.6 million tokens per 50-trial run. Future work could investigate smaller specialized models or distilled reflection policies while preserving the inspectability of the search trajectory.
Finally, warm-start effectiveness depends on how well the meta-training benchmarks represent the target distribution. Our OOD experiment shows that meta-learned initialization can remain useful under distribution shift, though direct-transfer quality is weaker than for the in-distribution targets. Expanding the diversity of meta-training datasets and incorporating uncertainty into warm-start selection are important directions for improving robustness.
Additional Experimental Setup
Meta-Learner Instantiation
To construct the warm-start prior, we evaluate three offline meta-learning models across 11 public IR benchmarks (Appendix § 7.1.1) using Group -Fold cross-validation grouped by dataset ID to prevent cross-domain leakage. Offline training data are generated by evaluating 36 candidate retrieval configurations on each source dataset (Appendix § 7.1.2).
Each dataset is represented by 13 corpus- and query-level meta-features (Appendix § 7.1.3), which are concatenated with 14 configuration-specific features to form a 27-dimensional representation for each dataset–configuration pair. The meta-learner is trained to score candidate configurations according to the target retrieval metric (i.e., nDCG@10), aligning predicted seed configurations with the downstream optimization objective.
Among the evaluated models, XGBRanker achieves one of the highest out-of-fold Spearman- correlations and is selected as the final meta-learner (Appendix § 7.1.4). It is subsequently retrained using the full 11-dataset meta-training corpus. For an unseen target dataset at inference time, the system extracts its 13-dimensional meta-feature representation, scores the 36 candidate anchor configurations, and returns the top-3 ranked configurations for warm-start initialization.
Meta-Training Benchmarks
The meta-training corpus comprises 11 public IR benchmarks spanning diverse domains and scales. These include SciDocs ([9] (Thakur et al. 2021)), ArguAna ((Wachsmuth et al., 2018; Thakur et al., 2021)), NFCorpus ((Boteva et al., 2016; Thakur et al., 2021)), Vaswani ((Vaswani and Cameron, 1970)), Cranfield ([8]), and six CQADupStack subforums ([13] (Thakur et al. 2021)): Physics, GIS, Programmers, Android, Mathematica, and Webmasters.
Offline Training Data Collection
To generate ground-truth utility targets, we evaluate each meta-training benchmark across a targeted 36-configuration subset (cfg_01–cfg_36 in Table 16). These configurations were selected to provide broad coverage of the retrieval search space, spanning six physical index collections, multiple dense embedding models and document chunking strategies, and varied numerical parameters (, ) and reranker architectures, while avoiding the computational cost of exhaustive evaluation. The resulting meta-training corpus contains 396 dataset–configuration pairs from 11 source datasets and 36 anchor configurations.
Meta-Feature and Configuration Representation
Each dataset is represented by a 13-dimensional meta-feature vector capturing corpus, query, and query–corpus characteristics. The features comprise: (i) Embedding Similarity Statistics (mean_similarity, similarity_std, and similarity_moe); (ii) Query–Corpus Interaction (query_vocab_overlap and avg_query_length); (iii) Document and Length Statistics (avg_doc_length_words, doc_len_std, doc_len_cv, char_per_word_ratio, and query_doc_len_ratio); and (iv) Lexical Diversity (vocab_entropy, type_token_ratio, and query_vocab_entropy).
Lexical statistics are computed from tokenized query and corpus text. Query–corpus vocabulary overlap is
where and denote the unique query and corpus vocabularies. Document-length coefficient of variation and the query–document length ratio are respectively
where and denote query and document lengths. Type–token ratio is , where is the total number of corpus tokens, and lexical entropy is computed as
Embedding-similarity features summarize query–corpus semantic similarities using their empirical mean, standard deviation, and margin-of-error estimate.
For each candidate configuration, the dataset representation is concatenated with 14 configuration-level features: fusion weight , retrieval depth , chunk size, three embedding-model indicators, four reranker indicators, and four chunking-strategy indicators. This yields a 27-dimensional feature vector for each dataset–configuration pair.
At inference, the target dataset’s 13-dimensional meta-feature vector is paired with each of the 36 anchor configurations. The trained meta-learner scores all 36 pairs, and the top configurations by predicted utility are used to initialize warm-started LATS-IR.
Cross-Validation and Model Selection
We compare three offline meta-learning models for predicting per-configuration retrieval utility: (1) XGBRanker with a rank:pairwise objective, (2) Random Forest Regressor, and (3) Ridge Regression. We note the meta-learning formulation is model-agnostic. In our experiments, configuration utility is defined by the downstream optimization metric (i.e., ).
To prevent cross-dataset leakage, hyperparameter tuning uses 5-fold Group -Fold cross-validation grouped by dataset ID, ensuring that all configuration evaluations from a dataset remain within the same fold.
Hyperparameters are selected by grid search using dataset-grouped cross-validation and out-of-fold Spearman rank correlation () between predicted and observed configuration utilities:
XGBRanker (54 combinations):
max_depth,learning_rate,n_estimators, andsubsample, withcolsample_bytree = 0.8and a pairwise ranking objective.Regression Baselines: Random Forest (27 combinations) and Ridge () are tuned using the same grouped cross-validation protocol.
Although Random Forest achieves a marginally higher cross-validation Spearman correlation (0.6352 vs. 0.6294 in Table 4), we select XGBRanker because its pairwise learning-to-rank objective directly matches the downstream task of ranking configurations for warm-start selection. The selected XGBRanker uses max_depth = 4, learning_rate = 0.1, n_estimators = 50, subsample = 1.0, and colsample_bytree = 0.8.
| Model | CV Spearman | Type |
| RandomForest | 0.6352 | Pointwise Regressor |
| XGBRanker (Selected) | 0.6294 | Pairwise Learning-to-Rank |
| Ridge | 0.3772 | Linear Regressor |
Table 4. Out-of-fold Spearman rank correlation () under dataset-grouped cross-validation for candidate meta-learners.
To further evaluate cross-dataset generalization, we perform leave-one-dataset-out evaluation of the selected XGBRanker, training on 10 source datasets and evaluating its configuration ranking on the held-out dataset. Table 5 reports Spearman correlation together with ranking-oriented metrics that directly evaluate the quality of the top-ranked warm-start candidates. Top-1 regret measures the utility gap between the highest-ranked predicted configuration and the oracle-best configuration; Regret@3 uses the best configuration among the top three predictions. Hit Rate@3 indicates whether the oracle-best configuration appears in the predicted top three, while Recall@3 measures overlap between the predicted and oracle top-three sets.
Table 5: Leave-one-dataset-out evaluation of XGBRanker across the 11 meta-training datasets. Metrics evaluate predicted configuration rankings; Config. nDCG@3 denotes configuration-ranking nDCG rather than retrieval nDCG.
| Target Dataset | Spearman | Top-1 Regret | Regret@3 | Hit Rate@3 | Recall@3 | Config. nDCG@3 |
| arguana | 0.4432 | 0.0782 | 0.0000 | 1.0000 | 0.3333 | 0.8684 |
| cqadupstack/android | 0.9197 | 0.0000 | 0.0000 | 1.0000 | 0.6667 | 0.9977 |
| cqadupstack/gis | 0.9794 | 0.0000 | 0.0000 | 1.0000 | 0.6667 | 0.9968 |
| cqadupstack/mathematica | 0.9369 | 0.0000 | 0.0000 | 1.0000 | 1.0000 | 1.0000 |
| cqadupstack/physics | 0.9799 | 0.0000 | 0.0000 | 1.0000 | 1.0000 | 1.0000 |
| cqadupstack/programmers | 0.9669 | 0.0000 | 0.0000 | 1.0000 | 0.6667 | 0.9989 |
| cqadupstack/webmasters | 0.8396 | 0.0000 | 0.0000 | 1.0000 | 0.6667 | 0.9908 |
| nfcorpus | 0.8108 | 0.0000 | 0.0000 | 1.0000 | 0.6667 | 0.9888 |
| scidocs | 0.8583 | 0.0000 | 0.0000 | 1.0000 | 0.3333 | 0.9451 |
| cranfield | 0.6166 | 0.0243 | 0.0239 | 0.0000 | 0.0000 | 0.9391 |
| vaswani | 0.2462 | 0.0909 | 0.0536 | 0.0000 | 0.0000 | 0.8616 |
| Mean | 0.7816 | 0.0176 | 0.0070 | 0.8182 | 0.5455 | 0.9625 |
Table 5.
Leave-one-dataset-out evaluation yields a mean Spearman correlation of 0.7816 and configuration-ranking nDCG@3 of 0.9625. The low mean Regret@3 (0.0070) further indicates that high-utility configurations are generally retained near the top of the predicted ranking, which is particularly relevant because IRAH uses the top-3 ranked configurations for warm-start initialization.
Feature-importance analysis (Table 6–Table 7) shows that the meta-learner relies on both dataset characteristics and configuration choices when ranking warm-start candidates. Dataset-level features account for 48.9% of cumulative normalized gain importance, while continuous configuration parameters, embedding choice, reranker choice, and chunking strategy account for the remainder. At the individual-feature level, the fusion weight has the highest importance, although importance is distributed across multiple dataset and configuration features rather than being dominated by a single factor.
Table 6: Top-12 feature importances for the trained XGBRanker, using the default XGBoost sklearn feature_importances values (normalized gain for tree-based boosters).
| Feature | Normalized Gain Importance |
| alpha ( fusion weight) | 0.0988 |
| emb_bge-large-en-v1.5 | 0.0666 |
| reranker_none | 0.0607 |
| emb_bge-m3 | 0.0604 |
| query_doc_len_ratio | 0.0593 |
| type_token_ratio | 0.0479 |
| query_vocab_overlap | 0.0464 |
| doc_len_cv | 0.0427 |
| mean_similarity | 0.0404 |
| top_k | 0.0402 |
| similarity_std | 0.0379 |
| query_vocab_entropy | 0.0372 |
Table 6.
Table 7: Aggregate feature importance by category for the XGBRanker meta-learner.
| Feature Group | # | Cumulative Importance | Mean Importance |
| Continuous Parameters (, , chunk size) | 3 | 0.1636 (16.4%) | 0.0545 |
| Embedding Model | 3 | 0.1572 (15.7%) | 0.0524 |
| Dataset Structural / Lexical Features | 11 | 0.4359 (43.6%) | 0.0396 |
| Reranker Model | 4 | 0.1323 (13.2%) | 0.0331 |
| Similarity / Diagnostics | 2 | 0.0530 (5.3%) | 0.0265 |
| Chunking Strategy | 4 | 0.0579 (5.8%) | 0.0145 |
| Total / Average | 27 | 1.0000 (100.0%) | 0.0370 |
Table 7.
LATS-IR Instantiation
LATS ((Zhou et al., 2023)) applies tree search and reflection to language-agent task-solving trajectories. In contrast, LATS-IR operates over complete IR pipeline configurations, with actions corresponding to structural or numerical configuration mutations. The main formulation is described in § 3.2; here we provide the phase-specific search space and implementation details used in our experiments.
Search Space and Phase-Specific Refinement
The joint pipeline search space combines categorical macro-structural choices with numerical parameters . Table 8 specifies the discrete base space used during structural search. Phase 2 subsequently permits localized off-grid refinement of selected numerical parameters.
Table 8: Base pipeline search space used for macro-structural search. Phase 2 performs localized numerical refinement beyond the discrete values listed here, as described below.
| Pipeline Parameter | Option Space / Parameter Bounds |
| Chunking Strategy | none (size: 0, overlap: 0)<br>token (256/25, 512/50, 512/100)<br>sentence (256/25, 512/50, 512/100)<br>recursive character (1500/150, 3000/300, 3000/600) |
| Dense Embedding Model | multi-qa-mpnet-base-cos-v1 (Song et al., 2020)<br>BAAI/bge-large-en-v1.5 (Xiao et al., 2023)<br>BAAI/bge-m3 [6] |
| Cross-Encoder Reranker | ms-marco-MiniLM-L-6-v2 (Reimers and Gurevych, 2019)<br>BAAI/bge-reranker-large (Xiao et al., 2023)<br>BAAI/bge-reranker-v2-m3 [6]<br>none (no reranking) |
| Retrieval Depth () | Phase 1 grid: |
| Fusion Weight () | Phase 1 grid: |
Table 8.
In all experiments, LATS-IR evaluates a fixed budget of 50 trials. Tree depth is capped at 18 as a safety guardrail, while node selection within the
tree is governed by UCT. The two search phases differ in the type and range of configuration mutations permitted:
Phase 1 (Macro-Structural Search): The optimizer searches over categorical pipeline structure, including chunking strategy, embedding model, and reranker. Numerical parameters are restricted to the coarse grids and . All categorical choices in Table 8 are eligible for mutation.
Phase 2 (Micro-Numerical Refinement): The categorical identity of the best-performing Phase 1 architecture is held
fixed, while LATS-IR performs localized numerical refinement around its Phase 1 parameters. Chunk size may vary within a token/character window around the Phase 1 value and is not restricted to the discrete values in Table 8. Chunk overlap is not independently optimized; it is obtained from the canonical table when available or otherwise set using . Retrieval depth is adjusted within a window around the Phase 1 value, bounded to . Fusion weight is refined using an initial local window that narrows to after three refinement cycles, with step size .
Adaptive UCT Exploration Weight
Rather than using a fixed exploration weight in Equation (2), LATS-IR recomputes before each node selection using a bias-corrected Exponentially Weighted Moving Average (EWMA) of observed trial rewards, with decay factor . The resulting weight controls the UCT exploration–exploitation trade-off within each search phase and is distinct from the structural-versus-numerical distinction between Phase 1 and Phase 2.
Let denote the bias-corrected EWMA reward. We use phase-specific schedules:
Phase 1 (Macro-Structural Search):
The comparatively large exploration weight encourages broad traversal of alternative structural configurations, while gradually placing greater emphasis on higher-valued branches as observed rewards increase.
Phase 2 (Micro-Numerical Refinement):
The lower exploration range emphasizes local refinement around the selected Phase 1 architecture. When recent reward estimates are lower, increases toward , allocating relatively more UCT exploration among candidate refinement paths.
Reflection and Mutation Implementation
At each evaluated node, LATS-IR uses an LLM-based reflection and mutation procedure to inform subsequent candidate generation. After evaluating a configuration, the reflection prompt provides the model with the evaluated configuration, its observed retrieval scores, the branch trajectory leading to the node, and the rationale associated with the proposed configuration. For non-root nodes, the trajectory includes the parent configuration and reward together with the observed parent-relative outcome; root nodes are treated as initial expansions.
The reflection prompt follows the template:
You are evaluating an IR retrieval configuration for a search optimization experiment.
Config: {embedding}, {chunker}, , , {reranker}.
Scores: {metric values}.
Search branch that led here: {branch trajectory and prior critiques}.
Rationale: {candidate-generation rationale}.
Write exactly two sentences: (1) summarize what the scores suggest about the configuration, and (2) suggest what the next expansion should try or avoid.
The resulting reflection is stored with the search node and can be reused as context during later candidate generation. Mutation prompts condition on the UCT-selected branch, previous evaluations, the active search phase, and the valid choices or bounds for that phase. The shared context includes:
Branch trajectory: {selected root-to-leaf path with reflections}.
Trial history: {previously evaluated configurations and scores}.
Observed patterns: {summary of prior outcomes}.
Objective: {optimization objective}.
Target score: {target reward}.
During Phase 1, the prompt permits both structural mutations and coarse numerical changes:
Phase 1: Structural and Coarse Numerical Exploration. Explore combinations of chunking strategy, embedding model, and reranker, together with the allowed coarse chunk-size, , and values. Use only the valid choices provided for this phase, and do not intentionally repeat an exact configuration from the trial history.
During Phase 2, the selected categorical structure is held fixed and the prompt restricts proposals to local numerical refinement:
Phase 2: Local Numerical Refinement. Keep the selected chunking strategy, embedding model, and reranker fixed. Refine within the active local window, vary near the selected value, and adjust chunk size within the allowed range. Avoid previously tried Phase 2 configurations.
Candidate configurations are emitted through a structured tool-call interface rather than free-form text:
propose_candidate_eve(embedding_model,
reranker_model, alpha, top_k,
chunk_strategy, chunk_size,
chunk_overlap, reasoning)
After proposing the requested candidates, the model issues a finalization call. Each proposed configuration is validated against the enumerated model choices, chunking recipes, and numerical bounds before retrieval evaluation. Invalid tool arguments are rejected and may be corrected within the bounded tool-call budget.
Implementation constants. Each expansion requests candidate configurations. The proposal agent is allowed up to six tool-call iterations, permitting invalid tool arguments to be corrected before finalization. Invalid configurations are not passed to retrieval evaluation. Stochastic optimization runs use a maximum budget of 50 evaluated configurations and three random seeds.
Duplicate handling. The candidate-generation prompt includes the prior trial history and explicitly instructs the model not to repeat previously evaluated configurations. Duplicate rejection is not additionally enforced as a hard constraint by the proposal tool.
All reflection and mutation calls use openai/gpt-oss-120b (OpenAI, 2025). Reflection and proposal generation uses temperature 0.
Optimization Baselines
We benchmark IRAH against two standard optimization baselines, a fixed non-exhaustive Grid Search and Tree-structured Parzen Estimator (TPE) (Bergstra et al., [3]) optimization implemented with Optuna (Akiba et al., [2]).
The Grid Search evaluates 108 predefined pipeline configurations spanning 18 physical index collections. Each collection is determined by a specific dense embedding model and chunking configuration, while reranker choice, retrieval depth , and fusion weight vary across configurations. The complete specification of all 108 configurations is provided in Table 16.
TPE is evaluated under the same 50-trial budget as IRAH and searches over the same pipeline components, including chunking strategy, embedding model, reranker, retrieval depth , and fusion weight . Categorical choices follow those listed in Table 8, while numerical parameters are sampled over their allowed ranges: and . Cold-started TPE begins without meta-learned initialization, whereas TPE (Warm) receives the same meta-learned seed initialization used by IRAH (Warm).
Retrieval Engine and Evaluation Protocol
All candidate retrieval configurations are evaluated using the same managed enterprise retrieval engine. Retrieval depth denotes the number of retrieved passage chunks rather than aggregated documents. The base search grid uses , while optimization methods may propose additional values within their method-specific numerical bounds (Section A.2.1).
Hybrid lexical-semantic retrieval is controlled by a fusion weight , where corresponds to pure lexical BM25 retrieval and corresponds to pure dense vector retrieval; intermediate values combine the two retrieval signals.
Additional Results
Search Dynamics and Phase Contributions
Optimization Trajectories
Figure 3 complements the final held-out results in Table 2 by showing how each stochastic optimizer evolves over the full 50-configuration evaluation budget. Each curve reports the best optimization-set nDCG@10 observed up to a given evaluation, averaged across three seeds, with shading indicating one standard deviation. Thus, the trajectories characterize both the quality of the initialization and the additional improvement obtained through downstream search, rather than only the final selected configuration.
Figure 3. Best-so-far optimization-set nDCG@10 over the 50-evaluation budget, averaged across three seeds; shading denotes ± one standard deviation. Warm variants use the same meta-learned initialization. Held-out queries are used only for final evaluation in Table 2.
The trajectories show that warm-start initialization often places the search in a strong region early in the optimization process. This effect is particularly visible on FiQA-2018 and CQADupStack–STATs, where the warm variants begin substantially above their cold-start counterparts. Subsequent search yields additional gains, but the magnitude and timing of improvement vary across datasets. On CoSQA, the out-of-distribution target, IRAH (Warm) rapidly improves during the first several evaluations despite the weaker direct transfer of the meta-learned initialization reported in Appendix § 8.§ 2. This illustrates how downstream search can compensate for a less reliable warm start under distribution shift.
The figure also shows that optimization-set performance does not directly determine the final held-out ranking. On SciFact and FiQA-2018, TPE (Warm) reaches a slightly higher best-so-far optimization score, whereas IRAH (Warm) achieves the strongest held-out performance in Table 2. Conversely, on CQADupStack-STATS, IRAH (Warm) reaches a comparable or slightly higher optimization-set incumbent while TPE (Warm) performs better on the held-out queries. On CoSQA, the two warm methods converge to very similar optimization-set performance, while TPE (Warm) retains a small held-out advantage.
These trajectories provide a process-level view of the optimization behavior that is not visible from final held-out scores alone. They show how meta-learned initialization and downstream search interact under a fixed evaluation budget, while also demonstrating that the relative quality of the configurations selected by the optimizers cannot be inferred solely from their best observed score on .
Improvement Beyond Warm-Start Initialization
Table 9 quantifies the improvement obtained by LATS-IR beyond the configurations supplied by the offline meta-learner. Warm-started IRAH improves over its strongest initialization on all four benchmarks, with optimization-set gains ranging from 0.0039 on SciFact to 0.0156 on CoSQA. Thus, the performance of warm-started IRAH is improved by the downstream hierarchical search which continues to identify stronger configurations after initialization.
| Dataset | Best seed | Final best | Search gain |
| SciFact | 0.7577 .0000 | 0.7616 .0025 | +0.0039 .0025 |
| FiQA-2018 | 0.4725 .0000 | 0.4795 .0003 | +0.0070 .0003 |
| CQADupStack-STATS | 0.3795 .0003 | 0.3835 .0002 | +0.0040 .0003 |
| CoSQA | 0.3249 .0000 | 0.3405 .0038 | +0.0156 .0038 |
Table 9. Optimization-set improvement of warm-started IRAH beyond the meta-learned initialization.
The largest improvement occurs on the OOD CoSQA benchmark, where the best meta-learned seed achieves 0.3249 nDCG@10 and downstream search increases the incumbent to 0.3405. This is consistent with the trajectory analysis above. Although direct transfer is weaker under the larger distribution shift, subsequent search can substantially improve upon the transferred initialization.
Phase 2 Refinement Gains
Table 10 quantifies the additional optimization-set improvement observed after Phase 1 structural and coarse numerical search. Phase 2 refinement provides its largest and most consistent gains on FiQA-2018, improving the Phase 1 incumbent by 0.0068 and 0.0070 nDCG@10 for cold- and warm-started IRAH, respectively. Warm-started IRAH also gains 0.0040 on CQADupStack-STATS. Refinement yields smaller improvements on SciFact and no additional improvement on CoSQA, where the eventual incumbent is already identified during Phase 1.
| Dataset | IRAH | Best after P1 | Final best | P2 gain |
| SciFact | Cold | 0.7705 .0041 | 0.7708 .0046 | +0.0004 .0006 |
| SciFact | Warm | 0.7605 .0035 | 0.7616 .0025 | +0.0011 .0014 |
| FiQA-2018 | Cold | 0.4694 .0081 | 0.4762 .0028 | +0.0068 .0054 |
| FiQA-2018 | Warm | 0.4725 .0000 | 0.4795 .0003 | +0.0070 .0003 |
| CQADupStack-STATS | Cold | 0.3757 .0078 | 0.3764 .0078 | +0.0007 .0008 |
| CQADupStack-STATS | Warm | 0.3795 .0003 | 0.3835 .0002 | +0.0040 .0003 |
| CoSQA | Cold | 0.3333 .0074 | 0.3333 .0074 | +0.0000 .0000 |
| CoSQA | Warm | 0.3405 .0038 | 0.3405 .0038 | +0.0000 .0000 |
Table 10. Optimization-set improvement from Phase 2 local numerical refinement.
These results indicate that the contribution of local numerical refinement is dataset dependent. On some benchmarks, Phase 1 already identifies the eventual incumbent, whereas on others Phase 2 yields measurable additional gains without revisiting the selected categorical structure. Because warm- and cold-started runs use different phase allocations, these results characterize the observed search process rather than a controlled phase ablation; different allocations or larger phase-specific budgets could change the relative contribution of each stage.
Table 9: Optimization-set improvement of warm-started IRAH beyond the meta-learned initialization. “Best seed” is the strongest of the three meta-learned seed configurations, and “Final best” is the strongest configuration identified over the complete search. Values are mean standard deviation over three runs; gain is computed within each run before aggregation.
Table 10: Optimization-set improvement from Phase 2 local numerical refinement. “Best after P1” is the highest nDCG@10 observed by the end of Phase 1, and “Final best” is the best value observed over the complete search. Values are mean standard deviation over three seeds. Phase 2 gain is computed within each run before aggregation.
Meta-Learned Initialization Quality
Table 11: Meta-feature distance (), nearest meta-training benchmark, and held-out performance of the top-ranked XGBRanker configuration. denotes the difference relative to the held-out performance of the configuration selected by Grid Search on the optimization split.
| Dataset | Nearest Source | Seed nDCG@10 | ||
| SciFact | 0.679 | NFCorpus | 0.7479 | +0.0158 |
| FiQA-2018 | 0.354 | Webmasters | 0.4883 | +0.0476 |
| CQADupStack-STATS | 0.396 | Mathematica | 0.3610 | +0.0008 |
| CoSQA | 2.092 | SciDocs | 0.2593 | −0.0757 |
Table 11. Meta-feature distance (), nearest meta-training benchmark, and held-out performance of the top-ranked XGBRanker configuration.
Table 11 characterizes the quality of meta-learned initialization for the three in-distribution targets and the OOD CoSQA benchmark. The nearest meta-training dataset under the 13-dimensional representation (Appendix § 7.1.3) is reported to contextualize each target’s meta-feature distance; the meta-learner itself is trained jointly across all 11 source datasets.
For the three in-distribution targets, the top-ranked XGBRanker configuration provides strong held-out retrieval performance and exceeds the held-out performance of the Grid-selected configuration by 0.0158, 0.0476, and 0.0008 nDCG@10, respectively. In contrast, direct transfer is substantially weaker on the OOD CoSQA target. The top-ranked seed achieves 0.2593 and trails the Grid-selected configuration by 0.0757. This contrast is consistent with the substantially larger meta-feature distance observed for CoSQA and illustrates a limitation of direct meta-learned transfer under distribution shift.
Although the XGBRanker seed is selected from the 36 anchor configurations contained within the larger 108-configuration Grid Search space, the two approaches use different selection criteria. Grid Search selects the configuration with the highest performance on the optimization split, whereas the meta-learner ranks anchor configurations using target dataset characteristics and relationships learned across source benchmarks. The top-ranked meta-learned seed achieves higher held-out performance than the Grid-selected configuration on all three in-distribution targets, suggesting that the learned prior can identify initialization candidates that generalize well to held-out queries. The CoSQA result, however, shows that this direct transfer can degrade under larger distribution shifts, motivating continued downstream exploration by LATS-IR.
IRAH Search Dynamics
Figure 4 illustrates the trial-level search dynamics for FiQA-2018 Run 0. The trajectories show a qualitative difference in the candidate configurations evaluated by the warm- and cold-started searches. IRAH (Cold) exhibits substantial variation, including a drop to approximately 0.242 nDCG@10 at
Figure 4. Trial-level optimization trajectories on FiQA-2018 (Run 0). The shaded regions indicate the Phase 1/Phase 2 schedule for IRAH (Warm) (Trials 0–11 and 12–49, respectively). IRAH (Cold) uses a longer Phase 1 schedule (Trials 0–29), transitioning to Phase 2 at Trial 30. TPE trajectories are shown over the same 50-trial budget for comparison.
Trial 18, while the TPE trajectories also contain intermittent low-performing evaluations throughout the search. In contrast, IRAH (Warm) enters Phase 2 with a high-performing configuration and subsequently evaluates configurations concentrated near nDCG@10 .
For this run, the pattern is consistent with meta-learned initialization placing LATS-IR in a promising region before local Phase 2 refinement. The resulting late-stage stability also suggests a potential direction for future work on convergence-based early stopping when additional evaluations produce little improvement.
Runtime and Index Construction
Grid Search Timeout Failures Across the complete Grid Search evaluation space (108 configurations 4 datasets = 432 configuration–dataset evaluations), 50 evaluations failed due to search-backend socket timeouts: 5 on SciFact, 15 on FiQA-2018, 17 on CQADupStack-STATS, and 13 on CoSQA. Timed-out configurations were treated as non-viable under the evaluation budget.
Selected Configurations Table 12 reports the configuration with the highest optimization-set nDCG@10 for each method, together with its optimization-set and corresponding held-out performance. For stochastic methods, we report the selected configuration from the run achieving the highest optimization-set score. This table is intended to characterize the configurations found by each optimizer; aggregate held-out performance across runs is reported in Table 2.
The selected configurations also reveal several qualitative patterns. Within a dataset, methods often agree on major structural components while differing in chunking and numerical settings. For example, all methods select bge-large-en-v1.5 on SciFact, while all selected CoSQA configurations use the same embedding and no reranker. IRAH additionally selects off-grid chunk sizes such as 2760 on FiQA-2018 and 832 on CQADupStack-STATS, illustrating the effect of Phase 2 local numerical refinement beyond the coarse Phase 1 grid. Finally, optimization-set and held-out scores do not always rank runs identically, reinforcing the use of aggregate held-out performance in Table 2 for method comparison.
Index-Build Overhead Because index-defining choices such as the embedding model, chunking strategy, and chunk size may require a distinct physical retrieval index, optimization runtime is not determined solely by the number of evaluated trials. In contrast, changes to retrieval depth , fusion weight , and reranking can reuse an existing physical index in our implementation.
Table 13 therefore reports the number of unique physical index collections required by each method. Grid Search deterministically spans 18 collections, whereas stochastic methods report the mean and standard deviation across three runs.
| Dataset | Method | Mean Std. | Range |
| SciFact | Grid Search | 18.0 | [18,18] |
| TPE (Cold) | 20.0 1.0 | [19,21] | |
| TPE (Warm) | 19.3 1.2 | [18,20] | |
| IRAH (Cold) | 17.3 2.1 | [15,19] | |
| IRAH (Warm) | 11.7 0.6 | [11,12] | |
| FiQA-2018 | Grid Search | 18.0 | [18,18] |
| TPE (Cold) | 20.7 1.5 | [19,22] | |
| TPE (Warm) | 18.0 2.0 | [16,20] | |
| IRAH (Cold) | 16.7 2.1 | [15,19] | |
| IRAH (Warm) | 21.3 5.5 | [16,27] | |
| CQADupStack-STATS | Grid Search | 18.0 | [18,18] |
| TPE (Cold) | 19.7 2.3 | [17,21] | |
| TPE (Warm) | 20.7 1.5 | [19,22] | |
| IRAH (Cold) | 20.7 5.0 | [16,26] | |
| IRAH (Warm) | 15.7 3.2 | [12,18] | |
| CoSQA | Grid Search | 18.0 | [18,18] |
| TPE (Cold) | 20.3 2.3 | [19,23] | |
| TPE (Warm) | 19.0 1.0 | [18,20] | |
| IRAH (Cold) | 19.7 3.1 | [17,23] | |
| IRAH (Warm) | 20.7 0.6 | [20,21] |
Table 13.
Table 13: Unique physical index collections required by each optimization method. Stochastic methods report mean std. and range across three runs; Grid Search is deterministic. Lowest mean collection count per dataset is shown in bold.
The index-collection counts help contextualize the method-runtime results in Table 3. On SciFact, IRAH (Warm) requires only 11.7 0.6 unique collections on average, compared with 18 for Grid Search and 17.3 2.1 for IRAH (Cold), coinciding with a reduction in optimization time to 3.37 0.11 hours from 6.75 and 5.99 1.15 hours, respectively. On CQADupStack-STATS, IRAH (Warm) similarly reduces both collection count (15.7 3.2 versus 20.7 5.0) and runtime (33.28 2.53 versus 59.92 2.75 hours) relative to IRAH (Cold).
Collection count alone, however, does not determine runtime cost. On CoSQA, IRAH (Warm) evaluates slightly more collections than IRAH (Cold) (20.7 0.6 versus 19.7 3.1), while still reducing the reported optimization time from 5.69 2.93 to 2.89 0.95 hours.
Runtime Diagnostics To further examine this relationship, Table 14 reports runtime diagnostics for FiQA-2018, where IRAH (Warm) exhibits a large method-runtime overhead.
| Method | Method Runtime (h) | Pure Search / Eval. (h) | Collections |
| TPE (Warm) | 44.71 | 3.73 | 18.0 |
| IRAH (Cold) | 47.41 | 9.71 | 16.7 |
| IRAH (Warm) | 74.49 | 24.91 | 21.3 |
Table 14.
Table 14: Runtime diagnostics on FiQA-2018. Method runtime corresponds to the optimization-loop wall time, while pure search time measures retrieval/evaluation time excluding explicit collection-build cost. Values are means across three runs.
On FiQA-2018, IRAH (Warm) has a method runtime of 74.49 hours, compared with 44.71 hours for TPE (Warm), while its pure search/evaluation time is also substantially larger (24.91 versus 3.73 hours). Together with the larger number of physical collections (21.3 versus 18.0), these diagnostics indicate that the FiQA runtime increase is not explained by collection count alone.
Across the three IRAH (Warm) runs, selected recursive-character chunk sizes include 3120, 2950, and 2760, illustrating how Phase 2 explores configurations outside the discrete Phase 1 grid. Thus, the higher FiQA-2018 runtime appears to reflect both additional index-related work and increased search/evaluation cost rather than collection count alone. Decentralizing Phase 2 chunk-size proposals, improving index reuse, or incorporating systems cost into the mutation policy are potential directions for reducing this overhead.
IRAH Agentic Token Cost
Table 15: Agentic LLM token usage for IRAH across three seeds. Values report mean total tokens per run standard deviation. Reduction denotes the decrease from Cold to Warm initialization. Grid Search and TPE do not invoke an LLM during optimization and are omitted.
Table 15 reports the LLM token overhead associated with reflection and mutation in IRAH. Warm-started IRAH uses fewer total tokens than its cold-started counterpart on all four benchmarks, with reductions ranging from 2.6% on CoSQA to 10.7% on SciFact, and an average reduction of approximately 5.8%. Because both variants use the same retrieval-evaluation budget, these differences reflect changes in agentic reasoning and context overhead rather than fewer retrieval evaluations. These reductions are consistent with the combined effects of meta-learned initialization and the warm-start phase allocation, which alter the search trajectory and associated LLM context overhead while preserving the same retrieval-evaluation budget.
| Dataset | IRAH (Cold) | IRAH (Warm) | Reduction |
| SciFact | 594.1k 9.8k | 530.6k 6.1k | 10.7% |
| FiQA-2018 | 589.3k 6.1k | 568.3k 7.3k | 3.6% |
| CQADupStack-STATS | 595.9k 29.9k | 559.5k 2.7k | 6.1% |
| CoSQA | 581.1k 10.4k | 565.9k 6.2k | 2.6% |
| Average | 590.1k | 556.1k | 5.8% |
Table 15.
References
- [1]Anum Afzal, Xueru Zheng, and Florian Matthes. 2026. Beyond grid search: Leveraging Bayesian optimization for accelerating RAG pipeline optimization. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), pages 263–277. Association for Computational Linguistics.
- [2]Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623–2631.arxiv.org/abs/1907.10902
- [3]James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, volume 24.
- [5]Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):1–43.DOI
- [6]Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bangkok, Thailand. Association for Computational Linguistics.arxiv.org/abs/2402.03216
- [7]Pengzhou Chen and Tao Chen. 2026. Cds4rag: Cyclic dual-sequential hyperparameter optimization for rag. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26, pages 4188–4196. International Joint Conferences on Artificial Intelligence Organization. Main Track.arxiv.org/abs/2605.08333
- [8]Cyril W Cleverdon. 1967. The cranfield tests on index language devices. Technical report, Aslib Proceedings.DOI
- [9]Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020. Specter: Document-level representation learning using citation-informed transformers. In ACL.arxiv.org/abs/2004.07180
- [10]Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. Advances in Neural Information Processing Systems (NeurIPS), 28.
- [11]Jia Fu, Xiaoting Qin, Fangkai Yang, Lu Wang, Jue Zhang, Qingwei Lin, Yubo Chen, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. 2024. AutoRAG-HP: Automatic online hyper-parameter tuning for retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3875–3891. Association for Computational Linguistics.arxiv.org/abs/2406.19251
- [12]Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.arxiv.org/abs/2312.10997
- [13]Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015. CQADupStack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian Document Computing Symposium, pages 3:1–3:8. ACM.


