SearchMaster: Grounded and Regulated Self-Play for Search Agents

Introduction

LLM-based search agents improve knowledge-intensive question answering by retrieving evidence from the web [22, 47, 14, 43]. Recent work has improved this capability either through reinforcement learning for tool use [10, 33, 53, 41, 31] or by constructing multi-turn search data for supervised training [54, 15, 6, 13]. Despite different training objectives, both rely on high-quality search data: questions that require cross-document retrieval and trajectories that acquire evidence through targeted search. Obtaining such data is expensive, often requiring human annotation or expert demonstrations.

Self-play appears to be a natural solution: the model can generate search tasks and learn by solving them. Existing search self-play methods have explored this direction by constructing questions from predefined answers [18] or through open-ended exploration [49]. While these approaches reduce reliance on externally written tasks, search-agent self-play has a distinct failure mode: self-generated tasks and rollouts can look useful while providing weak or misleading supervision. For example, the model may browse multiple documents yet still write a question answerable from one of them alone. Moreover, prior search self-play methods typically proxy task difficulty by rollout accuracy, often favoring tasks with moderate success rates; however, this signal may fail to distinguish shallow lookup tasks from those requiring multi-step search. Finally, document opening often exposes useful evidence, but the model may gradually over-rely on it, revisiting already-seen documents instead of acquiring targeted evidence. Thus, the central challenge is not merely generating more data, but stabilizing the self-play loop so that its training signals reinforce evidence-grounded tasks, multi-step search, and efficient tool use.

To address this challenge, we propose SearchMaster, a grounded and regulated self-play framework. SearchMaster uses a shared trainable policy for two roles: a Proposer, which generates multi-hop search tasks within a local search environment, and a Solver, which answers these tasks through browser-tool rollouts. A frozen Verifier evaluates task quality and solution accuracy, turning verified rollouts into rewards for GRPO training [30].

SearchMaster operationalizes this grounded-and-regulated principle through three mechanisms that jointly ground self-generated tasks and regulate the search behavior they induce (Figure 1). The Evidence-Chain Generator (ECG) requires the Proposer to incrementally build an explicit cross-document evidence chain during exploration and derive the question from the full chain. The Search-Depth Reward (SDR) scores a task by the search depth of its successful rollouts, encouraging the Proposer to generate tasks that require deeper retrieval. The Over-Opening Penalty (OOP) constrains the open-to-search ratio, discouraging redundant opening without penalizing tasks that legitimately require inspecting multiple documents.

Overall, our main contribution is to formulate search-agent self-play as a problem of grounding and regulating self-generated supervision. This perspective turns a potentially noisy self-play loop into one that promotes cross-document task grounding, search-depth-aware task selection, and balanced tool use. Our experiments show that SearchMaster substantially improves the Qwen3.5-9B backbone (Qwen Team [28]) across six deep-search benchmarks, raising average accuracy from 38.19% to 51.52% (+13.3 points), with an especially large 30.1-point gain on BrowseComp-Plus (Chen et al. [5]) (30.12% to 60.24%). Notably, this improvement relies solely on a local search environment, without human-labeled data or expert supervision. We will release the code, trained checkpoints, and self-generated training data.

Related Work

Search Agent Training

Early search agents primarily relied on prompting or human demonstrations for web tool use (Guu et al. [8]; Lewis et al. [11]; Schick et al. [29]). For instance, WebGPT (Nakano et al. [22]) trained language models to browse and cite web evidence using human demonstrations, while ReAct (Yao et al. [47]) used prompting to interleave reasoning with tool usage. These approaches show that external evidence improves answer reliability; however, achieving stable long-horizon search behavior demands further training.

Recent efforts to improve search-agent training broadly fall into two categories. The first focuses on optimizing search behavior through reinforcement learning and reward design on existing QA tasks (Tan et al. [35]; Mei et al. [20]). Search-R1 (Jin et al. [10]) and R1-Searcher (Song et al. [33]) use outcome-based rewards to encourage search during reasoning. Recognizing the limitations of final-answer rewards in guiding long trajectories, subsequent methods incorporate richer process-level signals: R-Search (Zhao et al. [53]) introduces multiple rewards for search and reasoning quality, StepSearch (Wang et al. [41]) applies step-wise feedback to improve retrieval and reduce redundancy, AutoRefine (Shi et al. [31]) iteratively refines retrieved knowledge across search calls, and AutoSearch (Sun et al. [34]) adapts search depth to balance accuracy and efficiency.

The second line of work aims to construct higher-quality training data by generating multi-turn search trajectories and challenging tasks. OpenResearcher (Li et al. [15]) builds an offline search environment and leverages a teacher model to produce multi-turn search-and-browse trajectories. WebSailor (Li et al. [13]) emphasizes task difficulty by creating high-uncertainty web tasks through structured sampling and information obfuscation. Extensions include synthetic data generation, multi-agent collaboration, and staged post-training (Chu et al. [6]; Liu et al. [17]; Yao et al. [48]). Despite these advances, acquiring search-worthy tasks and trajectories remains costly, often relying on human annotation, expert models, or complex generation pipelines.

Overall, while these approaches advance search-agent training, obtaining scalable multi-hop search data remains a challenge. This motivates self-play, where models generate and learn from their own search experiences.

Self-Play for Search Agent Training

Self-play offers a promising avenue to reduce reliance on external labeled data. In reasoning and tool-use domains, prior work shows that models can bootstrap learning by generating tasks and solutions themselves when validity and correctness can be verified (Huang et al. [9]; Zhao et al. [52]; Acikgoz et al. [1]). However, self-play for search agents is more challenging, since generated tasks must be grounded in evidence, require genuine multi-hop search, and induce meaningful rather than shallow tool-use trajectories.

Search Self-Play (Lu et al. [18]) is an early attempt at self-play for search-agent training: it samples a target answer from a predefined set, and the Proposer searches for supporting evidence to formulate a question pointing to it. While this enables self-play, task generation remains constrained by predefined answers rather than open-ended exploration. Dr. Zero (Yue et al. [49]) instead lets the Proposer generate questions directly through external search, rewarding task difficulty by the Solver’s success rate. This moves toward more autonomous task generation, but success rate remains a coarse signal: it reflects whether the Solver answers correctly, not whether the task requires nontrivial search beyond a shallow lookup. Beyond this difficulty-estimation issue, pseudo multi-hop questions and over-opening drift remain underexplored. SearchMaster addresses these search-specific issues by making evidence construction, task difficulty, and tool-use balance explicit parts of the self-play loop, enabling more reliable self-evolution of search agents.

Method

Problem Setup and Framework Overview

Figure 2 illustrates the SearchMaster framework. It trains a single policy $\pi_\theta$ within a search environment $\mathcal{E}$ constructed over document collection $\mathcal{D}$. The policy interacts with $\mathcal{E}$ through browser tools $\mathcal{T} = \{\text{search}, \text{open}, \text{find}\}$, where search returns ranked documents for a query, open reveals a document’s content, and find locates exact matches within opened documents.

During training, the same policy $\pi_\theta$ acts as both the Proposer $P_\theta$ and the Solver $S_\theta$. The Verifier $V$ is a frozen copy of the initial model and remains fixed.

Each self-play iteration samples unlabeled seed documents $d$ from $\mathcal{D}$. For each seed, the Proposer samples $K_P$ task-generation rollouts, each maintaining an ECG evidence chain $c$ and producing a candidate task $(c, q, y^\star)$. Quality filters remove invalid candidates before Solver rollouts. For each passing task, the Solver receives only $q$ and produces $K_S$ answer rollouts. The frozen Verifier judges each answer, producing correctness scores $z_i$, and we records the search depth $h_i$ of each rollout as the number of unique search queries. These signals define rewards for both roles: $z_i$ rewards Solver correctness, successful-rollout depths estimate task difficulty for the Proposer through SDR, and OOP is subtracted from both rewards to discourage excessive opening. The resulting samples form a Proposer GRPO group over the $K_P$ task-generation rollouts and a Solver GRPO group over the $K_S$ answer rollouts of the highest-reward passing task. Since one seed can produce multiple passing tasks and up to $K_P \times K_S$ Solver rollouts, using all Solver rollouts would make Solver samples dominate the update; we therefore keep only the $K_S$ Solver rollouts of the highest-reward passing task.

Evidence-Chain Task Generation

Generating genuine multi-hop search tasks is challenging because pseudo multi-hop tasks can arise even when the Proposer is explicitly instructed to generate questions requiring cross-document evidence. In one failure mode, the Proposer stays close to the seed, searching for surface-level facts and asking a question answerable from the seed alone. In another, it reaches related documents but does not encode the cross-document links into the question, leaving the final answer recoverable from just one visited document. Examples of both cases are provided in the Supplementary Material.

To address this, the Evidence-Chain Generator (ECG) turns task proposal into explicit chain construction. Starting from a seed document $d$, the Proposer repeatedly follows entity or fact links across documents: it selects a salient entity, searches for a connected document, identifies a new entity or fact, and extends the chain. The question is then generated with explicit requirements for full-chain necessity and no single-document answer.

Formally, for each $d$, the Proposer samples $K_P$ independent rollouts:

where $\mathcal{P}_{\mathrm{prop}}$ is the Proposer prompt (Supplementary Material), $\tau_j^P$ the tool-use trace, $c_j = e_{j,1} \rightarrow e_{j,2} \rightarrow\cdots\rightarrow e_{j,n_j}$ the evidence chain of entities or facts, $q_j$ the question generated from $c_j$, and $y_j^\star$ its reference answer. By deriving $q_j$ from this constrained chain rather than a single document, ECG biases task generation toward questions that require combining evidence across documents.

Before launching Solver rollouts, SearchMaster applies four filters to each candidate task:

• Tool-use filter: treats candidates with too few search or open actions as shallow.

• Format filter: checks that $(c_j, q_j, y_j^\star)$ are present and that $c_j$ contains at least two evidence items.

• Validity filter: asks the Verifier whether $q_j$ is valid, $y_j^\star$ is supported by $\tau_j^P$, and the chain is logically sound.

• Parametric-knowledge filter: removes questions the Verifier can answer without tools.

A candidate that fails any filter receives a low Proposer reward, whereas those passing all four are sent to the Solver.

Solver Rollouts and Search Depth

After a task passes the filters, the Solver receives only the question $q_j$ and produces $K_S$ independent answer rollouts

where $P_{\mathrm{solve}}$ is the Solver prompt (Supplementary Material), $\tau_i^S$ is the tool-use trace, and $\hat{y}_i$ is the final answer. The frozen Verifier compares $\hat{y}_i$ with the reference answer $y_j^\star$ and outputs a correctness score $z_i \in\{0,1\}$. We also compute the search depth $h_i$ as the number of unique search queries in $\tau_i^S$, and let $C = \sum_{i=1}^{K_S} z_i$ denote the number of successful rollouts. These signals play different roles: $z_i$ rewards Solver correctness, while the search depths of successful rollouts estimate how much search the proposed task requires for the Proposer reward.

Reward Design and Over-Opening Penalty

The Proposer reward favors tasks that are solvable but still challenging for the current Solver. Candidates that fail the quality filters receive low rewards. Passing tasks with $C = 0$ or $C = K_S$ receive only a low base reward $r_0 = 0.2$, since they are respectively too hard or too easy. If all $K_P$ tasks generated from a seed receive rewards that do not exceed $r_0$, we discard all data generated from that seed.

For the remaining tasks with $0 < C < K_S$, success rate alone does not distinguish shallow lookup from deeper search. We therefore measure task difficulty by the easiest successful search depth,

rather than by the mean or maximum search depth. If any successful rollout solves the task with shallow search, the task admits a shortcut and should not receive a high difficulty reward. We then compute the Search-Depth Reward (SDR):

where $H_{\mathrm{sdr}}$ is the target depth at which the reward saturates. This cap prevents SDR from rewarding arbitrarily many searches, while still favoring tasks whose easiest successful solution requires nontrivial search depth.

We now summarize the base Proposer reward as follows, where the first three cases correspond to the tool-use, format, and validity/parametric filters introduced above:

For the Solver, the base reward encourages answering correctly with tool-supported evidence:

As training proceeds, the model can drift toward excessive open calls. Early in training, opening documents often provides useful information that helps the model answer correctly, so such behavior is reinforced by answer rewards. Over time, however, the policy may over-rely on open: many opens revisit already-seen documents rather than acquiring new evidence, producing long but shallow trajectories. A concrete over-opening case is provided in the Supplementary Material. Accordingly, we introduce the Over-Opening Penalty (OOP) to regularize this behavior in both roles, since the Proposer and Solver share the same policy $\pi_\theta$. OOP penalizes the ratio of open to search actions rather than the raw number of opened documents, so it discourages redundant opening without suppressing legitimate multi-document exploration. Specifically, let $n_{\mathrm{open}}(\tau)$ and $n_{\mathrm{search}}(\tau)$ count open and search calls in a trajectory $\tau$. For $n_{\mathrm{search}}(\tau)>0$, define:

where $\alpha_{\mathrm{oop}}$ and $\beta_{\mathrm{oop}}$ are the open-to-search ratios at which the penalty starts and reaches its maximum, so $\rho_{\mathrm{oop}}$ rises linearly from $0$ at $\alpha_{\mathrm{oop}}$ to $1$ at $\beta_{\mathrm{oop}}$. The final rewards then subtract OOP with coefficient $\lambda_{\mathrm{oop}}$:

Training and Optimization

After rewards are assigned, each retained seed document contributes two GRPO groups: a Proposer group containing the $K_P$ task-generation rollouts, and a Solver group containing the $K_S$ answer rollouts of the highest-reward passing task. Seeds whose highest task reward does not exceed $r_0$ are discarded. Because of this seed-level gate, the final training data matches our high-quality search-data goal: retained tasks are grounded for cross-document retrieval, successful rollouts exhibit nontrivial search depth, and OOP regularizes tool use toward targeted search.

Within each retained group, rewards are normalized into advantages $\hat{A}$ by subtracting the group mean and dividing by the group standard deviation with a small numerical constant. The same advantage is assigned to every generated token in the sample. We then update the shared policy with a token-level clipped objective and a KL regularizer,

where the expectation is over unmasked generated tokens $t$, $\gamma_t = \pi_\theta(a_t \mid s_t)/\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)$ is the per-token importance ratio, and $(\epsilon_{\mathrm{low}}, \epsilon_{\mathrm{high}})$ is the asymmetric clip range. Tool observations remain in the context but are masked out of the loss, so gradients flow only through model-generated tokens. Proposer and Solver samples are optimized jointly under the shared policy $\pi_\theta$. Pseudocode for the full self-play iteration is provided in the Supplementary Material.

Experiments

Experimental Setup

Implementation Details. The local search database $\mathcal{D}$ is the offline corpus released by OpenResearcher [15] (about 15M documents, $\sim11$B tokens), indexed with a Qwen3-Embedding-8B [46] FAISS retriever. We use Qwen3.5-9B (Qwen Team 2026) as the default backbone and train it with GRPO at a constant learning rate of $1 \times10^{-6}$, clip range $(\epsilon_{\mathrm{low}}, \epsilon_{\mathrm{high}}) = (0.2, 0.28)$, and KL weight $\beta_{\mathrm{KL}} = 0.001$. Each iteration randomly samples seed documents from $\mathcal{D}$ and, after the four filters and the seed-level quality gate, retains 64 qualified seed documents for training. For each seed document, the Proposer samples $K_P = 8$ rollouts, and each passing candidate launches $K_S = 8$ independent Solver rollouts for verification and reward computation, with up to 200 tool calls per rollout. Although Solver rollouts are generated for all passing candidates to compute rewards, only the Solver group of the highest-reward passing task is used for optimization, so each retained seed contributes $K_P + K_S$ training samples. Each iteration therefore contributes a global optimization batch size of $64 \times(K_P + K_S) = 1{,}024$. For reward design, we set $r_0 = 0.2$, $H_{\mathrm{sdr}} = 10$, and OOP parameters $\alpha_{\mathrm{oop}} = 1.5$, $\beta_{\mathrm{oop}} = 2.5$, and $\lambda_{\mathrm{oop}} = 0.5$. Rollouts use a 256K-token context and decoding temperature 1.0. We train for 20 iterations, so SearchMaster learns from $20 \times64 = 1{,}280$ qualified seed documents in total. Prompt templates, filter thresholds, and additional training hyperparameters are provided in the Supplementary Material.

Benchmarks. We evaluate SearchMaster on both offline and online search benchmarks. The offline benchmark is BrowseComp-Plus [5], with 830 browsing questions over an offline retrieval corpus drawn from a different source than our OpenResearcher training corpus. For transfer to live web search, we use five online search benchmarks: BrowseComp [42] (1,266 hard-to-find factual questions), GAIA [21] (103 text-only tool-use assistant tasks), SEAL-0 [27] (111 search-intensive long-tail questions), WebWalkerQA [45] (680 multi-page navigation questions), and XBench-DeepSearch [4] (100 web-search questions). During evaluation, we do not use any context-management method; the context length is 256K tokens, decoding temperature is 1.0, and outputs are scored by GPT-5 [32] against the ground-truth answers, with the judging prompt provided in the Supplementary Material.

Main Results

Table 1 reports the main results. On BrowseComp-Plus, which supports multiple retrievers, we report all methods under its Qwen3-Embedding-8B [46] retriever setting for fair comparison.

Method	BrowseComp-Plus	BrowseComp	GAIA	SEAL-0	WebWalkerQA	XBench	Avg.
Proprietary models and systems
OpenAI o3 (OpenAI 2025c)	63.49	49.70	70.50	15.30	71.70	67.00	–
OpenAI o4-mini (OpenAI 2025c)	–	28.30	60.00	–	–	–	–
GPT-5-high (Singh et al. 2025)	70.12	54.90	76.40	43.20	–	77.80	–
GPT-4.1 (OpenAI 2025b)	35.42	–	–	–	–	–	–
Claude Sonnet 4 (Anthropic 2025)	36.75	12.20	68.30	–	61.70	65.00	–
Claude Opus 4 (Anthropic 2025)	36.14	–	–	–	–	–	–
Gemini 2.5 Pro (Gemini Team 2025)	28.67	–	–	–	–	–	–
Gemini 2.5 Flash (Gemini Team 2025)	33.01	–	–	–	–	–	–
OpenAI DeepResearch (OpenAI 2025a)	–	51.50	67.40	–	–	–	–
Open-source models and systems
GLM-4.5 (Zeng et al. 2025)	–	26.40	66.00	–	65.60	70.00	–
Kimi K2 (Team et al. 2025a)	–	14.10	57.70	–	63.00	50.00	–
DeepSeek-V3.1 (Liu et al. 2024)	–	30.00	63.10	–	61.20	71.00	–
Qwen3-32B (Yang et al. 2025)	10.36	–	–	–	–	–	–
SearchR1-32B (Jin et al. 2025)	10.36	–	–	–	–	–	–
gpt-oss-120B-high (Agarwal et al. 2025)	42.89	–	–	–	–	–	–
WebThinker-32B (Li et al. 2026a)	–	2.80	48.50	–	46.50	24.00	–
WebExplorer-8B (Liu et al. 2025)	–	15.70	50.00	–	62.70	53.70	–
WebDancer-QwQ (Wu et al. 2026)	–	3.80	51.50	–	47.90	–	–
WebShaper-72B (Tao et al. 2025b)	–	–	60.10	–	52.20	–	–
DeepDive-32B (Lu et al. 2025b)	–	15.30	–	25.50	–	51.80	–
OffSeeker-8B (Zhou et al. 2026)	–	12.80	51.50	–	61.70	49.00	–
WebSailor-72B (Li et al. 2025b)	–	12.00	55.40	–	–	55.00	–
WebSailor-V2-30B (Li et al. 2025a)	–	35.30	74.10	–	–	73.70	–
WebLeaper-C (Tao et al. 2025a)	–	38.80	73.20	48.60	–	72.00	–
BrowseMaster (Pang et al. 2025)	–	30.00	68.00	–	62.10	66.00	–
OpenResearcher-30B (Li et al. 2026b)	54.80	26.30	64.10	–	–	65.00	–
REDSearcher-30B (Chu et al. 2026)	–	42.10	80.10	–	–	–	–
Tongyi-DR-30B (Team et al. 2025c)	–	43.40	70.90	–	72.20	75.00	–
MiroThinker-72B (Team et al. 2025b)	–	47.10	81.90	51.00	62.10	77.80	–
Ours (Qwen3.5-9B backbone)
Qwen3.5-9B (Qwen Team 2026)	30.12	20.93	50.49	26.13	41.47	60.00	38.19
SearchMaster	60.24	28.75	57.28	35.14	59.71	68.00	51.52

Self-play yields substantial gains on BrowseComp-Plus. SearchMaster raises the Qwen3.5-9B backbone from 30.12% to 60.24%, an absolute gain of 30.1 points. This lifts the 9B model above much larger open-source systems such as gpt-oss-120B-high (42.89%) and proprietary models such as GPT-4.1 (35.42%) and Claude Opus 4 (36.14%), while narrowing the gap to stronger reasoning systems such as OpenAI o3 (63.49%). Notably, this is achieved purely through self-play over a local search environment, without human-written questions or advanced teacher models. These results show that a single model can bootstrap strong deep-search ability entirely from its own generated tasks. The Supplementary Material examines the source of this gain: the base model already issues valid tool calls but searches ineffectively, whereas SearchMaster learns to search more deeply and follow evidence across documents.

The learned search behavior generalizes to live web search. Although SearchMaster is trained entirely in an offline search environment, it also performs strongly on online benchmarks that query the live, open web. It improves the backbone on all five online search benchmarks, with gains ranging from +6.8 on GAIA to +18.2 on WebWalkerQA. This transfer from offline training to live web search shows that SearchMaster equips the model with search behavior that generalizes beyond its training environment.

Ablation Study

In this section, we study the contribution of ECG, SDR, and OOP. All variants use the same training budget and filtering pipeline unless otherwise noted. We first train a naive self-play baseline that keeps the same task filters and Solver verification as SearchMaster, but uses rollout accuracy as the task-difficulty signal, via a success-rate term $1 - \lvert2C/K_S - 1\rvert$, where $C$ is the number of successful Solver rollouts among $K_S$ attempts, so that the reward peaks when half of the rollouts succeed. Starting from this baseline, ECG changes the Proposer prompt into evidence-chain construction, SDR replaces the success-rate difficulty signal with the search-depth reward, and OOP adds the over-opening penalty.

Table 2 reports the results on BrowseComp-Plus. Naive self-play alone lifts the backbone from 30.12% to 45.18%, showing that self-play over a local corpus already provides a useful signal. Adding ECG, SDR, and OOP improves accuracy monotonically, and the full SearchMaster reaches 60.24%. These mechanisms improve complementary parts of the self-play loop: chain-based task construction (ECG), search-depth-aware task selection (SDR), and tool-use regularization (OOP). The rest of this section examines whether each mechanism produces its intended behavioral effect, beyond the final accuracy.

Variant	ECG	SDR	OOP	BrowseComp-Plus
Qwen3.5-9B	-	-	-	30.12
naive self-play				45.18
+ ECG	✓			53.25
+ SDR		✓		52.89
+ ECG + SDR	✓	✓		57.71
SearchMaster	✓	✓	✓	60.24

Does ECG Reduce Pseudo Multi-Hop Tasks? ECG makes evidence-chain construction explicit before question generation, so we directly test whether it reduces pseudo multi-hop tasks. We compare three settings on the same 1,000 seed documents: Qwen3.5-9B with the naive self-play prompt, Qwen3.5-9B with the ECG prompt, and the trained SearchMaster with ECG. Each setting generates 10 tasks per seed document. A GLM-5 judge labels each task as True Multi-Hop if it is answerable only by combining evidence across several documents, Pseudo Multi-Hop if it superficially involves multiple entities but is solvable by a single lookup, and Invalid if it leaks the answer or is unsupported by the retrieved evidence.

Table 3 shows that adding ECG to the base model increases true multi-hop tasks from 24.2% to 45.6% and reduces invalid tasks from 28.7% to 15.8%. After SearchMaster training, the true multi-hop rate further rises to 78.6%, while pseudo multi-hop drops to 15.0%. The Supplementary Material further compares Qwen3.5-9B with the naive and ECG prompts on the same seeds. In one case, the naive prompt searches around seed-level facts and asks a seed-answerable question; in another, it reaches related documents but leaves their links unused in the final question. These examples suggest that ECG helps in two ways: it drives the model to explore beyond seed-level facts, and it requires the explored evidence to be organized into an explicit chain before question generation. Together, these effects turn loosely related browsing into task-relevant cross-document dependencies.

Method	True Multi-Hop ↑	Pseudo ↓	Invalid ↓
Qwen3.5-9B + naive	24.2	47.1	28.7
Qwen3.5-9B + ECG	45.6	38.6	15.8
SearchMaster + ECG	78.6	15.0	6.4

Does SDR Keep the Training Tasks Search-Intensive?

SDR scores a task by $h_{\min}$, the minimum search depth among successful Solver rollouts (Eq. 4). Because $h_{\min}$ reflects the easiest successful solution, a large value indicates that the task cannot be solved by a shallow lookup. To test whether SDR keeps retained tasks search-intensive during training, we compare the naive self-play baseline, which uses a success-rate difficulty signal, with the +SDR setting from Table 2. We track $h_{\min}$ across iterations in Figure 3, discarding a few tasks with extremely large $h_{\min}$ for visualization.

Early in training, both settings produce tasks with increasing $h_{\min}$, suggesting that self-play initially discovers tasks requiring more search. Later, the two runs diverge. Under the success-rate signal, the Proposer has no direct incentive to preserve search depth: as the Solver improves, retained tasks become solvable with fewer searches, and $h_{\min}$ settles around 2–3. SDR instead rewards tasks whose easiest successful solution still requires substantial search, so $h_{\min}$ remains in the 8–10 range even as the Solver strengthens. This shows that SDR prevents the training tasks from decaying into shallow lookup tasks and maintains a multi-step search signal.

Does OOP Control Open Overuse?

To evaluate whether OOP regulates tool use, we compare two settings from Table 2: +ECG+SDR without OOP and the full SearchMaster with OOP. Since the Proposer and Solver share the same policy, we track the average open-to-search ratio of both roles across training iterations (Figure 4).

Without OOP, the open-to-search ratio of both roles increases steadily, indicating a drift toward opening documents rather than issuing targeted searches. To inspect this behavior, the Supplementary Material analyzes late-training Solver rollouts with at least twice as many opens as searches. In these over-opening rollouts, 56.3% of the extra open calls simply re-open already-seen documents rather than acquiring new evidence. With OOP, the ratio of both roles remains low, showing that the penalty curbs over-opening while still allowing the searches and opens needed for genuine multi-document tasks.

Conclusion

We introduce SearchMaster, a self-play framework for training search agents from tasks discovered in a local search environment. The central challenge is not merely question generation, but transforming autonomous exploration into useful training data for search behavior. SearchMaster addresses this challenge through ECG for evidence-grounded task construction, SDR for search-depth difficulty calibration, and OOP for preventing over-opening in tool-use trajectories. Across six deep-search benchmarks, SearchMaster raises the same 9B backbone’s average accuracy from 38.19% to 51.52%, with a 30.1-point gain on BrowseComp-Plus.

While SearchMaster reduces dependence on annotated QA data, it does not eliminate all costs. Training still requires a searchable environment, multiple Solver rollouts per proposed task, and Verifier calls for grounding, parametric filtering, and correctness. The current implementation operates in a local search environment, which improves reproducibility, though it may limit coverage of the open web’s full diversity and volatility. Future work can extend SearchMaster beyond local corpora to more diverse and dynamic web environments, while reducing the rollout and verification cost of self-play training.