betaFeedback

WorksComputer Science

Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?

Abstract

Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule. We study this problem through self-distillation, where a frozen teacher has access to task guidance unavailable to the student, and conduct a three-stage controlled study across three models and four interactive environments. We find that student-induced histories paired with teacher-generated action prefixes generally provide the strongest standalone training signal, whereas following the student at both levels generally performs worst. Nevertheless, fully on-policy data becomes beneficial when incorporated into a mixture with other configurations. Moreover, mixtures with identical teacher/student proportions at each level can yield substantially different performance, showing that how the two levels are paired matters beyond the overall amount of on-policy data. Finally, front-loading fully teacher-generated contexts and delaying fully on-policy contexts improves over a static mixture. These findings provide practical principles for deciding where, how much, and when agent distillation should follow the student.

Introduction

Knowledge distillation (KD) [5] is widely used to post-train large language model (LLM) agents [22], transferring task-solving behavior from a teacher to a student [23, 7]. In privileged-information (PI) distillation [12] (Alakuijala et al. 2026), the teacher receives task guidance unavailable to the student, and distillation trains the student to reproduce the resulting behavior without that guidance at deployment.

A central design choice is which policy generates the contexts used for distillation. Off-policy approaches train on teacher-generated trajectories [6, 10], providing coherent demonstrations but limited exposure to states encountered by the student. On-policy distillation (OPD) [1, 11] instead obtains teacher supervision on student-generated experience, reducing training–deployment mismatch. However, student errors can produce histories or partial actions from which even the teacher has difficulty providing useful continuations [3]. Both approaches can be effective in agent training [10, 18, 19] (Alakuijala et al. 2026), raising a basic question: should agent distillation be on-policy?

We argue that the trajectory-level notion of on-pol iciness commonly used to characterize distillation methods is incomplete for interactive agents. An agent’s training context contains two nested levels. At the history level, preceding interactions determine the context and environment state before the current action. At the action-prefix level, previously generated tokens within the current action determine the context for next-token prediction. Either level can independently come from the teacher or the student. For example, a teacher can generate an action from a student-induced history, while a student can generate an action from a teacher-induced history. The relevant question is therefore not only whether training is on-policy, but at which level and to what extent should distillation follow the student?

We introduce hierarchical agent distillation, which independently specifies the history source and action-prefix source. Choosing teacher or student at each level yields four configurations, TTTT, STST, TSTS, and SSSS, under a common distillation objective. This formulation separates three design choices: where the student is followed, how much each source configuration contributes to training, and when different configurations are emphasized during learning.

We study these choices through a controlled three-stage investigation. Stage 1 isolates the effects of history and action-prefix sources and their interaction. Stage 2 examines how the four configurations should be paired and mixed under a fixed training budget. Stage 3 studies how their allocation should change over training. We use self-distillation, initializing teacher and student from the same checkpoint while providing task guidance only to the frozen teacher, thereby controlling for architecture, model size, and initialization. Across three backbones and four interactive environments, we obtain the following findings:

  • The value of on-policy training depends on the level. Student-induced histories paired with teacher-generated action prefixes (STST) generally provide the strongest standalone performance, while fully on-policy training (SSSS) performs worst across the tested settings. Student histories expose the teacher to more opportunities for corrective supervision, whereas student-generated prefixes can constrain the usefulness of teacher continuations.

  • Pairing matters beyond the amount of on-policy data. Mixtures with identical proportions of student-generated histories and action prefixes can perform differently depending on how the two sources are paired. The strongest tested mixtures emphasize hybrid configurations while retaining contributions from both fully teacher-generated and fully on-policy contexts. In particular, a modest allocation to SSSS can improve mixed training despite its weak standalone performance.

  • The timing of data exposure matters. Front-loading TTTT accelerates early learning, whereas delaying SSSS improves later and final validation performance. Combining both schedules outperforms a static allocation throughout training and achieves the strongest final validation performance among the schedules tested.

Hierarchical Agent Distillation

Preliminaries

An interactive LLM agent alternates between generating actions and receiving environment feedback. Let hh denote the interaction history before the current action, including the task instruction and preceding actions and observations. The action is generated autoregressively, and uu denotes a token prefix within it. The student predicts the next token according to πθ(⋅∣h,u)\pi_{\theta}(\cdot\mid h,u). The teacher additionally receives privileged task guidance gg and predicts πτ(⋅∣h,u,g)\pi_{\tau}(\cdot\mid h,u,g). Distillation trains the student to match this informed teacher while operating without gg at deployment.

The contexts (h,u)(h,u) used for distillation may be generated by either policy. Teacher-trajectory context distillation uses teacher-generated histories and action prefixes (Alaküjala et al., 2026), whereas fully on-policy distillation uses student-generated histories and prefixes and queries the teacher for supervision on these contexts [1, 16]. We study self-distillation, initializing teacher and student from the same checkpoint and keeping the teacher frozen. The fully on-policy case therefore corresponds to on-policy self-distillation (OPSD) [24].

Hierarchical agent distillation framework

We decompose on-pol iciness into two nested choices: which policy generates the interaction history, and which policy generates the current-action prefix conditioned on that history. This decomposition allows the two sources to be controlled independently, combined in different proportions, and scheduled over training.

Illustration of the four source configurations

Figure 1. Illustration of the four source configurations. The first letter denotes the interaction-history source and the second the current-action-prefix source. In all configurations, the PI-conditioned teacher provides distillation supervision, while gradients update only the student.

Two Levels of On-Policiness

Let H∈{T,S}H \in\{T,S\} denote the history source and A∈{T,S}A \in\{T,S\} the action-prefix source, where TT and SS denote the teacher and current student. For a fixed data-collection round, let dH(h,g)d_H(h,g) denote the distribution of histories induced by source HH, and let rA(u∣h,g)r_A(u \mid h,g) denote the distribution of action prefixes generated by source AA from history hh. Student generation ignores gg. The history source determines the preceding interaction trajectory, while the prefix source generates tokens only within the selected current action; the prefix therefore always remains conditioned on the chosen history.

Training is on-policy at the history level when H=SH=S and at the action-prefix level when A=SA=S. The four combinations are summarized in Figure 1. TTTT corresponds to teacher-trajectory context distillation, while SSSS recovers fully on-policy self-distillation. STST obtains teacher continuations at student-induced histories, analogous to expert supervision on learner-visited states in dataset aggregation [13]. TSTS instead generates student prefixes from teacher-induced histories, as in prefix replay. In all four configurations, the PI-conditioned teacher provides the distillation target.

A Common Training Objective

For configuration HAHA, the student minimizes

LHA(θ)=E(h,g)∼dHEu∼rA(⋅∣h,g)[D(πT(⋅∣h,u,g),πθ(⋅∣h,u))],H∈{T,S}, A∈{T,S}.(1)\mathcal{L}_{HA}(\theta) = \mathbb{E}_{(h,g)\sim d_H}\mathbb{E}_{u\sim r_A(\cdot\mid h,g)}\left[D\left(\pi_T(\cdot\mid h,u,g),\pi_\theta(\cdot\mid h,u)\right)\right],\quad H\in\{T,S\},\ A\in\{T,S\}. \tag*{(1)}

where DD is the token-level distillation divergence. Forward KL uses D(p,q)=DKL(p∥q)D(p,q)=D_{\mathrm{KL}}(p\|q), and reverse KL uses D(p,q)=DKL(q∥p)D(p,q)=D_{\mathrm{KL}}(q\|p). The divergence and the two context sources are separate choices.

Operationally, we select a history from source HH, generate the current action from source AA, and evaluate teacher–student agreement along its token prefixes. The sampled context and teacher outputs are treated as fixed during each update; gradients are applied only to the student’s next-token distribution. The same supervision rule therefore applies to all four configurations.

Mixtures and Curricula

Let wHA≥0w_{HA}\geq0 denote the weight of configuration HAHA, with ∑H,AwHA=1\sum_{H,A}w_{HA}=1. A training mixture minimizes

Lw(θ)=∑H,A∈{T,S}wHALHA(θ).(2)\mathcal{L}_w(\theta)=\sum_{H,A\in\{T,S\}}w_{HA}\mathcal{L}_{HA}(\theta). \tag*{(2)}

The marginal weight on student histories is α=wST+wSS\alpha= w_{ST} + w_{SS}, and that on student action prefixes is β=wTS+wSS\beta= w_{TS} + w_{SS}. Holding α\alpha and β\beta fixed while changing the joint weights changes only how the two source choices are paired. For example, when α=β=12\alpha= \beta= \frac{1}{2}, equal weights on TTTT and SSSS form a diagonal mixture, equal weights on STST and TSTS form a crossed mixture, and equal weights on all four configurations form an independent mixture. These mixtures have identical marginal on-policiness but different source pairings.

Finally, allowing the weights to vary with training progress, w=w(k)w = w^{(k)}, defines a curriculum. Schedules with matched cumulative configuration weights isolate when each type of context is most useful. The framework therefore separates three choices: where to follow the student, how much each source configuration contributes, and when each should be emphasized.

Setup of Controlled Study

Models and Benchmarks

We use Qwen3-8B, Qwen3-14B (Yang et al. [20]), and Ministral-3-14B-Instruct-2512 (Liu et al. [9]), abbreviated as Ministral-3. For each backbone, teacher and student are initialized from the same checkpoint; the teacher remains frozen throughout training. We disable thinking mode for Qwen3.

We evaluate on ToolQA (Zhuang et al. [25]), OfficeBench (Wang et al. [17]), ALFWorld (Shridhar et al. [14]), and WebShop (Yao et al. [21]). Following Alakuijala et al. (2026), ToolQA and OfficeBench use train/validation/test splits of 646/107/323 and 191/9/86 tasks. For ALFWorld, we stratify by task type to sample 3,000 training and 200 validation tasks from the original training pool and retain the original evaluation split. For WebShop, we sample 3,000 training and 200 validation tasks and use the full 500-task test set. Further details are in Appendix § 10.1.2.

Privileged information construction. The teacher receives task guidance describing general solution strategies, tool-use tips, and common failure modes. For ToolQA and OfficeBench, we initialize guidance from Alakuijala et al. (2026) and refine it with Qwen3-32B using failed guidance-conditioned rollouts. For ALFWorld, we manually draft guidance per task type from failed student trajectories, use it to obtain successful Qwen3-32B trajectories, and let GPT-5.4 refine the guidance from both successes and failures. For WebShop, we manually draft guidance from 50 failed trajectories, organize it by decision stage and constraint pattern, and refine it with GPT-5.4 using corresponding successful Qwen3-32B trajectories. Examples are provided in Appendix § 10.1.1.

Training Setup

Agent architecture. Agents follow a ReAct-style interaction format (Yao et al. [22]); at each step, the model generates a reasoning segment followed by an operation conditioned on the interaction history. Operations invoke tools, execute Python code, or issue environment-specific commands.

Data collection and refresh. Before training, we collect one trajectory from the initial student and cache three trajectories from the frozen teacher for each training and validation task. Student trajectories provide SSSS examples; STST retains their histories but regenerates the current action with the teacher. A randomly selected cached teacher trajectory provides TTTT examples and the histories for TSTS, whose current actions are generated by the student. After each training round, we refresh all student-dependent data using the latest student: we regenerate student trajectories for SSSS, teacher continuations at those histories for STST, and student continuations at cached teacher histories for TSTS. TTTT continues to sample from the three cached teacher trajectories.

Training exercises. Each exercise consists of an interaction history and one current action. We uniformly sample eight action positions per source trajectory, with replacement when necessary, so each task contributes the same number of exercises. Distillation conditions on the history and supervises tokens in the current action.

Optimization. We apply rank-128 LoRA adapters to all linear layers. Training runs for eight rounds, refreshing student-dependent data between rounds. Each round contains 32, 16, 96, and 96 gradient updates for ToolQA, OfficeBench, ALFWorld, and WebShop, respectively, with global batch size 64. Full hyperparameters are in Appendix A.1.3.

Evaluation

We evaluate each trained student on every test task with three trials. The maximum action counts for ToolQA, OfficeBench, ALFWorld, and WebShop are 15, 15, 32, and 15, respectively. We report mean success rate (SR), pass@3 (the fraction of tasks solved in at least one trial), and mean action count over all trajectories. For WebShop, we additionally report the official Task Score, averaged across trials, which gives partial credit for satisfying task requirements. Decoding settings are listed in Table 4.

ModelTemperature (teacher)Top-pp (teacher)Temperature (student)Top-pp (student)
Qwen3-8B0.70.950.70.8
Qwen3-14B0.70.950.70.8
Ministral-30.091.00.081.0

Table 4. Decoding parameters for the teacher (data collection) and the student (data collection+evaluation).

Stage 1: Separating the Two Levels of On-Policiness

To isolate the effects of history- and action-prefix-level on-policiness, we train one student per configuration, matching initialization, training tasks, and the number of exercises per task within each backbone–benchmark setting.

Main Results

Table 1 reports results with forward KL. All four configurations substantially improve over the initial student, but their relative performance reveals a consistent distinction between the two levels.

The effect of student histories depends on the action-prefix source. Replacing teacher histories with student histories generally improves success rate when prefixes are teacher-generated (TT→STTT \rightarrow ST), but consistently hurts when prefixes are student-generated (TS→SSTS \rightarrow SS). Likewise, teacher prefixes provide little advantage on teacher histories (TTTT vs. TSTS: at most 1.9 points), but substantially outperform student prefixes on student histories (STST vs. SSSS: 3.6–14.4 points). Thus, student-induced histories are most effective when paired with teacher-generated actions.

STST performs best overall, while SSSS is weakest standalone. STST achieves the highest success rate in 10 of 12 settings, the highest pass@3 in all 12, and the highest WebShop Task Score for all three backbones. The two exceptions are ToolQA, where TTTT exceeds STST by 0.6 points for Qwen3-8B and Ministral-3, although STST still attains higher pass@3. In contrast, SSSS has the lowest success rate and pass@3 among distilled variants in every setting. Its average success-rate deficit relative to STST is larger for Qwen3-8B than Qwen3-14B (10.2 vs. 7.7 points), indicating greater sensitivity of the smaller model to student-generated prefixes.

The pattern is robust to KL direction. Reverse-KL results in Appendix § 10.§ 2.1 preserve the main source-level trends. Forward KL is generally similar or better, except in some SSSS settings, and is used throughout the remaining experiments.

Analysis

We next examine why the source choices interact and whether configurations that are weaker standalone nevertheless learn complementary behaviors. Annotation protocols are described in Appendix § 10.1.4.

Supervision during data collection

Student histories provide richer corrective supervision (TTTT vs. STST). For each benchmark, we sample 100 training tasks and two exercises per task. We measure Unresolved, the fraction of histories containing an uncorrected earlier error; Identified, the fraction of unresolved cases where the current action recognizes the error; and Addressed, the fraction of identified cases receiving a resolution or appropriate corrective step. Unresolved errors occur in 82.7–85.0% of STST exercises but only 14.0–19.1% of TTTT exercises (Table 6). Although identification is lower under STST, identified errors are addressed at comparable rates, giving STST substantially more opportunities for corrective supervision. Across training rounds, unresolved errors decrease while the teacher identifies a larger fraction of those remaining (Figure 5).

Table 6: Corrective supervision in TTTT and STST training exercises. Unresolved measures the percentage of histories containing an uncorrected earlier error. Identified is conditional on an unresolved error; Addressed is conditional on identification and includes either resolution or an appropriate corrective step.

ModelConfigurationUnresolved(%)Identified(%)Addressed(%)
Qwen3-8BST85.077.183.0
Qwen3-8BTT19.190.281.2
Qwen3-14BST82.780.784.5
Qwen3-14BTT14.089.386.0
Ministral-3ST83.380.682.9
Ministral-3TT15.788.182.9

Table 6.

Figure 5. Evolution of corrective supervision across training rounds for Qwen3-14B on ALFWorld.

Student prefixes constrain teacher continuations (STST vs. SSSS). We select 100 student-history exercises with unresolved errors per benchmark and compare a fully teacher-generated action with teacher continuations conditioned on the student’s reasoning, or on its reasoning plus half of its operation. We measure continuation Executability and Appropriateness. Both decline as more student-generated prefix is retained (Table 8); with half the operation retained, appropriateness drops from 80.2% to 56.2% for Qwen3-8B and from 83.8% to 60.2% for Qwen3-14B. Thus, student prefixes can constrain the teacher into less useful continuations even at the same history.

Table 8: Teacher-continuation quality at matched student histories with unresolved errors. The teacher generates either the entire current action or a continuation after the retained student reasoning, optionally followed by the first half of the student’s operation.

ModelRetained student prefixExecutability (%)Appropriateness (%)
Qwen3-8BNone (STST)92.080.2
Qwen3-8BReasoning81.070.7
Qwen3-8BReasoning + half operation73.256.2
Qwen3-14BNone (STST)94.783.8
Qwen3-14BReasoning85.276.0
Qwen3-14BReasoning + half operation76.060.2

Table 8.

History sources change where teacher and student disagree (TSTS vs. SSSS). At rounds 1, 3, 5, and 8, we sample 100 exercises per configuration containing an explicit error without subsequent self-correction (see details in Appendix § 10.1.4). We compute mean per-token KL before, within, and after the first annotated error span, then normalize the three regional means by their sum. Under TSTS, disagreement concentrates on the error span, whereas SSSS assigns relatively more disagreement before the error (Figure 8). Teacher histories therefore produce more error-localized disagreement, while student histories distribute disagreement more broadly across the action.

Figure 8. Regional teacher–student disagreement around the first annotated error in SSSS and TSTS training actions. At each selected round, we compute mean per-token KL before, within, and after the error span and normalize the three regional means by their sum.

Behavior of trained students

Configurations learn different behavioral strengths. For 80 test tasks per benchmark, we annotate one trajectory per configuration using five nonexclusive error categories: repeated failure, premature completion, rule inconsistency, operational mistakes despite correct plans, and high-level planning mistakes. SSSS exhibits fewer repeated failures and planning mistakes, whereas TTTT follows explicit rules more reliably (Table 7).

Table 7: Error profiles of trained students. Error categories are nonexclusive, so a trajectory may receive multiple labels.

ModelConfigurationRepeated failurePremature completionRule inconsistencyOperational mistakePlanning mistake
Qwen3-8BTT20%7%2%2%11%
Qwen3-8BST9%5%7%2%7%
Qwen3-8BTS17%7%7%5%7%
Qwen3-8BSS22%24%7%12%20%
Qwen3-14BTT14%7%1%2%9%
Qwen3-14BST7%2%6%2%5%
Qwen3-14BTS15%7%5%1%7%
Qwen3-14BSS20%17%9%11%19%
Ministral-3TT17%9%4%1%10%
Ministral-3ST7%4%4%2%5%
Ministral-3TS15%9%5%1%12%
Ministral-3SS22%20%9%9%22%

Table 7.

STST is strongest at recovering from student errors. For Qwen3-14B, we resume all four trained policies from the same 100 erroneous initial-model histories per benchmark on ToolQA, ALFWorld, and WebShop. STST achieves the highest rates of both error correction and eventual task success (Figure 7). Although TTTT and TSTS outperform SSSS from task starts, their eventual success from erroneous histories is similar to SSSS. Their standard-evaluation advantage therefore does not translate into stronger recovery. STST uniquely combines exposure to student mistakes with teacher-generated corrective actions.

Figure 7. Recovery evaluation with Qwen3-14B on shared erroneous histories from initial-model failures. We evaluate 100 histories per benchmark. “corrected” measures correction of the existing issue, and “succ.” measures eventual task success.

Configurations also differ in learning dynamics. For Qwen3-14B on ToolQA, ALFWorld, and WebShop, TTTT leads during rounds 1–3, STST overtakes it by round 5 and finishes highest, while SSSS improves more gradually and continues gaining later in training (Figure 6). These complementary behaviors motivate mixture design in Stage 2 and scheduling in Stage 3.

Figure 6. Validation success rate for Qwen3-14B on ToolQA, ALFWorld, and WebShop before training and after rounds 1, 3, 5, and 8.

Figure 2. Pairing at fixed marginal on-policiness with Qwen3-14B. Dashed lines indicate standalone STST performance from Stage 1.

Stage 2: Pairing and training-mixture study

Stage1Stage 1 favors STST as a standalone configuration, but weaker configurations may provide complementary supervision when mixed. We therefore study how the four configurations should be paired and allocated under a fixed training budget. All mixtures match the total number of exercises and training rounds and reuse the initial Stage 1 exercise pools. We order mixture weights as (TT,ST,TS,SS)(TT, ST, TS, SS).

Pairing Matters at Fixed On-Policiness

We compare three mixtures with identical marginal on-policiness: wDiagonal=(0.5,0,0,0.5)w_{\mathrm{Diagonal}} = (0.5, 0, 0, 0.5), wCrossed=(0,0.5,0.5,0)w_{\mathrm{Crossed}} = (0, 0.5, 0.5, 0), and wIndependent=(0.25,0.25,0.25,0.25)w_{\mathrm{Independent}} = (0.25, 0.25, 0.25, 0.25). Each assigns 50% of examples to student histories and 50% to student action prefixes, differing only in how the two sources are paired.

Figure 2 shows Qwen3-14B results; other backbones are in Appendix A.2.4. The independent mixture performs best across all shown metrics and benchmarks and improves success rate over standalone STST by 0.5–3.3 points. Thus, marginal on-policiness alone is insufficient: the joint pairing of history and prefix sources also matters.

From Complementarity to Mixture Composition

Retain both TTTT and SSSS. Holding STST and TSTS at 0.25 each, we compare TT-more, (0.5,0.25,0.25,0)(0.5, 0.25, 0.25, 0), and SS-more, (0,0.25,0.25,0.5)(0, 0.25, 0.25, 0.5), against the independent mixture. Across all settings, the independent mixture outperforms both, while TT-more outperforms SS-more (Figure 3, Figure 11, and Figure 12). Thus, retaining both diagonal configurations is more effective than assigning their combined budget entirely to either one. In particular, adding some SSSS improves over TT-more despite SSSS’s weak standalone performance. The TT-more–SS-more gap is smaller for Qwen3-14B than for Qwen3-8B, consistent with the smaller model’s greater sensitivity to fully on-policy training.

Figure 3. Retaining both diagonal configurations improves success rate with Qwen3-14B. Holding STST and TSTS at 0.25 each, the remaining allocation is assigned to TTTT only (TT-more), SSSS only (SS-more), or equally to both (Independent)

Figure 11. Retaining both diagonal configurations improves success rate with Qwen3-8B. Holding STST and TSTS at 0.25 each, the remaining allocation is assigned to TTTT only (TT-more), SSSS only (SS-more), or equally to both (Independent)

Figure 12. Retaining both diagonal configurations improves success rate with Ministral-3. Holding STST and TSTS at 0.25 each, the remaining allocation is assigned to TTTT only (TT-more), SSSS only (SS-more), or equally to both (Independent)

Emphasize hybrid configurations without removing the diagonals. We increase the combined weight of STST and TSTS from 0.5 to 0.75, yielding the hybrid-heavy mixture (0.125, 0.375, 0.375, 0.125). It improves success rate over the independent mixture in every setting (Table 9). However, removing TTTT and SSSS entirely, as in the crossed mixture, performs worse than the independent mixture. These results favor emphasizing hybrid configurations while retaining complementary TTTT and SSSS examples.

ModelToolQAOfficeBenchALFWorldWebShop
Qwen3-8B0.10.71.90.7
Qwen3-14B0.31.41.60.5
Ministral-30.30.60.70.9

Table 9. Success-rate gains, in percentage points, from replacing the Independent mixture (0.25, 0.25, 0.25, 0.25) with the hybrid-heavy mixture (0.125, 0.375, 0.375, 0.125).

Favor student histories over student action prefixes. Finally, we ask how to divide the hybrid allocation. Keeping TTTT and SSSS at 0.125 each, we compare ST-more, (0.125, 0.5, 0.25, 0.125), against TS-more, (0.125, 0.25, 0.5, 0.125). These mixtures exchange the student-history and student-prefix proportions, while preserving the combined hybrid allocation. ST-more yields higher success rate in every setting (Figure 13), extending the directional preference observed from Stage 1 to this mixed training.

Comparison of ST-more and TS-more across Qwen3-8B, Qwen3-14B, and Ministral-3-14B

Figure 13. Comparing the two hybrid directions. ST-more uses weights (0.125, 0.5, 0.25, 0.125), while TS-more uses (0.125, 0.25, 0.5, 0.125), ordered as (TTTT, STST, TSTS, SSSS).

The resulting principle of these comparisons is to emphasize student histories paired with teacher actions without excluding configurations that are weaker in isolation.

Stage 3: Scheduling the Training Mixture

Stage 2 shows that mixture composition matters, but should the same allocation be maintained throughout learning? The dynamics analysis in Stage 1 suggest otherwise: TTTT supports rapid initial progress, whereas SSSS improves more slowly and continues gaining later. We therefore investigate whether emphasizing these configurations at different stages improves mixed training.

We use the independent mixture, (0.25, 0.25, 0.25, 0.25), as a balanced reference and compare its static allocation with three schedules over 8 training rounds. TT-first places approximately 80% of the total TTTT allocation in rounds 1–4 and the remaining 20% in rounds 5–8. SS-later reverses this split for SSSS, allocating 20% early and 80% late. TT-first + SS-later combines both schedules. Each targeted configuration retains its total allocation across training. We measure validation success rate after every round on ToolQA, ALFWorld, and WebShop (Figure 4).

Figure 4. Validation success rate across eight training rounds under static and scheduled allocations of the independent mixture. initial denotes the initial student.

TTTT-first accelerates initial learning on all three benchmarks: by round 4, its success rate exceeds the static baseline by 8.7–13.2 percentage points. Its final advantage is smaller, at 0.7–2.0 points. In contrast, SSSS-later initially trails the static baseline on ToolQA and WebShop and closely tracks it on ALFWorld, but overtakes it during the second half of training and finishes 2.0–2.9 points higher. The two schedules therefore offer distinct benefits: front-loading TTTT improves early progress, while delaying SSSS yields a larger improvement in final performance.

The combined schedule outperforms the static baseline at all evaluated training rounds and achieves the highest final validation success rate among the tested schedules on all three benchmarks. Although TTTT-first alone often leads early, the combined schedule ultimately surpasses it. These results suggest complementary temporal roles: teacher-generated contexts support rapid initial learning, while greater exposure to fully on-policy contexts is more beneficial later. They favor front-loading TTTT and delaying SSSS within a mixture, rather than maintaining a uniform allocation throughout training.

Conclusions across the three stages. Across the three stages, effective on-polici­ness depends on where, how much, and when student-generated context is used. Student histories paired with teacher actions provide the strongest standalone performance and recovery, while mixed training benefits from retaining configurations that are weaker in isolation. Scheduling TTTT earlier and SSSS later further improves over a static mixture. Thus, effective agent distillation depends not on maximizing on-policy data, but on how it is sourced, combined, and scheduled.

Related Work

Distillation and privileged information for agents. Knowledge distillation transfers knowledge from a teacher model to a student model [5]. Demonstration-based approaches improve LLM agents through supervised training on interaction trajectories [23], while context distillation transfers behaviors elicited by additional instructions or reasoning context into a model that no longer requires such context at inference time [15, 4]. For interactive agents, Alakuijala et al. (2026) internalize task guidance and corrective feedback through iterative distillation, whereas Wang et al. [16] use trajectory-derived skills as privileged teacher context for on-policy self-distillation. Our work addresses the complementary question of how teacher- and student-generated contexts should be constructed, combined, and scheduled when distilling privileged guidance.

On-policy distillation and imitation learning. On-policy distillation (OPD) [1] mitigates training–deployment mismatch by obtaining teacher supervision on student-generated sequences, and has become increasingly common in LLM post-training [11, 24]. For multi-turn agents, however, student-generated interactions may reach states for which the teacher cannot provide reliable supervision. Concurrent work, ReOPD [8], addresses this issue by generating student actions from replayed teacher histories, enabling teacher supervision without requiring new environment interactions during student training. Related challenges arise in imitation learning [13], where expert supervision is collected on learner-visited states to mitigate sequential distribution shift. Our framework distinguishes history-level and action-prefix-level on-policiness as separate but coupled design choices, thereby unifying teacher-generated, student-generated, and hybrid contexts under a common objective. This formulation enables controlled analysis of their individual effects, their interactions in mixed training regimes, and their allocation over the course of training.

Conclusion

We introduced hierarchical agent distillation framework that formalizes on-policiness at the history and action-prefix levels. Across three backbones and four interactive environments, student-induced histories paired with teacher-generated actions generally provide the strongest standalone performance. Mixture comparisons show that pairing matters even when marginal on-policiness is fixed, and that configurations weaker in isolation can improve mixed training. Scheduling experiments further show that emphasizing fully teacher-generated contexts early and fully on-policy contexts later improves validation performance over a static uniform mixture. Together, these findings move agent distillation beyond a binary on-policy/off-policy choice: effective training depends on where the student is followed, how teacher and student contexts are paired, and when each configuration is emphasized.

Limitation. Our experiments focus on privileged-information-based self-distillation with frozen teachers initialized from the same checkpoints as the students. Generalization to substantially larger models remains to be established. Also, the mixture study identifies composition principles through targeted comparisons rather than a universally optimal allocation.

AI USE STATEMENT

We used large language models to assist with manuscript writing and editing, as well as in parts of the experimental pipeline, including annotating agent trajectories and constructing task guidance. AI-assisted outputs used in the experiments were generated following the procedures described in the paper and appendix and were reviewed or validated as appropriate. The authors remain responsible for the study design, experimental analysis, interpretation of results, and final manuscript.

Appendix

Additional details on experiment setup

Privileged information examples

We show the prompt used in constructing privileged information and PI examples.

Prompt

You are an expert at improving guidance for an LLM agent that must complete interactive tasks. Your goal is to optimize a set of seed hints/guidance using examples of successful and failed agent trajectories. The optimized hints will later be provided to another agent to help it solve new tasks of the same kind.

Inputs: Task / Task Type

TASK DESCRIPTION

Seed Hints: SEED HINTS

Successful Trajectories: SUCCESSFUL TRAJECTORIES

Failed Trajectories: FAILED TRAJECTORIES

Your Task

Analyze the successful and failed trajectories and revise the seed hints into a better set of guidance for solving this type of task.

Use the trajectories contrastively:

  • Identify strategies, decisions, action sequences, or checks that consistently contribute to success.

  • Identify mistakes, inefficient behaviors, incorrect assumptions, or missing checks that contribute to failure.

  • Pay particular attention to situations where successful and failed trajectories encounter similar states but take different actions.

  • Determine whether any existing seed hint is incomplete, ambiguous, misleading, overly specific, or unnecessary.

  • Preserve useful information from the seed hints while correcting or extending it when supported by the trajectories.

The resulting guidance should help an agent make better decisions during execution, rather than merely describe what happened in the demonstrations.

Requirements for the Optimized Hints

Generalize beyond the demonstrations.

Extract reusable strategies that apply to new tasks of the same type. Do not memorize task-specific object names, locations, answers, or exact action sequences unless they represent a general rule.

Be actionable.

State what the agent should do, when it should do it, and, when useful, what observations or conditions it should check before acting.

Prioritize failure prevention.

Explicitly address recurring mistakes observed in failed trajectories when they can be prevented by a general rule.

Be concise. Include only information that is likely to improve task success. Merge redundant rules and remove low-value commentary.

Reasoning Procedure

Before producing the final hints, internally:

  1. Compare successful and failed trajectories.

  2. Identify important decision points and recurring failure modes.

  3. Determine which seed hints already address them.

  4. Add, remove, merge, or revise hints as necessary.

  5. Check that every final hint is both useful and sufficiently general.

Output

Return only the optimized guidance in the following format:

.....

Do not summarize the trajectories, mention individual demonstrations, or explain how you derived the hints. The final output should be directly usable as guidance for an agent solving a new task.

Guidance Example

Figure 1.

Steps to Answer the Question:

  1. Read the question carefully.

  2. Choose appropriate tools. Read the documentation of the chosen tools to understand the functions of them.

  3. Your final answer should be a string only containing the answers to the question without explanatory text.

Specific hint: To solve the question asking about the total number of papers authored by A plus those written by the collaborators of A, you should follow:

  1. Identify the co-author(s) of A. (Answer: B).

  2. For A and B, get their joint paper by checking the edge attributes.

  3. Find out additional co-authors of B.

  4. Retrieve the papers jointly authored by B and each of B’s additional co-authors.

  5. Combine papers and remove duplicates to determine the total count

Guidance Example

Figure 2.

This is a heat-then-place task. Required subgoals:

  1. Locate and acquire the requested object.

  2. Heat that object using a microwave.

  3. Preserve the identity of the heated object.

  4. Deliver the heated object to the requested receptacle.

Useful action patterns: ......

Reasoning guidance:

  • The final placement alone is insufficient: the object must first satisfy the “hot” state requirement.

  • Search observations for the exact target object.

  • Once a target instance has been picked up, continue manipulating that same instance.

  • Do not search for the destination until necessary if the target object has not yet been acquired.

Benchmarks

We evaluate on four interactive agent benchmarks spanning tool-augmented question answering, office automation, embodied household tasks, and web interaction.

  • ToolQA. ToolQA [25] evaluates an agent’s ability to answer questions by retrieving and reasoning over information from external data sources with heterogeneous formats, including text documents, graphs, and tabular databases. Following the processed version of Alakuijala et al. (2026), we consider six task groups: DBLP, Agenda, Yelp, Airbnb, Flight, and Coffee. The agent has access to nine tools for querying and processing the corresponding data sources, while individual task groups typically require only a subset of these tools.

  • OfficeBench. OfficeBench [17] evaluates an agent’s ability to complete multi-step office workflows involving six applications: Email, Calendar, Word, Excel, OCR, and PDF Reader. The agent has access to 20 tools for interacting with these applications. Tasks can involve one to three applications, requiring the agent to plan across multiple steps, select appropriate tools, and transfer information between applications.

  • ALFWorld. ALFWorld [14] is a text-based embodied environment in which an agent follows natural-language instructions to perform household tasks through sequential interaction with objects and receptacles. We consider its six commonly used task categories: Pick & Place, Pick Two & Place, Clean & Place, Heat & Place, Cool & Place, and Examine in Light. Completing a task requires the agent to explore the environment,

locate relevant objects, and execute an appropriate sequence of navigation and manipulation actions, such as picking up, placing, cleaning, heating, or cooling objects.

  • WebShop. WebShop [21] is a simulated e-commerce environment that evaluates agents on interactive web navigation and product search. Given a natural-language shopping instruction specifying desired product properties and constraints, the agent must search for candidate products, navigate between search and product pages, inspect product information and available options, and purchase an item that best satisfies the instruction. The environment exposes two main types of actions, search[query] and click[option], requiring the agent to perform query formulation, information gathering, comparison, and multi-step decision making.

Training data statistics and hyperparameter settings

We provide the training data statistics for each benchmark in Table 2.

ParameterToolQAOfficeBenchALFWorldWebShop
Training tasks6461913,0003,000
Training rounds8888
Task visits per round256128768768
Total task visits2,0481,0246,1446,144
Equivalent task passes3.175.362.052.05
Action examples per task visit8888
Global batch size64646464
Optimizer updates per round32169696
Total optimizer updates256128768768

Table 2. Training data statistics

We run all experiments on 8 H100 80 GB GPUs and 4 AMD MI250X GPUs. We use the Adam optimizer throughout all training. Hyperparameter settings are presented in Table 3.

ParameterQwen3Ministral-3
Numerical precisionBF16BF16
LoRA rank128128
Learning rate10−510^{-5}2×10−52 \times10^{-5}
Weight decay-0.01
Warmup updates (ToolQA, OfficeBench, ALFWorld, WebShop)13, 7, 39, 39 (linear)-
Schedule after warmupConstantConstant
Distillation temperature1.01.0
Maximum generated tokens per action2,0482,048

Table 3. Training hyperparameters for the controlled study. All settings are shared across source configurations.

Trajectory Annotation Protocols

We use a two-stage annotation protocol. The first stage identifies exercises that satisfy the prerequisite for a given analysis, such as containing an issue that remains unresolved in the history. The second stage performs issue- or action-level annotation on the retained exercises. We primarily use Qwen3-235B-A22B as the annotator and use GPT-5.4 to adjudicate cases for which the primary annotator returns Uncertain.

Identifying exercises with unresolved issues. For each sampled exercise, we provide the complete interaction history to Qwen3-235B-A22B and ask it to identify all issues encountered during the trajectory and summarize how the agent responds to each issue. An issue may correspond, for example, to an incorrect intermediate decision, a failed operation, or a constraint revealed by environment feedback. Based on this analysis, the annotator determines whether at least one identified issue remains unresolved at the point of interest. It outputs one of three labels: Yes, No, or Uncertain, together with the identified issues and a brief summary of the attempted or completed resolution for each issue. Exercises labeled Uncertain are reassessed by GPT-5.4, which receives the original exercise together with the complete annotation produced by Qwen3-235B-A22B.

Annotating issue-specific agent behavior. After selecting exercises satisfying the prerequisite of the corresponding analysis, we perform more targeted annotations of the agent’s behavior. Depending on the diagnostic, we use the following procedures:

  • Issue recognition and corrective behavior. We ask Qwen3-235B-A22B to determine whether the reasoning segment of the current action explicitly recognizes an issue present in the preceding history. If an issue is recognized, the annotator further determines whether the action (i) resolves the issue or (ii) takes an appropriate corrective step toward resolving it. We additionally provide the task-specific privileged information used in our distillation framework as reference information to help the annotator interpret the task requirements and the intended solution procedure. Cases labeled Uncertain are reassessed by GPT-5.4.

  • Quality of the proposed correction. For annotations requiring a judgment about the quality of an action—for example, whether a proposed correction is executable and appropriate, or whether the action constitutes a valid self-correction—we use a solution-first annotation procedure. Before evaluating the agent’s action, the annotator is first asked to independently derive an appropriate next step given the task history and available task information. It then evaluates the agent’s proposed action relative to this independently derived solution. We found this decomposition to produce more reliable judgments than directly asking the annotator to assess action quality.

Experiment results

Reverse KL

To compare with the main results in Table 1 with forward KL, in Table 5, we show the difference after applying reverse KL.

ModelToolQA SRToolQA pass@3ToolQA Action CountOfficeBench SROfficeBench pass@3OfficeBench Action CountALFWorld SRALFWorld pass@3ALFWorld Action CountWebShop SRWebShop pass@3WebShop Task ScoreWebShop Action Count
Qwen3-8B17.22.717.2_{2.7}20.111.37.00.97.0_{0.9}8.114.237.21.737.2_{1.7}48.230.16.61.06.6_{1.0}9.438.11.538.1_{1.5}11.3
+ TT81.70.981.7_{0.9}89.26.370.93.370.9_{3.3}76.79.169.72.369.7_{2.3}77.416.770.20.770.2_{0.7}78.078.50.878.5_{0.8}7.9
+ ST81.11.581.1_{1.5}91.36.874.82.174.8_{2.1}84.98.271.72.071.7_{2.0}79.915.173.30.273.3_{0.2}83.282.21.082.2_{1.0}7.1
+ TS79.81.179.8_{1.1}87.96.471.72.571.7_{2.5}80.28.769.31.569.3_{1.5}75.515.469.10.969.1_{0.9}79.275.00.375.0_{0.3}7.5
+ SS74.51.774.5_{1.7}82.47.565.91.865.9_{1.8}73.39.660.81.860.8_{1.8}67.920.058.91.158.9_{1.1}66.873.10.973.1_{0.9}9.9
Qwen3-14B19.82.819.8_{2.8}24.111.77.80.47.8_{0.4}10.513.142.32.642.3_{2.6}52.926.99.10.79.1_{0.7}13.850.60.750.6_{0.7}12.5
+ TT84.91.284.9_{1.2}91.36.075.20.975.2_{0.9}84.98.372.11.972.1_{1.9}80.716.275.60.875.6_{0.8}81.282.70.582.7_{0.5}7.4
+ ST85.21.485.2_{1.4}92.66.277.51.477.5_{1.4}89.57.674.51.474.5_{1.4}83.915.379.41.379.4_{1.3}86.885.00.685.0_{0.6}7.2
+ TS84.30.784.3_{0.7}88.96.974.81.374.8_{1.3}82.68.572.62.072.6_{2.0}80.315.976.01.276.0_{1.2}83.081.91.281.9_{1.2}7.7
+ SS79.11.279.1_{1.2}85.17.170.90.870.9_{0.8}79.18.967.82.367.8_{2.3}76.619.467.90.967.9_{0.9}77.078.80.478.8_{0.4}8.5
Ministral-315.41.915.4_{1.9}21.110.08.51.38.5_{1.3}10.512.038.41.638.4_{1.6}49.629.19.70.89.7_{0.8}12.638.91.238.9_{1.2}10.8
+ TT82.40.982.4_{0.9}90.15.873.30.673.3_{0.6}81.48.771.01.371.0_{1.3}79.917.272.60.472.6_{0.4}79.679.91.179.9_{1.1}8.0
+ ST81.81.381.8_{1.3}93.25.576.01.076.0_{1.0}86.08.174.31.074.3_{1.0}81.815.875.51.075.5_{1.0}83.480.81.080.8_{1.0}7.6
+ TS80.90.880.9_{0.8}85.45.173.61.873.6_{1.8}82.68.571.51.171.5_{1.1}79.617.771.91.071.9_{1.0}80.077.20.877.2_{0.8}7.8
+ SS78.21.178.2_{1.1}84.86.471.30.971.3_{0.9}79.19.266.51.666.5_{1.6}75.918.664.41.164.4_{1.1}74.473.40.673.4_{0.6}8.4

Table 1. Stage 1 results with forward KL. Higher SR, pass@3, and Task Score is better. Bold and underlined values mark the best and second-best performance among the four distilled variants of each backbone. Subscripts denote standard error over three trials.

ModelToolQA SRToolQA pass@3ToolQA Action CountOfficeBench SROfficeBench pass@3OfficeBench Action CountALFWorld SRALFWorld pass@3ALFWorld Action CountWebShop SRWebShop Task ScoreWebShop Action Count
Qwen3-8B + TT↓ 0.1↓ 0.4↓ 0.2↓ 0.5↓ 0.6↑ 0.3↓ 0.4↓ 0.4↓ 0.5↓ 0.6↓ 0.7↓ 1.1
Qwen3-8B + ST↓ 0.1↓ 0.2↑ 0.4↓ 0.2↓ 0.4↑ 0.2↑ 0.2↓ 0.2↓ 0.1↑ 0.5↓ 0.3↓ 0.3
Qwen3-8B + TS↓ 0.3↓ 0.1↓ 0.4↓ 0.2↓ 0.5↓ 0.1↓ 0.5↓ 0.3↓ 0.1↓ 0.6↓ 0.7↓ 0.3
Qwen3-8B + SS↑ 0.2↑ 0.4↓ 0.3↓ 0.1↓ 0.6↑ 0.2↑ 0.2↓ 0.3↓ 0.1↑ 0.3↓ 0.4↑ 0.2
Qwen3-14B + TT↓ 0.2↓ 0.4↑ 0.3↓ 0.1↓ 0.7↑ 0.2↓ 0.5↑ 0.2↓ 0.6↓ 0.2↓ 0.4↓ 0.6
Qwen3-14B + ST↓ 0.3↓ 0.5↑ 0.1↓ 0.3↓ 0.4↓ 0.1↓ 0.3↓ 0.1↓ 0.3↓ 0.5↓ 0.7↓ 0.7
Qwen3-14B + TS↓ 0.8↓ 0.6↓ 0.2↓ 0.1↓ 0.6↑ 0.2↓ 0.2↓ 0.4↑ 0.4↓ 0.1↓ 0.5↓ 0.6
Qwen3-14B + SS↓ 0.1↑ 0.1↓ 0.3↑ 0.2↑ 0.3↑ 0.3↓ 0.4↓ 0.6↑ 0.1↓ 0.2↓ 0.1↓ 0.8
Ministral-3 + TT↓ 0.4↓ 0.6↑ 0.2↓ 0.1↓ 0.5↑ 0.2↓ 0.4↓ 0.4↓ 0.1↓ 0.2↓ 0.5↓ 0.3
Ministral-3 + ST↓ 0.2↓ 0.3↑ 0.2↓ 0.4↓ 0.6↑ 0.2↓ 0.1↓ 0.3↑ 0.2↓ 0.1↓ 0.4↓ 0.5
Ministral-3 + TS↓ 0.1↓ 0.5↑ 0.1↓ 0.4↓ 0.7↑ 0.3↓ 0.2↓ 0.5↓ 0.1↓ 0.3↓ 0.8↓ 0.5
Ministral-3 + SS↓ 0.2↓ 0.2↓ 0.1↑ 0.1↑ 0.4↑ 0.2↓ 0.2↓ 0.5↑ 0.2↓ 0.1↓ 0.3↓ 0.3

Table 5. Comparison with results in Table 1. Arrows denote the value change.

Stage 1: Analysis

Here we present results on trajectories and supervision analysis in Stage 1 in Table 6, Table 7, Table 8, Figure 5, Figure 6, Figure 8 and Figure 7.

TT example

Stage 2 Main results (cont.)

Table 9 and tbl:10 show results on Qwen3-8B and Ministral-3, respectively.

Stage 2: TT and SS contributions decomposition

Stage 2: ST vs TS

Stage 2: Hybrid configuration

Figure 9. Pairing at fixed marginal on-policiness with Qwen3-8B. Dashed lines indicate standalone STST performance from Stage 1.

Figure 10. Pairing at fixed marginal on-policiness with Minis­tra-3. Dashed lines indicate standalone STST performance from Stage 1.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pp. 21246–21263, 2024.arxiv.org/abs/2306.13649
  2. [3]Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562, 2026.arxiv.org/abs/2603.25562
  3. [4]Ya Gao, Kalle Kujanpää, Pekka Marttinen, Harri Valpola, and Alexander Ilin. Edit knowledge, not just facts via multi-step reasoning over background stories. arXiv preprint arXiv:2602.02028, 2026.
  4. [5]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
  5. [6]Minki Kang, Jongwon Jeong, Seanie Lee, Jaewoung Cho, and Sung Ju Hwang. Distilling llm agent into small models with retrieval and code tools. Advances in Neural Information Processing Systems, 38:106501–106538, 2026.
  6. [7]Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.
  7. [8]Baohua Liao, Hanze Dong, Christof Monz, Xinxing Xu, Li Dong, and Furu Wei. Multi-turn on-policy distillation with prefix replay. arXiv preprint arXiv:2607.04763, 2026.arxiv.org/abs/2607.04763
  8. [9]Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadek, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. Ministral 3. arXiv preprint arXiv:2601.08584, 2026a.arxiv.org/abs/2601.08584
  9. [10]Jun Liu, Zhenglun Kong, Peiyan Dong, Changdi Yang, Tianqi Li, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Pu Zhao, Xue Lin, Dong Huang, and Yanzhi Wang. Structured agent distillation for large language model, 2026b. URL https://arxiv.org/abs/2505.13820.
  10. [11]Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation.DOI
  11. [12]Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=ebZcMQImhG.arxiv.org/abs/2602.04942
  12. [13]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627–635. JMLR Workshop and Conference Proceedings, 2011.arxiv.org/abs/1011.0686
  13. [14]Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. {ALFW}orld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=O1OXO1YcCdTm.arxiv.org/abs/2010.03768
  14. [15]Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint arXiv:2209.15189, 2022.arxiv.org/abs/2209.15189
  15. [16]Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, et al. Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents. arXiv preprint arXiv:2604.10674, 2026.arxiv.org/abs/2604.10674
  16. [17]Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang. Officebench: Benchmarking language agents across multiple applications for office automation. arXiv preprint arXiv:2407.19056, 2024.arxiv.org/abs/2407.19056
  17. [18]Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026.arxiv.org/abs/2601.02780
  18. [19]Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026.arxiv.org/abs/2606.19348
  19. [20]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengeng Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  20. [21]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022a.arxiv.org/abs/2207.01206
  21. [22]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b.arxiv.org/abs/2210.03629
  22. [23]Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agent-Tuning: Enabling generalized agent abilities for LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 3053–3077, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.181. URL https://aclanthology.org/2024.findings-acl.181/.DOI
  23. [24]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=Jpxfof0EaS.arxiv.org/abs/2601.18734
  24. [25]Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36:50117–50143, 2023.arxiv.org/abs/2306.13304

Paper details

vlnc:W_u69w3je9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

large language models

View statement
Looking for related papers…
Contents