WorksComputer Science
Rethinking Supervision Scaling in On-Policy Distillation: From Dense Supervision to Stochastic Horizons
Abstract
On-policy distillation (OPD) provides dense token-level supervision along student-generated trajectories, yet the learning benefits of longer trajectories may not scale proportionally with their computational cost. We investigate how supervision utility evolves with trajectory length. Controlled experiments reveal diminishing returns from additional supervised tokens, while showing that later positions remain learnable and exhibit substantial cross-position supervision transfer. Gradient analysis further identifies shared update components across response positions and diverse position-specific residuals whose cross-rollout consistency decreases at greater depths. Motivated by these findings, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision by randomly varying rollout horizons while preserving access to deeper response states. SH-OPD samples a shared horizon for each training batch, enabling efficient parallel generation without per-sequence horizon scheduling overhead. We characterize the resulting reweighting bias and derive a compute-reweighting trade-off under standard optimization assumptions. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD achieves slightly higher mathematical reasoning accuracy at approximately 45% of the per-update cost of standard OPD, while maintaining competitive code generation performance at around 60% of the baseline cost. These results highlight the benefits of efficient supervision allocation across response depths.
Figure 1. Batch-level stochastic horizon sampling in SH-OPD. Just as mini-batch SGD reduces the per-update computation of full-batch gradient descent (GD) by sampling training examples, SH-OPD reduces the rollout cost of standard OPD by sampling a shared horizon for each batch. All trajectories are generated in parallel, with each response terminating at EOS or the shared horizon . By frequently sampling shorter horizons, SH-OPD reduces the computational overhead of repeatedly reinforcing overlapping supervision, while stochastic resampling across updates preserves access to potentially complementary supervision at deeper response positions. All generated tokens are optimized using the standard token-mean OPD objective.
Code: https://github.com/chen123CtrlS/SH-OPD
Introduction
Corresponding author(s): [email protected]
Introduction
On-policy distillation (OPD) has emerged as an effective paradigm for language model post-training [1, 6, 7, 18, 23]. Its core principle is to generate trajectories using the student policy and provide dense token-level supervision from a stronger teacher at each visited state. By supervising student-generated trajectories, OPD mitigates the train–inference distribution mismatch inherent in off-policy distillation. Unlike outcome-based reinforcement learning, which relies primarily on sequence-level feedback, OPD provides token-level supervision [16] opportunities along a trajectory of length , contributing to its remarkable data efficiency [5, 14]. However, longer trajectories incur greater costs in generation, teacher evaluation, and student optimization. More fundamentally, more supervised tokens do not necessarily yield proportionally more useful learning signals. This raises a central question: How does effective supervision in OPD scale with trajectory length?
Recent studies suggest that full-trajectory supervision is not always necessary for effective OPD. Prefix distillation and horizon-control methods achieve competitive performance with substantially shorter or progressively extended rollouts [32, 33]. Position-aware analyses attribute the reduced utility of later tokens to accumulated teacher–student discrepancies [26], while token-selection methods show that a small subset of informative tokens can preserve much of the benefit of dense supervision [28]. Beyond supervision allocation, Fu et al. [5] investigate state-space coverage and teacher–student alignment, while Shen et al. [20] and Yu et al. [30] characterize the geometric structure of OPD parameter updates. Together, these studies reveal the non-uniform benefits of token-level supervision and the structured nature of OPD updates. However, why additional supervised tokens yield diminishing returns remains insufficiently understood. In particular, supervision from different trajectory positions may induce overlapping parameter updates, limiting the marginal benefits of extending the supervision horizon. Meanwhile, position-specific gradients may capture additional information that varies across sampled trajectories. These possibilities motivate a closer examination of how supervision interacts across trajectory positions and whether additional tokens provide genuinely new learning signals.
To investigate these diminishing returns, we examine how supervision utility changes with trajectory length and position (§ 4.2 and § 4.4). Fixed-prefix replay experiments show that increasing the number of supervised tokens yields progressively smaller gains in downstream performance. However, equal-budget position controls reveal that later trajectory blocks remain highly learnable, and supervision on one block substantially improves teacher–student agreement in other, unsupervised blocks. These findings suggest that diminishing returns arise, at least in part, from substantial supervision overlap across trajectory positions, rather than an inherent lack of useful signals in later tokens.
To explain this cross-position transfer, we examine the geometry of blockwise OPD gradients (§ 4.5). We find substantial gradient alignment across trajectory positions, indicating a shared update component. Beyond this shared component, position-specific residuals exhibit diverse directions but become less consistent across sampled trajectories as response depth increases. These findings suggest that longer trajectories repeatedly reinforce shared updates while introducing increasingly trajectory-dependent variations. This provides an explanation for the diminishing returns of dense supervision and motivates reducing redundant supervision while preserving access to deeper response states.
Building on these findings, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision while preserving access to deeper response states (§ 5). SH-OPD samples a batch-level rollout horizon from a power-survival distribution, favoring shorter trajectories while occasionally exploring longer ones. We further analyze the resulting horizon-induced reweighting bias through the Gram matrix of expected depth-specific gradient residuals. Under standard optimization assumptions, our theoretical analysis characterizes the trade-off between compute-adjusted stochastic optimization error and the distortion of stable depth-specific supervision. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD substantially reduces training cost while maintaining competitive performance. On mathematical reasoning, SH-OPD achieves slightly higher accuracy than standard OPD while using less than half the per-update token-compute and wall-time costs. On code generation, it maintains competitive performance with approximately 60% of the baseline cost. These results highlight the benefits of allocating supervision more efficiently across response depths.
Our main contributions are summarized as follows:
Supervision scaling. We reveal diminishing returns from increasing supervision length and demonstrate substantial cross-position transfer, showing that additional supervised tokens do not necessarily provide proportional learning benefits.
Gradient geometry and theoretical analysis. We characterize shared and position-specific gradient components and their cross-rollout consistency. Building on this geometric structure, we analyze horizon-induced reweighting bias and establish a compute–reweighting trade-off under standard optimization assumptions.
Efficient method. We propose SH-OPD, a simple stochastic horizon strategy that reduces redundant supervision while preserving access to deeper response states, achieving substantial computational savings with competitive performance across tasks.
Preliminaries, Definitions, and Notation
On-Policy Distillation. Let denote an input prompt, the student policy parameterized by , and a fixed teacher policy. Given , the student generates a response , where denotes the number of response tokens. We use to denote the autoregressive state before generating token . On-Policy Distillation [1, 7] (OPD) minimizes the reverse KL divergence between the student and teacher distributions evaluated on prefixes sampled from the student policy:
For each student-sampled token, OPD defines a token-level teacher preference signal . In practice, the corresponding policy-gradient-style [29] OPD surrogate can be written as:
The loss function (Eq. (1)) assigns positive or negative token-level credit according to the teacher’s relative preference, thereby steering the student toward the teacher’s behavior at each state. For convenience, define the per-token OPD loss as , such that
Blockwise Decomposition. To examine how the utility of dense supervision evolves along a response, we partition each response into contiguous token blocks. Let denote the target block size and define as the number of blocks. The token indices belonging to the -th block are , . Let denote the number of tokens contained in block . We define the average OPD loss within block as . The full-response OPD loss admits the exact decomposition . When all blocks have equal size, i.e., and , Eq. (2) reduces to
This formulation preserves the standard full-response OPD objective while providing a natural unit for controlling supervision at the block level.
Empirical Anatomy of Dense Supervision
On-policy distillation appears unusually data-efficient partly because a single rollout generates supervision at every autoregressive prefix. A response of length does not provide one training target, but a sequence of visited states, , , each paired with a token-level teacher signal.
Prior work [5] shows that repeated rollouts from even a single query can cover a surprisingly broad region of the state space, and that OPD can continue learning for hundreds of steps even when the training states are fixed. However, state count is not necessarily supervision count. Two visited prefixes may be distinct as states while inducing nearly the same parameter update [20, 30]; conversely, different tokens may produce distinct gradient directions, but some of these directions may reflect trajectory-specific variation rather than supervision that consistently appears across rollouts [21]. Thus, the raw supervised-token counts can substantially overestimate the reusable learning signal in a trajectory. We therefore shift from measuring state-space coverage to measuring supervision in gradient space. Rather than asking how many states a rollout visits, we ask a more fundamental question: How much novel and reproducible learning signal does each additional supervised token actually contribute?
General Setup
Experimental setup. We study mathematical reasoning and code generation with DeepSeek-R1-Distill-Qwen-1.5B [8] as the student. For mathematics, we use JustRL-DeepSeek-1.5B [9] as the teacher, DAPO-Math-17K [31] for training, and AMC 2023 [12] and AIME 2025 [2] for evaluation. For code generation, we use DeepCoder-1.5B-Preview [17] as the teacher, TACO [13] for training, and LiveCodeBench v6 [10] (LCB v6) for evaluation.
Hyperparameter settings. We follow the OPD training pipeline of Fu et al. [5] and hold all optimization hyperparameters fixed across comparisons (details are in Appendix § 9.§ 2). Specifically, we use rollout and mini-batch sizes of 64 with token-mean loss aggregation. Maximum prompt lengths are 1,024 tokens for mathematics and 4,096 for code, with a maximum response length of 7,168 tokens for both domains. We report avg@4 accuracy on AMC 2023 and AIME 2025, and avg@4 execution accuracy on LCB v6.
Fixed-prefix Replay Reveals Rapid Saturation of Supervision Utility
Fixed-prefix replay. To investigate how the utility of token-level supervision scales with trajectory length, we follow Fu et al. [5] and construct a controlled fixed-prefix replay setting. Zhao et al. [36] have also shown that distillation on offline student rollouts can remain effective, providing further support for the use of fixed trajectories in our controlled analysis. We focus on the medium-difficulty query1 used by Fu et al. [5]. This setting provides a non-trivial learning trajectory, where the student improves substantially during training while avoiding early saturation or near-zero success throughout optimization.
Position Control Reveals Cross-Block Supervision Transfer
For each one-shot query, we sample a rollout batch of trajectories from the initial student policy and keep this trajectory set fixed throughout optimization. Specifically, the replay buffer consists of sampled trajectories: , where each response is sampled independently from . We then control the amount of accessible supervision by truncating every frozen trajectory at different response prefix lengths, , and apply the OPD objective (Eq. (2)) only to supervised positions within the retained prefix. These prefix conditions are strictly nested: . Therefore, increasing only exposes additional later token-level supervision from the same set of trajectories, while keeping the query, rollout samples, model initialization, optimization budget, and all other hyperparameters identical. Although the replayed states are fixed, the student distribution and teacher–student discrepancy are recomputed under the updated parameters at each optimization step. As references, we compare against the original on-policy One-Shot OPD, which resamples trajectories from the evolving student policy, and full-data OPD, which samples queries from the full training distribution. Unlike these baselines, fixed-prefix replay removes changes in state visitation and isolates the contribution of additional supervised positions along a fixed set of trajectories. This design enables a direct test of whether dense token-level supervision scales proportionally with trajectory length: Does exposing more supervised token positions continue to provide proportional learning benefits, or does supervision utility rapidly saturate?
We first investigate whether increasing the amount of token-level supervision along a fixed trajectory leads to proportional improvements in downstream performance. Figure 2 compares fixed-prefix replay with prefix lengths ranging from 128 to 7168 tokens. As the prefix length increases, the student is exposed to progressively more supervised token positions from the same frozen trajectories. However, the validation performance does not improve proportionally with the number of supervised tokens. In particular, extending the prefix beyond 1024 tokens yields only marginal additional gains on the averaged AMC 2023 and AIME 2025 accuracy, despite introducing substantially more token-level supervision. This effect is not caused by insufficient optimization: all prefix conditions achieve rapid actor loss reduction and similar convergence behavior. Thus, additional supervised tokens yield diminishing downstream returns. These results establish a basic empirical observation: token count validation utility .

Figure 2. Fixed-prefix replay reveals diminishing returns as supervised tokens increase. Left: validation accuracy averaged over AMC 2023 and AIME 2025 under different prefix lengths. Right: corresponding actor loss trajectories. Although longer prefixes expose the student to substantially more supervised token positions, they provide limited additional validation improvement, with later fluctuations reflecting the estimation errors introduced by fixed one-shot replay.
Position Control Reveals Cross-Block Supervision Transfer
The saturation observed under fixed-prefix replay admits two non-exclusive explanations: (1) Later trajectory positions may provide weaker or less transferable supervision. (2) Supervision from different positions may substantially overlap, so adding later tokens contributes little new information even when those tokens remain individually learnable. To disentangle these effects, we hold the supervised-token budget fixed, vary only the supervised position, and measure whether optimizing one region of the trajectory also improves teacher–student agreement on other regions.
Equal-budget position control. We reuse the frozen rollout pool from § 4.2 for the same query and retain the trajectories whose responses contain at least 4,096 tokens. Fixing this common trajectory subset across all conditions ensures that positional comparisons are not confounded by differences in trajectory length or sample composition. Within each retained trajectory, we then partition the first 4,096 response tokens into four consecutive, non-overlapping blocks of equal size: , , . Each training condition applies the OPD loss to exactly one block. Tokens preceding the supervised block are retained as causal context but receive no direct supervision. All conditions start from the same initialization and are optimized for updates under identical settings. Each update therefore contains exactly supervised tokens, ensuring matched supervision and update budgets across conditions.
Measuring cross-block transfer. Let denote the parameters after training on block . For each evaluation block , we measure the teacher–student discrepancy and its relative reduction from initialization (Positive indicates that training on reduces the discrepancy on ):
The discrepancy measures the average absolute difference between the student and teacher log-probabilities, evaluated on the same sampled tokens at fixed autoregressive states in block . The relative reduction then quantifies the proportionate decrease in this discrepancy on block after optimizing the student with supervision from block , normalized by its initial value. Accordingly, diagonal entries characterize within-block fitting, whereas off-diagonal entries capture cross-block transfer. Note that this metric quantifies changes in sampled-token log-probability gaps at fixed autoregressive states, rather than measuring the divergence between the full next-token distributions of the student and teacher.
(1) Later block remains learnable. The diagonal entries in Table 1 are uniformly high: training on any single block reduces its own teacher–student discrepancy by 91.7%–92.7%. Thus, supervision does not vanish at later trajectory positions; even can be fitted almost as effectively as under the same token and optimization budgets. Yet downstream accuracy varies across blocks, showing that local learnability does not imply equal validation utility. (2) Supervision transfers substantially across positions. All off-diagonal entries are positive, with training on one block reducing the discrepancy on unsupervised blocks by 46.5%–70.4%. This transfer extends beyond adjacent windows and operates in both directions, indicating that supervision from different trajectory regions is not independent. At the same time, within-block reductions remain consistently larger than cross-block reductions, suggesting a combination of cross-position transferable signal and block-specific information.
| Supervised block | Avg@4 Acc. (%) | Eval. | Eval. | Eval. | Eval. |
| 56.46 | 92.7 | 63.9 | 53.5 | 46.5 | |
| 54.38 | 66.8 | 92.2 | 65.3 | 59.9 | |
| 55.42 | 60.5 | 70.4 | 92.0 | 65.8 | |
| 52.60 | 54.2 | 66.1 | 67.2 | 91.7 |
Table 1. Cross-block transfer under matched supervision budgets. Rows denote supervised blocks and columns evaluation blocks; entries report . Bold diagonal entries show within-block fitting and off-diagonal entries cross-block transfer. Accuracy is the avg@4 across AMC 2023 and AIME 2025.
Taken together, these results rule out a simple “weak late-token” explanation for prefix saturation. Later blocks remain highly learnable, yet a substantial fraction of their supervisory signal can already be captured through training elsewhere in the trajectory. Thus, adding later tokens increases the number of supervised positions more than it increases the amount of functionally distinct supervision. While these results establish behavioral transfer across response positions, they do not directly reveal whether such transfer arises from geometrically aligned parameter updates. Nor do they determine whether the remaining position-specific information is reproducible across independent rollouts. We therefore turn to the gradient-space geometry of blockwise OPD supervision.
Gradient Geometry Reveals Shared Updates and Heterogeneous Residuals
The cross-block transfer observed in § 4.4 suggests that supervision from different response positions induces substantially overlapping parameter updates. However, strong transfer does not imply that position-specific supervision is entirely redundant. Different positions may share a dominant update direction while still contributing distinct local corrections. Moreover, the presence of such corrections does not guarantee that they are equally reproducible across independently sampled trajectories. To disentangle these effects, we examine the geometry of blockwise OPD gradients along three dimensions: shared directional alignment, position-dependent residual structure, and cross-sample reproducibility.
Blockwise gradient decomposition. We analyze blockwise OPD gradients at the common initialization for both mathematics and code generation. For each domain, we construct a frozen batch containing one sampled trajectory per distinct prompt, retaining responses with at least 4,096 valid tokens. We partition the first 4,096 tokens into consecutive blocks of 128 tokens. Let denote the batch-averaged OPD gradient for block , computed with detached teacher preference weights and before gradient clipping or optimizer transformations. To distinguish the shared update from position-dependent variation, we decompose each block gradient into its token-weighted mean and a centered residual:
Since all blocks have equal length, coincides with the full-horizon token-mean gradient up to numerical precision. We compare pairwise cosine similarities of the raw gradients and centered residuals, together with the eigenspectra of their respective Gram matrices. Importantly, centering removes the same mean vector from every block; it does not project the gradients onto the orthogonal complement of .
❶ A dominant shared direction persists across positions and prompts. Figure 3(a,Figure 3d) shows substantial positive alignment among raw block gradients throughout the response. The mean pairwise cosine similarity is 0.526 for mathematics and 0.680 for code, while even the minimum similarity remains positive (0.177 and 0.194, respectively). These results indicate that OPD supervision across different response positions contains a strong common update component. Notably, this alignment is observed in batches constructed from distinct prompts, suggesting that the shared direction is not merely an artifact of repeated rollouts of the same question. This provides a gradient-space explanation for the cross-block transfer observed in § 4.2 and § 4.4: optimizing one trajectory region can improve teacher–student agreement elsewhere because different regions induce partially aligned parameter updates.
Figure 3. Blockwise OPD gradient geometry at initialization using 128-token blocks over the first 4,096 response tokens. Top: mathematics; bottom: code. Raw block gradients are positively aligned across all measured positions, and their spectra contain a dominant component, indicating a global update direction that persists in multi-prompt batches. After subtracting the token-weighted mean gradient, residual cosine matrices exhibit heterogeneous positive and negative structure and substantially flatter spectra. Negative residual cosine denotes opposing deviations from the shared mean, not conflict between the complete block gradients. The annotated values are the largest eigenvalues of the unnormalized raw and residual Gram matrices; plotted spectra are normalized within each matrix.
② Position-specific supervision remains geometrically diverse. Subtracting the shared mean reveals a substantially different structure. As shown in Figure 3(b,Figure 3e), the residual cosine matrices exhibit heterogeneous positive and negative similarities, with minimum values of -0.174 in mathematics and -0.350 in code. These negative similarities reflect opposing deviations from the common mean, rather than conflicts between the complete block gradients, which remain positively aligned. The residual eigenspectra in Figure 3(c,Figure 3f) are also considerably flatter than their raw counterparts. After removing the shared mean, gradient energy is distributed across many directions rather than concentrated in a single dominant mode. Thus, dense supervision cannot be characterized as merely repeated copies of one gradient direction. A substantial amount of position-dependent gradient variation remains beyond the common update. This is consistent with the incomplete cross-block transfer in § 4.4, where supervision from one block improves other blocks but does not fully replace their direct supervision. However, substantial residual variation does not necessarily imply equally reproducible supervision. While some residual directions may capture systematic, position-specific corrections, others may be highly dependent on individual sampled trajectories. We therefore examine whether these residual directions remain consistent across independent samples.
❸ Position-specific residuals exhibit heterogeneous reproducibility. We split each replay pool into disjoint halves (A) and (B) and independently estimate the blockwise gradients. For each position (), we measure the alignment of both raw gradients and their deviations from the corresponding half-specific full-horizon gradient: . For independent rollouts of the same mathematics prompt, the first 128-token residual exhibits strong reproducibility, with . This demonstrates that subtracting the shared mean does not eliminate all stable, position-specific structure. In contrast, beyond 1,024 tokens, the mean residual cosine decreases to 0.071, despite a substantial average residual norm of 1.65. Over the same region, raw-gradient cosine remains 0.563. A similar separation appears across disjoint code prompts, where the depth-averaged raw and residual cosine similarities are 0.590 and 0.083, respectively. These results show that substantial position-specific gradient variation does not necessarily imply reproducible supervision. While raw gradients retain strong alignment, residual directions are less consistent across samples, suggesting substantial trajectory-dependent variation. Importantly, low residual alignment does not imply that these local corrections are uninformative.
Case study: Shared reasoning strategies and divergent execution. To contextualize the gradient observations, we inspect independent rollouts of the same mathematical reasoning problem: summing over nonnegative integer triples satisfying the even-parity constraint . Early responses exhibit highly similar structures: all begin with either “Okay” or “Alright,” followed by a restatement of the problem. Importantly, this consistency extends beyond linguistic framing. Within the first 512 tokens, several trajectories mention generating functions, and some explicitly identify the correct parity decomposition. Thus, early alignment is consistent with shared, task-specific reasoning strategies rather than merely superficial formatting. However, shared strategies do not guarantee consistent execution. One trajectory correctly identifies the parity cases but subsequently reasons about an invalid case with exactly one odd exponent. As generation proceeds, trajectories also diverge in reasoning progress: at the same token depth, some are still exploring solution methods while others have advanced to numerical calculations. Representative excerpts from the sampled trajectories are provided in Table 2. This suggests that weak late-position residual alignment may partly reflect differences in reasoning paths, errors, corrections, and semantic stages, rather than an absence of informative supervision. Together, these observations highlight a fundamental distinction between the amount of position-specific gradient variation and its reproducibility. To scale supervision efficiently, the goal is not to discard later positions, but to reduce redundant reinforcement of shared updates while preserving coverage of potentially useful position-specific corrections across trajectories.
| Stage | Sampled trajectory | Excerpt | Qualitative observation |
| Early | ID 1, [128, 512) | “The problem reminds me of generating functions where you can encode conditions on the exponents . . . Because the condition is is even, it splits into two cases: either all of , , are even, or exactly two of them are odd.” | Shared task-specific strategy and correct parity decomposition. |
| Early | ID 4, [256, 448) | “Case 2: Exactly two of are odd . . . For Case 2: For the case where exactly one is odd, we have , , or variations.” | A correct decomposition immediately branches into an invalid one-odd case. |
| Middle | ID 20, [1792, 1984) | “Hmmm. Hmmm confusing. Wait no, maybe I should model separately . . . How can we relate to ? . . . Wait, hold on . . . Wait, no.” | Still reorganizing the parity argument. |
| Middle | ID 41, [1792, 2048) | “25/12 + 5/3 = 45/12 = 15/4, which is correct . . . Thus, the answer is 25/12. Therefore, , , so .” | Already completes the numerical consistency check at a comparable depth. |
| Late | ID 12, [3328, 3584) | “Recall that = even part + odd part . . . probably get 25/12 . . . is 25 + 12 = 37.” | Reaches the correct final result after extended verification. |
| Late | ID 44, [3200, 3456) | “, . . . Thus, the sum is 31/16 . . . .” | Reaches a self-consistent but incorrect branch and final answer. |
Table 2. Qualitative excerpts from independent rollouts of the same medium-difficulty query from the initial student.
Stochastic Horizon Supervision: Algorithm and Theory
The analysis in § 4 reveals two complementary properties of dense OPD supervision. First, gradients from different response positions share substantial update directions, suggesting that supervising every position in every rollout may repeatedly reinforce overlapping information. Second, position-specific residuals remain geometrically diverse, indicating that supervision from earlier positions cannot fully substitute for later supervision. Moreover, the decreasing cross-rollout reproducibility of these residuals suggests that additional depth-specific corrections are not uniformly stable across sampled trajectories, although their low reproducibility does not necessarily imply low utility.
These findings motivate two key design principles: reducing redundant reinforcement of shared supervision while preserving access to heterogeneous depth-specific corrections. Fixed short-horizon supervision can exploit the former but permanently excludes deeper states, whereas full-horizon supervision preserves the latter at substantial computational cost. To balance these competing considerations, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which randomly varies the generation horizon across training updates. By frequently sampling shorter horizons while retaining access to longer trajectories, SH-OPD reduces the cost of repeatedly observing overlapping supervision without permanently excluding potentially complementary depth-specific signals. Importantly, SH-OPD adjusts supervision frequency across response depths via stochastic horizon sampling, without explicitly identifying or reweighting residual directions or altering the original token-level OPD objective.
Stochastic Horizons On-Policy Distillation
Prior work has explored horizon control to improve OPD efficiency. POPD progressively expands the rollout horizon, while TOPD adopts a fixed truncated horizon [33]. More recently, STOPD [34] introduces per-rollout stochastic truncation with inverse-probability weighting to compensate for truncation-induced gradient bias. In contrast, SH-OPD adopts a batch-level stochastic horizon shared across rollout workers, avoiding per-sequence horizon sampling and scheduling overhead.
Beyond this implementation difference, SH-OPD takes a different perspective on supervision allocation. Rather than explicitly correcting truncation-induced reweighting to recover the full-horizon update, SH-OPD retains the original token-mean OPD loss and studies the trade-off between reduced computation and the distortion of depth-specific supervision. This choice is motivated by the gradient structure identified in § 4: substantial cross-position overlap makes uniformly dense supervision potentially redundant, while heterogeneous residuals motivate maintaining access to deeper states. The resulting reweighting bias is characterized in § 5.2.
Definition 1 (Power-Survival Horizon Distribution). Let denote the maximum response length in standard OPD, and let be a base horizon such that is a positive integer. We define the set of admissible horizons as . For a parameter , the stochastic horizon follows a discrete power-survival distribution defined by
Equivalently, its probability mass function is
Remark 1. The parameter controls a trade-off between rollout depth and supervision allocation. At , SH-OPD recovers standard OPD, with . For any finite , all admissible horizons have nonzero probability, while increasing progressively shifts probability mass toward shorter rollouts, eventually concentrating on as . For the considered range of , the horizon probability mass exhibits an approximately U-shaped distribution, favoring shorter and longer horizons over intermediate ones. The expected horizon is , yielding an expected nominal generation cost of per batch, compared with under standard OPD[2]. For example, with , , and , the expected horizon is approximately 4,114 tokens, reducing the nominal generation cost by 42.6%. Importantly, stochastic horizon sampling also implicitly reweights supervision across response positions. Assuming all trajectories reach their sampled horizons, the expected weight assigned to block is , where earlier blocks receive higher expected weights due to their inclusion in more sampled horizons. This reduces to uniform weighting, , under standard OPD.
Algorithm 1 Stochastic Horizons On-Policy Distillation (SH-OPD)
Require: Student , teacher , dataset , batch size , horizons , exponent
; construct horizon distribution
for each training step do
Sample prompts
Batch-level horizon sampling
Sample one shared horizon: ,
Parallel rollout generation
Broadcast to all rollout workers
for in parallel do
Terminate at EOS or after tokens
end for
Standard OPD update
Compute token losses
Update using
end for
SH-OPD: Batch-Level Parallel Rollout Generation. As summarized in Algorithm 1, SH-OPD samples a single horizon per training batch and broadcasts it to all rollout workers. All trajectories are then generated in parallel, stopping at either EOS or the shared horizon , rather than generating full-length responses before truncation. The horizon serves only as a generation cap, without conditioning the student policy. All generated tokens are supervised using the standard token-mean OPD loss. This batch-level design avoids per-sequence horizon scheduling overhead while reducing generation cost. Although shorter horizons could enable larger batches under a fixed token budget, we keep the batch size fixed to isolate the effect of stochastic horizon sampling.
Theoretical Perspective: A Compute–Reweighting Trade-off
Section 3.4 empirically reveals a shared gradient component and position-specific residuals across response blocks. Building on this observation, we extend the decomposition to population-level expected gradients under the current student policy. This allows us to characterize how stochastic horizon sampling reweights supervision across response positions and induces a systematic bias relative to full-horizon OPD.
Assumption 1 (On-Policy Gradient Decomposition). At optimization step , SH-OPD samples a horizon and generates trajectories using the current student policy . Let denote the resulting
stochastic gradient of the token-mean OPD surrogate defined in Eq. (1). The student is updated according to
Let denote the training history before sampling the current batch. We define the conditional expected update direction and its deviation from full-horizon OPD as , , where is the expected full-horizon OPD update evaluated at the same current parameters . Defining , we obtain
The decomposition separates the reference OPD update, the systematic bias induced by stochastic horizons with in Definition 1, and the stochastic gradient noise. Importantly, all three components are evaluated at the evolving parameters , preserving the on-policy nature of training.
Based on Assumption 1, we characterize the horizon-induced bias as follows:
Lemma 1 (Geometry of the Reweighting Bias). For clarity, consider equal-length blocks and trajectories that reach their sampled horizons. Let be the expected gradient of block , with the expectation taken over current-policy trajectories and the derivative understood as the detached OPD surrogate gradient. Under this setting, . Following § 4.5, define the population-level centered residual . Let denote the implicit block weights described in Remark 1, and let . Because both weighting schemes sum to one, their shared gradient component cancels: . Define the stable residual Gram matrix by , denotes the matrix eigenvalue. The squared systematic bias then admits the exact characterization:
This result links the gradient geometry in § 4 to the bias induced by stochastic horizon sampling. Since the block weights sum to one, the shared gradient component cancels, and the reweighting bias depends solely on the weighted depth-specific residuals. Unlike the empirical analysis in § 4.5, which evaluates gradients at initialization, our theoretical characterization is defined at arbitrary current parameters, allowing to evolve throughout on-policy optimization. Thus, the reweighting bias is jointly determined by the changes in block weights and the geometry of depth-specific residuals, characterizing the trade-off between computational efficiency and preserving depth-specific supervision.
The preceding lemma characterizes the systematic gradient distortion induced by horizon reweighting. We next examine how this distortion interacts with stochastic optimization under a finite training budget.
Lemma 2 (Stationarity Bound under Horizon Reweighting). Let be an -smooth, lower-bounded reference potential satisfying . We assume that its gradient coincides with the expected full-horizon OPD update direction, i.e., . Furthermore, assume that the conditional variance of the stochastic update is uniformly bounded: . Under the conditions of Lemma 1, for any constant stepsize ,
where and . This separates the optimization error into three terms. The first decreases with the number of updates, the second captures stochastic gradient noise, and the third measures the distortion of stable depth-specific gradients. Importantly, the reweighting bias is evaluated along the evolving optimization trajectory rather than at a fixed parameter point.
Definition 2. The benefit of SH-OPD arises from reducing the expected rollout cost per update. By Definition 1, . With a fixed rollout batch size , the expected nominal generation cost per update is . Under a total nominal token budget , the corresponding number of updates is
Corollary 1 (Compute-Normalized Trade-off). Under the conditions of Lemma 2 and Definition 2, assume and . Choosing , where if , yields
where , and is independent of and .
How to Choose ? Corollary 1 establishes a compute–reweighting trade-off: shorter expected horizons permit more on-policy updates under a fixed compute budget, potentially reducing stochastic optimization error, but may also distort reproducible depth-specific gradient structure, as characterized by Lemma 1. While recovers full-horizon OPD with zero reweighting bias (), increasing is beneficial when the reduction in compute-adjusted stochastic cost outweighs the additional bias (Appendix § 10.§ 5). Thus, governs a principled balance between computational efficiency and the preservation of stable supervision structure. In the next part, we investigate the choice of empirically.
Empirical Evaluation
Experimental setup. We evaluate SH-OPD on mathematical reasoning and code generation with and . We consider for both mathematics and code generation. All other experimental settings follow § 4.1, with the horizon distribution being the only factor varied within each task. Notably, exactly recovers standard full-horizon OPD. Additional results extending SH-OPD to multi-teacher on-policy distillation (MOPD) and Instruction following tasks are presented in the Appendix § 9.§ 6.
Efficiency metrics. We measure training efficiency using two complementary per-update costs. The token-compute proxy counts the total prompt and response tokens processed by the rollout and training pipeline per update. It approximates sequence-processing workload but does not account for attention complexity or hardware utilization. The wall-time cost measures the end-to-end duration of each training update, including rollout generation, student and teacher scoring, and actor optimization, but excluding validation. Both metrics are averaged over logged training updates and normalized to standard OPD within each domain. In contrast, the cumulative supervised tokens in Figure 4(a,Figure 4d) count only valid generated response tokens contributing to the OPD loss, accounting for natural EOS termination.
Figure 4. Performance–cost trade-off under stochastic horizons. Mathematics (top) and code generation (bottom) are evaluated across different horizon exponents . The left column shows validation accuracy against cumulative supervised response tokens, the middle column shows accuracy against optimization steps, and the right column reports per-update token-compute proxy and wall-time cost relative to standard OPD (). SH-OPD substantially reduces training cost while maintaining competitive performance. Mathematics favors , whereas code generation prefers , highlighting the task-dependent trade-off between supervision efficiency and depth coverage.
SH-OPD improves supervision efficiency. Figure 4 demonstrates that stochastic horizons substantially reduce training cost while preserving competitive downstream performance. On mathematics, $\alpha = 1.0 achieves the strongest overall trade-off. At approximately 45 million supervised response tokens, it reaches 63.21% accuracy, compared with 60.36% for standard OPD at a similar token budget (44.6 million tokens). By the end of training, it also achieves a higher final accuracy than standard OPD (63.21% vs. 62.50%), while reducing the per-update token-compute proxy to 45.1% and wall time to 45.5% of the standard OPD baseline. These results demonstrate that reducing redundant supervision can improve learning efficiency without sacrificing solution quality.
The preferred exponent differs across tasks. Code generation favors a more conservative horizon distribution. Among the tested SH-OPD settings, achieves a higher peak accuracy than (26.00% vs. 25.00%) and a higher mean accuracy over the final five evaluations (25.26% vs. 24.49%). Although their final checkpoints are close, the overall learning trajectory favors . This setting reduces the per-update token-compute proxy to 61.8% and wall time to 60.0% of standard OPD, demonstrating substantial efficiency gains while maintaining competitive code-generation performance.
More aggressive horizon reduction is not always beneficial. The mathematics sweep further reveals that increasing beyond 1.0 does not consistently improve performance. Although and further reduce the cost per update, their final accuracies decrease to 58.93% and 58.21%, respectively. Thus, shorter horizons do not necessarily provide better supervision efficiency: excessively concentrating supervision on early response positions can compromise learning quality.
Choosing through Gradient Geometry. The task-dependent preference for horizon exponents is consistent with the compute–reweighting trade-off in § 5.2. By exploiting shared gradients, SH-OPD reduces redundant supervision, while aggressive horizon reduction may distort complementary depth-specific updates. Lemma 1 shows that this distortion is governed by the Gram matrix of expected residual gradients. Empirically, mathematics favors , whereas code generation prefers , retaining more frequent deep supervision. This difference is qualitatively consistent with the larger residual Gram eigenvalue observed for code (16.08) than for mathematics (5.57), suggesting greater sensitivity to depth reweighting. However, these eigenvalues are measured from finite-batch gradients over the first 4,096 tokens at initialization and do not directly quantify the population-level bias over the full training horizon. Thus, the observed gradient geometry provides a plausible explanation for the task-dependent trade-off rather than a quantitative prediction of the optimal exponent.
Overall, the results demonstrate that effective OPD supervision does not require uniformly dense coverage of every response position. SH-OPD achieves substantial computational savings by reducing the frequency of full-horizon supervision while retaining access to deeper states. The different preferences for horizon reduction across tasks further highlight the importance of balancing computational efficiency with the preservation of complementary depth-specific supervision.
Related Work
OPD and Learning Dynamics. On-policy distillation (OPD) transfers teacher capabilities through token-level supervision on student-generated trajectories, mitigating the distribution mismatch of off-policy distillation [1, 7]. Recent studies have examined the factors governing its effectiveness. Li et al. [14] identify teacher–student compatibility and novel teacher knowledge as important conditions for successful distillation. Ding and Zhang [3] question the role of teacher knowledge transfer in OPD, showing that substantial performance gains can persist under noisy teacher supervision and can be partially reproduced through teacher-free self-improvement. Fu et al. [5] show that repeated rollouts from a single query can cover a substantial fraction of the states visited during full-data training, while the absorption of teacher supervision remains slow. Other works investigate the reliability of the distillation objective. Entropy-Aware OPD and Hybrid Policy Distillation adjust divergence objectives to improve distribution matching and training stability [11, 38]. Fu et al. [4] analyze the bias and variance of sampled-token supervision and identify failure modes arising from long-horizon prefix drift. Similarly, PowerOPD addresses unstable token-level learning signals through bounded reward transformations [35]. These studies characterize the conditions and optimization mechanisms underlying OPD. In contrast, our work focuses on how supervision from different trajectory positions interacts and contributes to learning.
Supervision Allocation in OPD. Recent studies have explored selective supervision to improve OPD efficiency: TIP identifies informative tokens using student entropy and teacher–student divergence [28], while teachability-aware selection and information-efficiency criteria further account for local learnability and gradient-estimation reliability [22, 25]. Position-aware methods also adjust token weights according to accumulated teacher–student discrepancies [26]. These approaches primarily optimize which generated tokens receive supervision. A complementary line of work reduces rollout generation costs through horizon control. FastOPD demonstrates that reasoning-prefix distillation can match full-trajectory OPD with substantially lower computation [32]. POPD and TOPD adopt progressive and fixed rollout truncation, respectively [33], while temporal curriculum methods progressively extend trajectory depth for multi-turn agents [24]. More recent approaches adapt the horizon to training conditions: KAT terminates rollouts exhibiting persistent low-KL agreement [27], while ADWIN selects prefix windows using delayed full-rollout probes and prefix–full gradient alignment [15]. TurnOPD further introduces adaptive rollout-depth and turn-balanced supervision for long-horizon agents [37].
Gradient Geometry and Supervision Efficiency. Recent studies have investigated the geometry of OPD updates, revealing subspace locking [20], gradient sparsity and spectral concentration [30], and token-level gradient-estimation reliability [22]. ADWIN further exploits prefix–full gradient alignment for adaptive horizon selection [15]. Beyond these analyses, we investigate cross-position gradient structure, identifying shared update components, heterogeneous position-specific residuals, and their consistency across sampled trajectories. We further characterize horizon-induced reweighting bias through the Gram matrix of expected depth-specific gradient residuals, establishing a theoretical connection between gradient geometry and the trade-off between compute-adjusted optimization error and supervision reweighting bias. While prior work has improved OPD efficiency through selective supervision and horizon control, our work connects cross-position supervision transfer and gradient residual structure to the bias–compute trade-off of stochastic horizon allocation.
Conclusion

Figure 5 (PDF p. 16).
In this work, we revisit the scaling of token-level supervision in on-policy distillation. Through controlled experiments and gradient-space analysis, we show that increasing supervision density yields diminishing returns, as gradients across response positions share substantial update directions while retaining heterogeneous, less reproducible depth-specific components. Dense supervision should be reduced where supervision overlaps, while complementary depth-specific supervision should remain accessible rather than be permanently truncated. Motivated by these findings, we introduce Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision by randomly varying rollout horizons while preserving access to deeper response states. Our theoretical analysis further characterizes the trade-off between compute-adjusted stochastic optimization cost and the distortion of stable depth-specific gradient structure. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD substantially reduces training cost while maintaining competitive or improved downstream performance. The preferred horizon exponent differs across tasks, consistent with their heterogeneous gradient structures. These findings suggest that effective supervision scaling is not simply about increasing the number of supervised tokens, but about allocating supervision efficiently across response depths. More broadly, SH-OPD offers a simple approach to improving the compute efficiency of on-policy distillation without requiring uniformly dense supervision. Misra, Divyanshu Kakwani, Dmitry Belenko, Dominic Manno, Dongyang Fan, Dor Tsur, Duncan Riach, Dustin VanStee, Dylan Lu, Edgar Minasyan, Ehsan Hosseini-Asl, Eileen Long, Elena Lantz, Ellie Evans, Elliott Ning, Elvis Chen, Emanuel Scoullos, Eric Chung, Eric Harper, Eric Pham-Hung, Eric Tramel, Eric Yang, Erik Arakelyan, Evelina Bakhturina, Ewa Dobrowolska, Fadi Saady, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Frankie Siino, Frederic Lafrance, Fuxiao Liu, Gantavya Bhatt, Garvit Kulshreshtha, George Armstrong, Giulio Lovisotto, Gomathy Venkata Krishnan, Gosia Lazuka, Grace Lam, Greg Heinrich, Greg Pauloski, Grzegorz Chlebus, Guilin Liu, Guo Chen, Guoming Zhang, Guyue Huang, Haifeng Qian, Hanrong Ye, Hao Gao, Hao Zhang, Haoqin Tu, Haran Kumar Shiv Kumar, Haribhau Hud, Harris Nover, Harrison Saturley-Hall, Harshil Kotamreddy, Hemil Desai, Henry Estela, Hexin Wang, Himanshu Singh, Hui Wang, Hung-Yueh Chiang, Igor Gitman, Ilia Karmanov, Ilya Loshchilov, Ingo Guehring, Itay Neeman, Ivan Moshkov, Jacky Kwok, Jakub Slowikowski, James Shen, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeff Willette, Jenna Diamond, Jenny Chen, Jesse Oliver, Jessica Medina, Jiacheng Xu, Jialin Song, Jian Hu, Jian Tu, Jian Zhang, Jiantao Jiao, Jiaqi Zeng, Jie-Fang Zhang, Jimmy Zhang, Jin Xu, Jingyu Xin, Jinhang Choi, Joey Conway, Joey Guman, John Kamalu, John Liu, John Roberts, Johnny Greco, Jonah Alben, Jonas Yang, Jonathan Cohen, Jonathan Li, Jonathan Raiman, Joseph Jennings, Juhi Mittal, Julia Gusak, Julien Veron Vialard, June Yang, Junkeun Yi, Jupinder Parmar, Ka Shun Shum, Kaiwen Shi, Kajal Jain, Kanishk Singla, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Huang, Katie Lyons, Keith Wyss, Keshav Santhanam, Keval Morabia, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kiran Praveen, Konstantinos Krommydas, Krishna Puvvada, Krzysztof Pawelec, Kushal Shah, Li Ding, Lily Lee, Linda Liu, Linglin Jing, Luis Vega, Maarten Van Segbroeck, Mahan Fathi, Mahdi Nazemi, Maka Dong, Makesh Narsimhan Sreedhar, Maor Ashkenazi, Marc Cuevas, Marcin Chochowski, Marco Pavone, Markus Kliegl, Marta Stepniewska-Dziubinska, Martina Gonzalez Vilas, Martyna Patelka, Matan Rusanovsky, Matthieu Le, Matty Machczynski, Matvei Novikov, Mauricio Ferrato, Meenakshi Sharma, Mehrzad Samadi, Melissa Corpuz, Melissa Saelzer, Meng Xin, Mengru Wang, Mengxi Wu, Meriem Boubdir, Michael Boone, Michael Callahan, Michael Elad, Michael Lightstone, Michal Bien, Michal Zawalski, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Ranzinger, Mike Wojcikiewicz, Min-Hung Chen, Ming Kang, Mingjie Liu, Mingyuan Ma, Misha Nekorystnov, MJ Mikulski, Mohammad Qazim Bhat, Mohammad Shoeybi, Morgan Purcell, Mostofa Patwary, Nabin Mulepati, Nancy Agarwal, Narimane Hennouni, Nathan Sala, Nave Assaf, Nayeon Lee, Neel Vaidya, Negar Habibi, Netanel Haber, Nick Gupta, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nik Spirin, Niket Agarwal, Nikolai Ludwig, Nima Tajbakhsh, Nithin Sunny, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Omer Ullman Argov, Pablo Ribalta, Padmavathy Subramanian, Parth Mannan, Pavlo Molchanov, Pawel Morkisz, Peter Jin, Petr Lapukhov, Petro Junior Milan, Philip Monk, Pinky Xu, Piotr Januszewski, Piotr Laszkiewicz, Po-Han Huang, Praateek Mahajan, Pradeep Thalasta, Prakash Gurumurthy, Pranav Prashant Thombre, Prasoon Varshney, Pratyusha Maiti, Preeti Chadha, Pritam Biswas, Pritam Gundecha, Przemek Strzelczyk, Puhui Meng, Pulkit Kumar, Rabe e h Karimi Mahabadi, Rachel Oberman, Rajpreet Thethy, Ramani Akkati, Rameshwar Shivbhakta, Ran Zilberstein, Randy Gelhausen, Rasoul Shafipour, Raviraj Joshi, Ray Macalisang, Rayen Tian, Reka Kovacs, Renjie Pi, Resha Shetty, Ria C, Rima Shahbazyan, Rishabh Garg, Rita Fernandes Neves, Ritika Borkar, Ritu Gala, Riya Talwar, Riyad Islam, Robert Clark, Robert Kirby, Roger Waleffe, Rohan Joshi, Rohan Pillai, Rohit Jena, Rohit Watve, Roi Garnett, Roi Koren, Rongguang Ye, Russell J. Hewett, Ryan Prenger, Ryan Stewart, Ryota Egashira, Sadegh Mahdavi, Sai Nishanth Padala, Salika Dave, Sandhya Rani Narravula, Sandip Bhaskar, Sangkuk Lym, Sanjeev Satheesh, Sarah Amirlani, Saran Vikas Murari, Saurabh Mishra, Saurabh Srivastava, Saurav Muralidharan, Sean Narenthiran, Sean Wang, Seokju Weijia Gao, Shangyi Geng, Jie Gong, Linhu Gong, Shengao Gong, Xiaochen Gong, Qizheng Gu, Yicheng Gu, Shuhao Guan, Haiqing Guo, Shiqi Guo, Xiang Guo, Zhengyan Guo, Beixi Hao, Wenxin Hao, Xiaoru Hao, Dailan He, Haotian He, Lehan He, Qi He, Weiran He, Xinran He, Xinyi He, Yibo He, Yunjia He, Chao Hong, Tiange Hong, Hao Hu, Jiaxi Hu, Ruikun Hu, Weiming Hu, Yangyang Hu, Zhenxing Hu, Liang Hua, Jinbin Huang, Ke Huang, Ruiyuan Huang, Siying Huang, Weixiao Huang, Yan Huang, Zhengjie Huang, Zhiqi Huang, Yulong Hui, Chaobo Jia, Yutong Jiang, Zhejun Jiang, Zuoyou Jiang, Wenyi Jin, Xinyi Jin, Yu Jing, Huanjun Kong, Guokun Lai, Aidi Li, Cheng Li, Chengyuan Li, Cong Li, Fang Li, Guanyu Li, Haoyang Li, Jia Li, Junxiong Li, Lei Li, Letian Li, Lincan Li, Weihong Li, Wentao Li, Xintong Li, Yang Li, Yishen Li, Yiwei Li, Yuxiao Li, Zhaowei Li, Zhaoxi Li, Zheming Li, Zhengxiao Li, Zhiyuan Li, Jiawei Lin, Xiaohan Lin, Yibo Lin, Zichao Lin, Ziyan Lin, Bill Liu, Boxiao Liu, Chuan Liu, Liang Liu, Shaowei Liu, Shudong Liu, Shuran Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yanming Liu, Yibo Liu, Yipeng Liu, Zhengying Liu, Zhiheng Liu, Enzhe Lu, Haoyu Lu, Linqiang Lu, Tingzhan Lu, Zhiyuan Lu, Aotian Luo, G. Luo, Junyu Luo, Yifan Luo, B. Lyu, Wenzhou Lyu, Shaoguang Mao, Yuan Mei, Xin Men, Minqing Ni, Yixuan Niu, Siyuan Pan, Shujun Peng, Zhangyang Qi, Ruoyu Qin, ZeChao Qin, Zeyu Qin, Haiquan Qiu, Jianxin Qiu, Jiezhong Qiu, Bowen Qu, Yuhao Qu, Zeyu Shang, Youbo Shao, Han Shen, Jincheng Shi, Juanfeng Shi, Lidong Shi, Shengyuan Shi, Wingchun Siu, Pengwei Song, Xiaoxi Song, Jianlin Su, Yunfeng Su, Zhaochen Su, Lin Sui, Jingsong Sun, Junyao Sun, Shaoning Sun, Shuzhe Sun, Tongyu Sun, Yujun Sun, Yunpeng Tai, Chuning Tang, Heyi Tang, Sirui Tang, Zecheng Tang, Chaoran Tian, Rongpeng Tian, Yu Tian, Wei Tu, Chensi Wang, Chuang Wang, Chunjie Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Hao Wang, Hao Wang, Huaqing Wang, Hui Wang, Jiayi Wang, Jinglong Wang, Jinhong Wang, Jiuzheng Wang, Linian Wang, Shaobo Wang, Shenzhi Wang, Shuyi Wang, Si Wang, Siyuan Wang, Tianfu Wang, Wenjue Wang, Xingran Wang, Xinmei Wang, Xinyuan Wang, Xusheng Wang, Yalin Wang, Yangkun Wang, Yao Wang, Yaoyu Wang, Yejie Wang, Yiqin Wang, Yucheng Wang, Yuzhi Wang, Zhaoji Wang, Zhaowei Wang, Zhengtao Wang, Zhenhao Wang, Zhongsheng Wang, Zifan Wang, Chu Wei, Ming Wei, Shouxin Wei, Zichen Wen, Fan Wu, Haoning Wu, Rucong Wu, Wenhao Wu, Xiaoxue Wu, Yingcong Wu, Yongqi Wu, Yuxin Wu, Zijian Wu, Xinglang Xian, Chenxuan Xiang, Yuye Xiang, Bocheng Xiao, Chenjun Xiao, Xin Xiao, Jin Xie, Xiaotong Xie, Yifeng Xie, Zhe Xie, Bowei Xing, Yiming Xiong, Baosheng Xu, Boyu Xu, Jiale Xu, Jianfan Xu, Jing Xu, Jinjing Xu, L. H. Xu, Qingtao Xu, Shuyao Xu, Suting Xu, Tiantian Xu, Tianxiang Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ye Xu, Yueni Xu, Ziyao Xu, Haonan Xue, Junjie Yan, Yaoyao Yan, Fan Yang, Guangyao Yang, Hao Yang, Junwei Yang, Ruoyu Yang, Wenjie Yang, Xiaofei Yang, Xinyu Yang, Yi Yang, Yiling Yang, Ying Yang, Yuchen Yang, Zhen Yang, Zhilin Yang, Zian Yang, Zuhao Yang, Haotian Yao, Dan Ye, Haoran Ye, Wenjie Ye, Zhanbo Ye, Bohong Yin, Haoxiang Yin, Xietong Yin, Chengzhen Yu, Haozhen Yu, Longhui Yu, Shengnan Yu, Shuying Yu, Tianxiang Yu, Enming Yuan, Mengjie Yuan, Tongtian Yue, Wei Yue, Yang Yue, Dunyuan Zha, Haobing Zhan, B. H. Zhang, Dehao Zhang, Fei Zhang, Hao Zhang, Haoyuan Zhang, Huanyu Zhang, Jiapei Zhang, Jiaxuan Zhang, Jin Zhang, Kaiyi Zhang, Miaozhen Zhang, Puqi Zhang, Qinglei Zhang, Rong Zhang, Rui Zhang, Shaoshuai Zhang, Shiyi Zhang, Xiaobin Zhang, Xiaoyun Zhang, Y. Zhang, Yangkun Zhang, Ye Zhang, Yichi Zhang, Yikun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Zijimg Zhang, Bin Zhao, Chengguang Zhao, Feifan Zhao, Jinglun Zhao, Jinxiang Zhao, Shuai Zhao, Wenshuo Zhao, Xiangyu Zhao, Xuanle Zhao, Yikai Zhao, Zijia Zhao, Haozhi Zheng, Huabin Zheng, Ruihan Zheng, Shaojie Zheng, Tengyang Zheng, Haofeng Zhong, Lei Zhong, Longguang Zhong, M. Zhou, Qiankang Zhou, Runjie Zhou, Ruozhang Zhou, Xinyu Zhou, Yiqiao Zhou, Zaida Zhou, Jinguo Zhu, Liya Zhu, Xinhao Zhu, Yangjunfeng Zhu, Yuxuan Zhu, Zhen Zhu, Chen Zhuang, Weiyu Zhuang, and Xinxing Zu. Kimi k3: Open frontier intelligence, 2026.
Extension to Experiments
Default hyperparameters for OPD
Training and evaluation. We follow the default OPD configuration of Fu et al. [5], using AdamW with a learning rate of , weight decay 0.01, a rollout and mini-batch size of 64, sampling temperature 1.0, top-p=1.0, token-mean loss aggregation and gradient clipping at 1.0. We use maximum prompt lengths of 1,024 and 4,096 tokens for mathematics and code, respectively, and a maximum response length of 7,168 tokens for both domains. We evaluate each problem with four sampled responses, reporting avg@4 accuracy on AMC 2023 and AIME 2025 and avg@4 execution accuracy on LiveCodeBench v6. All comparisons use identical optimization hyperparameters and evaluation budgets. All experiments are conducted on 4 NVIDIA H20 GPUs.
Medium query
Query selection and difficulty. We adopt the medium-difficulty mathematical reasoning query (ID: t22) from the One-Shot OPD study by Fu et al. [5]. The query is drawn from DAPO-Math-17K [31], where its difficulty is determined by the initial student’s pass rate over eight independently sampled responses before training. Specifically, the medium query is solved in 4 out of 8 attempts, corresponding to an initial pass rate of 50%. The original study further shows that this query exhibits a gradual improvement in training success rate, from approximately 0.36 to 0.89 over 300 optimization steps, whereas the easy query is consistently solved and the hard query remains unsolved. We select the medium query because it provides a non-trivial learning regime with sufficient room for improvement, allowing us to examine how supervision from different response positions contributes to downstream learning without being dominated by early saturation or persistent failure.
Full query. The selected query is: Let be the set of triples of non-negative integers such that is even. Determine the value of
This sum can be expressed as , where and are relatively prime positive integers. Compute .
Fixed-prefix replay construction. Following the fixed-state analysis of Fu et al. [5], we construct a frozen replay pool by sampling independent trajectories from the initial student policy on this query, using a sampling temperature of 1.0 and a maximum response length of 7,168 tokens. The sampled trajectories are stored and reused throughout optimization, so the visited autoregressive states remain fixed while the student parameters evolve. For the fixed-prefix experiment in § 4.2, we consider supervision horizons of tokens. Each condition uses the same replay pool but restricts the OPD loss to valid token positions within the corresponding prefix. Trajectories shorter than a specified prefix contribute only their available valid tokens. Thus, the supervision sets are nested, and increasing the prefix length introduces additional later-position supervision without changing the underlying trajectories. At each optimization step, the teacher–student log-probability discrepancy is evaluated on the retained states using the updated student parameters.
Common trajectory subset for position-controlled experiments. For the cross-block transfer analysis in § 4.4, we further restrict the replay pool to trajectories containing at least 4,096 valid response tokens. Denoting the number of retained trajectories by , we partition the first 4,096 tokens of each retained trajectory into four non-overlapping blocks of 1,024 tokens. All position-controlled conditions share this identical trajectory subset, ensuring that differences in training outcomes arise from the supervised positions rather than trajectory length or sample composition. Each condition optimizes one block for 250 updates, with preceding tokens preserved as causal context but excluded from direct supervision.
Scope of the analysis. Fu et al. [5] show that the one-shot OPD phenomenon is robust to variations in query difficulty, response-length cap, and sampling temperature. Our experiments build on this controlled setting but address a different question: rather than examining how many distinct states are sufficient for learning, we investigate how much additional supervision is contributed by different token positions within a fixed set of trajectories.
Qualitative trajectory excerpts
Table 2: Qualitative excerpts from independent rollouts of the same medium-difficulty query from the initial student. Trajectory IDs are zero-based row indices in the frozen pool, and intervals are half-open response-token ranges. Excerpts are verbatim apart from ellipses and light mathematical typesetting.
Table 2 complements the case study in § 4.5 with excerpts from the frozen rollout pool. The examples illustrate three recurring patterns: a shared generating-function strategy early in generation, trajectory-specific mistakes despite initially correct parity reasoning, and different levels of progress at comparable response depths.
Horizon Distribution Statistics
Power-survival horizon distribution. We use and , yielding seven admissible horizons. Under the power-survival distribution, the expected horizons for are 7,168, 4,114, 2,655, 1,928, and 1,062 tokens, respectively, corresponding to nominal generation-budget reductions of 0%, 42.6%, 63.0%, 73.1%, and 85.2% relative to full-horizon OPD. The probability of sampling the maximum horizon decreases from 100% at to 37.8%, 14.3%, 5.4%, and 0.006%, respectively. Thus, increasing progressively concentrates sampling on shorter horizons while retaining nonzero probability of deeper supervision. These statistics characterize nominal horizon budgets; actual generated lengths may be shorter due to EOS termination.
Table 3: Statistics of the power-survival horizon distribution with and . Token reduction is computed relative to full-horizon OPD (), assuming trajectories reach their sampled horizons.
As shown in Table 3, increasing progressively shifts probability mass toward shorter horizons, substantially reducing the expected generation budget. Notably, moderate values of retain meaningful probabilities of sampling the maximum horizon, whereas concentrates 96.88% of the probability mass on the shortest horizon, effectively approaching fixed short-horizon supervision.
| Token Reduction | ||||
| 0 | 7,168 | 0.0% | 0.00% | 100.00% |
| 0.5 | 4,114 | 42.6% | 29.29% | 37.80% |
| 1.0 | 2,655 | 63.0% | 50.00% | 14.29% |
| 1.5 | 1,928 | 73.1% | 64.64% | 5.40% |
| 5.0 | 1,062 | 85.2% | 96.88% | 0.006% |
Table 3.
Empirical sampled and realized response lengths. Table 4 reports statistics from the full-data training runs used in Figure 4. A single horizon is sampled per global update and shared by all trajectories in that rollout batch. We define the empirical sampled horizon as the arithmetic mean of these per-update samples, and the realized response length as the mean number of valid generated response tokens after natural termination. The empirical sampled horizons are close to their theoretical expectations, with finite-run deviations caused by sampling variability. Realized response lengths are lower than the sampled horizons because some trajectories terminate at EOS.
| Domain | Updates | EOS term. | ||||
| Math | 0 | 279 | 7,168 | 7,168 | 5,764 | 54.34% |
| Math | 0.5 | 279 | 4,114 | 4,169 | 3,569 | 25.25% |
| Math | 1.0 | 279 | 2,655 | 2,859 | 2,546 | 13.88% |
| Math | 1.5 | 279 | 1,928 | 2,184 | 1,999 | 8.70% |
| Math | 5.0 | 279 | 1,062 | 1,057 | 1,055 | 0.58% |
| Code | 0 | 384 | 7,168 | 7,168 | 6,452 | 23.67% |
| Code | 0.5 | 384 | 4,114 | 4,091 | 3,783 | 11.58% |
| Code | 1.0 | 384 | 2,655 | 2,811 | 2,655 | 6.67% |
Table 4.
Table 4: Empirical horizon and response-length statistics for the full-data runs. is the theoretical mean sampled horizon, is its empirical mean across logged updates, and is the mean realized response length after EOS termination.
Multi-Teacher On-Policy Distillation
Setup. We extend SH-OPD to multi-teacher on-policy distillation (MOPD) with a single DeepSeek-R1-Distill-Qwen-1.5B student. Each example is routed by its ability label to a frozen domain-specific teacher: JustRL-DeepSeek-1.5B for 17,917 DAPO-Math training examples, DeepCoder-1.5B-Preview for 24,701 TACO code examples, and UltraData-IF-1.5B [5] for 4,501 first-turn Multi-IF instruction-following examples [19]. We compare using and . Mathematics and code are evaluated during training with avg@1 on AMC 2023 (40 problems), AIME 2025 (30 problems), and LiveCodeBench v6 (175 problems), using a 31,744-token response cap. Instruction following is evaluated on all 4,445 Multi-IF conversations, each containing three turns in one of eight languages.
| Token cost | Wall time | AMC23 | AIME25 | LCB v6 | Multi-IF mean | Multi-IF turn 3 | |
| 0 | 100.0% | 100.0% | 85.83 | 36.67 | 25.33 | 37.89 | 26.79 |
| 0.5 | 61.2% | 59.9% | 83.00 | 32.22 | 25.33 | 39.26 | 27.81 |
| 1.5 | 36.4% | 35.6% | 82.50 | 35.56 | 23.05 | 38.73 | 26.96 |
Table 5. MOPD efficiency and downstream performance. Per-update token-compute and wall-time costs are averaged over updates 1–300 and normalized to full-horizon MOPD. Math and code results are the mean avg@1 over the last three checkpoints, while Multi-IF scores are reported at checkpoint 300.
Results. SH-OPD remains effective under multi-teacher distillation. With , SH-OPD reduces per-update token-compute and wall-time costs to 61.2% and 59.9% of full-horizon MOPD, respectively. It matches the full-horizon result on LCB v6 and improves the Multi-IF mean from 37.89 to 39.26, while incurring moderate decreases on the two mathematics benchmarks. The more aggressive setting reduces both costs by roughly 64%, but exhibits less consistent performance across tasks. Overall, stochastic horizons retain useful multi-domain performance at substantially lower training cost, extending the efficiency benefits of SH-OPD beyond the single-teacher setting. Since Multi-IF training and evaluation share source conversations, the Multi-IF results should be interpreted as in-sample performance comparisons rather than evidence of held-out generalization.
Omitted Proofs and Additional Results
Proof of Lemma 1
Proof. We derive the population-level reweighting identity and establish the corresponding quadratic-form characterization of the systematic bias.
Consider equal-length blocks, each containing tokens. Let
denote the probability of sampling a horizon comprising blocks. As assumed in the lemma, trajectories reach their sampled horizons, and horizon sampling preserves the distribution of the observed prefixes. Consequently, the expected gradient of block , whenever included, is .
For a sampled horizon , the token-mean gradient is the average of the first block gradients. Taking the expectation over sampled horizons yields
Here, represents the effective weight assigned to block under the sampled horizon distribution. These weights are normalized, since
When , the horizon is fixed at . Thus, for all , and
Now define the weight perturbation . By normalization, . Recalling the centered population residual , we obtain
The last equality follows from the zero-sum property of the weight perturbations, which eliminates the shared gradient component.
By the definition of the stable residual Gram matrix,
the squared bias can be expressed as
Since is a Gram matrix, it is positive semidefinite. Applying the Rayleigh quotient bound gives
This establishes Eq. (6) and completes the proof of Lemma 1. □
Proof of Lemma 2
Reference potential assumption. The expected full-horizon update direction is defined by averaging the OPD surrogate gradients over trajectories sampled from the current student policy, thereby preserving the on-policy nature of training. For the stationarity analysis, we additionally assume the existence of an -smooth, lower-bounded reference potential satisfying . This assumption is not required for the reweighting-bias characterization in Lemma 1 and is not automatically guaranteed by the detached OPD surrogate. Under this assumption, Lemma 2 characterizes how horizon-induced reweighting bias and stochastic gradient variance jointly affect stationarity under an idealized stochastic gradient update model. Proof. Recall the update rule
Under the assumptions of Lemma 2, the conditional gradient decomposition is
where
and
For brevity, write
and denote conditional expectation given by .
Step 1: One-step descent. By the -smoothness of ,
Substituting the update rule gives
Conditioned on , both and are fixed, while has zero conditional mean. Consequently,
and
Taking conditional expectation in Eq. (16) yields
Since , we have . Therefore,
Combining 19 and 20 establishes
Step 2: Telescoping over the training trajectory.
Taking total expectation of Eq. (21), rearranging, and summing over gives
The first sum telescopes:
Using , we obtain
Dividing both sides by yields
Finally, applying Eq. (6) at each gives
Substitution into Eq. (25) proves Lemma 2.
Proof of Corollary 1
Starting from Lemma 2, for any ,
Step 1: Balancing the optimization and stochastic terms.
Consider
For , its unconstrained minimizer satisfies
Hence,
If , substitution yields
To cover the constraint , choose
When , we set .
If , then . Using , we obtain
Therefore, in either case,
Combining Eqs. Eq. (26) and Eq. (29),
Comparison with Full-Horizon OPD.
Step 2: Converting update count to expected compute.
By Definition 1,
With a fixed rollout batch size , define
For an expected nominal token budget , the number of updates is
For ,
Consequently,
Substituting into Eq. (30) gives
Finally, using , we obtain
Eq. 33 provides a finite-budget upper bound, retaining both the stochastic term and the optimization term. For a compact characterization of the leading-order compute–reweighting trade-off, we suppress the lower-order term and write
Here, denotes a leading-order representation rather than an exact finite-budget inequality. is independent of and . This recovers the simplified expression in Corollary 1.
Comparison with Full-Horizon OPD
We further characterize when stochastic horizons yield a tighter compute-normalized stationarity bound than standard full-horizon OPD. Recall that the complete finite-budget bound in Eq. (33) is
To ensure that the finite-budget bound applies to both SH-OPD and full-horizon OPD, we consider a common nominal compute budget .
At , the horizon distribution concentrates on , and the reweighting bias vanishes: . Consequently,
For any , comparing with shows that holds if and only if
This condition makes explicit the compute–reweighting trade-off: the reduction in compute-adjusted stochastic optimization error must outweigh the systematic bias introduced by horizon reweighting. In particular, shorter expected horizons do not necessarily yield a tighter bound, as the gradient variance and reweighting bias may also vary with . The comparison concerns theoretical upper bounds under a common expected nominal compute budget and does not guarantee improved empirical performance.
References
- [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, pages 21246–21263, 2024.arxiv.org/abs/2306.13649
- [2]Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.23281
- [3]Yi Ding and Ruqi Zhang. Does on-policy distillation really distill? from noisy teacher to self-improvement. arXiv preprint arXiv:2608.31046, 2026.
- [4]Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes. In Third Conference on Language Modeling, 2026.arxiv.org/abs/2603.25562
- [5]Zixuan Fu, Bingxiang He, Yuxin Zuo, Haohuan Huang, Jinqian Zhang, Ruhang Xiao, Cheng Qian, Qinyu Luo, Huan ang Gao, Yudong Wang, Zhiyuan Liu, Ning Ding, and Chaojun Xiao. Rethinking on-policy distillation of large language models ii: One training example. arXiv preprint arXiv:2609.04172, 2026.arxiv.org/abs/2609.04172
- [6]GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzhen Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Guo, Jingsen Wang, Jingzhao Du, Jinzhu Wu, Kedong Wang, Lei Li, Lin Fan, Lucen Zhong, Mingdao Liu, Mingming Zhao, Pengfan Du, Qian Dong, Rui Lu, Shuang-Li, Shulin Cao, Song Liu, Ting Jiang, Xiaodong Chen, Xiaohan Zhang, Xuancheng Huang, Xuezhen Dong, Yabo Xu, Yao Wei, Yifan An, Yilin Niu, Yitong Zhu, Yuanhao Wen, Yukuo Cen, Yushi Bai, Zhongpei Qiao, Zihan Wang, Zikang Wang, Zilin Zhu, Ziqiang Liu, Zixuan Li, Bojie Wang, Bosi Wen, Can Huang, Changpeng Cai, Chao Yu, Chen Li, Chengwei Hu, Chenhui Zhang, Dan Zhang, Daoyan Lin, Dayong Yang, Di Wang, Ding Ai, Erle Zhu, Fangzhou Yi, Feiyu Chen, Guohong Wen, Hailong Sun, Haisha Zhao, Haiyi Hu, Hanchen Zhang, Hanrui Liu, Hanyu Zhang, Hao Peng, Hao Tai, Haobo Zhang, He Liu, Hongwei Wang, Hongxi Yan, Hongyu Ge, Huan Liu, Huanpeng Chu, Jia’ni Zhao, Jiachen Wang, Jiajing Zhao, Jiamin Ren, Jiapeng Wang, Jiaxin Zhang, Jiayi Gui, Jiayue Zhao, Jijie Li, Jing An, Jing Li, Jingwei Yuan, Jinhua Du, Jinxin Liu, Junkai Zhi, Junwen Duan, Kaiyue Zhou, Kangjian Wei, Ke Wang, Keyun Luo, Laiqiang Zhang, Leigang Sha, Liang Xu, Lindong Wu, Lintao Ding, Lu Chen, Minghao Li, Nianyi Lin, Pan Ta, Qiang Zou, Rongjun Song, Ruiqi Yang, Shangqing Tu, Shangtong Yang, Shaoxiang Wu, Shengyan Zhang, Shijie Li, Shuang Li, Shuiyi Fan, Wei Qin, Wei Tian, Weining Zhang, Wenbo Yu, Wenjie Liang, Xiang Kuang, Xiangmeng Cheng, Xiangyang Li, Xiaoquan Yan, Xiaowei Hu, Xiaoying Ling, Xing Fan, Xingye Xia, Xinyuan Zhang, Xinze Zhang, Xirui Pan, Xu Zou, Xunkai Zhang, Yadi Liu, Yandong Wu, Yanfu Li, Yidong Wang, Yifan Zhu, Yijun Tan, Yilin Zhou, Yiming Pan, Ying Zhang, Yinpei Su, Yipeng Geng, Yong Yan, Yonglin Tan, Yuean Bi, Yuhan Shen, Yuhao Yang, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yurong Wu, Yutao Zhang, Yuxi Duan, Yuxuan Zhang, Zezhen Liu, Zhengtao Jiang, Zhenhe Yan, Zheyu Zhang, Zhixiang Wei, Zhuo Chen, Zhuoer Feng, Zijun Yao, Ziwei Chai, Ziyuan Wang, Zuzhou Zhang, Bin Xu, Minlie Huang, Hongning Wang, Ju anzi Li, Yuxiao Dong, and Jie Tang. Glm-5: from vide coding to agentic engineering, 2026. URL https://arxiv.org/abs/2602.15763.
- [7]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, pages 32694–32717, 2024.DOI
- [8]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianz u Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z.
- [9]Bingxiang He, Zekai Qu, Zeyuan Liu, Yinghao Chen, Yuxin Zuo, Cheng Qian, Kaiyan Zhang, Weize Chen, Chaojun Xiao, Ganqu Cui, Ning Ding, and Zhiyuan Liu. Justrl: Scaling a 1.5b llm with a simple rl recipe. arXiv preprint arXiv:2512.16649, 2025.arxiv.org/abs/2512.16649
- [10]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=chfJJYC3iL.
- [11]Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2603.07079
- [12]Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletsky, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13 (9):9, 2024.
- [13]Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852, 2023.arxiv.org/abs/2312.14852
- [14]Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. In ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning, 2026. URL https://openreview.net/forum?id=aaDDXXDGXB.arxiv.org/abs/2604.13016
- [15]Kun Liang, Chenming Tang, Clive Bai, Weijie Liu, Saiyong Yang, and Yunfang Wu. ADWIN: Adaptive windows for horizon-aware on-policy distillation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 10371–10389, 2026.arxiv.org/abs/2605.28396
- [16]Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation.DOI
- [17]Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level. https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349 2025. Notion Blog.
- [18]NVIDIA, ; Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Ghabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Fiček, Alex Gronskyi, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shepherd, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Mირჾ? Misra, Divyanshu Kakwani, Dmitry Belenko, Dominic Manno, Dongyang Fan, Dor Tsur, Duncan Riach, Dustin VanStee, Dylan Lu, Edgar Minasyan, Ehsan Hosseini-Asl, Eileen Long, Elena Lantz, Ellie Evans, Elliott Ning, Elvis Chen, Emanuel Scoullos, Eric Chung, Eric Harper, Eric Pham-Hung, Eric Tramel, Eric Yang, Erik Arkelyan, Evelina Bakhurtina, Ewa Dobrowolska, Fadi Saady, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Frankie Siino, Frederic Lafrance, Fuxiao Liu, Gantavya Bhatt, Garvit Kulshreshtha, George Armstrong, Giulio Lovisotto, Gomathy Venkata Krishnan, Gosia Lazuka, Grace Lam, Greg Heinrich, Greg Pauloski, Grzegorz Chlebus, Guilin Liu, Guo Chen, Guoming Zhang, Guyue Huang, Haifeng Qian, Hanrong Ye, Hao Gao, Hao Zhang, Haoqin Tu, Haran Kumar Shiv Kumar, Harib-hau Hud, Harris Nover, Harrison Saturley-Hall, Harshil Kotamreddy, Hemil Desai, Henry Estela, Hexin Wang, Himanshu Singh, Hui Wang, Hung-Yueh Chiang, Igor Gitman, Ilia Karmanov, Ilya Loshchilov, Ingo Guehring, Itay Neeman, Ivan Moshkov, Jacky Kwok, Jakub Slowikowski, James Shen, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeff Willette, Jenna Diamond, Jenny Chen, Jesse Oliver, Jessica Medina, Jiacheng Xu, Jialin Song, Jian Hu, Jian Tu, Jian Zhang, Jiantao Jiao, Jiaqi Zeng, Jie-Fang Zhang, Jimmy Zhang, Jin Xu, Jingyu Xin, Jinhang Choi, Joey Conway, Joey Guman, John Kamalu, John Liu, John Roberts, Johnny Greco, Jonah Alben, Jonas Yang, Jonathan Cohen, Jonathan Li, Jonathan Raiman, Joseph Jennings, Juhi Mittal, Julia Gusak, Julien Veron Vialard, June Yang, Junkeun Yi, Jupinder Parmar, Ka Shun Shum, Kaiwen Shi, Kajal Jain, Kanishk Singla, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Huang, Katie Lyons, Keith Wyss, Keshav Santhanam, Keval Morabia, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kiran Praveen, Konstantinos Krommydas, Krishna Puvvada, Krzysztof Pawelec, Kushal Shah, Li Ding, Lily Lee, Linda Liu, Linglin Jing, Luis Vega, Maarten Van Segbroeck, Mahan Fathi, Mahdi Nazemi, Maka Dong, Makesh Narsimhan Sreedhar, Maor Ashkenazi, Marc Cuevas, Marcin Chochowski, Marco Pavone, Markus Kliegl, Marta Stepniewska-Dziubinska, Martina Gonzalez Vilas, Martyna Patelka, Matan Rusanovsky, Matthieu Le, Matty Machczynski, Matvei Novikov, Mauricio Ferrato, Meenakshi Sharma, Mehrzad Samadi, Melissa Corpuz, Melissa Saelzer, Meng Xin, Mengru Wang, Mengxi Wu, Meriem Boubdir, Michael Boone, Michael Callahan, Michael Elad, Michael Lightstone, Michal Bien, Michal Zawalski, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Ranzinger, Mike Woj-cikiewicz, Min-Hung Chen, Ming Kang, Mingjie Liu, Mingyuan Ma, Misha Nekorystnov, MJ Mikulski, Mohammad Qazim Bhat, Mohammad Shoeybi, Morgan Purcell, Mostofa Patwary, Nabin Mulepati, Nancy Agarwal, Narimane Hennouni, Nathan Sala, Nave Assaf, Nayeon Lee, Neal Vaidya, Negar Habibi, Netanel Haber, Nick Gupta, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nik Spirin, Niket Agarwal, Nikolai Ludwig, Nima Tajbakhsh, Nithin Sunny, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Omer Ullman Argov, Pablo Ribalta, Padmavathy Subramanian, Parth Mannan, Pavlo Molchanov, Pawel Morkisz, Peter Jin, Petr Lapukhov, Petro Junior Milan, Philip Monk, Pinky Xu, Piotr Januszewski, Piotr Laszkiewicz, Po-Han Huang, Praateek Mahajan, Pradeep Thalasta, Prakash Gurumurthy, Pranav Prashant Thombre, Prasoon Varshney, Pratyusha Maiti, Preeti Chadha, Pritam Biswas, Pritam Gundecha, Przemek Strzelczyk, Puhui Meng, Pulkit Kumar, Rabe e h Karimi Mahabadi, Rachel Oberman, Rajpreet Thethy, Ramani Akkati, Rameshwar Shivbhakta, Ran Zilberstein, Randy Gellhausen, Rasoul Shafipour, Raviraj Joshi, Ray Macalisang, Rayen Tian, Reka Kovacs, Renjie Pi, Resha Shetty, Ria C, Rima Shahbazyan, Rishabh Garg, Rita Fernandes Neves, Ritika Borkar, Ritu Gala, Riya Talwar, Riyad Islam, Robert Clark, Robert Kirby, Roger Waleffe, Rohan Joshi, Rohan Pillai, Rohit Jena, Rohit Watwe, Roi Garnett, Roi Koren, Rongguang Ye, Russell J. Hewett, Ryan Prenger, Ryan Stewart, Ryota Egashira, Sadegh Mahdavi, Sai Nishanth Padala, Salika Dave, Sandhya Rani Narravula, Sandip Bhaskar, Sangkug Lym, Sanjeev Satheesh, Sarah Amirlani, Saran Vikas Murari, Saurabh Mishra, Saurabh Srivastava, Saurav Muralidharan, Sean Narenthiran, Sean Wang, Seokju Cho, Seong Hee Lee, Seonjin Na, Seonmyeong Bak, Seph Mard, Serge Panev, Sergei Kolchenko, Sergio Perez, Seth Henneman, Seth Poulos, Shahar Mor, Shantanu Acharya, Shaokun Zhang, Shaona Ghosh, Sharath Turukevere Sreenivas, Shaun Kotek, Shaya Gharghabi, Sheng-Chieh Lin, Shengliang Xu, Shi Chen, Shihao Wang, Shilpa Amireddy, Shiqing Fan, Shiyang Chen, Shrima Prabhumoye, Shriram Venugopal, Shuang Yu, Shubham Gandhi, Shubham Pachori, Shubham Toshniwal, Shuoyang Ding, Shwetha Krishnamurthy, Simeng Sun, Sivakumar Arayandi Thottakara, Somshubra Majumdar, Soumye Singhal, Sridurga Krithivasan, Sriharsha Niverty, Stephen Ge, Sugam Dipak Devare, Sukrit Rao, Sumeet Kumar Barua, Sungsoo Ha, Surya Nistala, Suseella Panguluri, Sweta Priyadarshi, Syeda Nahida Akter, Talor Abramovich, Tanay Varshney, Tandeep Singh Chandehoke, Teo Ene, Terry Kong, Thanh Do, Thinh Le, Tian Zhang, Tijmen Blankevoort, Tim Romanski, Tiyasa Mitra, Tom Balough, Tom Huang, Tomasz Grzegorzek, Tomer Asida, Tomer Bar Natan, Traian Rebedea, Trenton Starkey, Tugrul Konuk, Tuomas Rintamaki, Twinkle Vashisht, Tyler Kniess, Tyler Poon, Udi Karpas, Ushnish De, Utkarsh Vaidya, Vahid Noroozi, Vanshil Atul Shah, Venkat Srinivasan, Venkata Karri, Venmugil Elango, Vibhor Agrawal, Vibhu Jawa, Victor Cui, Victoria Edmonds, Vijay Korthikanti, Vikas Mehta, Vikranth Jeyakumar, Vinay Rao, Vineeth Kalluru Srinivas, Vitaly Kurin, Vitaly Lavrukhin, Wahidur Rahman, Wanli Jiang, Wasi Uddin Ahmad, Wataru Ishihara, Wei Du, Wei Ping, Wei-Ming Chen, Weiheng Chai, Weijia Chen, Wendy Quan, Wenliang Dai, Wenyan Cong, William Zhang, Wojciech Prazuch, Xiaowei Ren, Xin Yu, Xiwen Yu, Yan Wang, Yang Chen, Yang Yu, Yangyi Chen, Yaniv Galron, Yash Jain, Yashaswi Karnati, Ye Yu, Yejin Choi, Yemi Ibrahim, Yev Meyer, Yi Dong, Yi-Fu Wu, Yian Zhang, Yifan Zhang, Ying Lin, Yiyang Zhou, Yoav Miron, Yogendra Pandey, Yonatan Geifman, Yonggan Fu, Yoshi Suhara, Youngeun Kwon, Yu-Cheng Chou, Yu-Chiang Frank Wang, Yuanguo Kuang, Yuekai Zhang, Yugi Guwawala, Yuki Huang, Yunheng Zou, Yunsheng Liu, Zhenming Wang, Zhiding Yu, Zhiqi Li, Zhiyu Cheng, Zhongbo Zhu, Zhuolin Yang, Zichao Yue, Zihan Liu, Zijie Yan, Zirui Liu, Zoe Faul, and Zuhair Ahmed. Nemotron 3.5 super and lightning: Open, efficient mixture-of-experts hybrid mamba-transformer models for agentic ai, 2026. URL https://arxiv.org/abs/2610.38095.
- [19]OpenBMB. Ultradata-sft-2605, 2026. URL https://huggingface.co/datasets/openbmb/UltraData-SFT-2605.
- [20]Zhennan Shen, Yanshu Li, Qingyu Yin, Chak Tou Leong, Zhilin Wang, Yanxu Chen, Rongduo Han, Sunbowen Lee, and Yi R. Fung. On the geometry of on-policy distillation. arXiv preprint arXiv:2606.07082, 2026.
- [21]Huanxin Sheng, Zhiling Ye, Haonan Wang, Jian Wang, Jinjie Gu, and Jian Kang. 1 arXiv preprint arXiv:2609.24432, 2026.arxiv.org/abs/2609.24432
- [22]Huanxin Sheng, Zhiling Ye, Haonan Wang, Jian Wang, Jinjie Gu, and Jian Kang. 1% of tokens can be enough: On gradient estimation in on-policy distillation. arXiv preprint arXiv:2609.24432, 2026.arxiv.org/abs/2609.24432
- [23]Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, M. C., Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y. Charles, H. S. Che, Guanduo Chen, Guangyu Chen, Guanzheng Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kexin Chen, Peng Chen, Ruijue Chen, Wentao Chen, Xin Chen, Yang Chen, Yanru Chen, Yifei Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Dazhi Cheng, Yean Cheng, Jialei Cui, Jingbing Cui, Anqi Dai, Jiaqi Deng, Hao Ding, Rui Ding, Shaofeng Ding, Mengfan Dong, Mengnan Dong, Yuhao Dong, Yuxin Dong, Angang Du, Chenzhuang Du, Dikang Du, Jus en Du, Yulun Du, Yu Fan, Jing Feng, Qiulin Feng, Yichen Feng, Kelin Fu, Qiang Fu, Fuxuan Gao, Hongcheng Gao, Jingyue Gao, Tong Gao, Weijia Gao, Shangyi Geng, Jie Gong, Linhu Gong, Shengao Gong, Xiaochен Gong, Qizheng Gu, Yicheng Gu, Shuhao Guan, Haiqing Guo, Shiqi Guo, Xiang Guo, Zhengyan Guo, Beixi Hao, Wenxin Hao, Xiaoru Hao, Dailan He, Haotian He, Lehan He, Qi He, Weiran He, Xinran He, Xinyi He, Yibo He, Yunjia He, Chao Hong, Tiange Hong, Hao Hu, Jiaxi Hu, Ruikun Hu, Weiming Hu, Yangyang Hu, Zhenxing Hu, Liang Hua, Jinbin Huang, Ke Huang, Ruiyuan Huang, Siying Huang, Weixiao Huang, Yan Huang, Zhengjie Huang, Zhiqi Huang, Yulong Hui, Chaobo Jia, Yutong Jiang, Zhejun Jiang, Zuoyou Jiang, Wenyi Jin, Xinyi Jin, Yu Jing, Huanjun Kong, Guokun Lai, Aidi Li, Cheng Li, Chengyuan Li, Cong Li, Fang Li, Guanyu Li, Haoyang Li, Jia Li, Junxiong Li, Lei Li, Letian Li, Lincan Li, Weihong Li, Wentao Li, Xintong Li, Yang Li, Yishen Li, Yiwei Li, Yuxiao Li, Zhaowei Li, Zhaoxi Li, Zheming Li, Zhengxiao Li, Zhiyuan Li, Jiawei Lin, Xiaohan Lin, Yibo Lin, Zichao Lin, Ziyan Lin, Bill Liu, Boxiao Liu, Chuan Liu, Liang Liu, Shaowei Liu, Shudong Liu, Shuran Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yanning Liu, Yibo Liu, Yipeng Liu, Zhengying Liu, Zhiheng Liu, Enzhe Lu, Haoyu Lu, Linqiang Lu, Tingzhan Lu, Zhiyuan Lu, Aotian Luo, G. Luo, Junyu Luo, Yifan Luo, B. Lyu, Wenzhou Lyu, Shaoguang Mao, Yuan Mei, Xin Men, Minqing Ni, Yixuan Niu, Siyuan Pan, Shujun Peng, Zhangyang Qi, Ruoyu Qin, ZeChao Qin, Zeyu Qin, Haiquan Qiu, Jianxin Qiu, Jiez h ong Qiu, Bowen Qu, Yuhao Qu, Zeyu Shang, Youbo Shao, Han Shen, Jincheng Shi, Juanfeng Shi, Lidong Shi, Shengyuan Shi, Wingchun Siu, Pengwei Song, Xiaoxi Song, Jianlin Su, Yunfeng Su, Zhaoc hen Su, Lin Sui, Jingsong Sun, Junyao Sun, Shaoning Sun, Shuzhe Sun, Tongyu Sun, Yujun Sun, Yunpeng Tai, Chuning Tang, Heyi Tang, Sirui Tang, Zecheng Tang, Chaoran Tian, Rongpeng Tian, Yu Tian, Wei Tu, Chensi Wang, Chuang Wang, Chunjie Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Hao Wang, Hao Wang, Huaqing Wang, Hui Wang, Jiayi Wang, Jinglong Wang, Jinhong Wang, Jiuzheng Wang, Linian Wang, Shaobo Wang, Shenzhi Wang, Shuyi Wang, Si Wang, Siyuan Wang, Tianfu Wang, Wenjue Wang, Xingran Wang, Xinmei Wang, Xinyuan Wang, Xusheng Wang, Yalin Wang, Yangkun Wang, Yao Wang, Yaoyu Wang, Yejie Wang, Yiqin Wang, Yucheng Wang, Yuzhi Wang, Zhaoji Wang, Zhaowei Wang, Zhengtao Wang, Zhenhao Wang, Zhongsheng Wang, Zifan Wang, Chu Wei, Ming Wei, Shouxin Wei, Zichen Wen, Fan Wu, Haoning Wu, Rucong Wu, Wenhao Wu, Xiaoxue Wu, Yingcong Wu, Yongqi Wu, Yuxin Wu, Zijian Wu, Xinglang Xian, Chenxuan Xiang, Yue Xiang, Bocheng Xiao, Chenjun Xiao, Xin Xiao, Jin Xie, Xiaotong Xie, Yifeng Xie, Zhe Xie, Bowei Xing, Yiming Xiong, Baosheng Xu, Boyu Xu, Jiale Xu, Jianfan Xu, Jing Xu, Jin jing Xu, L. H. Xu, Qingtao Xu, Shuyao Xu, Suting Xu, Tiantian Xu, Tianxiang Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ye Xu, Yueni Xu, Ziyao Xu, Haonan Xue, Junjie Yan, Yaoyao Yan, Fan Yang, Guangyao Yang, Hao Yang, Junwei Yang, Ruoyu Yang, Wenjie Yang, Xiaofei Yang, Xinyu Yang, Yi Yang, Yiling Yang, Ying Yang, Yuchen Yang, Zhen Yang, Zhilin Yang, Zian Yang, Zuhao Yang, Haotian Yao, Dan Ye, Haoran Ye, Wenjie Ye, Zhanbo Ye, Bohong Yin, Haoxiang Yin, Xietong Yin, Chengzhen Yu, Haozhen Yu, Longhui Yu, Shengnan Yu, Shuying Yu, Tianxiang Yu, Enming Yuan, Mengjie Yuan, Tongtian Yue, Wei Yue, Yang Yue, Dunyuan Zha, Haobing Zhan, B. H. Zhang, Dehao Zhang, Fei Zhang, Hao Zhang, Haoyuan Zhang, Huanyu Zhang, Jiapei Zhang, Jiaxuan Zhang, Jin Zhang, Kaiyi Zhang, Miaozhen Zhang, Puqi Zhang, Qinglei Zhang, Rong Zhang, Rui Zhang, Shaoshuai Zhang, Shiyi Zhang, Xiaobin Zhang, Xiaoyun Zhang, Y. Zhang, Yangkun Zhang, Ye Zhang, Yichi Zhang, Yikun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Zijin Zhang, Bin Zhao, Chenguang Zhao, Feifan Zhao, Jinglun Zhao, Jinxiang Zhao, Shuai Zhao, Wenshuo Zhao, Xiangyu Zhao, Xuanle Zhao, Yikai Zhao, Zijia Zhao, Haozhi Zheng, Huabin Zheng, Ruihan Zheng, Shaojie Zheng, Tengyang Zheng, Haofeng Zhong, Lei Zhong, Longguang Zhong, M. Zhou, Qiankang Zhou, Runjie Zhou, Ruozhang Zhou, Xinyu Zhou, Yiqiao Zhou, Zaida Zhou, Jinguo Zhu, Liya Zhu, Xinhao Zhu, Yangjunfeng Zhu, Yuxuan Zhu, Zhen Zhu, Chen Zhuang, Weiyu Zhuang, and Xinxing Zu. Kimi k3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653.
- [24]Jiaqi WANG, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. In Third Conference on Language Modeling, 2026.arxiv.org/abs/2604.24005
- [25]Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, and Hongxia Yang. Not all disagreement is learnable: Token teachability in on-policy distillation. arXiv preprint arXiv:2605.26844, 2026.arxiv.org/abs/2605.26844
- [26]Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, and Yifei Wang. On the position bias of on-policy distillation. arXiv preprint arXiv:2606.22600, 2026.arxiv.org/abs/2606.22600
- [27]Haoran Xin, Anhao Zhao, Ying Sun, Jin Li, Xiaoyu Shen, and Hui Xiong. Escaping the KL agreement trap in on-policy distillation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 41418–41431, 2026.arxiv.org/abs/2606.09471
- [28]Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026.arxiv.org/abs/2604.14084
- [29]Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026.arxiv.org/abs/2602.12125
- [30]Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, and Han-Jia Ye. Dense supervision, sparse updates: On the sparsity and geometry of on-policy distillation. arXiv preprint arXiv:2606.13657, 2026.arxiv.org/abs/2606.13657
- [31]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, juncai liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale. In Advances in Neural Information Processing Systems, pages 113222–113244, 2025.arxiv.org/abs/2503.14476
- [32]Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, and Robert E. Tillman. Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026, pages 25553–25569, July 2026.
- [33]Yaocheng Zhang, Jiajun Chai, Yuqian Fu, Songjun Tu, Xiaohan Wang, Wei Lin, Guojun Yin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Are full rollouts necessary for on-policy distillation? arXiv preprint arXiv:2605.31490, 2026.arxiv.org/abs/2605.31490
- [34]Zhengxin Zhang, Maosen Tang, Ning Wang, Changhao Li, Jingren Zhou, Bo Dai, Sainyam Galhotra, Alex Townsend, and Claire Cardie. Does on-policy distillation need to complete every rollout? arXiv preprint arXiv:2610.21337, 2026.arxiv.org/abs/2610.21337
- [35]Anhao Zhao, Junlong Tong, Yingqi Fan, Ping Nie, Wenjie Li, and Xiaoyu Shen. PowerOPD: Stabilizing on-policy distillation with bounded power transformation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 48414–48427, 2026.arxiv.org/abs/2606.17199
- [36]Siyan Zhao, Yonggan Fu, Jindong Jiang, Shih-Yang Liu, Song Bian, Byung-Kwan Lee, Sharath Turuvekere Sreenivas, Wenliang Dai, Hanrong Ye, Aditya Grover, and Pavlo Molchanov. When do we need on-policy distillation? distilling on offline student rollouts is often better. arXiv preprint arXiv:2610.11291, 2026.arxiv.org/abs/2610.11291
- [37]Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, and Jingjing Chen. Turnopd: Making on-policy distillation turn-aware for efficient long-horizon agent training. arXiv preprint arXiv:2607.05804, 2026.arxiv.org/abs/2607.05804
- [38]Wenhong Zhu, Ruobing Xie, Rui Wang, and Pengfei Liu. Hybrid policy distillation for LLMs. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2604.20244


