betaFeedback

WorksComputer Science

Rethinking Supervision Scaling in On-Policy Distillation: From Dense Supervision to Stochastic Horizons

Abstract

On-policy distillation (OPD) provides dense token-level supervision along student-generated trajectories, yet the learning benefits of longer trajectories may not scale proportionally with their computational cost. We investigate how supervision utility evolves with trajectory length. Controlled experiments reveal diminishing returns from additional supervised tokens, while showing that later positions remain learnable and exhibit substantial cross-position supervision transfer. Gradient analysis further identifies shared update components across response positions and diverse position-specific residuals whose cross-rollout consistency decreases at greater depths. Motivated by these findings, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision by randomly varying rollout horizons while preserving access to deeper response states. SH-OPD samples a shared horizon for each training batch, enabling efficient parallel generation without per-sequence horizon scheduling overhead. We characterize the resulting reweighting bias and derive a compute-reweighting trade-off under standard optimization assumptions. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD achieves slightly higher mathematical reasoning accuracy at approximately 45% of the per-update cost of standard OPD, while maintaining competitive code generation performance at around 60% of the baseline cost. These results highlight the benefits of efficient supervision allocation across response depths.

Figure 1. Batch-level stochastic horizon sampling in SH-OPD. Just as mini-batch SGD reduces the per-update computation of full-batch gradient descent (GD) by sampling training examples, SH-OPD reduces the rollout cost of standard OPD by sampling a shared horizon for each batch. All trajectories are generated in parallel, with each response terminating at EOS or the shared horizon HtH_t. By frequently sampling shorter horizons, SH-OPD reduces the computational overhead of repeatedly reinforcing overlapping supervision, while stochastic resampling across updates preserves access to potentially complementary supervision at deeper response positions. All generated tokens are optimized using the standard token-mean OPD objective.

Code: https://github.com/chen123CtrlS/SH-OPD

Introduction

Corresponding author(s): [email protected]

Introduction

On-policy distillation (OPD) has emerged as an effective paradigm for language model post-training [1, 6, 7, 18, 23]. Its core principle is to generate trajectories using the student policy and provide dense token-level supervision from a stronger teacher at each visited state. By supervising student-generated trajectories, OPD mitigates the train–inference distribution mismatch inherent in off-policy distillation. Unlike outcome-based reinforcement learning, which relies primarily on sequence-level feedback, OPD provides O(N)O(N) token-level supervision [16] opportunities along a trajectory of length NN, contributing to its remarkable data efficiency [5, 14]. However, longer trajectories incur greater costs in generation, teacher evaluation, and student optimization. More fundamentally, more supervised tokens do not necessarily yield proportionally more useful learning signals. This raises a central question: How does effective supervision in OPD scale with trajectory length?

Recent studies suggest that full-trajectory supervision is not always necessary for effective OPD. Prefix distillation and horizon-control methods achieve competitive performance with substantially shorter or progressively extended rollouts [32, 33]. Position-aware analyses attribute the reduced utility of later tokens to accumulated teacher–student discrepancies [26], while token-selection methods show that a small subset of informative tokens can preserve much of the benefit of dense supervision [28]. Beyond supervision allocation, Fu et al. [5] investigate state-space coverage and teacher–student alignment, while Shen et al. [20] and Yu et al. [30] characterize the geometric structure of OPD parameter updates. Together, these studies reveal the non-uniform benefits of token-level supervision and the structured nature of OPD updates. However, why additional supervised tokens yield diminishing returns remains insufficiently understood. In particular, supervision from different trajectory positions may induce overlapping parameter updates, limiting the marginal benefits of extending the supervision horizon. Meanwhile, position-specific gradients may capture additional information that varies across sampled trajectories. These possibilities motivate a closer examination of how supervision interacts across trajectory positions and whether additional tokens provide genuinely new learning signals.

To investigate these diminishing returns, we examine how supervision utility changes with trajectory length and position (§ 4.2 and § 4.4). Fixed-prefix replay experiments show that increasing the number of supervised tokens yields progressively smaller gains in downstream performance. However, equal-budget position controls reveal that later trajectory blocks remain highly learnable, and supervision on one block substantially improves teacher–student agreement in other, unsupervised blocks. These findings suggest that diminishing returns arise, at least in part, from substantial supervision overlap across trajectory positions, rather than an inherent lack of useful signals in later tokens.

To explain this cross-position transfer, we examine the geometry of blockwise OPD gradients (§ 4.5). We find substantial gradient alignment across trajectory positions, indicating a shared update component. Beyond this shared component, position-specific residuals exhibit diverse directions but become less consistent across sampled trajectories as response depth increases. These findings suggest that longer trajectories repeatedly reinforce shared updates while introducing increasingly trajectory-dependent variations. This provides an explanation for the diminishing returns of dense supervision and motivates reducing redundant supervision while preserving access to deeper response states.

Building on these findings, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision while preserving access to deeper response states (§ 5). SH-OPD samples a batch-level rollout horizon from a power-survival distribution, favoring shorter trajectories while occasionally exploring longer ones. We further analyze the resulting horizon-induced reweighting bias through the Gram matrix of expected depth-specific gradient residuals. Under standard optimization assumptions, our theoretical analysis characterizes the trade-off between compute-adjusted stochastic optimization error and the distortion of stable depth-specific supervision. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD substantially reduces training cost while maintaining competitive performance. On mathematical reasoning, SH-OPD achieves slightly higher accuracy than standard OPD while using less than half the per-update token-compute and wall-time costs. On code generation, it maintains competitive performance with approximately 60% of the baseline cost. These results highlight the benefits of allocating supervision more efficiently across response depths.

Our main contributions are summarized as follows:

  • Supervision scaling. We reveal diminishing returns from increasing supervision length and demonstrate substantial cross-position transfer, showing that additional supervised tokens do not necessarily provide proportional learning benefits.

  • Gradient geometry and theoretical analysis. We characterize shared and position-specific gradient components and their cross-rollout consistency. Building on this geometric structure, we analyze horizon-induced reweighting bias and establish a compute–reweighting trade-off under standard optimization assumptions.

  • Efficient method. We propose SH-OPD, a simple stochastic horizon strategy that reduces redundant supervision while preserving access to deeper response states, achieving substantial computational savings with competitive performance across tasks.

Preliminaries, Definitions, and Notation

On-Policy Distillation. Let x∼Dx \sim\mathcal{D} denote an input prompt, πθ\pi_\theta the student policy parameterized by θ\theta, and πt\pi_t a fixed teacher policy. Given xx, the student generates a response y=(y1,…,yN)∼πθ(⋅∣x)y = (y_1, \ldots, y_N) \sim\pi_\theta(\cdot\mid x), where N:=∣y∣N := |y| denotes the number of response tokens. We use si:=(x,y<i)s_i := (x, y_{<i}) to denote the autoregressive state before generating token yiy_i. On-Policy Distillation [1, 7] (OPD) minimizes the reverse KL divergence between the student and teacher distributions evaluated on prefixes sampled from the student policy:

K(θ)=Ex∼D, y∼πθ(⋅∣x)[∑i=1N(log⁡πθ(yi∣si)−log⁡πt(yi∣si))].\mathcal{K}(\theta) = \mathbb{E}_{x \sim\mathcal{D},\, y \sim\pi_\theta(\cdot\mid x)} \left[\sum_{i=1}^{N} \left(\log\pi_\theta(y_i \mid s_i) - \log\pi_t(y_i \mid s_i)\right)\right].

For each student-sampled token, OPD defines a token-level teacher preference signal Ai:=stopgrad⁡(log⁡πt(yi∣si)πθ(yi∣si))A_i := \operatorname{stopgrad}\left(\log\frac{\pi_t(y_i \mid s_i)}{\pi_\theta(y_i \mid s_i)}\right). In practice, the corresponding policy-gradient-style [29] OPD surrogate can be written as:

LOPD(θ)=Ex∼D, y∼πθ(⋅∣x)[−1N∑i=1NAilog⁡πθ(yi∣si)].(1)\mathcal{L}_{\mathrm{OPD}}(\theta) = \mathbb{E}_{x \sim\mathcal{D},\, y \sim\pi_\theta(\cdot\mid x)} \left[-\frac{1}{N} \sum_{i=1}^{N} A_i \log\pi_\theta(y_i \mid s_i)\right]. \tag*{(1)}

The loss function (Eq. (1)) assigns positive or negative token-level credit according to the teacher’s relative preference, thereby steering the student toward the teacher’s behavior at each state. For convenience, define the per-token OPD loss as ℓi(θ;x,y):=−Ailog⁡πθ(yi∣si)\ell_i(\theta; x, y) := -A_i \log\pi_\theta(y_i \mid s_i), such that

LOPD(θ)=Ex∼D, y∼πθ(⋅∣x)[1N∑i=1Nℓi(θ;x,y)].(2)\mathcal{L}_{\mathrm{OPD}}(\theta) = \mathbb{E}_{x \sim\mathcal{D},\, y \sim\pi_\theta(\cdot\mid x)} \left[\frac{1}{N} \sum_{i=1}^{N} \ell_i(\theta; x, y)\right]. \tag*{(2)}

Blockwise Decomposition. To examine how the utility of dense supervision evolves along a response, we partition each response into contiguous token blocks. Let mm denote the target block size and define K=⌈Nm⌉K = \left\lceil\frac{N}{m} \right\rceil as the number of blocks. The token indices belonging to the kk-th block are Ik={(k−1)m+1,…,min⁡(km,N)}\mathcal{I}_k = \{(k-1)m+1,\ldots,\min(km,N)\}, k=1,…,Kk=1,\ldots,K. Let nk:=∣Ik∣n_k := |\mathcal{I}_k| denote the number of tokens contained in block kk. We define the average OPD loss within block kk as Lk(θ;x,y):=1nk∑i∈Ikℓi(θ;x,y)\mathcal{L}_k(\theta;x,y) := \frac{1}{n_k}\sum_{i\in\mathcal{I}_k}\ell_i(\theta;x,y). The full-response OPD loss admits the exact decomposition L^OPD(θ;x,y)=∑k=1KnkNLk(θ;x,y)\widehat{\mathcal{L}}_{\mathrm{OPD}}(\theta;x,y) = \sum_{k=1}^{K}\frac{n_k}{N}\mathcal{L}_k(\theta;x,y). When all blocks have equal size, i.e., nk=mn_k=m and N=KmN=Km, Eq. (2) reduces to

L^OPD(θ)=Ex∼D, y∼πθ(⋅∣x)[1K∑k=1KLk(θ;x,y)].(3)\widehat{\mathcal{L}}_{\mathrm{OPD}}(\theta) = \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot\mid x)}\left[\frac{1}{K}\sum_{k=1}^{K}\mathcal{L}_k(\theta;x,y)\right]. \tag*{(3)}

This formulation preserves the standard full-response OPD objective while providing a natural unit for controlling supervision at the block level.

Empirical Anatomy of Dense Supervision

On-policy distillation appears unusually data-efficient partly because a single rollout generates supervision at every autoregressive prefix. A response of length NN does not provide one training target, but a sequence of NN visited states, si:=(x,y<i)s_i := (x,y_{<i}), i=1,…,Ni=1,\ldots,N, each paired with a token-level teacher signal.

Prior work [5] shows that repeated rollouts from even a single query can cover a surprisingly broad region of the state space, and that OPD can continue learning for hundreds of steps even when the training states are fixed. However, state count is not necessarily supervision count. Two visited prefixes may be distinct as states while inducing nearly the same parameter update [20, 30]; conversely, different tokens may produce distinct gradient directions, but some of these directions may reflect trajectory-specific variation rather than supervision that consistently appears across rollouts [21]. Thus, the raw supervised-token counts can substantially overestimate the reusable learning signal in a trajectory. We therefore shift from measuring state-space coverage to measuring supervision in gradient space. Rather than asking how many states a rollout visits, we ask a more fundamental question: How much novel and reproducible learning signal does each additional supervised token actually contribute?

General Setup

Experimental setup. We study mathematical reasoning and code generation with DeepSeek-R1-Distill-Qwen-1.5B [8] as the student. For mathematics, we use JustRL-DeepSeek-1.5B [9] as the teacher, DAPO-Math-17K [31] for training, and AMC 2023 [12] and AIME 2025 [2] for evaluation. For code generation, we use DeepCoder-1.5B-Preview [17] as the teacher, TACO [13] for training, and LiveCodeBench v6 [10] (LCB v6) for evaluation.

Hyperparameter settings. We follow the OPD training pipeline of Fu et al. [5] and hold all optimization hyperparameters fixed across comparisons (details are in Appendix § 9.§ 2). Specifically, we use rollout and mini-batch sizes of 64 with token-mean loss aggregation. Maximum prompt lengths are 1,024 tokens for mathematics and 4,096 for code, with a maximum response length of 7,168 tokens for both domains. We report avg@4 accuracy on AMC 2023 and AIME 2025, and avg@4 execution accuracy on LCB v6.

Fixed-prefix Replay Reveals Rapid Saturation of Supervision Utility

Fixed-prefix replay. To investigate how the utility of token-level supervision scales with trajectory length, we follow Fu et al. [5] and construct a controlled fixed-prefix replay setting. Zhao et al. [36] have also shown that distillation on offline student rollouts can remain effective, providing further support for the use of fixed trajectories in our controlled analysis. We focus on the medium-difficulty query1 used by Fu et al. [5]. This setting provides a non-trivial learning trajectory, where the student improves substantially during training while avoiding early saturation or near-zero success throughout optimization.

Position Control Reveals Cross-Block Supervision Transfer

For each one-shot query, we sample a rollout batch of B=64B = 64 trajectories from the initial student policy and keep this trajectory set fixed throughout optimization. Specifically, the replay buffer consists of BB sampled trajectories: {(x,y(b))}b=1B\{(x, y^{(b)})\}_{b=1}^{B}, where each response y(b)y^{(b)} is sampled independently from πθ0(⋅∣x)\pi_{\theta_0}(\cdot|x). We then control the amount of accessible supervision by truncating every frozen trajectory at different response prefix lengths, N∈{128,512,1024,2048,4096,7168}N \in\{128, 512, 1024, 2048, 4096, 7168\}, and apply the OPD objective (Eq. (2)) only to supervised positions within the retained prefix. These prefix conditions are strictly nested: S128⊂S512⊂⋯⊂S7168\mathcal{S}_{128} \subset\mathcal{S}_{512} \subset\cdots\subset\mathcal{S}_{7168}. Therefore, increasing NN only exposes additional later token-level supervision from the same set of BB trajectories, while keeping the query, rollout samples, model initialization, optimization budget, and all other hyperparameters identical. Although the replayed states are fixed, the student distribution and teacher–student discrepancy are recomputed under the updated parameters at each optimization step. As references, we compare against the original on-policy One-Shot OPD, which resamples BB trajectories from the evolving student policy, and full-data OPD, which samples BB queries from the full training distribution. Unlike these baselines, fixed-prefix replay removes changes in state visitation and isolates the contribution of additional supervised positions along a fixed set of trajectories. This design enables a direct test of whether dense token-level supervision scales proportionally with trajectory length: Does exposing more supervised token positions continue to provide proportional learning benefits, or does supervision utility rapidly saturate?

We first investigate whether increasing the amount of token-level supervision along a fixed trajectory leads to proportional improvements in downstream performance. Figure 2 compares fixed-prefix replay with prefix lengths ranging from 128 to 7168 tokens. As the prefix length increases, the student is exposed to progressively more supervised token positions from the same frozen trajectories. However, the validation performance does not improve proportionally with the number of supervised tokens. In particular, extending the prefix beyond 1024 tokens yields only marginal additional gains on the averaged AMC 2023 and AIME 2025 accuracy, despite introducing substantially more token-level supervision. This effect is not caused by insufficient optimization: all prefix conditions achieve rapid actor loss reduction and similar convergence behavior. Thus, additional supervised tokens yield diminishing downstream returns. These results establish a basic empirical observation: token count ↑↛\uparrow\not\to validation utility ↑\uparrow.

Accuracy and actor loss trajectories for fixed-prefix replay

Figure 2. Fixed-prefix replay reveals diminishing returns as supervised tokens increase. Left: validation accuracy averaged over AMC 2023 and AIME 2025 under different prefix lengths. Right: corresponding actor loss trajectories. Although longer prefixes expose the student to substantially more supervised token positions, they provide limited additional validation improvement, with later fluctuations reflecting the estimation errors introduced by fixed one-shot replay.

Position Control Reveals Cross-Block Supervision Transfer

The saturation observed under fixed-prefix replay admits two non-exclusive explanations: (1) Later trajectory positions may provide weaker or less transferable supervision. (2) Supervision from different positions may substantially overlap, so adding later tokens contributes little new information even when those tokens remain individually learnable. To disentangle these effects, we hold the supervised-token budget fixed, vary only the supervised position, and measure whether optimizing one region of the trajectory also improves teacher–student agreement on other regions.

Equal-budget position control. We reuse the frozen rollout pool from § 4.2 for the same query and retain the BcB_c trajectories whose responses contain at least 4,096 tokens. Fixing this common trajectory subset across all conditions ensures that positional comparisons are not confounded by differences in trajectory length or sample composition. Within each retained trajectory, we then partition the first 4,096 response tokens into four consecutive, non-overlapping blocks of equal size: Wk=Ik={(k−1)m+1,…,km}W_k = \mathcal{I}_k = \{(k-1)m+1,\ldots,km\}, m=1024m = 1024, k∈{1,2,3,4}k \in\{1,2,3,4\}. Each training condition applies the OPD loss to exactly one block. Tokens preceding the supervised block are retained as causal context but receive no direct supervision. All conditions start from the same initialization θ0\theta_0 and are optimized for T=250T = 250 updates under identical settings. Each update therefore contains exactly BcmB_cm supervised tokens, ensuring matched supervision and update budgets across conditions.

Measuring cross-block transfer. Let θT(k)\theta_T^{(k)} denote the parameters after training on block WkW_k. For each evaluation block WjW_j, we measure the teacher–student discrepancy and its relative reduction from initialization (Positive Rk→jR_{k\to j} indicates that training on WkW_k reduces the discrepancy on WjW_j):

Dj(θ)=1Bcm∑b=1Bc∑i∈Ij∣log⁡πθ(yi(b)∣si(b))πt(yi(b)∣si(b))∣,Rk→j=1−Dj(θT(k))Dj(θ0).(4)D_j(\theta) = \frac{1}{B_cm}\sum_{b=1}^{B_c}\sum_{i\in\mathcal{I}_j}\left|\log\frac{\pi_\theta(y_i^{(b)}\mid s_i^{(b)})}{\pi_t(y_i^{(b)}\mid s_i^{(b)})}\right|,\qquad R_{k\to j}=1-\frac{D_j(\theta_T^{(k)})}{D_j(\theta_0)}. \tag*{(4)}

The discrepancy Dj(θ)D_j(\theta) measures the average absolute difference between the student and teacher log-probabilities, evaluated on the same sampled tokens at fixed autoregressive states in block jj. The relative reduction Rk→jR_{k\to j} then quantifies the proportionate decrease in this discrepancy on block jj after optimizing the student with supervision from block kk, normalized by its initial value. Accordingly, diagonal entries characterize within-block fitting, whereas off-diagonal entries capture cross-block transfer. Note that this metric quantifies changes in sampled-token log-probability gaps at fixed autoregressive states, rather than measuring the divergence between the full next-token distributions of the student and teacher.

(1) Later block remains learnable. The diagonal entries in Table 1 are uniformly high: training on any single block reduces its own teacher–student discrepancy by 91.7%–92.7%. Thus, supervision does not vanish at later trajectory positions; even W4W_4 can be fitted almost as effectively as W1W_1 under the same token and optimization budgets. Yet downstream accuracy varies across blocks, showing that local learnability does not imply equal validation utility. (2) Supervision transfers substantially across positions. All off-diagonal entries are positive, with training on one block reducing the discrepancy on unsupervised blocks by 46.5%–70.4%. This transfer extends beyond adjacent windows and operates in both directions, indicating that supervision from different trajectory regions is not independent. At the same time, within-block reductions remain consistently larger than cross-block reductions, suggesting a combination of cross-position transferable signal and block-specific information.

Supervised blockAvg@4 Acc. (%)Eval. W1W_1Eval. W2W_2Eval. W3W_3Eval. W4W_4
W1W_156.4692.763.953.546.5
W2W_254.3866.892.265.359.9
W3W_355.4260.570.492.065.8
W4W_452.6054.266.167.291.7

Table 1. Cross-block transfer under matched supervision budgets. Rows denote supervised blocks and columns evaluation blocks; entries report 100Rk→j100R_{k\to j}. Bold diagonal entries show within-block fitting and off-diagonal entries cross-block transfer. Accuracy is the avg@4 across AMC 2023 and AIME 2025.

Taken together, these results rule out a simple “weak late-token” explanation for prefix saturation. Later blocks remain highly learnable, yet a substantial fraction of their supervisory signal can already be captured through training elsewhere in the trajectory. Thus, adding later tokens increases the number of supervised positions more than it increases the amount of functionally distinct supervision. While these results establish behavioral transfer across response positions, they do not directly reveal whether such transfer arises from geometrically aligned parameter updates. Nor do they determine whether the remaining position-specific information is reproducible across independent rollouts. We therefore turn to the gradient-space geometry of blockwise OPD supervision.

Gradient Geometry Reveals Shared Updates and Heterogeneous Residuals

The cross-block transfer observed in § 4.4 suggests that supervision from different response positions induces substantially overlapping parameter updates. However, strong transfer does not imply that position-specific supervision is entirely redundant. Different positions may share a dominant update direction while still contributing distinct local corrections. Moreover, the presence of such corrections does not guarantee that they are equally reproducible across independently sampled trajectories. To disentangle these effects, we examine the geometry of blockwise OPD gradients along three dimensions: shared directional alignment, position-dependent residual structure, and cross-sample reproducibility.

Blockwise gradient decomposition. We analyze blockwise OPD gradients at the common initialization θ0\theta_0 for both mathematics and code generation. For each domain, we construct a frozen batch containing one sampled trajectory per distinct prompt, retaining responses with at least 4,096 valid tokens. We partition the first 4,096 tokens into K=32K = 32 consecutive blocks of 128 tokens. Let gkg_k denote the batch-averaged OPD gradient for block kk, computed with detached teacher preference weights and before gradient clipping or optimizer transformations. To distinguish the shared update from position-dependent variation, we decompose each block gradient into its token-weighted mean and a centered residual:

gˉ=∑k=1Kwkgk,δk=gk−gˉ,wk=1K.\bar{g} = \sum_{k=1}^{K} w_k g_k,\qquad\delta_k = g_k - \bar{g},\qquad w_k = \frac{1}{K}.

Since all blocks have equal length, gˉ\bar{g} coincides with the full-horizon token-mean gradient up to numerical precision. We compare pairwise cosine similarities of the raw gradients and centered residuals, together with the eigenspectra of their respective Gram matrices. Importantly, centering removes the same mean vector from every block; it does not project the gradients onto the orthogonal complement of gˉ\bar{g}.

❶ A dominant shared direction persists across positions and prompts. Figure 3(a,Figure 3d) shows substantial positive alignment among raw block gradients throughout the response. The mean pairwise cosine similarity is 0.526 for mathematics and 0.680 for code, while even the minimum similarity remains positive (0.177 and 0.194, respectively). These results indicate that OPD supervision across different response positions contains a strong common update component. Notably, this alignment is observed in batches constructed from distinct prompts, suggesting that the shared direction is not merely an artifact of repeated rollouts of the same question. This provides a gradient-space explanation for the cross-block transfer observed in § 4.2 and § 4.4: optimizing one trajectory region can improve teacher–student agreement elsewhere because different regions induce partially aligned parameter updates.

Figure 3. Blockwise OPD gradient geometry at initialization using 128-token blocks over the first 4,096 response tokens. Top: mathematics; bottom: code. Raw block gradients are positively aligned across all measured positions, and their spectra contain a dominant component, indicating a global update direction that persists in multi-prompt batches. After subtracting the token-weighted mean gradient, residual cosine matrices exhibit heterogeneous positive and negative structure and substantially flatter spectra. Negative residual cosine denotes opposing deviations from the shared mean, not conflict between the complete block gradients. The annotated λmax⁡\lambda_{\max} values are the largest eigenvalues of the unnormalized raw and residual Gram matrices; plotted spectra are normalized within each matrix.

② Position-specific supervision remains geometrically diverse. Subtracting the shared mean reveals a substantially different structure. As shown in Figure 3(b,Figure 3e), the residual cosine matrices exhibit heterogeneous positive and negative similarities, with minimum values of -0.174 in mathematics and -0.350 in code. These negative similarities reflect opposing deviations from the common mean, rather than conflicts between the complete block gradients, which remain positively aligned. The residual eigenspectra in Figure 3(c,Figure 3f) are also considerably flatter than their raw counterparts. After removing the shared mean, gradient energy is distributed across many directions rather than concentrated in a single dominant mode. Thus, dense supervision cannot be characterized as merely repeated copies of one gradient direction. A substantial amount of position-dependent gradient variation remains beyond the common update. This is consistent with the incomplete cross-block transfer in § 4.4, where supervision from one block improves other blocks but does not fully replace their direct supervision. However, substantial residual variation does not necessarily imply equally reproducible supervision. While some residual directions may capture systematic, position-specific corrections, others may be highly dependent on individual sampled trajectories. We therefore examine whether these residual directions remain consistent across independent samples.

❸ Position-specific residuals exhibit heterogeneous reproducibility. We split each replay pool into disjoint halves (A) and (B) and independently estimate the blockwise gradients. For each position (kk), we measure the alignment of both raw gradients and their deviations from the corresponding half-specific full-horizon gradient: ρkraw=cos⁡(gkA,gkB),ρkres=cos⁡(gkA−gˉA,gkB−gˉB)\rho_k^{\mathrm{raw}} = \cos(g_k^A, g_k^B), \rho_k^{\mathrm{res}} = \cos(g_k^A - \bar{g}^A, g_k^B - \bar{g}^B). For independent rollouts of the same mathematics prompt, the first 128-token residual exhibits strong reproducibility, with ρ1res=0.787\rho_1^{\mathrm{res}} = 0.787. This demonstrates that subtracting the shared mean does not eliminate all stable, position-specific structure. In contrast, beyond 1,024 tokens, the mean residual cosine decreases to 0.071, despite a substantial average residual norm of 1.65. Over the same region, raw-gradient cosine remains 0.563. A similar separation appears across disjoint code prompts, where the depth-averaged raw and residual cosine similarities are 0.590 and 0.083, respectively. These results show that substantial position-specific gradient variation does not necessarily imply reproducible supervision. While raw gradients retain strong alignment, residual directions are less consistent across samples, suggesting substantial trajectory-dependent variation. Importantly, low residual alignment does not imply that these local corrections are uninformative.

Case study: Shared reasoning strategies and divergent execution. To contextualize the gradient observations, we inspect BcB_c independent rollouts of the same mathematical reasoning problem: summing 1/(2a3b5c)1/(2a^3b^5c) over nonnegative integer triples satisfying the even-parity constraint a+b+c≡0(mod2)a+b+c \equiv0 \pmod{2}. Early responses exhibit highly similar structures: all begin with either “Okay” or “Alright,” followed by a restatement of the problem. Importantly, this consistency extends beyond linguistic framing. Within the first 512 tokens, several trajectories mention generating functions, and some explicitly identify the correct parity decomposition. Thus, early alignment is consistent with shared, task-specific reasoning strategies rather than merely superficial formatting. However, shared strategies do not guarantee consistent execution. One trajectory correctly identifies the parity cases but subsequently reasons about an invalid case with exactly one odd exponent. As generation proceeds, trajectories also diverge in reasoning progress: at the same token depth, some are still exploring solution methods while others have advanced to numerical calculations. Representative excerpts from the sampled trajectories are provided in Table 2. This suggests that weak late-position residual alignment may partly reflect differences in reasoning paths, errors, corrections, and semantic stages, rather than an absence of informative supervision. Together, these observations highlight a fundamental distinction between the amount of position-specific gradient variation and its reproducibility. To scale supervision efficiently, the goal is not to discard later positions, but to reduce redundant reinforcement of shared updates while preserving coverage of potentially useful position-specific corrections across trajectories.

StageSampled trajectoryExcerptQualitative observation
EarlyID 1, [128, 512)“The problem reminds me of generating functions where you can encode conditions on the exponents . . . Because the condition is a+b+ca+b+c is even, it splits into two cases: either all of aa, bb, cc are even, or exactly two of them are odd.”Shared task-specific strategy and correct parity decomposition.
EarlyID 4, [256, 448)“Case 2: Exactly two of a,b,ca,b,c are odd . . . For Case 2: For the case where exactly one is odd, we have a=2a′+1a=2a' + 1, b=2b′b=2b', c=2c′c=2c' or variations.”A correct decomposition immediately branches into an invalid one-odd case.
MiddleID 20, [1792, 1984)“Hmmm. Hmmm confusing. Wait no, maybe I should model UU separately . . . How can we relate UU to TT? . . . Wait, hold on . . . Wait, no.”Still reorganizing the parity argument.
MiddleID 41, [1792, 2048)“25/12 + 5/3 = 45/12 = 15/4, which is correct . . . Thus, the answer is 25/12. Therefore, m=25m = 25, n=12n = 12, so m+n=37m+n = 37.”Already completes the numerical consistency check at a comparable depth.
LateID 12, [3328, 3584)“Recall that G(q)G(q) = even part + odd part . . . probably get 25/12 . . . m+nm+n is 25 + 12 = 37.”Reaches the correct final result after extended verification.
LateID 44, [3200, 3456)“Teven=31/16T_{\mathrm{even}} = 31/16, Todd=29/16T_{\mathrm{odd}} = 29/16 . . . Thus, the sum is 31/16 . . . m+n=47m+n = 47.”Reaches a self-consistent but incorrect branch and final answer.

Table 2. Qualitative excerpts from independent rollouts of the same medium-difficulty query from the initial student.

Stochastic Horizon Supervision: Algorithm and Theory

The analysis in § 4 reveals two complementary properties of dense OPD supervision. First, gradients from different response positions share substantial update directions, suggesting that supervising every position in every rollout may repeatedly reinforce overlapping information. Second, position-specific residuals remain geometrically diverse, indicating that supervision from earlier positions cannot fully substitute for later supervision. Moreover, the decreasing cross-rollout reproducibility of these residuals suggests that additional depth-specific corrections are not uniformly stable across sampled trajectories, although their low reproducibility does not necessarily imply low utility.

These findings motivate two key design principles: reducing redundant reinforcement of shared supervision while preserving access to heterogeneous depth-specific corrections. Fixed short-horizon supervision can exploit the former but permanently excludes deeper states, whereas full-horizon supervision preserves the latter at substantial computational cost. To balance these competing considerations, we propose Stochastic Horizons On-Policy Distillation (SH-OPD), which randomly varies the generation horizon across training updates. By frequently sampling shorter horizons while retaining access to longer trajectories, SH-OPD reduces the cost of repeatedly observing overlapping supervision without permanently excluding potentially complementary depth-specific signals. Importantly, SH-OPD adjusts supervision frequency across response depths via stochastic horizon sampling, without explicitly identifying or reweighting residual directions or altering the original token-level OPD objective.

Stochastic Horizons On-Policy Distillation

Prior work has explored horizon control to improve OPD efficiency. POPD progressively expands the rollout horizon, while TOPD adopts a fixed truncated horizon [33]. More recently, STOPD [34] introduces per-rollout stochastic truncation with inverse-probability weighting to compensate for truncation-induced gradient bias. In contrast, SH-OPD adopts a batch-level stochastic horizon shared across rollout workers, avoiding per-sequence horizon sampling and scheduling overhead.

Beyond this implementation difference, SH-OPD takes a different perspective on supervision allocation. Rather than explicitly correcting truncation-induced reweighting to recover the full-horizon update, SH-OPD retains the original token-mean OPD loss and studies the trade-off between reduced computation and the distortion of depth-specific supervision. This choice is motivated by the gradient structure identified in § 4: substantial cross-position overlap makes uniformly dense supervision potentially redundant, while heterogeneous residuals motivate maintaining access to deeper states. The resulting reweighting bias is characterized in § 5.2.

Definition 1 (Power-Survival Horizon Distribution). Let Hmax⁡H_{\max} denote the maximum response length in standard OPD, and let H0H_0 be a base horizon such that K=Hmax⁡/H0K = H_{\max}/H_0 is a positive integer. We define the set of admissible horizons as H={H0,2H0,…,KH0}\mathcal{H} = \{H_0, 2H_0, \ldots, KH_0\}. For a parameter α≥0\alpha\ge0, the stochastic horizon H∼PαH \sim P_\alpha follows a discrete power-survival distribution defined by

Pα(H≥kH0)=k−α,k=1,…,K.P_\alpha(H \ge kH_0) = k^{-\alpha}, \qquad k = 1,\ldots,K.

Equivalently, its probability mass function is

Pα(H=kH0)={k−α−(k+1)−α,k<K,K−α,k=K.P_\alpha(H = kH_0) = \begin{cases} k^{-\alpha} - (k+1)^{-\alpha}, & k < K, \\ K^{-\alpha}, & k = K. \end{cases}

Remark 1. The parameter α\alpha controls a trade-off between rollout depth and supervision allocation. At α=0\alpha= 0, SH-OPD recovers standard OPD, with P0(H=Hmax⁡)=1P_0(H = H_{\max}) = 1. For any finite α>0\alpha> 0, all admissible horizons have nonzero probability, while increasing α\alpha progressively shifts probability mass toward shorter rollouts, eventually concentrating on H0H_0 as α→∞\alpha\to\infty. For the considered range of α\alpha, the horizon probability mass exhibits an approximately U-shaped distribution, favoring shorter and longer horizons over intermediate ones. The expected horizon is E[Hα]=H0∑k=1Kk−α\mathbb{E}[H_{\alpha}] = H_0 \sum_{k=1}^{K} k^{-\alpha}, yielding an expected nominal generation cost of BE[Hα]B\mathbb{E}[H_{\alpha}] per batch, compared with BHmax⁡BH_{\max} under standard OPD[2]. For example, with H0=1024H_0 = 1024, Hmax⁡=7168H_{\max} = 7168, and α=0.5\alpha= 0.5, the expected horizon is approximately 4,114 tokens, reducing the nominal generation cost by 42.6%. Importantly, stochastic horizon sampling also implicitly reweights supervision across response positions. Assuming all trajectories reach their sampled horizons, the expected weight assigned to block jj is wα,j=∑k=jKpα(k)kw_{\alpha,j} = \sum_{k=j}^{K} \frac{p_{\alpha}(k)}{k}, where earlier blocks receive higher expected weights due to their inclusion in more sampled horizons. This reduces to uniform weighting, w0,j=1/Kw_{0,j} = 1/K, under standard OPD.

Algorithm 1 Stochastic Horizons On-Policy Distillation (SH-OPD)

Require: Student πθ\pi_{\theta}, teacher πt\pi_{\mathrm{t}}, dataset D\mathcal{D}, batch size BB, horizons (H0,Hmax⁡)(H_0, H_{\max}), exponent α\alpha

  1. K←Hmax⁡/H0K \leftarrow H_{\max}/H_0; construct horizon distribution pαp_{\alpha}

  2. for each training step tt do

  3. &nbsp;&nbsp;&nbsp;&nbsp;Sample prompts {xb}b=1B∼D\{x_b\}_{b=1}^{B} \sim\mathcal{D}

Batch-level horizon sampling

  1. &nbsp;&nbsp;&nbsp;&nbsp;Sample one shared horizon: kt∼pαk_t \sim p_{\alpha}, Ht←ktH0H_t \leftarrow k_tH_0

Parallel rollout generation

  1. &nbsp;&nbsp;&nbsp;&nbsp;Broadcast HtH_t to all rollout workers

  2. &nbsp;&nbsp;&nbsp;&nbsp;for b=1,…,Bb = 1, \ldots, B in parallel do

  3. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;yb←Generate⁡(πθ,xb)y_b \leftarrow\operatorname{Generate}(\pi_{\theta}, x_b) ▹\triangleright Terminate at EOS or after HtH_t tokens

  4. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Nb←∣yb∣N_b \leftarrow|y_b|

  5. &nbsp;&nbsp;&nbsp;&nbsp;end for

Standard OPD update

  1. &nbsp;&nbsp;&nbsp;&nbsp;Compute token losses ℓb,i(θ)\ell_{b,i}(\theta)

  2. &nbsp;&nbsp;&nbsp;&nbsp;Lt←∑b=1B∑i=1Nbℓb,i(θ)∑b=1BNb\mathcal{L}_t \leftarrow\dfrac{\sum_{b=1}^{B}\sum_{i=1}^{N_b}\ell_{b,i}(\theta)}{\sum_{b=1}^{B}N_b}

  3. &nbsp;&nbsp;&nbsp;&nbsp;Update θ\theta using ∇θLt\nabla_{\theta}\mathcal{L}_t

  4. end for

SH-OPD: Batch-Level Parallel Rollout Generation. As summarized in Algorithm 1, SH-OPD samples a single horizon per training batch and broadcasts it to all rollout workers. All trajectories are then generated in parallel, stopping at either EOS or the shared horizon HtH_t, rather than generating full-length responses before truncation. The horizon serves only as a generation cap, without conditioning the student policy. All generated tokens are supervised using the standard token-mean OPD loss. This batch-level design avoids per-sequence horizon scheduling overhead while reducing generation cost. Although shorter horizons could enable larger batches under a fixed token budget, we keep the batch size fixed to isolate the effect of stochastic horizon sampling.

Theoretical Perspective: A Compute–Reweighting Trade-off

Section 3.4 empirically reveals a shared gradient component and position-specific residuals across response blocks. Building on this observation, we extend the decomposition to population-level expected gradients under the current student policy. This allows us to characterize how stochastic horizon sampling reweights supervision across response positions and induces a systematic bias relative to full-horizon OPD.

Assumption 1 (On-Policy Gradient Decomposition). At optimization step tt, SH-OPD samples a horizon Ht∼PαH_t \sim P_{\alpha} and generates trajectories using the current student policy πθt\pi_{\theta_t}. Let g^tα\hat{g}_t^{\alpha} denote the resulting

stochastic gradient of the token-mean OPD surrogate defined in Eq. (1). The student is updated according to

θt+1=θt−ηg^tα,t=1,…,T.\theta_{t+1} = \theta_t - \eta\hat{g}_t^\alpha,\quad t = 1,\ldots,T.

Let Ft\mathcal{F}_t denote the training history before sampling the current batch. We define the conditional expected update direction and its deviation from full-horizon OPD as hα(θt):=E[g^tα∣Ft]h_\alpha(\theta_t) := \mathbb{E}[\hat{g}_t^\alpha\mid\mathcal{F}_t], bα(θt):=hα(θt)−h0(θt)b_\alpha(\theta_t) := h_\alpha(\theta_t) - h_0(\theta_t), where h0(θt)h_0(\theta_t) is the expected full-horizon OPD update evaluated at the same current parameters θt\theta_t. Defining ξtα=g^tα−hα(θt)\xi_t^\alpha= \hat{g}_t^\alpha- h_\alpha(\theta_t), we obtain

g^tα=h0(θt)+bα(θt)+ξtα,E[ξtα∣Ft]=0.(5)\begin{aligned} \hat{g}_t^\alpha&= h_0(\theta_t) + b_\alpha(\theta_t) + \xi_t^\alpha,\\ \mathbb{E}[\xi_t^\alpha\mid\mathcal{F}_t] &= 0. \tag*{(5)} \end{aligned}

The decomposition separates the reference OPD update, the systematic bias induced by stochastic horizons with α\alpha in Definition 1, and the stochastic gradient noise. Importantly, all three components are evaluated at the evolving parameters θt\theta_t, preserving the on-policy nature of training.

Based on Assumption 1, we characterize the horizon-induced bias bα(θ)b_\alpha(\theta) as follows:

Lemma 1 (Geometry of the Reweighting Bias). For clarity, consider equal-length blocks and trajectories that reach their sampled horizons. Let μk(θ):=Eτ∼πθ[∇θLk(θ;τ)]\mu_k(\theta) := \mathbb{E}_{\tau\sim\pi_\theta}[\nabla_\theta\mathcal{L}_k(\theta;\tau)] be the expected gradient of block kk, with the expectation taken over current-policy trajectories and the derivative understood as the detached OPD surrogate gradient. Under this setting, h0(θ)=1K∑k=1Kμk(θ)h_0(\theta) = \frac{1}{K}\sum_{k=1}^{K}\mu_k(\theta). Following § 4.5, define the population-level centered residual δk(θ)=μk(θ)−h0(θ)\delta_k(\theta) = \mu_k(\theta) - h_0(\theta). Let wα,kw_{\alpha,k} denote the implicit block weights described in Remark 1, and let Δwα,k=wα,k−1K\Delta w_{\alpha,k} = w_{\alpha,k} - \frac{1}{K}. Because both weighting schemes sum to one, their shared gradient component cancels: bα(θ)=∑k=1KΔwα,kμk(θ)=∑k=1KΔwα,kδk(θ)b_\alpha(\theta) = \sum_{k=1}^{K}\Delta w_{\alpha,k}\mu_k(\theta) = \sum_{k=1}^{K}\Delta w_{\alpha,k}\delta_k(\theta). Define the stable residual Gram matrix by [Gstable(θ)]ij=⟨δi(θ),δj(θ)⟩[G_{\mathrm{stable}}(\theta)]_{ij} = \langle\delta_i(\theta), \delta_j(\theta)\rangle, λ(⋅)\lambda(\cdot) denotes the matrix eigenvalue. The squared systematic bias then admits the exact characterization:

∥bα(θ)∥2=Δwα⊤Gstable(θ)Δwα≤λmax⁡(Gstable(θ))∥Δwα∥22(6)\lVert b_\alpha(\theta)\rVert^2 = \Delta w_\alpha^\top G_{\mathrm{stable}}(\theta)\Delta w_\alpha\le\lambda_{\max}\left(G_{\mathrm{stable}}(\theta)\right)\lVert\Delta w_\alpha\rVert_2^2 \tag*{(6)}

This result links the gradient geometry in § 4 to the bias induced by stochastic horizon sampling. Since the block weights sum to one, the shared gradient component cancels, and the reweighting bias depends solely on the weighted depth-specific residuals. Unlike the empirical analysis in § 4.5, which evaluates gradients at initialization, our theoretical characterization is defined at arbitrary current parameters, allowing Gstable(θ)G_{\mathrm{stable}}(\theta) to evolve throughout on-policy optimization. Thus, the reweighting bias is jointly determined by the changes in block weights and the geometry of depth-specific residuals, characterizing the trade-off between computational efficiency and preserving depth-specific supervision.

The preceding lemma characterizes the systematic gradient distortion induced by horizon reweighting. We next examine how this distortion interacts with stochastic optimization under a finite training budget.

Lemma 2 (Stationarity Bound under Horizon Reweighting). Let F(θ)F(\theta) be an LL-smooth, lower-bounded reference potential satisfying F(θ)≥Finf⁡F(\theta) \ge F_{\inf}. We assume that its gradient coincides with the expected full-horizon OPD update direction, i.e., h0(θ)=∇F(θ)h_0(\theta) = \nabla F(\theta). Furthermore, assume that the conditional variance of the stochastic update is uniformly bounded: E[∥ξtα∥2∣Ft]≤σα2\mathbb{E}[\lVert\xi_t^\alpha\rVert^2 \mid\mathcal{F}_t] \le\sigma_\alpha^2. Under the conditions of Lemma 1, for any constant stepsize 0<η≤1/L0 < \eta\le1/L,

1T∑t=1TE[∥∇F(θt)∥2]≤2ΔFηT⏟optimization+Lησα2⏟stochastic+Bα,T2⏟reweighting,(7)\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\left[\lVert\nabla F(\theta_t)\rVert^2\right] \le\underbrace{\frac{2\Delta_F}{\eta T}}_{\text{optimization}} + \underbrace{L\eta\sigma_\alpha^2}_{\text{stochastic}} + \underbrace{\mathcal{B}_{\alpha,T}^2}_{\text{reweighting}}, \tag*{(7)}

where ΔF:=F(θ1)−Finf⁡\Delta_F := F(\theta_1) - F_{\inf} and Bα,T2:=1T∑t=1TE[ΔwαGstable(θt)Δwα]\mathcal{B}^2_{\alpha,T} := \frac{1}{T} \sum_{t=1}^{T} \mathbb{E}\left[\Delta w_\alpha^{} G_{\mathrm{stable}}(\theta_t)\Delta w_\alpha\right]. This separates the optimization error into three terms. The first decreases with the number of updates, the second captures stochastic gradient noise, and the third measures the distortion of stable depth-specific gradients. Importantly, the reweighting bias is evaluated along the evolving optimization trajectory rather than at a fixed parameter point.

Definition 2. The benefit of SH-OPD arises from reducing the expected rollout cost per update. By Definition 1, E[Hα]=H0∑k=1Kk−α\mathbb{E}[H_\alpha] = H_0 \sum_{k=1}^{K} k^{-\alpha}. With a fixed rollout batch size BB, the expected nominal generation cost per update is cα=BE[Hα]c_\alpha= B\mathbb{E}[H_\alpha]. Under a total nominal token budget CC, the corresponding number of updates is

Tα=⌊CBE[Hα]⌋.T_\alpha= \left\lfloor\frac{C}{B\mathbb{E}[H_\alpha]} \right\rfloor.

Corollary 1 (Compute-Normalized Trade-off). Under the conditions of Lemma 2 and Definition 2, assume ΔF>0\Delta_F > 0 and C≥2cαC \ge2c_\alpha. Choosing ηα=min⁡{1L,2ΔFLσα2}\eta_\alpha= \min\left\{\frac{1}{L}, \sqrt{\frac{2\Delta_F}{L\sigma_\alpha^2}}\right\}, where ηα=1/L\eta_\alpha= 1/L if σα2=0\sigma_\alpha^2 = 0, yields

Eα(C)≲κσα2E[Hα]C⏟finite-compute stochastic cost+Bα,Tα2⏟reweighting bias,(8)\mathcal{E}_\alpha(C) \lesssim\underbrace{\sqrt{\frac{\kappa\sigma_\alpha^2\mathbb{E}[H_\alpha]}{C}}}_{\text{finite-compute stochastic cost}} + \underbrace{\mathcal{B}^2_{\alpha,T_\alpha}}_{\text{reweighting bias}}, \tag*{(8)}

where Eα(C):=1Tα∑t=1TαE∥∇F(θt)∥2\mathcal{E}_\alpha(C) := \frac{1}{T_\alpha}\sum_{t=1}^{T_\alpha}\mathbb{E}\lVert\nabla F(\theta_t)\rVert^2, and κ=16LΔFB\kappa= 16L\Delta_F B is independent of α\alpha and CC.

How to Choose α\alpha? Corollary 1 establishes a compute–reweighting trade-off: shorter expected horizons permit more on-policy updates under a fixed compute budget, potentially reducing stochastic optimization error, but may also distort reproducible depth-specific gradient structure, as characterized by Lemma 1. While α=0\alpha= 0 recovers full-horizon OPD with zero reweighting bias (Bα,Tα2=0\mathcal{B}^2_{\alpha,T_\alpha} = 0), increasing α\alpha is beneficial when the reduction in compute-adjusted stochastic cost outweighs the additional bias (Appendix § 10.§ 5). Thus, α\alpha governs a principled balance between computational efficiency and the preservation of stable supervision structure. In the next part, we investigate the choice of α\alpha empirically.

Empirical Evaluation

Experimental setup. We evaluate SH-OPD on mathematical reasoning and code generation with H0=1024H_0 = 1024 and Hmax⁡=7168H_{\max} = 7168. We consider α∈{0,0.5,1.0,1.5,5.0}\alpha\in\{0, 0.5, 1.0, 1.5, 5.0\} for both mathematics and code generation. All other experimental settings follow § 4.1, with the horizon distribution being the only factor varied within each task. Notably, α=0\alpha= 0 exactly recovers standard full-horizon OPD. Additional results extending SH-OPD to multi-teacher on-policy distillation (MOPD) and Instruction following tasks are presented in the Appendix § 9.§ 6.

Efficiency metrics. We measure training efficiency using two complementary per-update costs. The token-compute proxy counts the total prompt and response tokens processed by the rollout and training pipeline per update. It approximates sequence-processing workload but does not account for attention complexity or hardware utilization. The wall-time cost measures the end-to-end duration of each training update, including rollout generation, student and teacher scoring, and actor optimization, but excluding validation. Both metrics are averaged over logged training updates and normalized to standard OPD within each domain. In contrast, the cumulative supervised tokens in Figure 4(a,Figure 4d) count only valid generated response tokens contributing to the OPD loss, accounting for natural EOS termination.

Figure 4. Performance–cost trade-off under stochastic horizons. Mathematics (top) and code generation (bottom) are evaluated across different horizon exponents α\alpha. The left column shows validation accuracy against cumulative supervised response tokens, the middle column shows accuracy against optimization steps, and the right column reports per-update token-compute proxy and wall-time cost relative to standard OPD (α=0\alpha= 0). SH-OPD substantially reduces training cost while maintaining competitive performance. Mathematics favors α=1.0\alpha= 1.0, whereas code generation prefers α=0.5\alpha= 0.5, highlighting the task-dependent trade-off between supervision efficiency and depth coverage.

SH-OPD improves supervision efficiency. Figure 4 demonstrates that stochastic horizons substantially reduce training cost while preserving competitive downstream performance. On mathematics, $\alpha = 1.0 achieves the strongest overall trade-off. At approximately 45 million supervised response tokens, it reaches 63.21% accuracy, compared with 60.36% for standard OPD at a similar token budget (44.6 million tokens). By the end of training, it also achieves a higher final accuracy than standard OPD (63.21% vs. 62.50%), while reducing the per-update token-compute proxy to 45.1% and wall time to 45.5% of the standard OPD baseline. These results demonstrate that reducing redundant supervision can improve learning efficiency without sacrificing solution quality.

The preferred exponent differs across tasks. Code generation favors a more conservative horizon distribution. Among the tested SH-OPD settings, α=0.5\alpha= 0.5 achieves a higher peak accuracy than α=1.0\alpha= 1.0 (26.00% vs. 25.00%) and a higher mean accuracy over the final five evaluations (25.26% vs. 24.49%). Although their final checkpoints are close, the overall learning trajectory favors α=0.5\alpha= 0.5. This setting reduces the per-update token-compute proxy to 61.8% and wall time to 60.0% of standard OPD, demonstrating substantial efficiency gains while maintaining competitive code-generation performance.

More aggressive horizon reduction is not always beneficial. The mathematics sweep further reveals that increasing α\alpha beyond 1.0 does not consistently improve performance. Although α=1.5\alpha= 1.5 and α=5.0\alpha= 5.0 further reduce the cost per update, their final accuracies decrease to 58.93% and 58.21%, respectively. Thus, shorter horizons do not necessarily provide better supervision efficiency: excessively concentrating supervision on early response positions can compromise learning quality.

Choosing α\alpha through Gradient Geometry. The task-dependent preference for horizon exponents is consistent with the compute–reweighting trade-off in § 5.2. By exploiting shared gradients, SH-OPD reduces redundant supervision, while aggressive horizon reduction may distort complementary depth-specific updates. Lemma 1 shows that this distortion is governed by the Gram matrix of expected residual gradients. Empirically, mathematics favors α=1.0\alpha= 1.0, whereas code generation prefers α=0.5\alpha= 0.5, retaining more frequent deep supervision. This difference is qualitatively consistent with the larger residual Gram eigenvalue observed for code (16.08) than for mathematics (5.57), suggesting greater sensitivity to depth reweighting. However, these eigenvalues are measured from finite-batch gradients over the first 4,096 tokens at initialization and do not directly quantify the population-level bias over the full training horizon. Thus, the observed gradient geometry provides a plausible explanation for the task-dependent trade-off rather than a quantitative prediction of the optimal exponent.

Overall, the results demonstrate that effective OPD supervision does not require uniformly dense coverage of every response position. SH-OPD achieves substantial computational savings by reducing the frequency of full-horizon supervision while retaining access to deeper states. The different preferences for horizon reduction across tasks further highlight the importance of balancing computational efficiency with the preservation of complementary depth-specific supervision.

Related Work

OPD and Learning Dynamics. On-policy distillation (OPD) transfers teacher capabilities through token-level supervision on student-generated trajectories, mitigating the distribution mismatch of off-policy distillation [1, 7]. Recent studies have examined the factors governing its effectiveness. Li et al. [14] identify teacher–student compatibility and novel teacher knowledge as important conditions for successful distillation. Ding and Zhang [3] question the role of teacher knowledge transfer in OPD, showing that substantial performance gains can persist under noisy teacher supervision and can be partially reproduced through teacher-free self-improvement. Fu et al. [5] show that repeated rollouts from a single query can cover a substantial fraction of the states visited during full-data training, while the absorption of teacher supervision remains slow. Other works investigate the reliability of the distillation objective. Entropy-Aware OPD and Hybrid Policy Distillation adjust divergence objectives to improve distribution matching and training stability [11, 38]. Fu et al. [4] analyze the bias and variance of sampled-token supervision and identify failure modes arising from long-horizon prefix drift. Similarly, PowerOPD addresses unstable token-level learning signals through bounded reward transformations [35]. These studies characterize the conditions and optimization mechanisms underlying OPD. In contrast, our work focuses on how supervision from different trajectory positions interacts and contributes to learning.

Supervision Allocation in OPD. Recent studies have explored selective supervision to improve OPD efficiency: TIP identifies informative tokens using student entropy and teacher–student divergence [28], while teachability-aware selection and information-efficiency criteria further account for local learnability and gradient-estimation reliability [22, 25]. Position-aware methods also adjust token weights according to accumulated teacher–student discrepancies [26]. These approaches primarily optimize which generated tokens receive supervision. A complementary line of work reduces rollout generation costs through horizon control. FastOPD demonstrates that reasoning-prefix distillation can match full-trajectory OPD with substantially lower computation [32]. POPD and TOPD adopt progressive and fixed rollout truncation, respectively [33], while temporal curriculum methods progressively extend trajectory depth for multi-turn agents [24]. More recent approaches adapt the horizon to training conditions: KAT terminates rollouts exhibiting persistent low-KL agreement [27], while ADWIN selects prefix windows using delayed full-rollout probes and prefix–full gradient alignment [15]. TurnOPD further introduces adaptive rollout-depth and turn-balanced supervision for long-horizon agents [37].

Gradient Geometry and Supervision Efficiency. Recent studies have investigated the geometry of OPD updates, revealing subspace locking [20], gradient sparsity and spectral concentration [30], and token-level gradient-estimation reliability [22]. ADWIN further exploits prefix–full gradient alignment for adaptive horizon selection [15]. Beyond these analyses, we investigate cross-position gradient structure, identifying shared update components, heterogeneous position-specific residuals, and their consistency across sampled trajectories. We further characterize horizon-induced reweighting bias through the Gram matrix of expected depth-specific gradient residuals, establishing a theoretical connection between gradient geometry and the trade-off between compute-adjusted optimization error and supervision reweighting bias. While prior work has improved OPD efficiency through selective supervision and horizon control, our work connects cross-position supervision transfer and gradient residual structure to the bias–compute trade-off of stochastic horizon allocation.

Conclusion

Dense supervision exhibits diminishing returns; shared gradients + heterogeneous residuals; sparse deep supervision with stochastic horizons; compute–reweighting trade-off

Figure 5 (PDF p. 16).

In this work, we revisit the scaling of token-level supervision in on-policy distillation. Through controlled experiments and gradient-space analysis, we show that increasing supervision density yields diminishing returns, as gradients across response positions share substantial update directions while retaining heterogeneous, less reproducible depth-specific components. Dense supervision should be reduced where supervision overlaps, while complementary depth-specific supervision should remain accessible rather than be permanently truncated. Motivated by these findings, we introduce Stochastic Horizons On-Policy Distillation (SH-OPD), which reduces redundant supervision by randomly varying rollout horizons while preserving access to deeper response states. Our theoretical analysis further characterizes the trade-off between compute-adjusted stochastic optimization cost and the distortion of stable depth-specific gradient structure. Experiments on mathematical reasoning and code generation demonstrate that SH-OPD substantially reduces training cost while maintaining competitive or improved downstream performance. The preferred horizon exponent differs across tasks, consistent with their heterogeneous gradient structures. These findings suggest that effective supervision scaling is not simply about increasing the number of supervised tokens, but about allocating supervision efficiently across response depths. More broadly, SH-OPD offers a simple approach to improving the compute efficiency of on-policy distillation without requiring uniformly dense supervision. Misra, Divyanshu Kakwani, Dmitry Belenko, Dominic Manno, Dongyang Fan, Dor Tsur, Duncan Riach, Dustin VanStee, Dylan Lu, Edgar Minasyan, Ehsan Hosseini-Asl, Eileen Long, Elena Lantz, Ellie Evans, Elliott Ning, Elvis Chen, Emanuel Scoullos, Eric Chung, Eric Harper, Eric Pham-Hung, Eric Tramel, Eric Yang, Erik Arakelyan, Evelina Bakhturina, Ewa Dobrowolska, Fadi Saady, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Frankie Siino, Frederic Lafrance, Fuxiao Liu, Gantavya Bhatt, Garvit Kulshreshtha, George Armstrong, Giulio Lovisotto, Gomathy Venkata Krishnan, Gosia Lazuka, Grace Lam, Greg Heinrich, Greg Pauloski, Grzegorz Chlebus, Guilin Liu, Guo Chen, Guoming Zhang, Guyue Huang, Haifeng Qian, Hanrong Ye, Hao Gao, Hao Zhang, Haoqin Tu, Haran Kumar Shiv Kumar, Haribhau Hud, Harris Nover, Harrison Saturley-Hall, Harshil Kotamreddy, Hemil Desai, Henry Estela, Hexin Wang, Himanshu Singh, Hui Wang, Hung-Yueh Chiang, Igor Gitman, Ilia Karmanov, Ilya Loshchilov, Ingo Guehring, Itay Neeman, Ivan Moshkov, Jacky Kwok, Jakub Slowikowski, James Shen, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeff Willette, Jenna Diamond, Jenny Chen, Jesse Oliver, Jessica Medina, Jiacheng Xu, Jialin Song, Jian Hu, Jian Tu, Jian Zhang, Jiantao Jiao, Jiaqi Zeng, Jie-Fang Zhang, Jimmy Zhang, Jin Xu, Jingyu Xin, Jinhang Choi, Joey Conway, Joey Guman, John Kamalu, John Liu, John Roberts, Johnny Greco, Jonah Alben, Jonas Yang, Jonathan Cohen, Jonathan Li, Jonathan Raiman, Joseph Jennings, Juhi Mittal, Julia Gusak, Julien Veron Vialard, June Yang, Junkeun Yi, Jupinder Parmar, Ka Shun Shum, Kaiwen Shi, Kajal Jain, Kanishk Singla, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Huang, Katie Lyons, Keith Wyss, Keshav Santhanam, Keval Morabia, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kiran Praveen, Konstantinos Krommydas, Krishna Puvvada, Krzysztof Pawelec, Kushal Shah, Li Ding, Lily Lee, Linda Liu, Linglin Jing, Luis Vega, Maarten Van Segbroeck, Mahan Fathi, Mahdi Nazemi, Maka Dong, Makesh Narsimhan Sreedhar, Maor Ashkenazi, Marc Cuevas, Marcin Chochowski, Marco Pavone, Markus Kliegl, Marta Stepniewska-Dziubinska, Martina Gonzalez Vilas, Martyna Patelka, Matan Rusanovsky, Matthieu Le, Matty Machczynski, Matvei Novikov, Mauricio Ferrato, Meenakshi Sharma, Mehrzad Samadi, Melissa Corpuz, Melissa Saelzer, Meng Xin, Mengru Wang, Mengxi Wu, Meriem Boubdir, Michael Boone, Michael Callahan, Michael Elad, Michael Lightstone, Michal Bien, Michal Zawalski, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Ranzinger, Mike Wojcikiewicz, Min-Hung Chen, Ming Kang, Mingjie Liu, Mingyuan Ma, Misha Nekorystnov, MJ Mikulski, Mohammad Qazim Bhat, Mohammad Shoeybi, Morgan Purcell, Mostofa Patwary, Nabin Mulepati, Nancy Agarwal, Narimane Hennouni, Nathan Sala, Nave Assaf, Nayeon Lee, Neel Vaidya, Negar Habibi, Netanel Haber, Nick Gupta, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nik Spirin, Niket Agarwal, Nikolai Ludwig, Nima Tajbakhsh, Nithin Sunny, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Omer Ullman Argov, Pablo Ribalta, Padmavathy Subramanian, Parth Mannan, Pavlo Molchanov, Pawel Morkisz, Peter Jin, Petr Lapukhov, Petro Junior Milan, Philip Monk, Pinky Xu, Piotr Januszewski, Piotr Laszkiewicz, Po-Han Huang, Praateek Mahajan, Pradeep Thalasta, Prakash Gurumurthy, Pranav Prashant Thombre, Prasoon Varshney, Pratyusha Maiti, Preeti Chadha, Pritam Biswas, Pritam Gundecha, Przemek Strzelczyk, Puhui Meng, Pulkit Kumar, Rabe e h Karimi Mahabadi, Rachel Oberman, Rajpreet Thethy, Ramani Akkati, Rameshwar Shivbhakta, Ran Zilberstein, Randy Gelhausen, Rasoul Shafipour, Raviraj Joshi, Ray Macalisang, Rayen Tian, Reka Kovacs, Renjie Pi, Resha Shetty, Ria C, Rima Shahbazyan, Rishabh Garg, Rita Fernandes Neves, Ritika Borkar, Ritu Gala, Riya Talwar, Riyad Islam, Robert Clark, Robert Kirby, Roger Waleffe, Rohan Joshi, Rohan Pillai, Rohit Jena, Rohit Watve, Roi Garnett, Roi Koren, Rongguang Ye, Russell J. Hewett, Ryan Prenger, Ryan Stewart, Ryota Egashira, Sadegh Mahdavi, Sai Nishanth Padala, Salika Dave, Sandhya Rani Narravula, Sandip Bhaskar, Sangkuk Lym, Sanjeev Satheesh, Sarah Amirlani, Saran Vikas Murari, Saurabh Mishra, Saurabh Srivastava, Saurav Muralidharan, Sean Narenthiran, Sean Wang, Seokju Weijia Gao, Shangyi Geng, Jie Gong, Linhu Gong, Shengao Gong, Xiaochen Gong, Qizheng Gu, Yicheng Gu, Shuhao Guan, Haiqing Guo, Shiqi Guo, Xiang Guo, Zhengyan Guo, Beixi Hao, Wenxin Hao, Xiaoru Hao, Dailan He, Haotian He, Lehan He, Qi He, Weiran He, Xinran He, Xinyi He, Yibo He, Yunjia He, Chao Hong, Tiange Hong, Hao Hu, Jiaxi Hu, Ruikun Hu, Weiming Hu, Yangyang Hu, Zhenxing Hu, Liang Hua, Jinbin Huang, Ke Huang, Ruiyuan Huang, Siying Huang, Weixiao Huang, Yan Huang, Zhengjie Huang, Zhiqi Huang, Yulong Hui, Chaobo Jia, Yutong Jiang, Zhejun Jiang, Zuoyou Jiang, Wenyi Jin, Xinyi Jin, Yu Jing, Huanjun Kong, Guokun Lai, Aidi Li, Cheng Li, Chengyuan Li, Cong Li, Fang Li, Guanyu Li, Haoyang Li, Jia Li, Junxiong Li, Lei Li, Letian Li, Lincan Li, Weihong Li, Wentao Li, Xintong Li, Yang Li, Yishen Li, Yiwei Li, Yuxiao Li, Zhaowei Li, Zhaoxi Li, Zheming Li, Zhengxiao Li, Zhiyuan Li, Jiawei Lin, Xiaohan Lin, Yibo Lin, Zichao Lin, Ziyan Lin, Bill Liu, Boxiao Liu, Chuan Liu, Liang Liu, Shaowei Liu, Shudong Liu, Shuran Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yanming Liu, Yibo Liu, Yipeng Liu, Zhengying Liu, Zhiheng Liu, Enzhe Lu, Haoyu Lu, Linqiang Lu, Tingzhan Lu, Zhiyuan Lu, Aotian Luo, G. Luo, Junyu Luo, Yifan Luo, B. Lyu, Wenzhou Lyu, Shaoguang Mao, Yuan Mei, Xin Men, Minqing Ni, Yixuan Niu, Siyuan Pan, Shujun Peng, Zhangyang Qi, Ruoyu Qin, ZeChao Qin, Zeyu Qin, Haiquan Qiu, Jianxin Qiu, Jiezhong Qiu, Bowen Qu, Yuhao Qu, Zeyu Shang, Youbo Shao, Han Shen, Jincheng Shi, Juanfeng Shi, Lidong Shi, Shengyuan Shi, Wingchun Siu, Pengwei Song, Xiaoxi Song, Jianlin Su, Yunfeng Su, Zhaochen Su, Lin Sui, Jingsong Sun, Junyao Sun, Shaoning Sun, Shuzhe Sun, Tongyu Sun, Yujun Sun, Yunpeng Tai, Chuning Tang, Heyi Tang, Sirui Tang, Zecheng Tang, Chaoran Tian, Rongpeng Tian, Yu Tian, Wei Tu, Chensi Wang, Chuang Wang, Chunjie Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Hao Wang, Hao Wang, Huaqing Wang, Hui Wang, Jiayi Wang, Jinglong Wang, Jinhong Wang, Jiuzheng Wang, Linian Wang, Shaobo Wang, Shenzhi Wang, Shuyi Wang, Si Wang, Siyuan Wang, Tianfu Wang, Wenjue Wang, Xingran Wang, Xinmei Wang, Xinyuan Wang, Xusheng Wang, Yalin Wang, Yangkun Wang, Yao Wang, Yaoyu Wang, Yejie Wang, Yiqin Wang, Yucheng Wang, Yuzhi Wang, Zhaoji Wang, Zhaowei Wang, Zhengtao Wang, Zhenhao Wang, Zhongsheng Wang, Zifan Wang, Chu Wei, Ming Wei, Shouxin Wei, Zichen Wen, Fan Wu, Haoning Wu, Rucong Wu, Wenhao Wu, Xiaoxue Wu, Yingcong Wu, Yongqi Wu, Yuxin Wu, Zijian Wu, Xinglang Xian, Chenxuan Xiang, Yuye Xiang, Bocheng Xiao, Chenjun Xiao, Xin Xiao, Jin Xie, Xiaotong Xie, Yifeng Xie, Zhe Xie, Bowei Xing, Yiming Xiong, Baosheng Xu, Boyu Xu, Jiale Xu, Jianfan Xu, Jing Xu, Jinjing Xu, L. H. Xu, Qingtao Xu, Shuyao Xu, Suting Xu, Tiantian Xu, Tianxiang Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ye Xu, Yueni Xu, Ziyao Xu, Haonan Xue, Junjie Yan, Yaoyao Yan, Fan Yang, Guangyao Yang, Hao Yang, Junwei Yang, Ruoyu Yang, Wenjie Yang, Xiaofei Yang, Xinyu Yang, Yi Yang, Yiling Yang, Ying Yang, Yuchen Yang, Zhen Yang, Zhilin Yang, Zian Yang, Zuhao Yang, Haotian Yao, Dan Ye, Haoran Ye, Wenjie Ye, Zhanbo Ye, Bohong Yin, Haoxiang Yin, Xietong Yin, Chengzhen Yu, Haozhen Yu, Longhui Yu, Shengnan Yu, Shuying Yu, Tianxiang Yu, Enming Yuan, Mengjie Yuan, Tongtian Yue, Wei Yue, Yang Yue, Dunyuan Zha, Haobing Zhan, B. H. Zhang, Dehao Zhang, Fei Zhang, Hao Zhang, Haoyuan Zhang, Huanyu Zhang, Jiapei Zhang, Jiaxuan Zhang, Jin Zhang, Kaiyi Zhang, Miaozhen Zhang, Puqi Zhang, Qinglei Zhang, Rong Zhang, Rui Zhang, Shaoshuai Zhang, Shiyi Zhang, Xiaobin Zhang, Xiaoyun Zhang, Y. Zhang, Yangkun Zhang, Ye Zhang, Yichi Zhang, Yikun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Zijimg Zhang, Bin Zhao, Chengguang Zhao, Feifan Zhao, Jinglun Zhao, Jinxiang Zhao, Shuai Zhao, Wenshuo Zhao, Xiangyu Zhao, Xuanle Zhao, Yikai Zhao, Zijia Zhao, Haozhi Zheng, Huabin Zheng, Ruihan Zheng, Shaojie Zheng, Tengyang Zheng, Haofeng Zhong, Lei Zhong, Longguang Zhong, M. Zhou, Qiankang Zhou, Runjie Zhou, Ruozhang Zhou, Xinyu Zhou, Yiqiao Zhou, Zaida Zhou, Jinguo Zhu, Liya Zhu, Xinhao Zhu, Yangjunfeng Zhu, Yuxuan Zhu, Zhen Zhu, Chen Zhuang, Weiyu Zhuang, and Xinxing Zu. Kimi k3: Open frontier intelligence, 2026.

Extension to Experiments

Default hyperparameters for OPD

Training and evaluation. We follow the default OPD configuration of Fu et al. [5], using AdamW with a learning rate of 1×10−61 \times10^{-6}, weight decay 0.01, a rollout and mini-batch size of 64, sampling temperature 1.0, top-p=1.0, token-mean loss aggregation and gradient clipping at 1.0. We use maximum prompt lengths of 1,024 and 4,096 tokens for mathematics and code, respectively, and a maximum response length of 7,168 tokens for both domains. We evaluate each problem with four sampled responses, reporting avg@4 accuracy on AMC 2023 and AIME 2025 and avg@4 execution accuracy on LiveCodeBench v6. All comparisons use identical optimization hyperparameters and evaluation budgets. All experiments are conducted on 4 NVIDIA H20 GPUs.

Medium query

Query selection and difficulty. We adopt the medium-difficulty mathematical reasoning query (ID: t22) from the One-Shot OPD study by Fu et al. [5]. The query is drawn from DAPO-Math-17K [31], where its difficulty is determined by the initial student’s pass rate over eight independently sampled responses before training. Specifically, the medium query is solved in 4 out of 8 attempts, corresponding to an initial pass rate of 50%. The original study further shows that this query exhibits a gradual improvement in training success rate, from approximately 0.36 to 0.89 over 300 optimization steps, whereas the easy query is consistently solved and the hard query remains unsolved. We select the medium query because it provides a non-trivial learning regime with sufficient room for improvement, allowing us to examine how supervision from different response positions contributes to downstream learning without being dominated by early saturation or persistent failure.

Full query. The selected query is: Let SS be the set of triples (a,b,c)(a,b,c) of non-negative integers such that a+b+ca+b+c is even. Determine the value of

∑(a,b,c)∈S12a3b5c.\sum_{(a,b,c)\in S} \frac{1}{2^{a}3^{b}5^{c}}.

This sum can be expressed as m/nm/n, where mm and nn are relatively prime positive integers. Compute m+nm+n.

Fixed-prefix replay construction. Following the fixed-state analysis of Fu et al. [5], we construct a frozen replay pool by sampling B=64B=64 independent trajectories from the initial student policy on this query, using a sampling temperature of 1.0 and a maximum response length of 7,168 tokens. The sampled trajectories are stored and reused throughout optimization, so the visited autoregressive states remain fixed while the student parameters evolve. For the fixed-prefix experiment in § 4.2, we consider supervision horizons of {128,512,1024,2048,4096,7168}\{128,512,1024,2048,4096,7168\} tokens. Each condition uses the same replay pool but restricts the OPD loss to valid token positions within the corresponding prefix. Trajectories shorter than a specified prefix contribute only their available valid tokens. Thus, the supervision sets are nested, and increasing the prefix length introduces additional later-position supervision without changing the underlying trajectories. At each optimization step, the teacher–student log-probability discrepancy is evaluated on the retained states using the updated student parameters.

Common trajectory subset for position-controlled experiments. For the cross-block transfer analysis in § 4.4, we further restrict the replay pool to trajectories containing at least 4,096 valid response tokens. Denoting the number of retained trajectories by BcB_c, we partition the first 4,096 tokens of each retained trajectory into four non-overlapping blocks of 1,024 tokens. All position-controlled conditions share this identical trajectory subset, ensuring that differences in training outcomes arise from the supervised positions rather than trajectory length or sample composition. Each condition optimizes one block for 250 updates, with preceding tokens preserved as causal context but excluded from direct supervision.

Scope of the analysis. Fu et al. [5] show that the one-shot OPD phenomenon is robust to variations in query difficulty, response-length cap, and sampling temperature. Our experiments build on this controlled setting but address a different question: rather than examining how many distinct states are sufficient for learning, we investigate how much additional supervision is contributed by different token positions within a fixed set of trajectories.

Qualitative trajectory excerpts

Table 2: Qualitative excerpts from independent rollouts of the same medium-difficulty query from the initial student. Trajectory IDs are zero-based row indices in the frozen pool, and intervals are half-open response-token ranges. Excerpts are verbatim apart from ellipses and light mathematical typesetting.

Table 2 complements the case study in § 4.5 with excerpts from the frozen rollout pool. The examples illustrate three recurring patterns: a shared generating-function strategy early in generation, trajectory-specific mistakes despite initially correct parity reasoning, and different levels of progress at comparable response depths.

Horizon Distribution Statistics

Power-survival horizon distribution. We use H0=1024H_0 = 1024 and Hmax⁡=7168H_{\max} = 7168, yielding seven admissible horizons. Under the power-survival distribution, the expected horizons for α∈{0,0.5,1.0,1.5,5.0}\alpha\in\{0, 0.5, 1.0, 1.5, 5.0\} are 7,168, 4,114, 2,655, 1,928, and 1,062 tokens, respectively, corresponding to nominal generation-budget reductions of 0%, 42.6%, 63.0%, 73.1%, and 85.2% relative to full-horizon OPD. The probability of sampling the maximum horizon decreases from 100% at α=0\alpha= 0 to 37.8%, 14.3%, 5.4%, and 0.006%, respectively. Thus, increasing α\alpha progressively concentrates sampling on shorter horizons while retaining nonzero probability of deeper supervision. These statistics characterize nominal horizon budgets; actual generated lengths may be shorter due to EOS termination.

Table 3: Statistics of the power-survival horizon distribution with H0=1024H_0 = 1024 and Hmax⁡=7168H_{\max} = 7168. Token reduction is computed relative to full-horizon OPD (α=0\alpha= 0), assuming trajectories reach their sampled horizons.

As shown in Table 3, increasing α\alpha progressively shifts probability mass toward shorter horizons, substantially reducing the expected generation budget. Notably, moderate values of α\alpha retain meaningful probabilities of sampling the maximum horizon, whereas α=5.0\alpha= 5.0 concentrates 96.88% of the probability mass on the shortest horizon, effectively approaching fixed short-horizon supervision.

α\alphaE[H]\mathbb{E}[H]Token ReductionP(H=H0)P(H = H_0)P(H=Hmax⁡)P(H = H_{\max})
07,1680.0%0.00%100.00%
0.54,11442.6%29.29%37.80%
1.02,65563.0%50.00%14.29%
1.51,92873.1%64.64%5.40%
5.01,06285.2%96.88%0.006%

Table 3.

Empirical sampled and realized response lengths. Table 4 reports statistics from the full-data training runs used in Figure 4. A single horizon is sampled per global update and shared by all B=64B = 64 trajectories in that rollout batch. We define the empirical sampled horizon as the arithmetic mean of these per-update samples, and the realized response length as the mean number of valid generated response tokens after natural termination. The empirical sampled horizons are close to their theoretical expectations, with finite-run deviations caused by sampling variability. Realized response lengths are lower than the sampled horizons because some trajectories terminate at EOS.

Domainα\alphaUpdatesE[H]\mathbb{E}[H]H‾sampled\overline{H}_{\mathrm{sampled}}L‾response\overline{L}_{\mathrm{response}}EOS term.
Math02797,1687,1685,76454.34%
Math0.52794,1144,1693,56925.25%
Math1.02792,6552,8592,54613.88%
Math1.52791,9282,1841,9998.70%
Math5.02791,0621,0571,0550.58%
Code03847,1687,1686,45223.67%
Code0.53844,1144,0913,78311.58%
Code1.03842,6552,8112,6556.67%

Table 4.

Table 4: Empirical horizon and response-length statistics for the full-data runs. E[H]\mathbb{E}[H] is the theoretical mean sampled horizon, H‾sampled\overline{H}_{\mathrm{sampled}} is its empirical mean across logged updates, and L‾response\overline{L}_{\mathrm{response}} is the mean realized response length after EOS termination.

Multi-Teacher On-Policy Distillation

Setup. We extend SH-OPD to multi-teacher on-policy distillation (MOPD) with a single DeepSeek-R1-Distill-Qwen-1.5B student. Each example is routed by its ability label to a frozen domain-specific teacher: JustRL-DeepSeek-1.5B for 17,917 DAPO-Math training examples, DeepCoder-1.5B-Preview for 24,701 TACO code examples, and UltraData-IF-1.5B [5] for 4,501 first-turn Multi-IF instruction-following examples [19]. We compare α∈{0,0.5,1.5}\alpha\in\{0, 0.5, 1.5\} using H0=1024H_0 = 1024 and Hmax⁡=7168H_{\max} = 7168. Mathematics and code are evaluated during training with avg@1 on AMC 2023 (40 problems), AIME 2025 (30 problems), and LiveCodeBench v6 (175 problems), using a 31,744-token response cap. Instruction following is evaluated on all 4,445 Multi-IF conversations, each containing three turns in one of eight languages.

α\alphaToken costWall timeAMC23AIME25LCB v6Multi-IF meanMulti-IF turn 3
0100.0%100.0%85.8336.6725.3337.8926.79
0.561.2%59.9%83.0032.2225.3339.2627.81
1.536.4%35.6%82.5035.5623.0538.7326.96

Table 5. MOPD efficiency and downstream performance. Per-update token-compute and wall-time costs are averaged over updates 1–300 and normalized to full-horizon MOPD. Math and code results are the mean avg@1 over the last three checkpoints, while Multi-IF scores are reported at checkpoint 300.

Results. SH-OPD remains effective under multi-teacher distillation. With α=0.5\alpha= 0.5, SH-OPD reduces per-update token-compute and wall-time costs to 61.2% and 59.9% of full-horizon MOPD, respectively. It matches the full-horizon result on LCB v6 and improves the Multi-IF mean from 37.89 to 39.26, while incurring moderate decreases on the two mathematics benchmarks. The more aggressive α=1.5\alpha= 1.5 setting reduces both costs by roughly 64%, but exhibits less consistent performance across tasks. Overall, stochastic horizons retain useful multi-domain performance at substantially lower training cost, extending the efficiency benefits of SH-OPD beyond the single-teacher setting. Since Multi-IF training and evaluation share source conversations, the Multi-IF results should be interpreted as in-sample performance comparisons rather than evidence of held-out generalization.

Omitted Proofs and Additional Results

Proof of Lemma 1

Proof. We derive the population-level reweighting identity and establish the corresponding quadratic-form characterization of the systematic bias.

Consider KK equal-length blocks, each containing H0H_0 tokens. Let

pα(m)=Pα(H=mH0),m=1,…,K,p_{\alpha}(m) = \mathbb{P}_{\alpha}(H = mH_0), \qquad m = 1,\ldots,K,

denote the probability of sampling a horizon comprising mm blocks. As assumed in the lemma, trajectories reach their sampled horizons, and horizon sampling preserves the distribution of the observed prefixes. Consequently, the expected gradient of block kk, whenever included, is μk(θ)\mu_k(\theta).

For a sampled horizon H=mH0H = mH_0, the token-mean gradient is the average of the first mm block gradients. Taking the expectation over sampled horizons yields

hα(θ)=∑m=1Kpα(m)(1m∑k=1mμk(θ))=∑k=1K(∑m=kKpα(m)m)μk(θ).(9)\begin{aligned} h_{\alpha}(\theta) &= \sum_{m=1}^{K} p_{\alpha}(m) \left(\frac{1}{m}\sum_{k=1}^{m}\mu_k(\theta)\right) \\ &= \sum_{k=1}^{K}\left(\sum_{m=k}^{K}\frac{p_{\alpha}(m)}{m}\right)\mu_k(\theta). \tag*{(9)} \end{aligned}

Here, wα,kw_{\alpha,k} represents the effective weight assigned to block kk under the sampled horizon distribution. These weights are normalized, since

∑k=1Kwα,k=∑m=1Kpα(m)m∑k=1m1=∑m=1Kpα(m)=1.(10)\begin{aligned} \sum_{k=1}^{K} w_{\alpha,k} &= \sum_{m=1}^{K}\frac{p_{\alpha}(m)}{m}\sum_{k=1}^{m}1 \\ &= \sum_{m=1}^{K}p_{\alpha}(m)=1. \tag*{(10)} \end{aligned}

When α=0\alpha=0, the horizon is fixed at Hmax⁡=KH0H_{\max}=KH_0. Thus, w0,k=1/Kw_{0,k}=1/K for all kk, and

h0(θ)=1K∑k=1Kμk(θ).h_0(\theta)=\frac{1}{K}\sum_{k=1}^{K}\mu_k(\theta).

Now define the weight perturbation Δwα,k=wα,k−1/K\Delta w_{\alpha,k}=w_{\alpha,k}-1/K. By normalization, ∑k=1KΔwα,k=0\sum_{k=1}^{K}\Delta w_{\alpha,k}=0. Recalling the centered population residual δk(θ)=μk(θ)−h0(θ)\delta_k(\theta) = \mu_k(\theta) - h_0(\theta), we obtain

bα(θ)=hα(θ)−h0(θ)=∑k=1KΔwα,kμk(θ)=∑k=1KΔwα,k(h0(θ)+δk(θ))=∑k=1KΔwα,kδk(θ).(11)\begin{aligned} b_\alpha(\theta) &= h_\alpha(\theta) - h_0(\theta) \\ &= \sum_{k=1}^{K} \Delta w_{\alpha,k}\mu_k(\theta) \\ &= \sum_{k=1}^{K} \Delta w_{\alpha,k}\left(h_0(\theta) + \delta_k(\theta)\right) \\ &= \sum_{k=1}^{K} \Delta w_{\alpha,k}\delta_k(\theta). \tag*{(11)} \end{aligned}

The last equality follows from the zero-sum property of the weight perturbations, which eliminates the shared gradient component.

By the definition of the stable residual Gram matrix,

[Gstable(θ)]ij=⟨δi(θ),δj(θ)⟩,[G_{\mathrm{stable}}(\theta)]_{ij} = \langle\delta_i(\theta), \delta_j(\theta) \rangle,

the squared bias can be expressed as

∥bα(θ)∥2=∥∑k=1KΔwα,kδk(θ)∥2=∑i=1K∑j=1KΔwα,iΔwα,j⟨δi(θ),δj(θ)⟩=ΔwαTGstable(θ)Δwα.(12)\begin{aligned} \|b_\alpha(\theta)\|^2 &= \left\|\sum_{k=1}^{K} \Delta w_{\alpha,k}\delta_k(\theta)\right\|^2 \\ &= \sum_{i=1}^{K}\sum_{j=1}^{K}\Delta w_{\alpha,i}\Delta w_{\alpha,j}\langle\delta_i(\theta),\delta_j(\theta)\rangle\\ &= \Delta w_\alpha^{\mathsf{T}}G_{\mathrm{stable}}(\theta)\Delta w_\alpha. \tag*{(12)} \end{aligned}

Since Gstable(θ)G_{\mathrm{stable}}(\theta) is a Gram matrix, it is positive semidefinite. Applying the Rayleigh quotient bound gives

∥bα(θ)∥2=ΔwαTGstable(θ)Δwα≤λmax⁡(Gstable(θ))∥Δwα∥22.(13)\begin{aligned} \|b_\alpha(\theta)\|^2 &= \Delta w_\alpha^{\mathsf{T}}G_{\mathrm{stable}}(\theta)\Delta w_\alpha\\ &\le\lambda_{\max}\left(G_{\mathrm{stable}}(\theta)\right)\|\Delta w_\alpha\|_2^2. \tag*{(13)} \end{aligned}

This establishes Eq. (6) and completes the proof of Lemma 1. □

Proof of Lemma 2

Reference potential assumption. The expected full-horizon update direction h0(θ)h_0(\theta) is defined by averaging the OPD surrogate gradients over trajectories sampled from the current student policy, thereby preserving the on-policy nature of training. For the stationarity analysis, we additionally assume the existence of an LL-smooth, lower-bounded reference potential FF satisfying ∇F(θ)=h0(θ)\nabla F(\theta) = h_0(\theta). This assumption is not required for the reweighting-bias characterization in Lemma 1 and is not automatically guaranteed by the detached OPD surrogate. Under this assumption, Lemma 2 characterizes how horizon-induced reweighting bias and stochastic gradient variance jointly affect stationarity under an idealized stochastic gradient update model. Proof. Recall the update rule

θt+1=θt−ηg^tα.\theta_{t+1} = \theta_t - \eta\widehat{g}_t^\alpha.

Under the assumptions of Lemma 2, the conditional gradient decomposition is

g^tα=∇F(θt)+bα(θt)+ξtα,(14)\widehat{g}_t^\alpha= \nabla F(\theta_t) + b_\alpha(\theta_t) + \xi_t^\alpha, \tag*{(14)}

where

E[ξtα∣Ft]=0\mathbb{E}[\xi_t^\alpha\mid\mathcal{F}_t] = 0

and

E[∥ξtα∥2∣Ft]≤σα2.\mathbb{E}[\|\xi_t^\alpha\|^2 \mid\mathcal{F}_t] \le\sigma_\alpha^2.

For brevity, write

ut:=∇F(θt),bt:=bα(θt),u_t := \nabla F(\theta_t), \qquad b_t := b_\alpha(\theta_t),

and denote conditional expectation given Ft\mathcal{F}_t by Et[⋅]\mathbb{E}_t[\cdot].

Step 1: One-step descent. By the LL-smoothness of FF,

F(θt+1)≤F(θt)+⟨∇F(θt),θt+1−θt⟩+L2∥θt+1−θt∥2.(15)F(\theta_{t+1}) \le F(\theta_t) + \langle\nabla F(\theta_t), \theta_{t+1} - \theta_t \rangle+ \frac{L}{2}\|\theta_{t+1} - \theta_t\|^2. \tag*{(15)}

Substituting the update rule gives

F(θt+1)≤F(θt)−η⟨ut,g^tα⟩+Lη22∥g^tα∥2.(16)F(\theta_{t+1}) \le F(\theta_t) - \eta\langle u_t, \widehat{g}_t^\alpha\rangle+ \frac{L\eta^2}{2}\|\widehat{g}_t^\alpha\|^2. \tag*{(16)}

Conditioned on Ft\mathcal{F}_t, both utu_t and btb_t are fixed, while ξtα\xi_t^\alpha has zero conditional mean. Consequently,

Et[g^tα]=ut+bt,(17)\mathbb{E}_t[\widehat{g}_t^\alpha] = u_t + b_t, \tag*{(17)}

and

Et∥g^tα∥2=Et∥ut+bt+ξtα∥2=∥ut+bt∥2+Et∥ξtα∥2≤∥ut+bt∥2+σα2.(18)\begin{aligned} \mathbb{E}_t\|\widehat{g}_t^\alpha\|^2 &= \mathbb{E}_t\|u_t + b_t + \xi_t^\alpha\|^2 \\ &= \|u_t + b_t\|^2 + \mathbb{E}_t\|\xi_t^\alpha\|^2 \\ &\le\|u_t + b_t\|^2 + \sigma_\alpha^2. \tag*{(18)} \end{aligned}

Taking conditional expectation in Eq. (16) yields

Et[F(θt+1)]≤F(θt)−η⟨ut,ut+bt⟩+Lη22(∥ut+bt∥2+σα2).(19)\mathbb{E}_t[F(\theta_{t+1})] \le F(\theta_t) - \eta\langle u_t, u_t + b_t\rangle+ \frac{L\eta^2}{2}\left(\|u_t + b_t\|^2 + \sigma_\alpha^2\right). \tag*{(19)}

Since η≤1/L\eta\le1/L, we have Lη2≤ηL\eta^2 \le\eta. Therefore,

−η⟨ut,ut+bt⟩+Lη22∥ut+bt∥2≤−η⟨ut,ut+bt⟩+η2∥ut+bt∥2=−η2∥ut∥2+η2∥bt∥2.(20)-\eta\langle u_t,u_t+b_t\rangle+\frac{L\eta^2}{2}\lVert u_t+b_t\rVert^2 \le-\eta\langle u_t,u_t+b_t\rangle+\frac{\eta}{2}\lVert u_t+b_t\rVert^2 = -\frac{\eta}{2}\lVert u_t\rVert^2+\frac{\eta}{2}\lVert b_t\rVert^2. \tag*{(20)}

Combining 19 and 20 establishes

Et[F(θt+1)]≤F(θt)−η2∥ut∥2+η2∥bt∥2+Lη22σα2.(21)\mathbb{E}_t[F(\theta_{t+1})]\le F(\theta_t)-\frac{\eta}{2}\lVert u_t\rVert^2+\frac{\eta}{2}\lVert b_t\rVert^2+\frac{L\eta^2}{2}\sigma_\alpha^2. \tag*{(21)}

Step 2: Telescoping over the training trajectory.

Taking total expectation of Eq. (21), rearranging, and summing over t=1,…,Tt=1,\ldots,T gives

η2∑t=1TE∥ut∥2≤∑t=1T(EF(θt)−EF(θt+1))+η2∑t=1TE∥bt∥2+Lη2T2σα2.(22)\frac{\eta}{2}\sum_{t=1}^{T}\mathbb{E}\lVert u_t\rVert^2 \le\sum_{t=1}^{T}\left(\mathbb{E}F(\theta_t)-\mathbb{E}F(\theta_{t+1})\right) +\frac{\eta}{2}\sum_{t=1}^{T}\mathbb{E}\lVert b_t\rVert^2+\frac{L\eta^2T}{2}\sigma_\alpha^2. \tag*{(22)}

The first sum telescopes:

∑t=1T(EF(θt)−EF(θt+1))=F(θ1)−EF(θT+1).(23)\sum_{t=1}^{T}\left(\mathbb{E}F(\theta_t)-\mathbb{E}F(\theta_{t+1})\right)=F(\theta_1)-\mathbb{E}F(\theta_{T+1}). \tag*{(23)}

Using F(θT+1)≥FinfF(\theta_{T+1})\ge F_{\mathrm{inf}}, we obtain

η2∑t=1TE∥∇F(θt)∥2≤ΔF+η2∑t=1TE∥bα(θt)∥2+Lη2T2σα2.(24)\frac{\eta}{2}\sum_{t=1}^{T}\mathbb{E}\lVert\nabla F(\theta_t)\rVert^2 \le\Delta_F+\frac{\eta}{2}\sum_{t=1}^{T}\mathbb{E}\lVert b_\alpha(\theta_t)\rVert^2+\frac{L\eta^2T}{2}\sigma_\alpha^2. \tag*{(24)}

Dividing both sides by ηT/2\eta T/2 yields

1T∑t=1TE∥∇F(θt)∥2≤2ΔFηT+Lησα2+1T∑t=1TE∥bα(θt)∥2.(25)\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\lVert\nabla F(\theta_t)\rVert^2 \le\frac{2\Delta_F}{\eta T}+L\eta\sigma_\alpha^2+\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\lVert b_\alpha(\theta_t)\rVert^2. \tag*{(25)}

Finally, applying Eq. (6) at each θt\theta_t gives

1T∑t=1TE∥bα(θt)∥2=Bα,T2.\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\lVert b_\alpha(\theta_t)\rVert^2=\mathcal{B}_{\alpha,T}^2.

Substitution into Eq. (25) proves Lemma 2.

Proof of Corollary 1

Starting from Lemma 2, for any 0<η≤1/L0 < \eta\le1/L,

1T∑t=1TE∥∇F(θt)∥2≤2ΔFηT+Lησα2+Bα,T2.(26)\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\lVert\nabla F(\theta_t)\rVert^2 \le\frac{2\Delta_F}{\eta T}+L\eta\sigma_\alpha^2+\mathcal{B}_{\alpha,T}^2. \tag*{(26)}

Step 1: Balancing the optimization and stochastic terms.

Consider

RT(η)=2ΔFηT+Lησα2.R_T(\eta)=\frac{2\Delta_F}{\eta T}+L\eta\sigma_\alpha^2.

For σα2>0\sigma_\alpha^2>0, its unconstrained minimizer satisfies

∂RT(η)∂η=−2ΔFη2T+Lσα2=0.\frac{\partial R_T(\eta)}{\partial\eta}=-\frac{2\Delta_F}{\eta^2T}+L\sigma_\alpha^2=0.

Hence,

η⋆=2ΔFLσα2T.(27)\eta^\star=\sqrt{\frac{2\Delta_F}{L\sigma_\alpha^2T}}. \tag*{(27)}

If η⋆≤1/L\eta^\star\le1/L, substitution yields

RT(η⋆)=2ΔFη⋆T+Lη⋆σα2=22LΔFσα2T.(28)\begin{aligned} R_T(\eta^\star)&=\frac{2\Delta_F}{\eta^\star T}+L\eta^\star\sigma_\alpha^2\\ &=2\sqrt{\frac{2L\Delta_F\sigma_\alpha^2}{T}}. \tag*{(28)} \end{aligned}

To cover the constraint η≤1/L\eta\le1/L, choose

η=min⁡{1L,2ΔFLσα2T}.\eta=\min\left\{\frac{1}{L},\sqrt{\frac{2\Delta_F}{L\sigma_\alpha^2T}}\right\}.

When σα2=0\sigma_\alpha^2=0, we set η=1/L\eta=1/L.

If η⋆>1/L\eta^\star>1/L, then σα2<2LΔF/T\sigma_\alpha^2<2L\Delta_F/T. Using η=1/L\eta=1/L, we obtain

RT(1/L)=2LΔFT+σα2≤2LΔFT+22LΔFσα2T.R_T(1/L)=\frac{2L\Delta_F}{T}+\sigma_\alpha^2\le\frac{2L\Delta_F}{T}+2\sqrt{\frac{2L\Delta_F\sigma_\alpha^2}{T}}.

Therefore, in either case,

RT(η)≤22LΔFσα2T+2LΔFT.(29)R_T(\eta)\le2\sqrt{\frac{2L\Delta_F\sigma_\alpha^2}{T}}+\frac{2L\Delta_F}{T}. \tag*{(29)}

Combining Eqs. Eq. (26) and Eq. (29),

1T∑t=1TE∥∇F(θt)∥2≤22LΔFσα2T+2LΔFT+Bα,T2.(30)\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\lVert\nabla F(\theta_t)\rVert^2\le2\sqrt{\frac{2L\Delta_F\sigma_\alpha^2}{T}}+\frac{2L\Delta_F}{T}+\mathcal{B}_{\alpha,T}^2. \tag*{(30)}

Comparison with Full-Horizon OPD.

Step 2: Converting update count to expected compute.

By Definition 1,

E[Hα]=H0∑k=1Kk−α.(31)\mathbb{E}[H_\alpha] = H_0 \sum_{k=1}^{K} k^{-\alpha}. \tag*{(31)}

With a fixed rollout batch size BB, define

cα=BE[Hα].c_\alpha= B\mathbb{E}[H_\alpha].

For an expected nominal token budget CC, the number of updates is

Tα=⌊Ccα⌋.T_\alpha= \left\lfloor\frac{C}{c_\alpha} \right\rfloor.

For C≥2cαC \ge2c_\alpha,

Tα≥C2cα.T_\alpha\ge\frac{C}{2c_\alpha}.

Consequently,

1Tα≤2cαC,1Tα≤2cαC.\frac{1}{T_\alpha} \le\frac{2c_\alpha}{C}, \qquad\frac{1}{\sqrt{T_\alpha}} \le\sqrt{\frac{2c_\alpha}{C}}.

Substituting T=TαT=T_\alpha into Eq. (30) gives

Eα(C)≤22LΔFσα2Tα+2LΔFTα+Bα,Tα2\mathcal{E}_\alpha(C) \le2\sqrt{\frac{2L\Delta_F\sigma_\alpha^2}{T_\alpha}}+\frac{2L\Delta_F}{T_\alpha}+\mathcal{B}_{\alpha,T_\alpha}^2
≤4LΔFσα2cαC+4LΔFcαC+Bα,Tα2.(32)\le4\sqrt{\frac{L\Delta_F\sigma_\alpha^2c_\alpha}{C}}+\frac{4L\Delta_Fc_\alpha}{C}+\mathcal{B}_{\alpha,T_\alpha}^2. \tag*{(32)}

Finally, using cα=BE[Hα]c_\alpha=B\mathbb{E}[H_\alpha], we obtain

Eα(C)≤4LΔFσα2BE[Hα]C+4LΔFBE[Hα]C+Bα,Tα2.(33)\mathcal{E}_\alpha(C) \le4\sqrt{\frac{L\Delta_F\sigma_\alpha^2B\mathbb{E}[H_\alpha]}{C}}+\frac{4L\Delta_FB\mathbb{E}[H_\alpha]}{C}+\mathcal{B}_{\alpha,T_\alpha}^2. \tag*{(33)}

Eq. 33 provides a finite-budget upper bound, retaining both the O(C−1/2)O(C^{-1/2}) stochastic term and the O(C−1)O(C^{-1}) optimization term. For a compact characterization of the leading-order compute–reweighting trade-off, we suppress the lower-order O(C−1)O(C^{-1}) term and write

Eα(C)≲κσα2E[Hα]C+Bα,Tα2,κ=16LΔFB.(34)\mathcal{E}_\alpha(C) \lesssim\sqrt{\frac{\kappa\sigma_\alpha^2\mathbb{E}[H_\alpha]}{C}}+\mathcal{B}_{\alpha,T_\alpha}^2, \qquad\kappa= 16L\Delta_FB. \tag*{(34)}

Here, ≲\lesssim denotes a leading-order representation rather than an exact finite-budget inequality. κ=16LΔFB\kappa=16L\Delta_FB is independent of α\alpha and CC. This recovers the simplified expression in Corollary 1.

Comparison with Full-Horizon OPD

We further characterize when stochastic horizons yield a tighter compute-normalized stationarity bound than standard full-horizon OPD. Recall that the complete finite-budget bound in Eq. (33) is

Uα(C):=4LΔFBσα2E[Hα]C+4LΔFBE[Hα]C+Bα,Tα2.(35)U_\alpha(C) := 4\sqrt{\frac{L\Delta_F B\sigma_\alpha^2\mathbb{E}[H_\alpha]}{C}}+\frac{4L\Delta_F B\mathbb{E}[H_\alpha]}{C}+\mathcal{B}^2_{\alpha,T_\alpha}. \tag*{(35)}

To ensure that the finite-budget bound applies to both SH-OPD and full-horizon OPD, we consider a common nominal compute budget C≥2BHmax⁡C \ge2BH_{\max}.

At α=0\alpha=0, the horizon distribution concentrates on Hmax⁡H_{\max}, and the reweighting bias vanishes: B0,T02=0\mathcal{B}^2_{0,T_0}=0. Consequently,

U0(C)=4LΔFBσ02Hmax⁡C+4LΔFBHmax⁡C.(36)U_0(C)=4\sqrt{\frac{L\Delta_F B\sigma_0^2H_{\max}}{C}}+\frac{4L\Delta_F BH_{\max}}{C}. \tag*{(36)}

For any α>0\alpha>0, comparing Uα(C)U_\alpha(C) with U0(C)U_0(C) shows that Uα(C)<U0(C)U_\alpha(C)<U_0(C) holds if and only if

Bα,Tα2<4LΔFBC(σ02Hmax⁡−σα2E[Hα])+4LΔFBC(Hmax⁡−E[Hα]).(37)\mathcal{B}^2_{\alpha,T_\alpha}<4\sqrt{\frac{L\Delta_F B}{C}}\left(\sqrt{\sigma_0^2H_{\max}}-\sqrt{\sigma_\alpha^2\mathbb{E}[H_\alpha]}\right)+\frac{4L\Delta_F B}{C}\left(H_{\max}-\mathbb{E}[H_\alpha]\right). \tag*{(37)}

This condition makes explicit the compute–reweighting trade-off: the reduction in compute-adjusted stochastic optimization error must outweigh the systematic bias introduced by horizon reweighting. In particular, shorter expected horizons do not necessarily yield a tighter bound, as the gradient variance and reweighting bias may also vary with α\alpha. The comparison concerns theoretical upper bounds under a common expected nominal compute budget and does not guarantee improved empirical performance.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, pages 21246–21263, 2024.arxiv.org/abs/2306.13649
  2. [2]Mislav Balunovic, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.23281
  3. [3]Yi Ding and Ruqi Zhang. Does on-policy distillation really distill? from noisy teacher to self-improvement. arXiv preprint arXiv:2608.31046, 2026.
  4. [4]Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes. In Third Conference on Language Modeling, 2026.arxiv.org/abs/2603.25562
  5. [5]Zixuan Fu, Bingxiang He, Yuxin Zuo, Haohuan Huang, Jinqian Zhang, Ruhang Xiao, Cheng Qian, Qinyu Luo, Huan ang Gao, Yudong Wang, Zhiyuan Liu, Ning Ding, and Chaojun Xiao. Rethinking on-policy distillation of large language models ii: One training example. arXiv preprint arXiv:2609.04172, 2026.arxiv.org/abs/2609.04172
  6. [6]GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzhen Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Guo, Jingsen Wang, Jingzhao Du, Jinzhu Wu, Kedong Wang, Lei Li, Lin Fan, Lucen Zhong, Mingdao Liu, Mingming Zhao, Pengfan Du, Qian Dong, Rui Lu, Shuang-Li, Shulin Cao, Song Liu, Ting Jiang, Xiaodong Chen, Xiaohan Zhang, Xuancheng Huang, Xuezhen Dong, Yabo Xu, Yao Wei, Yifan An, Yilin Niu, Yitong Zhu, Yuanhao Wen, Yukuo Cen, Yushi Bai, Zhongpei Qiao, Zihan Wang, Zikang Wang, Zilin Zhu, Ziqiang Liu, Zixuan Li, Bojie Wang, Bosi Wen, Can Huang, Changpeng Cai, Chao Yu, Chen Li, Chengwei Hu, Chenhui Zhang, Dan Zhang, Daoyan Lin, Dayong Yang, Di Wang, Ding Ai, Erle Zhu, Fangzhou Yi, Feiyu Chen, Guohong Wen, Hailong Sun, Haisha Zhao, Haiyi Hu, Hanchen Zhang, Hanrui Liu, Hanyu Zhang, Hao Peng, Hao Tai, Haobo Zhang, He Liu, Hongwei Wang, Hongxi Yan, Hongyu Ge, Huan Liu, Huanpeng Chu, Jia’ni Zhao, Jiachen Wang, Jiajing Zhao, Jiamin Ren, Jiapeng Wang, Jiaxin Zhang, Jiayi Gui, Jiayue Zhao, Jijie Li, Jing An, Jing Li, Jingwei Yuan, Jinhua Du, Jinxin Liu, Junkai Zhi, Junwen Duan, Kaiyue Zhou, Kangjian Wei, Ke Wang, Keyun Luo, Laiqiang Zhang, Leigang Sha, Liang Xu, Lindong Wu, Lintao Ding, Lu Chen, Minghao Li, Nianyi Lin, Pan Ta, Qiang Zou, Rongjun Song, Ruiqi Yang, Shangqing Tu, Shangtong Yang, Shaoxiang Wu, Shengyan Zhang, Shijie Li, Shuang Li, Shuiyi Fan, Wei Qin, Wei Tian, Weining Zhang, Wenbo Yu, Wenjie Liang, Xiang Kuang, Xiangmeng Cheng, Xiangyang Li, Xiaoquan Yan, Xiaowei Hu, Xiaoying Ling, Xing Fan, Xingye Xia, Xinyuan Zhang, Xinze Zhang, Xirui Pan, Xu Zou, Xunkai Zhang, Yadi Liu, Yandong Wu, Yanfu Li, Yidong Wang, Yifan Zhu, Yijun Tan, Yilin Zhou, Yiming Pan, Ying Zhang, Yinpei Su, Yipeng Geng, Yong Yan, Yonglin Tan, Yuean Bi, Yuhan Shen, Yuhao Yang, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yurong Wu, Yutao Zhang, Yuxi Duan, Yuxuan Zhang, Zezhen Liu, Zhengtao Jiang, Zhenhe Yan, Zheyu Zhang, Zhixiang Wei, Zhuo Chen, Zhuoer Feng, Zijun Yao, Ziwei Chai, Ziyuan Wang, Zuzhou Zhang, Bin Xu, Minlie Huang, Hongning Wang, Ju anzi Li, Yuxiao Dong, and Jie Tang. Glm-5: from vide coding to agentic engineering, 2026. URL https://arxiv.org/abs/2602.15763.
  7. [7]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, pages 32694–32717, 2024.DOI
  8. [8]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianz u Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z.
  9. [9]Bingxiang He, Zekai Qu, Zeyuan Liu, Yinghao Chen, Yuxin Zuo, Cheng Qian, Kaiyan Zhang, Weize Chen, Chaojun Xiao, Ganqu Cui, Ning Ding, and Zhiyuan Liu. Justrl: Scaling a 1.5b llm with a simple rl recipe. arXiv preprint arXiv:2512.16649, 2025.arxiv.org/abs/2512.16649
  10. [10]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=chfJJYC3iL.
  11. [11]Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2603.07079
  12. [12]Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletsky, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13 (9):9, 2024.
  13. [13]Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852, 2023.arxiv.org/abs/2312.14852
  14. [14]Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. In ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning, 2026. URL https://openreview.net/forum?id=aaDDXXDGXB.arxiv.org/abs/2604.13016
  15. [15]Kun Liang, Chenming Tang, Clive Bai, Weijie Liu, Saiyong Yang, and Yunfang Wu. ADWIN: Adaptive windows for horizon-aware on-policy distillation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 10371–10389, 2026.arxiv.org/abs/2605.28396
  16. [16]Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation.DOI
  17. [17]Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level. https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Level-1cf81902c14680b3bee5eb349 2025. Notion Blog.
  18. [18]NVIDIA, ; Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Ghabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Fiček, Alex Gronskyi, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shepherd, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Mირჾ? Misra, Divyanshu Kakwani, Dmitry Belenko, Dominic Manno, Dongyang Fan, Dor Tsur, Duncan Riach, Dustin VanStee, Dylan Lu, Edgar Minasyan, Ehsan Hosseini-Asl, Eileen Long, Elena Lantz, Ellie Evans, Elliott Ning, Elvis Chen, Emanuel Scoullos, Eric Chung, Eric Harper, Eric Pham-Hung, Eric Tramel, Eric Yang, Erik Arkelyan, Evelina Bakhurtina, Ewa Dobrowolska, Fadi Saady, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Frankie Siino, Frederic Lafrance, Fuxiao Liu, Gantavya Bhatt, Garvit Kulshreshtha, George Armstrong, Giulio Lovisotto, Gomathy Venkata Krishnan, Gosia Lazuka, Grace Lam, Greg Heinrich, Greg Pauloski, Grzegorz Chlebus, Guilin Liu, Guo Chen, Guoming Zhang, Guyue Huang, Haifeng Qian, Hanrong Ye, Hao Gao, Hao Zhang, Haoqin Tu, Haran Kumar Shiv Kumar, Harib-hau Hud, Harris Nover, Harrison Saturley-Hall, Harshil Kotamreddy, Hemil Desai, Henry Estela, Hexin Wang, Himanshu Singh, Hui Wang, Hung-Yueh Chiang, Igor Gitman, Ilia Karmanov, Ilya Loshchilov, Ingo Guehring, Itay Neeman, Ivan Moshkov, Jacky Kwok, Jakub Slowikowski, James Shen, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeff Willette, Jenna Diamond, Jenny Chen, Jesse Oliver, Jessica Medina, Jiacheng Xu, Jialin Song, Jian Hu, Jian Tu, Jian Zhang, Jiantao Jiao, Jiaqi Zeng, Jie-Fang Zhang, Jimmy Zhang, Jin Xu, Jingyu Xin, Jinhang Choi, Joey Conway, Joey Guman, John Kamalu, John Liu, John Roberts, Johnny Greco, Jonah Alben, Jonas Yang, Jonathan Cohen, Jonathan Li, Jonathan Raiman, Joseph Jennings, Juhi Mittal, Julia Gusak, Julien Veron Vialard, June Yang, Junkeun Yi, Jupinder Parmar, Ka Shun Shum, Kaiwen Shi, Kajal Jain, Kanishk Singla, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Huang, Katie Lyons, Keith Wyss, Keshav Santhanam, Keval Morabia, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kiran Praveen, Konstantinos Krommydas, Krishna Puvvada, Krzysztof Pawelec, Kushal Shah, Li Ding, Lily Lee, Linda Liu, Linglin Jing, Luis Vega, Maarten Van Segbroeck, Mahan Fathi, Mahdi Nazemi, Maka Dong, Makesh Narsimhan Sreedhar, Maor Ashkenazi, Marc Cuevas, Marcin Chochowski, Marco Pavone, Markus Kliegl, Marta Stepniewska-Dziubinska, Martina Gonzalez Vilas, Martyna Patelka, Matan Rusanovsky, Matthieu Le, Matty Machczynski, Matvei Novikov, Mauricio Ferrato, Meenakshi Sharma, Mehrzad Samadi, Melissa Corpuz, Melissa Saelzer, Meng Xin, Mengru Wang, Mengxi Wu, Meriem Boubdir, Michael Boone, Michael Callahan, Michael Elad, Michael Lightstone, Michal Bien, Michal Zawalski, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Ranzinger, Mike Woj-cikiewicz, Min-Hung Chen, Ming Kang, Mingjie Liu, Mingyuan Ma, Misha Nekorystnov, MJ Mikulski, Mohammad Qazim Bhat, Mohammad Shoeybi, Morgan Purcell, Mostofa Patwary, Nabin Mulepati, Nancy Agarwal, Narimane Hennouni, Nathan Sala, Nave Assaf, Nayeon Lee, Neal Vaidya, Negar Habibi, Netanel Haber, Nick Gupta, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nik Spirin, Niket Agarwal, Nikolai Ludwig, Nima Tajbakhsh, Nithin Sunny, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Omer Ullman Argov, Pablo Ribalta, Padmavathy Subramanian, Parth Mannan, Pavlo Molchanov, Pawel Morkisz, Peter Jin, Petr Lapukhov, Petro Junior Milan, Philip Monk, Pinky Xu, Piotr Januszewski, Piotr Laszkiewicz, Po-Han Huang, Praateek Mahajan, Pradeep Thalasta, Prakash Gurumurthy, Pranav Prashant Thombre, Prasoon Varshney, Pratyusha Maiti, Preeti Chadha, Pritam Biswas, Pritam Gundecha, Przemek Strzelczyk, Puhui Meng, Pulkit Kumar, Rabe e h Karimi Mahabadi, Rachel Oberman, Rajpreet Thethy, Ramani Akkati, Rameshwar Shivbhakta, Ran Zilberstein, Randy Gellhausen, Rasoul Shafipour, Raviraj Joshi, Ray Macalisang, Rayen Tian, Reka Kovacs, Renjie Pi, Resha Shetty, Ria C, Rima Shahbazyan, Rishabh Garg, Rita Fernandes Neves, Ritika Borkar, Ritu Gala, Riya Talwar, Riyad Islam, Robert Clark, Robert Kirby, Roger Waleffe, Rohan Joshi, Rohan Pillai, Rohit Jena, Rohit Watwe, Roi Garnett, Roi Koren, Rongguang Ye, Russell J. Hewett, Ryan Prenger, Ryan Stewart, Ryota Egashira, Sadegh Mahdavi, Sai Nishanth Padala, Salika Dave, Sandhya Rani Narravula, Sandip Bhaskar, Sangkug Lym, Sanjeev Satheesh, Sarah Amirlani, Saran Vikas Murari, Saurabh Mishra, Saurabh Srivastava, Saurav Muralidharan, Sean Narenthiran, Sean Wang, Seokju Cho, Seong Hee Lee, Seonjin Na, Seonmyeong Bak, Seph Mard, Serge Panev, Sergei Kolchenko, Sergio Perez, Seth Henneman, Seth Poulos, Shahar Mor, Shantanu Acharya, Shaokun Zhang, Shaona Ghosh, Sharath Turukevere Sreenivas, Shaun Kotek, Shaya Gharghabi, Sheng-Chieh Lin, Shengliang Xu, Shi Chen, Shihao Wang, Shilpa Amireddy, Shiqing Fan, Shiyang Chen, Shrima Prabhumoye, Shriram Venugopal, Shuang Yu, Shubham Gandhi, Shubham Pachori, Shubham Toshniwal, Shuoyang Ding, Shwetha Krishnamurthy, Simeng Sun, Sivakumar Arayandi Thottakara, Somshubra Majumdar, Soumye Singhal, Sridurga Krithivasan, Sriharsha Niverty, Stephen Ge, Sugam Dipak Devare, Sukrit Rao, Sumeet Kumar Barua, Sungsoo Ha, Surya Nistala, Suseella Panguluri, Sweta Priyadarshi, Syeda Nahida Akter, Talor Abramovich, Tanay Varshney, Tandeep Singh Chandehoke, Teo Ene, Terry Kong, Thanh Do, Thinh Le, Tian Zhang, Tijmen Blankevoort, Tim Romanski, Tiyasa Mitra, Tom Balough, Tom Huang, Tomasz Grzegorzek, Tomer Asida, Tomer Bar Natan, Traian Rebedea, Trenton Starkey, Tugrul Konuk, Tuomas Rintamaki, Twinkle Vashisht, Tyler Kniess, Tyler Poon, Udi Karpas, Ushnish De, Utkarsh Vaidya, Vahid Noroozi, Vanshil Atul Shah, Venkat Srinivasan, Venkata Karri, Venmugil Elango, Vibhor Agrawal, Vibhu Jawa, Victor Cui, Victoria Edmonds, Vijay Korthikanti, Vikas Mehta, Vikranth Jeyakumar, Vinay Rao, Vineeth Kalluru Srinivas, Vitaly Kurin, Vitaly Lavrukhin, Wahidur Rahman, Wanli Jiang, Wasi Uddin Ahmad, Wataru Ishihara, Wei Du, Wei Ping, Wei-Ming Chen, Weiheng Chai, Weijia Chen, Wendy Quan, Wenliang Dai, Wenyan Cong, William Zhang, Wojciech Prazuch, Xiaowei Ren, Xin Yu, Xiwen Yu, Yan Wang, Yang Chen, Yang Yu, Yangyi Chen, Yaniv Galron, Yash Jain, Yashaswi Karnati, Ye Yu, Yejin Choi, Yemi Ibrahim, Yev Meyer, Yi Dong, Yi-Fu Wu, Yian Zhang, Yifan Zhang, Ying Lin, Yiyang Zhou, Yoav Miron, Yogendra Pandey, Yonatan Geifman, Yonggan Fu, Yoshi Suhara, Youngeun Kwon, Yu-Cheng Chou, Yu-Chiang Frank Wang, Yuanguo Kuang, Yuekai Zhang, Yugi Guwawala, Yuki Huang, Yunheng Zou, Yunsheng Liu, Zhenming Wang, Zhiding Yu, Zhiqi Li, Zhiyu Cheng, Zhongbo Zhu, Zhuolin Yang, Zichao Yue, Zihan Liu, Zijie Yan, Zirui Liu, Zoe Faul, and Zuhair Ahmed. Nemotron 3.5 super and lightning: Open, efficient mixture-of-experts hybrid mamba-transformer models for agentic ai, 2026. URL https://arxiv.org/abs/2610.38095.
  19. [19]OpenBMB. Ultradata-sft-2605, 2026. URL https://huggingface.co/datasets/openbmb/UltraData-SFT-2605.
  20. [20]Zhennan Shen, Yanshu Li, Qingyu Yin, Chak Tou Leong, Zhilin Wang, Yanxu Chen, Rongduo Han, Sunbowen Lee, and Yi R. Fung. On the geometry of on-policy distillation. arXiv preprint arXiv:2606.07082, 2026.
  21. [21]Huanxin Sheng, Zhiling Ye, Haonan Wang, Jian Wang, Jinjie Gu, and Jian Kang. 1 arXiv preprint arXiv:2609.24432, 2026.arxiv.org/abs/2609.24432
  22. [22]Huanxin Sheng, Zhiling Ye, Haonan Wang, Jian Wang, Jinjie Gu, and Jian Kang. 1% of tokens can be enough: On gradient estimation in on-policy distillation. arXiv preprint arXiv:2609.24432, 2026.arxiv.org/abs/2609.24432
  23. [23]Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, M. C., Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y. Charles, H. S. Che, Guanduo Chen, Guangyu Chen, Guanzheng Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kexin Chen, Peng Chen, Ruijue Chen, Wentao Chen, Xin Chen, Yang Chen, Yanru Chen, Yifei Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Dazhi Cheng, Yean Cheng, Jialei Cui, Jingbing Cui, Anqi Dai, Jiaqi Deng, Hao Ding, Rui Ding, Shaofeng Ding, Mengfan Dong, Mengnan Dong, Yuhao Dong, Yuxin Dong, Angang Du, Chenzhuang Du, Dikang Du, Jus en Du, Yulun Du, Yu Fan, Jing Feng, Qiulin Feng, Yichen Feng, Kelin Fu, Qiang Fu, Fuxuan Gao, Hongcheng Gao, Jingyue Gao, Tong Gao, Weijia Gao, Shangyi Geng, Jie Gong, Linhu Gong, Shengao Gong, Xiaochен Gong, Qizheng Gu, Yicheng Gu, Shuhao Guan, Haiqing Guo, Shiqi Guo, Xiang Guo, Zhengyan Guo, Beixi Hao, Wenxin Hao, Xiaoru Hao, Dailan He, Haotian He, Lehan He, Qi He, Weiran He, Xinran He, Xinyi He, Yibo He, Yunjia He, Chao Hong, Tiange Hong, Hao Hu, Jiaxi Hu, Ruikun Hu, Weiming Hu, Yangyang Hu, Zhenxing Hu, Liang Hua, Jinbin Huang, Ke Huang, Ruiyuan Huang, Siying Huang, Weixiao Huang, Yan Huang, Zhengjie Huang, Zhiqi Huang, Yulong Hui, Chaobo Jia, Yutong Jiang, Zhejun Jiang, Zuoyou Jiang, Wenyi Jin, Xinyi Jin, Yu Jing, Huanjun Kong, Guokun Lai, Aidi Li, Cheng Li, Chengyuan Li, Cong Li, Fang Li, Guanyu Li, Haoyang Li, Jia Li, Junxiong Li, Lei Li, Letian Li, Lincan Li, Weihong Li, Wentao Li, Xintong Li, Yang Li, Yishen Li, Yiwei Li, Yuxiao Li, Zhaowei Li, Zhaoxi Li, Zheming Li, Zhengxiao Li, Zhiyuan Li, Jiawei Lin, Xiaohan Lin, Yibo Lin, Zichao Lin, Ziyan Lin, Bill Liu, Boxiao Liu, Chuan Liu, Liang Liu, Shaowei Liu, Shudong Liu, Shuran Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yanning Liu, Yibo Liu, Yipeng Liu, Zhengying Liu, Zhiheng Liu, Enzhe Lu, Haoyu Lu, Linqiang Lu, Tingzhan Lu, Zhiyuan Lu, Aotian Luo, G. Luo, Junyu Luo, Yifan Luo, B. Lyu, Wenzhou Lyu, Shaoguang Mao, Yuan Mei, Xin Men, Minqing Ni, Yixuan Niu, Siyuan Pan, Shujun Peng, Zhangyang Qi, Ruoyu Qin, ZeChao Qin, Zeyu Qin, Haiquan Qiu, Jianxin Qiu, Jiez h ong Qiu, Bowen Qu, Yuhao Qu, Zeyu Shang, Youbo Shao, Han Shen, Jincheng Shi, Juanfeng Shi, Lidong Shi, Shengyuan Shi, Wingchun Siu, Pengwei Song, Xiaoxi Song, Jianlin Su, Yunfeng Su, Zhaoc hen Su, Lin Sui, Jingsong Sun, Junyao Sun, Shaoning Sun, Shuzhe Sun, Tongyu Sun, Yujun Sun, Yunpeng Tai, Chuning Tang, Heyi Tang, Sirui Tang, Zecheng Tang, Chaoran Tian, Rongpeng Tian, Yu Tian, Wei Tu, Chensi Wang, Chuang Wang, Chunjie Wang, Dinglu Wang, Feng Wang, Hailong Wang, Haiming Wang, Hao Wang, Hao Wang, Huaqing Wang, Hui Wang, Jiayi Wang, Jinglong Wang, Jinhong Wang, Jiuzheng Wang, Linian Wang, Shaobo Wang, Shenzhi Wang, Shuyi Wang, Si Wang, Siyuan Wang, Tianfu Wang, Wenjue Wang, Xingran Wang, Xinmei Wang, Xinyuan Wang, Xusheng Wang, Yalin Wang, Yangkun Wang, Yao Wang, Yaoyu Wang, Yejie Wang, Yiqin Wang, Yucheng Wang, Yuzhi Wang, Zhaoji Wang, Zhaowei Wang, Zhengtao Wang, Zhenhao Wang, Zhongsheng Wang, Zifan Wang, Chu Wei, Ming Wei, Shouxin Wei, Zichen Wen, Fan Wu, Haoning Wu, Rucong Wu, Wenhao Wu, Xiaoxue Wu, Yingcong Wu, Yongqi Wu, Yuxin Wu, Zijian Wu, Xinglang Xian, Chenxuan Xiang, Yue Xiang, Bocheng Xiao, Chenjun Xiao, Xin Xiao, Jin Xie, Xiaotong Xie, Yifeng Xie, Zhe Xie, Bowei Xing, Yiming Xiong, Baosheng Xu, Boyu Xu, Jiale Xu, Jianfan Xu, Jing Xu, Jin jing Xu, L. H. Xu, Qingtao Xu, Shuyao Xu, Suting Xu, Tiantian Xu, Tianxiang Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ye Xu, Yueni Xu, Ziyao Xu, Haonan Xue, Junjie Yan, Yaoyao Yan, Fan Yang, Guangyao Yang, Hao Yang, Junwei Yang, Ruoyu Yang, Wenjie Yang, Xiaofei Yang, Xinyu Yang, Yi Yang, Yiling Yang, Ying Yang, Yuchen Yang, Zhen Yang, Zhilin Yang, Zian Yang, Zuhao Yang, Haotian Yao, Dan Ye, Haoran Ye, Wenjie Ye, Zhanbo Ye, Bohong Yin, Haoxiang Yin, Xietong Yin, Chengzhen Yu, Haozhen Yu, Longhui Yu, Shengnan Yu, Shuying Yu, Tianxiang Yu, Enming Yuan, Mengjie Yuan, Tongtian Yue, Wei Yue, Yang Yue, Dunyuan Zha, Haobing Zhan, B. H. Zhang, Dehao Zhang, Fei Zhang, Hao Zhang, Haoyuan Zhang, Huanyu Zhang, Jiapei Zhang, Jiaxuan Zhang, Jin Zhang, Kaiyi Zhang, Miaozhen Zhang, Puqi Zhang, Qinglei Zhang, Rong Zhang, Rui Zhang, Shaoshuai Zhang, Shiyi Zhang, Xiaobin Zhang, Xiaoyun Zhang, Y. Zhang, Yangkun Zhang, Ye Zhang, Yichi Zhang, Yikun Zhang, Yizhi Zhang, Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Zijin Zhang, Bin Zhao, Chenguang Zhao, Feifan Zhao, Jinglun Zhao, Jinxiang Zhao, Shuai Zhao, Wenshuo Zhao, Xiangyu Zhao, Xuanle Zhao, Yikai Zhao, Zijia Zhao, Haozhi Zheng, Huabin Zheng, Ruihan Zheng, Shaojie Zheng, Tengyang Zheng, Haofeng Zhong, Lei Zhong, Longguang Zhong, M. Zhou, Qiankang Zhou, Runjie Zhou, Ruozhang Zhou, Xinyu Zhou, Yiqiao Zhou, Zaida Zhou, Jinguo Zhu, Liya Zhu, Xinhao Zhu, Yangjunfeng Zhu, Yuxuan Zhu, Zhen Zhu, Chen Zhuang, Weiyu Zhuang, and Xinxing Zu. Kimi k3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653.
  24. [24]Jiaqi WANG, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. In Third Conference on Language Modeling, 2026.arxiv.org/abs/2604.24005
  25. [25]Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, and Hongxia Yang. Not all disagreement is learnable: Token teachability in on-policy distillation. arXiv preprint arXiv:2605.26844, 2026.arxiv.org/abs/2605.26844
  26. [26]Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, and Yifei Wang. On the position bias of on-policy distillation. arXiv preprint arXiv:2606.22600, 2026.arxiv.org/abs/2606.22600
  27. [27]Haoran Xin, Anhao Zhao, Ying Sun, Jin Li, Xiaoyu Shen, and Hui Xiong. Escaping the KL agreement trap in on-policy distillation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 41418–41431, 2026.arxiv.org/abs/2606.09471
  28. [28]Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026.arxiv.org/abs/2604.14084
  29. [29]Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026.arxiv.org/abs/2602.12125
  30. [30]Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, and Han-Jia Ye. Dense supervision, sparse updates: On the sparsity and geometry of on-policy distillation. arXiv preprint arXiv:2606.13657, 2026.arxiv.org/abs/2606.13657
  31. [31]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, juncai liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale. In Advances in Neural Information Processing Systems, pages 113222–113244, 2025.arxiv.org/abs/2503.14476
  32. [32]Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, and Robert E. Tillman. Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026, pages 25553–25569, July 2026.
  33. [33]Yaocheng Zhang, Jiajun Chai, Yuqian Fu, Songjun Tu, Xiaohan Wang, Wei Lin, Guojun Yin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Are full rollouts necessary for on-policy distillation? arXiv preprint arXiv:2605.31490, 2026.arxiv.org/abs/2605.31490
  34. [34]Zhengxin Zhang, Maosen Tang, Ning Wang, Changhao Li, Jingren Zhou, Bo Dai, Sainyam Galhotra, Alex Townsend, and Claire Cardie. Does on-policy distillation need to complete every rollout? arXiv preprint arXiv:2610.21337, 2026.arxiv.org/abs/2610.21337
  35. [35]Anhao Zhao, Junlong Tong, Yingqi Fan, Ping Nie, Wenjie Li, and Xiaoyu Shen. PowerOPD: Stabilizing on-policy distillation with bounded power transformation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 48414–48427, 2026.arxiv.org/abs/2606.17199
  36. [36]Siyan Zhao, Yonggan Fu, Jindong Jiang, Shih-Yang Liu, Song Bian, Byung-Kwan Lee, Sharath Turuvekere Sreenivas, Wenliang Dai, Hanrong Ye, Aditya Grover, and Pavlo Molchanov. When do we need on-policy distillation? distilling on offline student rollouts is often better. arXiv preprint arXiv:2610.11291, 2026.arxiv.org/abs/2610.11291
  37. [37]Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, and Jingjing Chen. Turnopd: Making on-policy distillation turn-aware for efficient long-horizon agent training. arXiv preprint arXiv:2607.05804, 2026.arxiv.org/abs/2607.05804
  38. [38]Wenhong Zhu, Ruobing Xie, Rui Wang, and Pengfei Liu. Hybrid policy distillation for LLMs. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2604.20244

Paper details

vlnc:W_98mz9rq4
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents