betaFeedback

WorksComputer Science

Proposing Better Rollouts for On-Policy Distillation

Abstract

On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher–student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.

Introduction

On-policy distillation [10, 1, 34] has emerged as an important post-training paradigm for large language models. Unlike standard knowledge distillation [12, 13, 61, 16] or supervised finetuning [44, 4], which train the student on teacher- or expert-generated data, OPD lets the student sample its own trajectories and then distills token-level teacher supervision on those trajectories. This makes the training distribution better match the student’s inference-time behavior, and has led to strong results in summarization, translation, and mathematical reasoning tasks [54, 34, 49]. Recent large-scale models including Qwen3.5, MiMo-V2, and DeepSeek-V4 have therefore adopted on-policy distillation as a core component of their post-training pipelines [46, 51, 6].

The fine-grained, on-policy supervision is what makes OPD powerful, but it also exposes OPD to a potential teacher–student support mismatch. Since trajectories are sampled from the student, the teacher is expected to provide token-level supervision not only on responses it would naturally generate, but also on student trajectories that often have low probability under the teacher. On such low-reachability trajectories, the teacher’s conditional token distribution can become noisy or even misleading. This mismatch can be self-reinforcing across distillation rounds. If a degenerate pattern receives insufficient penalty in one round, it can remain in the student’s high-probability region, be sampled again in later rounds, and receive similarly weak or noisy teacher supervision [53]. Recent studies [31] have observed related failure modes in OPD-trained students, including repetition [8, 10] and entropy collapse [24, 26]. These findings suggest that the effectiveness of OPD depends not only on using on-policy samples, but also on whether the teacher can provide reliable supervision on the particular regions of the student distribution visited during training.

We verify this compounding effect empirically. At selected training steps, we use the teacher to complete a student trajectory and a teacher trajectory from the same prompt.2 Figure 1 shows two trends under OPD. First, the similarity between the two teacher completions drifts downward over training, with the 10th percentile dropping more sharply, indicating that a small but growing tail of student trajectories is rarely reachable under the teacher. Second, this drift also degrades completion quality. The fraction of prompts where the teacher’s completion from a student trajectory receives a lower PRM reward than its completion from its own trajectory grows steadily under both gap thresholds. Both trends are consistent with the mechanism above: once the student places probability on trajectories the teacher would rarely reach itself, subsequent rounds reinforce them, and the teacher–student mismatch persists across iterations.

Figure 1. Teacher–student mismatch compounds under OPD. Upper: 10th-percentile similarity between the two completions drops by 0.24 from its peak, while our method remains stable. Lower: fraction of prompts on which the former scores >0.10> 0.10 (solid) or >0.20> 0.20 (dashed) below the latter in PRM score; the former reaches ∼24%\sim24\% by the end of training, while our method keeps it near zero.

To break this loop, we propose to optimize the rollout policy directly, shifting probability toward trajectories the teacher is more likely to reach while remaining close enough to the current student to maintain the on-policy nature. Concretely, we introduce a reweighting function based on relative teacher support, which upweights trajectories the teacher supports more strongly than the current student. To stay close to on-policy behavior, we constrain this reweighting to a local trust region around the current student. Solving this problem yields a geometric mixture of the student and teacher policies, with a single scalar interpolating between on-policy and teacher-guided sampling. However, computing this mixture online requires running both the teacher and student at every decoding step, which is costly at scale. We therefore train a proposal policy to approximate it, so that rollouts need only a single forward pass. This gives our Mixture Proposal On-Policy Distillation (MPOPD), where the rollout policy is first derived as a geometric mixture of the student and teacher and then approximated by a proposal policy for efficient generation.

Prior efforts to improve OPD have mainly modified the distillation objective [21, 24, 57, 26], or adjusted the rollout policy via heuristic sampling [53, 10]. In contrast, we treat the rollout policy itself as an explicit optimization target, derived from a local trust-region principle and approximated by a learnable proposal policy for scalable generation. Figure 1 shows that our method avoids the compounding mismatch, with 10th-percentile similarity 0.30 higher and PRM score gap fraction 18 points lower than OPD. We further evaluate MPOPD on 16 math, knowledge, and medical reasoning benchmarks. Across these tasks, MPOPD consistently outperforms OPD and recent variants. On math, it improves average accuracy over OPD by +1.46 on Qwen3-1.7B-Base and +6.91 on Qwen3-4B-Base, while reducing generation length by 41% at the 4B scale.

Setup: The Rollout Policy in On-Policy Distillation

Let D\mathcal{D} denote the prompt distribution over input problems xx, and let V\mathcal{V} denote the vocabulary. A response trajectory is a token sequence τ=(y1,…,yj,…,y∣τ∣)\tau= (y_1,\ldots,y_j,\ldots,y_{|\tau|}) with yj∈Vy_j \in\mathcal{V}, and a policy π\pi defines the autoregressive distribution π(τ∣x)=∏j=1∣τ∣π(yj∣x,y<j)\pi(\tau\mid x) = \prod_{j=1}^{|\tau|} \pi(y_j \mid x, y_{<j}). We denote the teacher policy by π∗\pi^{*} and the student policy at the tt-th distillation round by πt\pi_t. At each round tt, OPD samples response trajectories from a rollout policy qtq_t, queries the teacher on the sampled responses, and updates the student against teacher supervision through a token-level objective:

Figure 2. Overview of MPOPD. Left: Standard on-policy distillation suffers from compounding teacher–student mismatch, where student rollouts continuously drift beyond the teacher-reachable region and receive unreliable supervision. Right: MPOPD derives a tilted target as a geometric mixture of the student and teacher, and trains a proposal policy to approximate it.

L(π;qt)=Ex∼D, τ∼qt(⋅∣x)[∑j=1∣τ∣KL(π(⋅∣x,y<j)∥π⋆(⋅∣x,y<j))].(1)\mathcal{L}(\pi;q_t)=\mathbb{E}_{x\sim\mathcal{D},\,\tau\sim q_t(\cdot\mid x)}\left[\sum_{j=1}^{|\tau|}\mathrm{KL}\left(\pi(\cdot\mid x,y_{<j})\mathrel{\|}\pi^\star(\cdot\mid x,y_{<j})\right)\right]. \tag*{(1)}

In standard OPD, the rollout policy is taken to be the current student, qt=πtq_t=\pi_t, so the student learns from its own behavior. While this choice aligns training trajectories with the student’s inference-time behavior, it also exposes the teacher to low-reachability student trajectories, producing the compounding mismatch over training rounds (Figure 1). We therefore treat the rollout policy as a design choice and seek a qtq_t that (i) upweights tokens that are more reachable under the teacher, and (ii) deviates from πt\pi_t only locally, so the modification acts as a correction rather than a replacement.

A natural way to implement (i) and (ii) structurally is to write the rollout policy as a multiplicative reweighting of the student policy. Specifically, for a prefix (x,y<j)(x,y_{<j}) and a candidate next token v∈Vv\in\mathcal{V},

qt(v∣x,y<j)∝πt(v∣x,y<j) h(v,x,y<j),(2)q_t(v\mid x,y_{<j})\propto\pi_t(v\mid x,y_{<j})\,h(v,x,y_{<j}), \tag*{(2)}

where hh is a nonnegative reweighting function. The factor πt\pi_t anchors qtq_t to the current student, while hh redistributes probability mass across candidate tokens at each generation step. A good hh should perform this redistribution so that the rollout shifts toward trajectories better aligned with the teacher’s own behavior. Building on this requirement, we design a local score that captures which tokens should be upweighted, and show that the resulting hh takes an exponential-tilt form, yielding qtq_t as a geometric mixture of πt\pi_t and π⋆\pi^\star (§ 3.1). To use this rollout policy efficiently on the fly, we then approximate it with a learnable proposal policy that samples rollouts in a single forward pass (§ 3.2).

Mixture Proposal for On-Policy Distillation

Reweighting by Relative Teacher Support

The reweighting framework in § 2 requires a metric that determines which tokens should receive more probability. Here, we choose the relative teacher support: for a token vv at prefix (x,y<j)(x,y_{<j}), define

st(v,x,y<j)=log⁡π⋆(v∣x,y<j)−log⁡πt(v∣x,y<j).s_t(v,x,y_{<j})=\log\pi^\star(v\mid x,y_{<j})-\log\pi_t(v\mid x,y_{<j}).

A high sts_t means that the teacher assigns larger probability to vv than the student does. Upweighting such tokens follows two intuitions. First, high teacher probability suggests that the teacher can provide more effective supervision, avoiding noisy supervision on tokens the teacher itself rarely generates. Second, low student probability suggests that the student has not yet fully learned these tokens; if the student already assigns high probability, further emphasizing it provides less benefit. Reweighting by sts_t therefore prioritizes tokens that are both well supported by the teacher and not yet captured by the student.

Having specified the reweighting signal, we next control how strongly it shifts the rollout distribution away from the student. We choose qtq_t by maximizing the expected relative teacher support within a local trust region around the student in KL divergence:

max⁡q(⋅∣x,y<j) Ev∼q(⋅∣x,y<j)[st(v,x,y<j)]s.t.KL⁡(q(⋅∣x,y<j)∥πt(⋅∣x,y<j))≤δ.(3)\max_{q(\cdot\mid x,y_{<j})}\ \mathbb{E}_{v\sim q(\cdot\mid x,y_{<j})}\left[s_t(v,x,y_{<j})\right]\quad\text{s.t.}\quad\operatorname{KL}\left(q(\cdot\mid x,y_{<j})\mathbin\Vert\pi_t(\cdot\mid x,y_{<j})\right)\le\delta. \tag*{(3)}

The objective encourages tokens with larger relative teacher support, while the KL constraint prevents the rollout distribution from moving too far from the current student policy. Solving this constrained problem yields a student-anchored exponential reweighting of πt\pi_t:

Proposition 3.1 (Geometric mixture of the tilted target). For a fixed prefix (x,y<j)(x,y_{<j}), there exists a multiplier η≥0\eta\ge0 such that the solution to Eq. (3) takes the form

qη,t∗(v∣x,y<j)=πt(v∣x,y<j)exp⁡(ηst(v,x,y<j))Z(x,y<j),(4)q_{\eta,t}^{*}(v \mid x,y_{<j})=\frac{\pi_t(v \mid x,y_{<j})\exp\left(\eta s_t(v,x,y_{<j})\right)}{Z(x,y_{<j})}, \tag*{(4)}

where η\eta is the inverse Lagrange multiplier associated with the KL constraint and Z(x,y<j)=∑v′∈Vπt(v′∣x,y<j)exp⁡(ηst(v′,x,y<j))Z(x,y_{<j})=\sum_{v'\in\mathcal{V}}\pi_t(v' \mid x,y_{<j})\exp\left(\eta s_t(v',x,y_{<j})\right). Equivalently, substituting the definition of sts_t gives

qη,t∗(v∣x,y<j)=πt(v∣x,y<j)1−ηπ∗(v∣x,y<j)η∑v′∈Vπt(v′∣x,y<j)1−ηπ∗(v′∣x,y<j)η.(5)q_{\eta,t}^{*}(v \mid x,y_{<j})=\frac{\pi_t(v \mid x,y_{<j})^{1-\eta}\pi^{*}(v \mid x,y_{<j})^{\eta}}{\sum_{v'\in\mathcal{V}}\pi_t(v' \mid x,y_{<j})^{1-\eta}\pi^{*}(v' \mid x,y_{<j})^{\eta}}. \tag*{(5)}

The proof is deferred to Appendix § 9.§ 1. The solution is a geometric mixture of the student and teacher policy, governed by a prefix-dependent scalar η\eta. In this case, the reweighting function in Eq. (2) becomes an exponential tilt, h(v,x,y<j)=exp⁡(ηst(v,x,y<j))h(v,x,y_{<j})=\exp(\eta s_t(v,x,y_{<j})). Here, sts_t determines which tokens are upweighted, while η\eta controls the strength of this reweighting. A further discussion, including a comparison with arithmetic mixture, is deferred to Appendix § 9.3. In practice, we can restrict η∈[0,1]\eta\in[0,1] and treat it as a tunable parameter, which yields a continuous interpolation between the two endpoints: η=0\eta=0 recovers πt\pi_t, while η=1\eta=1 recovers π∗\pi^{*}. Intermediate values balance student anchoring and teacher guidance, with larger η\eta placing more mass on tokens with higher relative teacher support.

The improvement in expected relative teacher support from tilting depends on the spread of sts_t across the vocabulary. To make this precise, for a fixed prefix (x,y<j)(x,y_{<j}), let Sj(q)=Ev∼q[st(v,x,y<j)]S_j(q)=\mathbb{E}_{v\sim q}[s_t(v,x,y_{<j})] denote the expected relative teacher support under a distribution qq, and let σj(x,y<j)=Var⁡v∼πt[st(v,x,y<j)]\sigma_j(x,y_{<j})=\sqrt{\operatorname{Var}_{v\sim\pi_t}[s_t(v,x,y_{<j})]} denote its standard deviation under the student. The following corollary quantifies this improvement under the trust-region constraint.

Corollary 3.1 (Local improvement from tilting). For a fixed prefix (x,y<j)(x,y_{<j}), consider the local trust-region problem in Eq. (3). For sufficiently small δ\delta,

Sj(qη∗)−Sj(πt)=σj(x,y<j)2δ+o(δ).(6)S_j(q_{\eta}^{*})-S_j(\pi_t)=\sigma_j(x,y_{<j})\sqrt{2\delta}+o(\sqrt{\delta}). \tag*{(6)}

The proof is deferred to Appendix § 9.§ 2. The improvement is governed by two quantities. The local standard deviation σj\sigma_j formalizes the spread above: when σj\sigma_j is small the student already agrees with the teacher up to a constant offset and tilting has little to redistribute, while a larger σj\sigma_j indicates that high- and low-score tokens coexist in the student policy and tilting can reallocate mass between them. The trust-region radius δ\delta controls the allowed deviation from the student, with δ\sqrt{\delta} scaling reflecting diminishing returns from larger local deviations. Appendix § 9.2 further shows that, under a structured teacher noise model, tilted rollouts reduce exposure to noisy teacher supervision provided a covariance condition holds.

Learning a Proposal Rollout Policy

The tilted rollout policy in Eq. (5) is expensive to sample from directly: each decoding step would require running both the student and teacher, combining their logits, and sampling from the resulting mixture. This online construction requires synchronizing two forward passes throughout generation, making it difficult to implement efficiently at scale.

To avoid this online cost, we introduce a proposal policy q^t\hat{q}_t that approximates the tilted target. At round tt, trajectories are sampled from q^t\hat{q}_t, while the student update is still defined against πt\pi_t and π∗\pi^{*}. The student is updated by

πt+1=arg⁡min⁡πEx∼D, τ∼q^t(⋅∣x)[∑j=1∣τ∣wt,j−1(τ)KL⁡(π(⋅∣x,y<j)∥π∗(⋅∣x,y<j))].(7)\pi_{t+1}=\arg\min_{\pi}\mathbb{E}_{x\sim\mathcal{D},\,\tau\sim\hat{q}_t(\cdot\mid x)}\left[\sum_{j=1}^{|\tau|}w_{t,j-1}(\tau)\operatorname{KL}\left(\pi(\cdot\mid x,y_{<j})\mathbin\Vert\pi^{*}(\cdot\mid x,y_{<j})\right)\right]. \tag*{(7)}

Algorithm 1 Mixture Proposal for On-Policy Distillation

Require: initial student π0\pi_0, teacher π⋆\pi^\star, training prompt distribution D\mathcal{D}, rounds RR, tilt strength η\eta.

  1. Initialize proposal policy q^0←π0\hat{q}_0 \leftarrow\pi_0

  2. for t=0,1,…,R−1t = 0, 1, \ldots, R - 1 do

  3. Sample a batch of prompts from D\mathcal{D}

  4. Generate trajectories τ∼q^t(⋅∣x)\tau\sim\hat{q}_t(\cdot\mid x) for each prompt xx

  5. Update the student πt+1\pi_{t+1} by the distillation objective with importance weights wt,j(τ)w_{t,j}(\tau) in Eq. (8):

πt+1=arg⁡min⁡π Ex∼D, τ∼q^t(⋅∣x)[∑j=1∣τ∣wt,j−1(τ)KL⁡(π(⋅∣x,y<j)∥π⋆(⋅∣x,y<j))]\pi_{t+1} = \underset{\pi}{\arg\min}\ \mathbb{E}_{x\sim\mathcal{D},\ \tau\sim\hat{q}_t(\cdot\mid x)}\left[\sum_{j=1}^{|\tau|} w_{t,j-1}(\tau)\operatorname{KL}\left(\pi(\cdot\mid x,y_{<j})\mathbin\Vert\pi^\star(\cdot\mid x,y_{<j})\right)\right]
  1. Define the tilted target qη,t+1⋆q^\star_{\eta,t+1} from πt+1\pi_{t+1} and π⋆\pi^\star using Eq. (9)

  2. Update q^t+1\hat{q}_{t+1} toward target qη,t+1⋆q^\star_{\eta,t+1} based on

q^t+1=arg⁡min⁡q^ Ex∼D, τ∼q^t(⋅∣x)[∑j=1∣τ∣KL⁡(q^(⋅∣x,y<j)∥qη,t+1⋆(⋅∣x,y<j))]\hat{q}_{t+1} = \underset{\hat{q}}{\arg\min}\ \mathbb{E}_{x\sim\mathcal{D},\ \tau\sim\hat{q}_t(\cdot\mid x)}\left[\sum_{j=1}^{|\tau|}\operatorname{KL}\left(\hat{q}(\cdot\mid x,y_{<j})\mathbin\Vert q^\star_{\eta,t+1}(\cdot\mid x,y_{<j})\right)\right]
  1. end for

  2. return final student πR\pi_R

where wt,j(τ)w_{t,j}(\tau) corrects for the gap between the proposal policy q^t\hat{q}_t and the student πt\pi_t. Following prior work [10, 29], we use the clipped one-step ratio

wt,j(τ)=clip⁡(πt(v∣x,y<j)q^t(v∣x,y<j),1/C,C)(8)w_{t,j}(\tau)=\operatorname{clip}\left(\frac{\pi_t(v\mid x,y_{<j})}{\hat{q}_t(v\mid x,y_{<j})},1/C,C\right) \tag*{(8)}

with wt,0(τ)=1w_{t,0}(\tau)=1 and clipping threshold C>1C>1. To learn the proposal policy for the next round, we form the tilted target at the updated student,

qη,t+1⋆(v∣x,y<j)=πt+1(v∣x,y<j)1−ηπ⋆(v∣x,y<j)η∑v′∈Vπt+1(v′∣x,y<j)1−ηπ⋆(v′∣x,y<j)η,(9)q^\star_{\eta,t+1}(v\mid x,y_{<j})=\frac{\pi_{t+1}(v\mid x,y_{<j})^{1-\eta}\pi^\star(v\mid x,y_{<j})^\eta}{\sum_{v'\in\mathcal{V}}\pi_{t+1}(v'\mid x,y_{<j})^{1-\eta}\pi^\star(v'\mid x,y_{<j})^\eta}, \tag*{(9)}

which serves as the target for updating q^t+1\hat{q}_{t+1}:

q^t+1=arg⁡min⁡q^ Ex∼D, τ∼q^t(⋅∣x)[∑j=1∣τ∣KL⁡(q^(⋅∣x,y<j)∥qη,t+1⋆(⋅∣x,y<j))].(10)\hat{q}_{t+1}=\underset{\hat{q}}{\arg\min}\ \mathbb{E}_{x\sim\mathcal{D},\ \tau\sim\hat{q}_t(\cdot\mid x)}\left[\sum_{j=1}^{|\tau|}\operatorname{KL}\left(\hat{q}(\cdot\mid x,y_{<j})\mathbin\Vert q^\star_{\eta,t+1}(\cdot\mid x,y_{<j})\right)\right]. \tag*{(10)}

The procedure alternates between two updates: the student update uses trajectories sampled from the proposal q^t\hat{q}_t, while the proposal update moves q^t\hat{q}_t toward the next tilted target qη,t+1⋆q^\star_{\eta,t+1}. As a result, the teacher is queried only during training to define the learning target, while rollout generation uses only q^t\hat{q}_t. The full procedure (Algorithm 1) provides an efficient on-the-fly approximation to the geometric mixture rollout while keeping student updates aligned with the OPD objective. Appendix § 8.2.§ 2 evaluates the quality of this approximation. Further discussion is in Appendix § 8.3.

Experiments

Experimental Setup

Models and datasets. We evaluate two student scales in the Qwen3 model family [55], Qwen3-1.7B-Base and Qwen3-4B-Base, distilled from Qwen3-4B-Instruct-2507 as the teacher. For the main math reasoning experiments, we train on DAPO-Math-17K [59] and evaluate on AIME-24, AIME-25, AIME-26, AMC-23, HMMT 2025 (Feb & Nov) [3], Minerva Math [30], and OlympiadBench [14]. We report avg@32 for all benchmarks except OlympiadBench, where we report avg@8 due to its large size. For additional evaluation, we distill Qwen3-4B-Base on two domains: knowledge QA and medical reasoning. In knowledge QA, the model is trained on ARPO-10k [7] and evaluated on

GPQA-Diamond [38], ARC-Challenge [5], and MMLU-Pro [47], with avg@8 reported for all three benchmarks. In medical reasoning, we use M3K [19] for training and evaluate on MedBench [23], MedQA [22], Lancet & NEJM [19], and MedX [68]. Following [19], we report avg@1. Additional details are provided in Appendix § 8.1.§ 1.

Baselines. In addition to standard OPD, we compare against three recent variants: ExOPD [57], which applies an extrapolative supervision relative to the student base model; EOPD [24], which augments OPD with an entropy-aware forward-KL term at high-entropy teacher positions; and VETO [21], which uses a geometric mixture of the student and teacher as the distillation target.

Implementation details. All methods are trained using the verl [43] framework with vLLM [28] as the rollout generation backend. We use one rollout per prompt, rollout temperature 0.8, and batch size 256 for math reasoning and 128 for the additional knowledge QA and medical reasoning experiments. Local-support methods use teacher top-16 support. The proposal policy q^t\hat{q}_t is a full model initialized from the current student at each round. All experiments are conducted on GH200 GPUs. Additional implementation details are provided in Appendix § 8.1.§ 2.

Main Results

Main results on math reasoning. Table 1 reports accuracy and average generation length across eight math benchmarks at two student scales. On Qwen3-4B-Base, our method achieves the best accuracy on all eight benchmarks, improving over the strongest baseline by +4.31 on average. On Qwen3-1.7B-Base, our method is best on 7 out of 8 benchmarks, with an average gain of +0.82. Appendix § 8.2.§ 3 further scales to a larger teacher and student. Overall, these results show that optimizing the rollout policy itself can further improve OPD beyond modifying the distillation objective or target distribution. The comparison with VETO is particularly informative: VETO also uses a geometric mixture of the student and teacher, but applies it as the distillation target rather than the rollout policy. The gap between VETO and our method thus highlights the benefit of reweighting the rollout distribution rather than the distillation target.

BenchmarkBaseOPDExOPDEOPDVETOMPOPDImp.
Qwen3-1.7B-Base
AMC2313.05 / 6.96k39.22 / 16.24k40.00 / 16.37k41.02 / 16.16k39.14 / 10.30k41.09 / 9.78k+0.07 / ↓39.5%
AIME241.67 / 8.00k11.46 / 16.38k12.92 / 16.38k11.56 / 16.35k10.62 / 14.90k12.50 / 14.65k−0.42 / ↓10.6%
AIME251.15 / 7.66k7.19 / 16.36k7.29 / 16.38k8.65 / 16.29k7.40 / 15.06k8.75 / 14.79k+0.10 / ↓9.2%
AIME260.73 / 7.73k7.71 / 16.33k7.71 / 16.37k7.81 / 16.31k7.92 / 15.00k9.06 / 14.85k+1.14 / ↓1.0%
H25F0.21 / 7.82k3.12 / 16.32k2.92 / 16.37k3.12 / 16.32k2.81 / 15.50k3.65 / 15.48k+0.53 / ↓5.2%
H25N0.63 / 6.78k1.46 / 16.34k1.35 / 16.37k1.25 / 16.34k1.35 / 15.88k2.29 / 15.74k+0.83 / ↓3.7%
Minerva8.96 / 6.90k24.35 / 15.81k25.57 / 15.14k25.88 / 15.74k26.86 / 8.57k28.25 / 7.02k+1.39 / ↓18.0%
Olymp.12.54 / 6.65k33.85 / 16.24k33.72 / 16.36k34.20 / 16.20k32.94 / 11.38k34.51 / 10.77k+0.31 / ↓33.5%
Avg.4.87 / 7.31k16.05 / 16.25k16.43 / 16.22k16.69 / 16.21k16.13 / 13.32k17.51 / 12.89k+0.82 / ↓20.5%
Qwen3-4B-Base
AMC2313.44 / 1.87k56.33 / 16.26k56.80 / 16.34k61.25 / 16.32k63.12 / 6.51k69.61 / 6.08k+6.49 / ↓6.6%
AIME242.50 / 1.72k23.02 / 16.37k22.50 / 16.34k22.19 / 16.31k23.85 / 12.47k28.65 / 12.32k+4.80 / ↓1.2%
AIME252.08 / 1.37k23.44 / 16.37k22.29 / 16.38k23.44 / 16.30k24.69 / 11.63k28.96 / 11.42k+4.27 / ↓1.8%
AIME260.83 / 1.77k19.69 / 16.33k21.15 / 16.33k20.73 / 16.31k20.83 / 11.27k25.00 / 11.27k+3.85 / ↓31.0%
H25F0.42 / 1.83k13.02 / 16.35k12.19 / 16.34k12.71 / 16.35k11.56 / 13.06k15.83 / 13.09k+2.81 / ↓20.0%
H25N2.50 / 1.92k12.19 / 16.37k12.81 / 16.38k12.19 / 16.35k10.73 / 13.70k15.42 / 13.32k+2.61 / ↓18.7%
Minerva9.88 / 2.26k32.74 / 16.28k33.80 / 16.27k34.79 / 16.27k40.26 / 2.52k42.65 / 2.43k+2.39 / ↓3.4%
Olymp.11.42 / 1.81k47.24 / 16.28k47.39 / 16.33k47.42 / 16.30k53.41 / 7.14k56.82 / 7.13k+3.41 / ↓0.1%
Avg.5.38 / 1.82k28.46 / 16.33k28.62 / 16.34k29.34 / 16.31k31.06 / 9.79k35.37 / 9.63k+4.31 / ↓1.6%

Table 1. Evaluation results on math benchmarks. Each cell reports accuracy (%) and average generation length. Imp. compares MPOPD with the best baseline in each row, showing the accuracy gain in percentage points and the relative length change. Best and second-best accuracies within each model block are bolded and underlined, respectively.

Table 1 also shows that our method reduces generation length. While some baselines often approach the max token limit, our method generates substantially shorter responses while improving accuracy. On Qwen3-4B-Base, our method reduces the average generation length by 41% relative to OPD while improving average accuracy by +6.91. This length gap mainly comes from the baselines producing degenerate responses that fail to terminate, rather than from our method truncating valid reasoning. Appendix § 8.2.§ 1 further confirms this with a finer-grained analysis. Figure 3 provides a representative example: our method produces a concise correct solution in 319 tokens, whereas OPD reaches the 16.4k limit with repeated final-answer patterns and EOPD reaches the same limit with spurious problem-rejection behavior, ultimately answering incorrectly. Both failure modes (benign repetition and active divergence) are consistent with the mechanism framed in § 2, where rollouts drifting beyond the teacher’s reachable region receive unreliable supervision. Together, these results suggest that the tilted proposal policy helps avoid the degenerate patterns observed in standard OPD [8, 10], rather than merely trading accuracy for shorter outputs.

Case study on Minerva Math (id 141)

Figure 3. Case study on Minerva Math (id 141). We show selected snippets from the raw model outputs. OPD is counted as correct but reaches our 16.4k-token generation limit with repeated final-answer patterns. EOPD also reaches this limit and fails with degenerate spurious behavior. Green, amber, and red indicate correct spans, repetition, and erroneous degeneration, respectively.

Generalization beyond math. To validate whether MPOPD generalizes beyond math reasoning, we further evaluate it on 3 knowledge QA and 5 medical reasoning benchmarks. As shown in Table 2, on Qwen3-4B-Base, our method achieves the best accuracy on all tasks, improving the strongest average baseline from 65.34 to 66.54 on knowledge QA and from 58.03 to 59.17 on medical reasoning. The same length-accuracy pattern also persists in these domains: compared with OPD, the average generation length drops from 10.91k to 2.65k on knowledge QA and from 8.00k to 0.88k on medical reasoning, while accuracy improves in both cases. These results indicate that MPOPD generalizes beyond mathematical reasoning to other domains.

Knowledge Question Answering
BenchmarkBaseOPDExOPDEOPDVETOMPOPDImp.
GPQA-D7.32 / 8.22k38.51 / 16.37k40.03 / 16.38k41.60 / 16.35k40.47 / 5.78k42.17 / 6.34k+0.57 / ↓61.3%
ARC-C24.82 / 3.92k88.52 / 8.18k85.38 / 8.19k87.95 / 8.17k91.45 / 0.33k92.37 / 0.32k+0.92 / ↓5.1%
MMLU-P9.30 / 5.58k62.45 / 8.19k63.75 / 8.20k62.82 / 8.15k64.10 / 1.42k65.08 / 1.31k+0.98 / ↓7.2%
Avg.13.81 / 5.91k63.16 / 10.91k63.05 / 10.92k64.12 / 10.89k65.34 / 2.51k66.54 / 2.65k+1.20 / ↑5.7%
Medical Reasoning
BenchmarkBaseOPDExOPDEOPDVETOMPOPDImp.
PubMed44.50 / 3.53k73.00 / 8.00k74.10 / 8.19k72.10 / 8.19k73.20 / 0.42k74.80 / 0.43k+0.70 / ↓94.7%
MedQA55.46 / 1.28k72.58 / 7.99k73.13 / 8.19k71.72 / 8.19k73.45 / 1.02k74.86 / 1.01k+1.41 / ↓0.8%
Lancet58.01 / 1.22k61.41 / 8.00k63.83 / 8.19k60.44 / 8.19k63.59 / 0.96k64.56 / 0.92k+0.73 / ↓88.8%
MedX10.35 / 3.86k16.01 / 8.00k15.73 / 8.19k16.01 / 8.19k14.77 / 1.26k16.29 / 1.20k+0.28 / ↓85.0%
NEJM55.56 / 0.88k62.19 / 8.00k63.35 / 8.19k63.35 / 8.17k63.52 / 0.96k65.34 / 0.84k+1.82 / ↓12.6%
Avg.44.78 / 2.15k57.04 / 8.00k58.03 / 8.19k56.72 / 8.19k57.71 / 0.92k59.17 / 0.88k+1.14 / ↓89.3%

Table 2. Evaluation results on knowledge QA and medical reasoning benchmarks on Qwen3-4B-Base. Each cell reports accuracy (%) and average generation length. Imp. compares MPOPD with the best baseline in each row, showing the accuracy gain in percentage points and the relative length change. Best and second-best accuracies within each section are bolded and underlined, respectively.

Training Efficiency

Since our method trains an additional proposal policy, we compare its training cost with existing OPD variants. Figure 4 reports the wall-clock efficiency and step-time breakdown. Although our method introduces an additional proposal policy update, it maintains a favorable accuracy-time trade-off throughout training, with only about 13% extra wall-clock time on average. Note that VETO is trained for one extra epoch to reach its optimal performance, which accounts for its longer wall-clock time. The step-time breakdown further shows where the extra cost comes from. Our method spends more time on model training due to the additional proposal-policy update, but it partially offsets this cost through faster rollout generation, as it samples fewer degenerate long responses.

Figure 4. Training efficiency comparison. Left: wall-clock training time across methods. Right: step-time breakdown into rollout generation, model training, rollout scoring, and other overhead.

Table 3 further examines whether the improvement is simply due to extra optimization compute. It reports the average performance on math benchmarks along with average per-round rollout generation time, model training time, and peak memory, comparing our method with OPD using one or two student update steps per round. Doubling the number of OPD student updates per round nearly doubles the training time from 75.00s to 150.01s, but only changes average accuracy from 28.46% to 28.59%. In contrast, our method reaches 35.37% average accuracy with 131.12s of training time per round, lower than OPD with two updates. The additional proposal policy adds only modest memory overhead (141.99 vs. ∼\sim124GB) and is partly offset by faster rollout generation (94.68 vs. ∼\sim130s) since MPOPD generates shorter responses. Together, these numbers indicate that the gain comes from improving the rollout distribution rather than merely spending more compute on student updates.

MetricOPD (K=1)OPD (K=2)Ours
Avg. Perf. (%)28.4628.5935.37
Rollout Gen. (s)128.37130.5894.68
Training (s)75.00150.01131.12
Peak Mem. (GB)123.65124.04141.99

Table 3. Comparison with OPD using extra student updates on Qwen3-4B-Base math benchmarks. KK denotes the number of model update steps performed in each training round.

Analysis

Training Diagnostics. Figure 5 provides diagnostics for the mechanism behind the improvements. Panels (a) and (b) first rule out the explanation that our method only shortens generations by suppressing reasoning. Our method obtains a higher training reward than OPD, while reducing the fraction of responses that reach the maximum length by 45 points. Thus, the shorter outputs mainly come from fewer degraded responses hitting the maximum length, rather than from suppressing useful reasoning.

Figure 5. Training diagnostics. We compare standard OPD and our tilted rollout proposal over training. (a) Training reward. (b) Fraction of responses clipped by the maximum generation length. (c) Average teacher log-probability on generated tokens. (d) Student–teacher entropy gap. (e) Fraction of token-level loss from low-confidence teacher positions.

Panels (c)–(e) further show that MPOPD changes the trajectory distribution used for student supervision. The generated trajectories receive higher teacher log-probability (+0.11), indicating that they remain more likely under the teacher policy. Following [31], we examine the student–teacher entropy gap, which shrinks by 0.04 under our method, indicating that the student’s predictive confidence is more aligned with the teacher’s. In (e), the fraction of loss from low-confidence teacher positions drops by 17 points, indicating that student updates are increasingly driven by confident teacher supervision. Together with Figure 1, these results clarify the failure mode of OPD: as rollouts drift away from trajectories likely under the teacher, training places more loss on low-confidence teacher tokens. The tilted proposal reduces this pattern, making student updates less dominated by such degraded rollouts.

Impact of the Tilt Strength η\eta. The tilt strength η\eta is a hyperparameter that controls how strongly the rollout policy deviates from the current student toward the teacher. A small η\eta keeps the rollout close to the student, while a large η\eta weakens on-policy training. As shown in Table 4, average accuracy varies within about 0.76 across the range, peaking at η=0.20\eta= 0.20, indicating that MPOPD is not highly sensitive to this hyperparameter. Appendix § 8.2.§ 4 further varies the teacher support size.

η\etaAIME26H25NMinervaAvg.
0.1023.3315.1042.3626.93
0.1524.1715.4242.6527.41
0.2025.0015.4242.6527.69
0.3024.6915.3142.4327.48

Table 4. Effect of the coefficient η\eta on selected Qwen3-4B-Base math benchmarks. Scores are reported as accuracy (%).

Related Work

Reasoning in Large Language Models. Recent progress on LLM reasoning [35, 60, 63] has been driven by two closely related post-training paradigms. Reinforcement learning with verifiable rewards optimizes the policy against sparse outcome-level feedback, and has emerged as the dominant recipe behind recent reasoning-focused models [42, 59, 64, 39, 13, 45], yielding strong gains on mathematical, coding, and agentic reasoning [7, 67, 52, 65, 66]. Distillation instead supervises the student with dense token-level signals from a stronger teacher [17, 41, 25], and has become a core component of reasoning-oriented post-training pipelines [51, 6, 46], often paired with distilled reasoning corpora [12, 61, 33, 2] or used to transfer reasoning across scales and modalities [4, 37].

On-Policy Distillation. On-policy distillation lets the student generate its own trajectories and receive token-level teacher supervision [1, 10, 34], addressing the train–test mismatch of off-policy distillation [27, 11, 48, 32]. Owing to its efficiency in transferring strong capabilities from large teachers into mid-scale students, OPD has been explored along several directions, including cross-family distillation [36], multi-teacher guidance aggregation [9, 50], black-box distillation [58], and self-distillation schemes that internalize the model’s in-context capabilities into its parameters [56, 15, 62, 20]. Despite this progress, recent studies have reported failure modes such as repetition, overlong generations, and entropy collapse [8, 24, 26, 31]. Existing remedies mainly modify the supervision target, via alternative divergences [1], extrapolative or entropy-aware objectives [57, 24], and target reformulations [21, 26], or adjust the rollout policy through heuristic teacher–student interleaving [10, 53]. In this work, we instead study the rollout policy itself in OPD.

Conclusion

We study OPD and identify a compounding teacher–student mismatch: student rollouts can drift into low teacher-reachability regions, where noisy supervision reinforces degenerate patterns across rounds. To address this, we introduce MPOPD, which formulates the rollout policy as a local trust-region problem under relative teacher support, yielding a geometric mixture of the student and teacher. We then approximate this mixture with a learnable proposal policy that generates rollouts in a single forward pass. Across 16 math, knowledge, and medical reasoning benchmarks, MPOPD consistently outperforms OPD and recent variants. Diagnostics show that these gains come from shifting rollouts toward higher teacher-reachable trajectories, suggesting that the rollout policy is a critical design choice for improving OPD.

Acknowledgments

This work is supported by National Science Foundation under Award No. IIS-2117902. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government. This research used the DeltaAI advanced computing and data resource, which is supported by the National Science Foundation (award OAC 2320345) and the State of Illinois. DeltaAI is a joint effort of the University of Illinois Urbana-Champaign and its National Center for Supercomputing Applications. This work used the DeltaAI system at the National Center for Supercomputing Applications through allocation CIS250246 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.

Appendix

More Experiment Details

Training and Evaluation Details

Training

General setup. Training is conducted on 2 GPUs with tensor-parallel rollout size 2, rollout count n=1n = 1, rollout temperature 0.8, and rollout top-pp 1.0. The maximum prompt length is 2048 tokens and the maximum training response length is 4096 tokens. We use a learning rate of 3×10−63 \times10^{-6} with weight decay 0.1.

Domain-specific settings. For math reasoning and knowledge QA, we use a training batch size of 256 with PPO mini-batch size 128, training on the full split of each dataset. For medical reasoning, we use a training batch size of 128 with PPO mini-batch size 64, training on a 5K subset. A summary is provided in Table 5.

SettingMathKnowledge QAMedical
Batch size256256128
PPO mini-batch12812864
Training datafull splitfull split5K subset
Learning rate3×10−63 \times10^{-6}
Weight decay0.1
Train temperature0.8
Train top-pp1.0
Train max response len.4096

Table 5. Training configuration across domains. Settings shared across all domains are marked with “–”.

Evaluation

Math reasoning. We use the prompt template

{problem} Please reason step by step, and put your final
answer within \boxed{}.

Completions are sampled with temperature 0.8, top-pp 0.9, and max_tokens= 16384. We draw n=32n = 32 completions per problem, except on OlympiadBench where n=8n = 8 due to its larger size. We report accuracy and average generated length.

Knowledge QA. For zero-shot MCQA on GPQA-Diamond and ARC-Challenge, we use the prompt template

Please solve the following multiple-choice question. Think step by
step. End your response with a single final answer line in one of
these exact formats: Final answer: C {"answer": "C"}. Use only
the choice letter, and do not add any text after the final answer
line. Question: {question} Options: {options}

For MMLU-Pro, we use subject-specific few-shot exemplars from the validation split and prompt the model to end with The answer is (X). For ARC-Challenge and MMLU-Pro, we randomly sample 500 questions from each test set for evaluation, while GPQA-Diamond is evaluated on its full test set. Completions are sampled with temperature 0.8 and top-pp 0.9, with n=8n = 8 per question. We use max_tokens= 16384 for GPQA-Diamond and 8192 for ARC-Challenge and MMLU-Pro.

Medical reasoning. We use the prompt template

Question: {question} Options: {options} Think step by step.
Please show your choice in the answer field with only the choice
letter, e.g., {"answer": "C"}.

We use deterministic MCQA-style decoding with temperature 0.0, top-pp 1.0, max_tokens=8192, and n=1n = 1. We report accuracy and average generated length.

Implementation Details

Rollouts and teacher queries. At each round tt, training trajectories are sampled from the proposal policy q^t\hat{q}_t (one rollout per prompt). The teacher π∗\pi^* is never used to generate trajectories at training time; it is queried only to obtain token-level reference probabilities, which are used both to construct the student distillation loss and to materialize the tilted target qη,t+1∗q^*_{\eta,t+1} for the proposal update. To keep teacher query costs comparable across methods, both the distillation KL and the construction of qη,t+1∗q^*_{\eta,t+1} are restricted to the teacher top-kk support (k=16k = 16).

Student and proposal updates. Each round consists of one student gradient step on the rollout batch using the clipped importance weights wt,j(τ)w_{t,j}(\tau) in Eq. (8) with clipping threshold C=5C = 5, followed by K=1K = 1 inner gradient steps of the proposal update toward the freshly constructed target qη,t+1∗q^*_{\eta,t+1}. The proposal q^0\hat{q}_0 is initialized from π0\pi_0, and at each subsequent round q^t+1\hat{q}_{t+1} is initialized from the updated student πt+1\pi_{t+1} before its inner updates.

Baseline parity. All baselines share our rollout backend, optimizer, chat template, training data, and evaluation infrastructure with MPOPD; we only vary method-specific objective terms and their coefficients.

Additional Experimental Results

A Closer Look at Accuracy and Generation Length

The main results show that MPOPD improves accuracy while reducing generation length. These aggregate improvements leave open where the reduction in length occurs and how it relates to the accuracy gains. We therefore examine three successive questions, corresponding to panels (a)–(c) of Table 6: whether the gains depend on the decoding budget, whether they are explained by termination behavior, and where the length reduction occurs.

(a) Accuracy at different decoding budgets
MethodAcc@4KAcc@8KAcc@16K
Teacher31.6046.5650.16
OPD21.8925.3228.46
VETO24.5228.2931.06
MPOPD26.8931.4235.37
(b) Termination and answer presence at 16K
MethodTerm. rateAcc. | term.Ans. presenceFirst ans. pos.
OPD0.6066.8659.654.85K
VETO53.2755.1356.174.68K
MPOPD55.0961.2559.184.79K
(c) No-answer termination and correct-response lengths
MethodNo-ans. term.# Tokens to first correct ans.# Tokens after first correct ans.
Teacher0.043.61K0.22K
OPD0.033.87K12.38K
VETO0.043.55K0.17K
MPOPD0.013.75K0.34K

Table 6. Decoding budgets and termination behavior. Results are macro-averaged over eight math benchmarks. Accuracy and rates are percentages; K denotes thousands of tokens. In (a), the best and second-best student accuracies are bolded and underlined, respectively.

Following the main setting of our paper, we investigate distilling Qwen3-4B-Base (student) from Qwen3-4B-Instruct-2507 (teacher) on math reasoning, trained on DAPO-Math-17K with training rollouts capped at 4K tokens. We compare MPOPD with OPD, the standard on-policy baseline it builds on, and VETO, the strongest baseline in this setting (Table 1), which applies the same geometric mixture as the distillation target rather than as the rollout policy. We also include the teacher as a reference for accuracy and correct-response length. All evaluations follow the protocol in Appendix § 8.1.§ 1.

(a) Do the gains depend on the decoding budget? Training rollouts are capped at 4K tokens, whereas the main evaluation allows up to 16K, so part of the gap could reflect how each method extrapolates beyond its training length. We therefore evaluate all methods under matched decoding budgets, where Acc@4K, Acc@8K, and Acc@16K denote accuracy with a maximum generation length of 4K, 8K, and 16K tokens. MPOPD outperforms OPD and VETO at all three budgets, with gains of 5.00 and 2.37 already present at the training-matched 4K budget. The improvement therefore cannot be attributed solely to extrapolation beyond the training length.

(b) Is the gap at 16K explained by termination? Since several baselines frequently reach the 16K limit, we next examine termination behavior at this budget. The termination rate (Term. rate) is the fraction of responses that end before the limit, and Acc. | term. is the accuracy among these terminated responses. Answer presence (Ans. presence) is the fraction of responses containing at least one complete boxed answer, whether or not they terminate, and first-answer position (First ans. pos.) is the average token position of the first complete boxed answer over answer-containing responses, regardless of correctness. MPOPD and VETO have similar termination rates (55.09% and 53.27%), yet MPOPD is more accurate both overall (35.37% vs. 31.06%) and among terminated responses (61.25% vs. 55.13%), so its advantage over VETO is not explained by terminating more often. OPD and MPOPD have similar answer presence and first-answer positions, making it unlikely that OPD’s lower accuracy arises from correct answers appearing later and being truncated. OPD’s higher accuracy conditional on termination is measured on only 0.60% of its responses and is therefore unstable.

(c) Where does the length reduction come from? With termination ruled out as the main explanation, we ask whether the shorter generations of MPOPD reflect premature stopping or the removal of redundant continuation. The no-answer termination rate (No-ans. term.) is the fraction of responses that end before the limit without a parseable final answer, which would rise if the model stopped prematurely. For the two length columns, we split each correct response at the end of its first correct boxed answer and report the average number of tokens before and after this point. Although MPOPD terminates on 55.09% of responses, only 0.01% end without a parseable answer, and its tokens to the first correct answer (3.75K) are comparable to those of OPD (3.87K) and the teacher (3.61K). In contrast, tokens generated after the first correct answer drop from 12.38K for OPD to 0.34K. On every benchmark, at least 94.6% of the correct-response length reduction relative to OPD occurs after the first correct answer, and 95.78% of correct OPD responses repeat the already scored boxed answer. These results indicate that the shorter generations come from removing post-answer repetition rather than from truncating useful reasoning.

Proposal Approximation and Effective KL Radius

Our method uses a learned proposal policy to approximate the tilted target. We examine this approximation after the proposal update at different training steps in the same 4B student/teacher setting on math benchmarks. For readability, we omit prefix and round indices and write π\pi for the student, q∗∝(π∗)1−η(π)ηq^{*} \propto(\pi^{*})^{1-\eta}(\pi)^{\eta} for the tilted target, and q^\hat{q} for the learned proposal. We measure both KL directions between q∗q^{*} and q^\hat{q} on the renormalized teacher top-16 support. We also compute Spearman’s ρ\rho across prefixes between KL⁡(q∗∥π)\operatorname{KL}(q^{*}\Vert\pi) and KL⁡(q^∥π)\operatorname{KL}(\hat{q}\Vert\pi), using both the top-16 support and full-vocabulary. This correlation measures whether the target and proposal induce a similar ordering of displacement from the student across prefixes.

Table 7(a) shows that both approximation KLs remain small across training, and at the final step the full-vocabulary KLs are also below 0.01, close to the top-16 values. The displacement correlations are consistently positive at all measured steps, showing that q^\hat{q} tends to move farther from the student where q∗q^* requests a larger shift, although they do not imply equal displacement magnitudes. Overall, the small KLs and stable rank correlations show that q^\hat{q} closely follows both the tilted target and its relative variation across prefixes, including beyond the truncated top-16 support. MPOPD samples from the learned q^\hat{q} as an efficient proxy, whereas the exact geometric-mixture and trust-region interpretation applies to the ideal target q∗q^*.

(a) Approximation to the tilted target
StepKL⁡(q∗∥q^)\operatorname{KL}(q^{*}\Vert\hat{q})KL⁡(q^∥q∗)\operatorname{KL}(\hat{q}\Vert q^{*})ρ\rho top-16ρ\rho full vocab.
200.0122990.0127040.8800.704
400.0097870.0099050.8620.746
600.0090070.0091220.8620.820
800.0090320.0091110.8880.763
1000.0083690.0084620.8820.789
1200.0089310.0090110.8880.766
1340.0082910.0084110.8940.710
(b) Effective radius of the tilted target
StepMedian70th percentile80th percentile
206.07×10−76.07 \times10^{-7}4.04×10−44.04 \times10^{-4}0.004136
601.21×10−61.21 \times10^{-6}0.0014400.005581
1003.10×10−73.10 \times10^{-7}8.19×10−48.19 \times10^{-4}0.005517
1341.33×10−71.33 \times10^{-7}1.83×10−41.83 \times10^{-4}0.008065

Table 7. Proposal approximation and effective KL radius. (a) Mean approximation KLs on the teacher top-16 support and rank correlations of displacement from the student. (b) Percentiles of KL⁡(q∗∥π)\operatorname{KL}(q^{*}\Vert\pi) on the full vocabulary.

Effective KL radius. For a fixed KL radius, Proposition 3.1 yields a prefix-dependent η\eta. Solving the constrained problem separately at every prefix would add online computation, so our implementation instead treats η\eta as a global tunable mixing coefficient. Because the student and teacher distributions vary across prefixes, a global η\eta induces a non-uniform effective KL radius KL⁡(q∗∥π)\operatorname{KL}(q^* \Vert\pi). To quantify this effect, Table 7(b) reports its median, 70th, and 80th percentiles on the full vocabulary at four training steps. Across training, the distribution remains concentrated near zero, with most prefixes receiving only a small shift and a smaller right tail receiving larger shifts. Thus, although a fixed η\eta induces a non-uniform, prefix-dependent radius, it does not produce a large deviation from the student for most prefixes, preserving student anchoring in practice.

Scaling to a Larger Teacher and Student

To examine whether the improvements extend to larger model scales, we use Qwen3-30B-A3B-Instruct-2507 as the teacher and Qwen3-4B-Base or Qwen3-8B-Base as the student. All methods are trained for two epochs with teacher top-16 support. The evaluation protocol is unchanged.

Figure 6 shows that MPOPD achieves the best average accuracy for both students, improving over OPD by 3.18 and 2.48 points at 4B and 8B, respectively, and over VETO by 2.17 and 1.36 points. These results indicate that MPOPD remains effective with a larger teacher and at a larger student scale.

Bar chart comparing OPD, VETO, and Ours for 4B and 8B students

Figure 6. Distillation with a larger teacher and student. Bars show macro-average accuracy (%) over eight math benchmarks after two training epochs with Qwen3-30B-A3B-Instruct-2507 as the teacher. “Ours” denotes MPOPD.

Impact of the Teacher Support Size

We next examine the teacher top-kk support used for the student distillation loss and the construction of the tilted target. We vary k∈{4,16,32,128}k \in\{4, 16, 32, 128\} in the original 4B student/teacher setting on math reasoning, with k=16k = 16 as the default.

Table 8 shows that reducing kk from 16 to 4 lowers average accuracy by 1.23. Increasing the support beyond 16 yields limited changes: k=16k = 16, 32, and 128 achieve 35.37, 35.33, and 35.66, respectively, a range of 0.33. The gains are not monotonic across support sizes. These results support top-16 as a practical choice in this setting.

kkAIME24AIME25AIME26AMC23H25FH25NMinervaOlymp.Avg.
427.1927.6023.7566.9514.6915.2142.3555.3634.14
1628.6528.9625.0069.6115.8315.4242.6556.8235.37
3230.0028.6524.9069.8414.7914.9042.6856.8835.33
12829.4828.9624.5870.3915.6215.9442.7057.6035.66

Table 8. Impact of the teacher support size. Scores are accuracy (%); Avg. is the macro-average over eight math benchmarks. Best and second-best accuracies are bolded and underlined, respectively.

Discussion

Why use a learned tilted rollout proposal? A possible alternative is to sample from the student πt\pi_t and reweight the resulting trajectories toward the tilted target qη,t∗q^*_{\eta,t}. However, reweighting only changes the contribution of trajectories that have already been sampled; it does not change which prefixes and tokens are visited during rollout generation. If completions with large teacher–student log-ratio have low probability under πt\pi_t, they may rarely appear in student rollouts, and importance weights cannot recover trajectories that were not collected. The learned proposal q^t\hat{q}_t instead changes the rollout distribution directly by generating trajectories from an approximation to the tilted target. The clipped weights in Eq. (7) are used only to keep the student update aligned with the OPD objective when training on proposal-generated rollouts.

Implementation considerations. We initialize q^0←π0\hat{q}_0 \leftarrow\pi_0, and at each subsequent round initialize q^t+1\hat{q}_{t+1} from the updated student πt+1\pi_{t+1} before training it toward the next tilted target qη,t+1∗q^*_{\eta,t+1}. This warm-start keeps the proposal close to the current student at the beginning of each round, so the proposal update primarily needs to absorb the teacher-induced tilt rather than re-learn the student distribution from scratch.

Beyond the fixed-η\eta scheme used in our experiments, it is natural to ask whether η\eta could instead be adapted across rounds, which would correspond to an implicit curriculum over rollout distributions. One conceivable design is to set η\eta so that the local KL between qη,t+1∗q^*_{\eta,t+1} and πt+1\pi_{t+1} matches a target radius, and to grow this radius only when the learned proposal tracks the tilted target stably—plausible diagnostics for such stability include the proposal-training loss, the proposal–student KL, and the clipping rate of the importance weights. Conceptually, this would tie the amount of teacher-guided deviation to the current training state rather than to a fixed schedule. We flag this as a possible extension rather than a component of our method, since our reported results use a fixed η\eta.

A separate direction concerns the parameterization of q^t\hat{q}_t itself. Rather than maintaining a full copy of the student, one could parameterize q^t\hat{q}_t with a lightweight adapter such as LoRA [18] on top of the frozen student backbone, in which case the proposal overhead would reduce to storing and updating a small set of adapter parameters. We expect this to be largely orthogonal to the rollout-policy design studied here, and leave a careful empirical comparison to future work.

Limitations. Our study has several limitations. The learnable proposal policy q^t\hat{q}_t introduces modest training-time overhead, which may motivate lighter parameterizations such as LoRA-based proposals at larger student scales. We also treat the tilt strength η\eta as a fixed scalar; adaptive schedules that tie η\eta to training-state signals are a natural extension we do not pursue here. The clipped importance weights wt,j(τ)w_{t,j}(\tau) also introduce a fixed clipping threshold CC, and a more principled treatment of the proposal–student gap could further tighten the alignment between rollout sampling and the OPD objective. Finally, our empirical study focuses on the Qwen3 family, with main evaluations at the 1.7B and 4B scales across math, knowledge QA, and medical reasoning, and additional math experiments extending to an 8B student in Appendix § 8.2.§ 3. We leave extensions to cross-family teacher–student pairs, still larger scales, and additional modalities to future work.

Declaration of LLM Usage and Broader Impact

Declaration of LLM Usage. This work studies large language model distillation, where LLMs are central experimental subjects throughout the paper: they are used as teacher and student policies, sampled to generate reasoning trajectories, queried for token-level distillation supervision, and evaluated on mathematical, knowledge-intensive, and medical reasoning benchmarks. LLMs were also used as editing tools to improve manuscript clarity, grammar, and presentation, without replacing the authors’ role in method design, analysis, or experimental interpretation.

Broader Impact. This paper introduces MPOPD, a rollout-policy design framework for improving on-policy distillation of reasoning-oriented language models. By guiding student rollouts toward higher-teacher support trajectories while retaining the scalability of standalone generation, our method can improve the effectiveness and budget efficiency of distilling reasoning capabilities into smaller models. This may make capable reasoning systems more accessible in domains such as mathematics, knowledge-intensive question answering, and medical reasoning benchmarks. At the same time, stronger distilled reasoning models may inherit general risks associated with LLM deployment, including over-reliance on automated reasoning, hallucinated or incorrect answers, and potential misuse in settings where generated outputs are treated as authoritative. Our work does not introduce a new high-risk application setting, collect human-subject data, or release a model intended for direct deployment; rather, it proposes a training method for existing reasoning-model distillation pipelines. We therefore view its broader societal risks as largely aligned with those of existing LLM reasoning and distillation systems, and encourage downstream users to apply appropriate verification, monitoring, and domain-specific safeguards before deployment in high-stakes settings.

Missing Proofs

Proof of Proposition 3.1

Proposition 3.1 (Geometric mixture of the tilted target). For a fixed prefix (x,y<j)(x, y_{<j}), there exists a multiplier η≥0\eta\ge0 such that the solution to Eq. (3) takes the form

qη,t∗(v∣x,y<j)=πt(v∣x,y<j)exp⁡(ηst(v,x,y<j))Z(x,y<j),(11)q_{\eta,t}^{*}(v \mid x, y_{<j}) = \frac{\pi_t(v \mid x, y_{<j}) \exp\left(\eta s_t(v, x, y_{<j})\right)}{Z(x, y_{<j})}, \tag*{(11)}

where η\eta is the inverse Lagrange multiplier associated with the KL constraint and Z(x,y<j)=∑v′∈Vπt(v′∣x,y<j)exp⁡(ηst(v′,x,y<j))Z(x, y_{<j}) = \sum_{v' \in\mathcal{V}} \pi_t(v' \mid x, y_{<j}) \exp\left(\eta s_t(v', x, y_{<j})\right). Equivalently, substituting the definition of sts_t gives

qη,t∗(v∣x,y<j)=πt(v∣x,y<j)1−ηπ∗(v∣x,y<j)η∑v′∈Vπt(v′∣x,y<j)1−ηπ∗(v′∣x,y<j)η.(12)q_{\eta,t}^{*}(v \mid x, y_{<j}) = \frac{\pi_t(v \mid x, y_{<j})^{1-\eta}\pi^{*}(v \mid x, y_{<j})^{\eta}}{\sum_{v' \in\mathcal{V}} \pi_t(v' \mid x, y_{<j})^{1-\eta}\pi^{*}(v' \mid x, y_{<j})^{\eta}}. \tag*{(12)}

Proof. Fix a prefix (x,y<j)(x, y_{<j}) and write

p(v)=πt(v∣x,y<j),s(v)=st(v,x,y<j)p(v) = \pi_t(v \mid x, y_{<j}), \qquad s(v) = s_t(v, x, y_{<j})

for brevity. The local trust-region problem in Eq. (3) is

max⁡q∑v∈Vq(v)s(v)s.t.KL⁡(q∥p)≤δ,∑v∈Vq(v)=1.\max_q \sum_{v \in\mathcal{V}} q(v)s(v) \quad\text{s.t.} \quad\operatorname{KL}(q\|p) \le\delta, \qquad\sum_{v \in\mathcal{V}} q(v) = 1.

We consider the nondegenerate case in which the KL constraint is active; the degenerate cases are recovered by η=0\eta= 0 or by a constant score on the student support. The Lagrangian is

L(q,λ,ν)=∑vq(v)s(v)−λ(∑vq(v)log⁡q(v)p(v)−δ)−ν(∑vq(v)−1),\mathcal{L}(q,\lambda,\nu) = \sum_v q(v)s(v) - \lambda\left(\sum_v q(v)\log\frac{q(v)}{p(v)}-\delta\right)-\nu\left(\sum_v q(v)-1\right),

where λ>0\lambda> 0 is the multiplier for the KL constraint and ν\nu enforces normalization. Taking the derivative with respect to q(v)q(v) and setting it to zero gives

s(v)−λ(log⁡q(v)p(v)+1)−ν=0.s(v) - \lambda\left(\log\frac{q(v)}{p(v)} + 1\right) - \nu= 0.

Therefore,

q(v)=p(v)exp⁡(s(v)λ)exp⁡(−1−νλ).q(v) = p(v)\exp\left(\frac{s(v)}{\lambda}\right)\exp\left(-1-\frac{\nu}{\lambda}\right).

The last factor is independent of vv and is absorbed into the normalization constant. Letting η=1/λ\eta= 1/\lambda, we obtain

qη∗(v)=p(v)exp⁡(ηs(v))∑v′∈Vp(v′)exp⁡(ηs(v′)).q_{\eta}^{*}(v) = \frac{p(v)\exp(\eta s(v))}{\sum_{v' \in\mathcal{V}}p(v')\exp(\eta s(v'))}.

Returning to the original notation yields

qη∗(v∣x,y<j)=πt(v∣x,y<j)exp⁡(ηst(v,x,y<j))Zη(x,y<j),q_{\eta}^{*}(v \mid x, y_{<j}) = \frac{\pi_t(v \mid x, y_{<j})\exp\left(\eta s_t(v, x, y_{<j})\right)}{Z_{\eta}(x, y_{<j})},

where

Zη(x,y<j)=∑v′∈Vπt(v′∣x,y<j)exp⁡(ηst(v′,x,y<j)).Z_{\eta}(x, y_{<j}) = \sum_{v' \in\mathcal{V}} \pi_t(v' \mid x, y_{<j})\exp\left(\eta s_t(v', x, y_{<j})\right).

Substituting

st(v,x,y<j)=log⁡π∗(v∣x,y<j)−log⁡πt(v∣x,y<j)s_t(v, x, y_{<j}) = \log\pi^{*}(v \mid x, y_{<j}) - \log\pi_t(v \mid x, y_{<j})

gives

πt(v∣x,y<j)exp⁡(ηst(v,x,y<j))=πt(v∣x,y<j)1−ηπ∗(v∣x,y<j)η.\pi_t(v \mid x, y_{<j})\exp\left(\eta s_t(v, x, y_{<j})\right) = \pi_t(v \mid x, y_{<j})^{1-\eta}\pi^{*}(v \mid x, y_{<j})^{\eta}.

Hence

qη∗(v∣x,y<j)=πt(v∣x,y<j)1−ηπ∗(v∣x,y<j)η∑v′∈Vπt(v′∣x,y<j)1−ηπ∗(v′∣x,y<j)η.q_{\eta}^{*}(v \mid x, y_{<j}) = \frac{\pi_t(v \mid x, y_{<j})^{1-\eta}\pi^{*}(v \mid x, y_{<j})^{\eta}}{\sum_{v' \in\mathcal{V}} \pi_t(v' \mid x, y_{<j})^{1-\eta}\pi^{*}(v' \mid x, y_{<j})^{\eta}}.

When the constraint is active, the multiplier η\eta is chosen so that KL⁡(qη∗∥p)=δ\operatorname{KL}(q_{\eta}^{*}\|p) = \delta.

Proof of Corollary 3.1

Corollary 3.1 (Local improvement from tilting). For a fixed prefix (x,y<j)(x, y_{<j}), consider the local trust-region problem in Eq. (3). For sufficiently small δ\delta,

Sj(qη∗)−Sj(πt)=σj(x,y<j)2δ+o(δ).(13)S_j(q_\eta^*) - S_j(\pi_t) = \sigma_j(x, y_{<j})\sqrt{2\delta} + o(\sqrt{\delta}). \tag*{(13)}

Proof. Fix a prefix (x,y<j)(x, y_{<j}) and write

p(v)=πt(v∣x,y<j),s(v)=st(v,x,y<j)p(v) = \pi_t(v \mid x, y_{<j}), \qquad s(v) = s_t(v, x, y_{<j})

for brevity. Let

μ=Ev∼p[s(v)],σ2=Var⁡v∼p[s(v)].\mu= \mathbb{E}_{v\sim p}[s(v)], \qquad\sigma^2 = \operatorname{Var}_{v\sim p}[s(v)].

By Proposition 3.1, the optimal local tilted distribution can be written as

qη∗(v)=p(v)exp⁡(ηs(v))Z(η),Z(η)=Ev∼p[exp⁡(ηs(v))].q_\eta^*(v) = \frac{p(v)\exp(\eta s(v))}{Z(\eta)}, \qquad Z(\eta) = \mathbb{E}_{v\sim p}[\exp(\eta s(v))].

As δ→0\delta\to0, the corresponding multiplier satisfies η→0\eta\to0. Define the centered cumulant generating function

Λ(η)=log⁡Ev∼p[exp⁡(η(s(v)−μ))].\Lambda(\eta) = \log\mathbb{E}_{v\sim p}[\exp(\eta(s(v)-\mu))].

Then

log⁡Z(η)=ημ+Λ(η),Sj(qη∗)=ddηlog⁡Z(η)=μ+Λ′(η).\log Z(\eta) = \eta\mu+ \Lambda(\eta), \qquad S_j(q_\eta^*) = \frac{d}{d\eta}\log Z(\eta) = \mu+ \Lambda'(\eta).

Since the vocabulary is finite, Λ\Lambda is smooth in a neighborhood of η=0\eta= 0. Therefore,

Λ(η)=σ22η2+O(η3),Λ′(η)=σ2η+O(η2).\Lambda(\eta) = \frac{\sigma^2}{2}\eta^2 + O(\eta^3), \qquad\Lambda'(\eta) = \sigma^2\eta+ O(\eta^2).

It follows that

Sj(qη∗)−Sj(πt)=Sj(qη∗)−μ=σ2η+O(η2).S_j(q_\eta^*) - S_j(\pi_t) = S_j(q_\eta^*) - \mu= \sigma^2\eta+ O(\eta^2).

It remains to relate η\eta to the KL radius δ\delta. For the exponential tilt,

KL⁡(qη∗∥p)=Eqη∗[log⁡qη∗(v)p(v)]=ηSj(qη∗)−log⁡Z(η).\operatorname{KL}(q_\eta^*\|p) = \mathbb{E}_{q_\eta^*}\left[\log\frac{q_\eta^*(v)}{p(v)}\right] = \eta S_j(q_\eta^*) - \log Z(\eta).

Using the identities above,

KL⁡(qη∗∥p)=η(μ+Λ′(η))−(ημ+Λ(η))=ηΛ′(η)−Λ(η).\operatorname{KL}(q_\eta^*\|p) = \eta(\mu+\Lambda'(\eta))-(\eta\mu+\Lambda(\eta)) = \eta\Lambda'(\eta)-\Lambda(\eta).

Substituting the Taylor expansions gives

δ=KL⁡(qη∗∥p)=σ22η2+O(η3).\delta= \operatorname{KL}(q_\eta^*\|p) = \frac{\sigma^2}{2}\eta^2 + O(\eta^3).

Thus, when σ>0\sigma> 0,

η=2δσ+O(δ).\eta= \frac{\sqrt{2\delta}}{\sigma} + O(\delta).

Substituting this relation into the improvement expansion yields

Sj(qη∗)−Sj(πt)=σ2(2δσ+O(δ))+O(δ)=σ2δ+O(δ).S_j(q_\eta^*) - S_j(\pi_t) = \sigma^2\left(\frac{\sqrt{2\delta}}{\sigma}+O(\delta)\right)+O(\delta)=\sigma\sqrt{2\delta}+O(\delta).

Returning to the original notation, σ=σj(x,y<j)\sigma= \sigma_j(x, y_{<j}), we obtain

Sj(qη∗)−Sj(πt)=σj(x,y<j)2δ+o(δ).S_j(q_\eta^*) - S_j(\pi_t) = \sigma_j(x, y_{<j})\sqrt{2\delta} + o(\sqrt{\delta}).

If σ=0\sigma= 0, then the score is constant on the support of πt(⋅∣x,y<j)\pi_t(\cdot\mid x, y_{<j}), so any finite-KL reweighting has the same expected score. The first-order improvement is therefore zero, which is consistent with the stated expansion.

Optimization Analysis under Structured Teacher Target Noise

We analyze a stochastic teacher–student setting. At round tt, the current student and proposal policies may be random through the training history; in the one-step analysis below, we condition on this history and suppress the conditioning notation. A token-level state is denoted by c=(x,j,y<j)c=(x,j,y_{<j}). For any rollout policy qq, let

Mq:=Ex∼D,τ∼q(⋅∣x)[∣τ∣]∈(0,∞).M_q := \mathbb{E}_{x\sim\mathcal{D},\tau\sim q(\cdot\mid x)}[|\tau|] \in(0,\infty).

Define the normalized token-level occupancy distribution dqd^q by

Ec∼dq[f(c)]:=1MqEx∼D,τ∼q(⋅∣x)[∑j=1∣τ∣f(x,j,y<j)].\mathbb{E}_{c\sim d^q}[f(c)] := \frac{1}{M_q}\mathbb{E}_{x\sim\mathcal{D},\tau\sim q(\cdot\mid x)}\left[\sum_{j=1}^{|\tau|}f(x,j,y_{<j})\right].

We introduce an ideal clean teacher target π⋆(⋅∣c)\pi_\star(\cdot\mid c) only for analysis. The teacher target actually used for distillation is a random conditional distribution π~(⋅∣c)\widetilde{\pi}(\cdot\mid c), which models stochastic or uncertain teacher supervision around the clean target.

To relate teacher reachability to supervision reliability, let rT(c)≥0r_T(c)\ge0 denote a reachability-risk score, with larger values assigned to prefixes that are less reachable under the teacher. For example, rT(c)r_T(c) can be chosen as a nondecreasing function of the teacher prefix negative log-likelihood,

mT(c)=−1max⁡{1,j−1}∑k=1j−1log⁡π⋆(yk∣x,y<k),rT(c)=ϕ(mT(c)),m_T(c)=-\frac{1}{\max\{1,j-1\}}\sum_{k=1}^{j-1}\log\pi_\star(y_k\mid x,y_{<k}),\qquad r_T(c)=\phi(m_T(c)),

where ϕ\phi is nondecreasing.

For a state cc, define the clean and observed teacher losses as

ℓc∘(θ)=KL⁡(πθ(⋅∣c)∥π⋆(⋅∣c)),ℓ~c(θ)=KL⁡(πθ(⋅∣c)∥π~(⋅∣c)).\ell_c^\circ(\theta)=\operatorname{KL}\left(\pi_\theta(\cdot\mid c)\mathbin{\|}\pi_\star(\cdot\mid c)\right),\qquad\widetilde{\ell}_c(\theta)=\operatorname{KL}\left(\pi_\theta(\cdot\mid c)\mathbin{\|}\widetilde{\pi}(\cdot\mid c)\right).

The average per-token reachability-risk exposure of a rollout policy qq is

Rt(q)=Ec∼dq[rT(c)].R_t(q)=\mathbb{E}_{c\sim d^q}[r_T(c)].

Assumption F.1 (Reachability-structured teacher target noise). For each token-level state cc, define

ζt(c)=∇θℓ~c(θt)−∇θℓc∘(θt).\zeta_t(c)=\nabla_\theta\widetilde{\ell}_c(\theta_t)-\nabla_\theta\ell_c^\circ(\theta_t).

The perturbation is conditionally unbiased relative to the clean teacher target, and its conditional second moment is controlled by reachability risk:

E[ζt(c)∣c]=0,E[∥ζt(c)∥2∣c]≤λrT(c)\mathbb{E}[\zeta_t(c)\mid c]=0,\qquad\mathbb{E}[\|\zeta_t(c)\|^2\mid c]\le\lambda r_T(c)

for some λ>0\lambda>0.

This is an analysis-only noise model: teacher supervision is unbiased relative to π⋆∘\pi_\star^\circ, but its gradient perturbation is allowed to be larger on lower-reachability states as measured by rT(c)r_T(c).

For the clean target, define the normalized per-token objective Ft,q∘(θ)=Ec∼dq[ℓc∘(θ)]F_{t,q}^\circ(\theta)=\mathbb{E}_{c\sim d^q}[\ell_c^\circ(\theta)], and let

Gt(q)=Ec∼dq[∥∇θℓc∘(θt)∥2].G_t(q)=\mathbb{E}_{c\sim d^q}\left[\|\nabla_\theta\ell_c^\circ(\theta_t)\|^2\right].

Proposition F.1 (One-step descent under reachability-structured teacher target noise). Assume that Ft,q∘F_{t,q}^\circ is LL-smooth in θ\theta for a fixed occupancy dqd^q. Consider one stochastic update using the observed teacher target:

θt+1=θt−α∇ℓ~c(θt),c∼dq.\theta_{t+1}=\theta_t-\alpha\nabla\widetilde{\ell}_c(\theta_t),\qquad c\sim d^q.

Under Assumption F.1,

E[Ft,q∘(θt)−Ft,q∘(θt+1)]≥α∥∇θFt,q∘(θt)∥2−Lα22(Gt(q)+λRt(q)),\mathbb{E}\left[F_{t,q}^\circ(\theta_t)-F_{t,q}^\circ(\theta_{t+1})\right]\ge\alpha\|\nabla_\theta F_{t,q}^\circ(\theta_t)\|^2-\frac{L\alpha^2}{2}\left(G_t(q)+\lambda R_t(q)\right),

where the expectation is over c∼dqc\sim d^q and the stochastic teacher target.

Proof. Let

g~c∘=∇θℓc∘(θt),g~c=∇θℓ~c(θt)=gc∘+ζt(c).\widetilde{g}^{\circ}_{c}=\nabla_{\theta}\ell^{\circ}_{c}(\theta_t),\qquad\widetilde{g}_{c}=\nabla_{\theta}\widetilde{\ell}_{c}(\theta_t)=g^{\circ}_{c}+\zeta_t(c).

By LL-smoothness,

Ft,q∘(θt+1)≤Ft,q∘(θt)−α⟨∇θFt,q∘(θt),g~c⟩+Lα22∥g~c∥2.F^{\circ}_{t,q}(\theta_{t+1})\leq F^{\circ}_{t,q}(\theta_t)-\alpha\left\langle\nabla_{\theta}F^{\circ}_{t,q}(\theta_t),\widetilde{g}_{c}\right\rangle+\frac{L\alpha^2}{2}\left\|\widetilde{g}_{c}\right\|^2.

Taking expectation over c∼dqc\sim d^q and the observed-target randomness, the conditional zero-mean property gives

E[g~c]=Ec∼dq[gc∘]=∇θFt,q∘(θt).\mathbb{E}[\widetilde{g}_{c}]=\mathbb{E}_{c\sim d^q}[g^{\circ}_{c}]=\nabla_{\theta}F^{\circ}_{t,q}(\theta_t).

Therefore the first-order term becomes

E[⟨∇θFt,q∘(θt),g~c⟩]=∥∇θFt,q∘(θt)∥2.\mathbb{E}\left[\left\langle\nabla_{\theta}F^{\circ}_{t,q}(\theta_t),\widetilde{g}_{c}\right\rangle\right]=\left\|\nabla_{\theta}F^{\circ}_{t,q}(\theta_t)\right\|^2.

For the second-order term,

E∥g~c∥2=E∥gc∘+ζt(c)∥2.\mathbb{E}\left\|\widetilde{g}_{c}\right\|^2=\mathbb{E}\left\|g^{\circ}_{c}+\zeta_t(c)\right\|^2.

The cross term vanishes because E[ζt(c)∣c]=0\mathbb{E}[\zeta_t(c)\mid c]=0, and Assumption F.1 implies

E∥g~c∥2≤Ec∼dq∥gc∘∥2+λEc∼dq[rT(c)]=Gt(q)+λRt(q).\mathbb{E}\left\|\widetilde{g}_{c}\right\|^2\leq\mathbb{E}_{c\sim d^q}\left\|g^{\circ}_{c}\right\|^2+\lambda\mathbb{E}_{c\sim d^q}\left[r_T(c)\right]=G_t(q)+\lambda R_t(q).

Substituting these two bounds into the smoothness inequality proves the claim.

Comparison with on-policy sampling. Proposition F.1 assigns each rollout policy qq the proposal-induced one-step lower bound

Bt(q)=α∥∇θFt,q∘(θt)∥2−Lα22(Gt(q)+λRt(q)).B_t(q)=\alpha\left\|\nabla_{\theta}F^{\circ}_{t,q}(\theta_t)\right\|^2-\frac{L\alpha^2}{2}\left(G_t(q)+\lambda R_t(q)\right).

This is not a guarantee on a fixed evaluation objective. It compares the descent bound for the clean objective induced by each rollout occupancy.

To compare the ideal tilted rollout policy qη,t⋆q_{\eta,t}^{\star} with standard on-policy sampling πt\pi_t, define

At=∥∇θFt,qη,t⋆∘(θt)∥2−∥∇θFt,πt∘(θt)∥2,Vt=Gt(qη,t⋆)−Gt(πt),Dt=Rt(πt)−Rt(qη,t⋆).A_t=\left\|\nabla_{\theta}F^{\circ}_{t,q_{\eta,t}^{\star}}(\theta_t)\right\|^2-\left\|\nabla_{\theta}F^{\circ}_{t,\pi_t}(\theta_t)\right\|^2,\qquad V_t=G_t(q_{\eta,t}^{\star})-G_t(\pi_t),\qquad D_t=R_t(\pi_t)-R_t(q_{\eta,t}^{\star}).

Then

Bt(qη,t⋆)−Bt(πt)=αAt−Lα22Vt+Lα2λ2Dt.B_t(q_{\eta,t}^{\star})-B_t(\pi_t)=\alpha A_t-\frac{L\alpha^2}{2}V_t+\frac{L\alpha^2\lambda}{2}D_t.

Therefore, the tilted rollout policy has a no-smaller proposal-induced one-step lower bound than on-policy sampling whenever

At+αLλ2Dt≥αL2Vt.A_t+\frac{\alpha L\lambda}{2}D_t\geq\frac{\alpha L}{2}V_t.

This decomposition separates three effects. The term AtA_t captures the change in clean descent signal under the rollout-induced state occupancy. The term VtV_t captures the change in clean-gradient second moment. The term DtD_t captures the reduction in reachability-risk exposure. Thus tilted sampling improves this proposal-induced lower bound when it increases the clean descent signal and/or reduces reachability-risk exposure without inducing too large a second-moment penalty.

When does tilting reduce reachability-risk exposure? The comparison above treats Dt=Rt(πt)−Rt(qη,t⋆)D_t=R_t(\pi_t)-R_t(q_{\eta,t}^{\star}) as a property of the induced rollout distribution. The geometric tilt does not automatically guarantee Dt>0D_t>0, because it is defined through relative teacher support rather than directly through the reachability-risk score. A local diagnostic can be expressed using a covariance condition.

For a prefix state cc, let

st(v,c)=log⁡π⋆(v∣c)−log⁡πt(v∣c)s_t(v,c)=\log\pi^{\star}(v\mid c)-\log\pi_t(v\mid c)

be the teacher–student log-ratio score used by the tilted rollout policy, and let ρT(v,c)\rho_T(v,c) denote the reachability-risk score of the next prefix obtained by appending token vv. For qη(v∣c)∝πt(v∣c)exp⁡(ηst(v,c))q_{\eta}(v\mid c)\propto\pi_t(v\mid c)\exp(\eta s_t(v,c)),

ddηEv∼qη(⋅∣c)[ρT(v,c)]∣η=0=Cov⁡v∼πt(⋅∣c)(ρT(v,c),st(v,c)).\left.\frac{d}{d\eta}\mathbb{E}_{v\sim q_{\eta}(\cdot\mid c)}[\rho_T(v,c)]\right|_{\eta=0}=\operatorname{Cov}_{v\sim\pi_t(\cdot\mid c)}(\rho_T(v,c),s_t(v,c)).

Therefore, if tokens with larger relative teacher support tend to lead to lower reachability-risk next prefixes, the covariance is negative and a small geometric tilt locally reduces reachability-risk exposure. This condition links the risk-reduction term DtD_t in the optimization bound to the proposed tilted rollout policy.

Geometric and Arithmetic Interpolation

Interpolation paths. A natural and commonly used way to interpolate between two policies is the arithmetic mixture (1−α)πt+απ⋆(1-\alpha)\pi_t+\alpha\pi^\star [10, 40]. This appendix contrasts it with the geometric interpolation used in our rollout policy and clarifies which proposal-design objective each path corresponds to.

For a fixed prefix c=(x,y<j)c=(x,y_{<j}), write

p(v)=πt(v∣c),p⋆(v)=π⋆(v∣c)p(v)=\pi_t(v\mid c),\qquad p^\star(v)=\pi^\star(v\mid c)

for the student and teacher conditionals. The geometric interpolation used in our method is

qηgeo(v∣c)=p(v)1−ηp⋆(v)η∑v′p(v′)1−ηp⋆(v′)η.q_\eta^{\mathrm{geo}}(v\mid c)=\frac{p(v)^{1-\eta}p^\star(v)^\eta}{\sum_{v'}p(v')^{1-\eta}p^\star(v')^\eta}.

A natural alternative is the arithmetic interpolation

qαarith(v∣c)=(1−α)p(v)+αp⋆(v).q_\alpha^{\mathrm{arith}}(v\mid c)=(1-\alpha)p(v)+\alpha p^\star(v).

Both paths interpolate between the student and teacher endpoints, but they arise from different information-geometric objectives.

Different variational objectives. The geometric path is the barycenter under the reverse-KL geometry:

qηgeo=arg min⁡q (1−η)KL⁡(q∥p)+ηKL⁡(q∥p⋆).q_\eta^{\mathrm{geo}}=\underset{q}{\operatorname{arg\,min}}\,(1-\eta)\operatorname{KL}(q\Vert p)+\eta\operatorname{KL}(q\Vert p^\star).

Thus it can be viewed as a product-of-experts [17] interpolation: probability mass is assigned according to the joint support of the student and teacher conditionals. This geometry matches the local trust-region formulation used in our rollout policy. Indeed, with the teacher–student log-ratio

st(v,c)=log⁡p⋆(v)−log⁡p(v),s_t(v,c)=\log p^\star(v)-\log p(v),

the same distribution can be written as

qηgeo(v∣c)∝p(v)exp⁡(ηst(v,c)).q_\eta^{\mathrm{geo}}(v\mid c)\propto p(v)\exp(\eta s_t(v,c)).

Therefore, the geometric rollout policy is the KL-regularized mirror-descent update that starts from the student conditional and reweights candidate tokens by relative teacher support.

The arithmetic path corresponds to a different variational problem:

qαarith=arg min⁡q (1−α)KL⁡(p∥q)+αKL⁡(p⋆∥q).q_\alpha^{\mathrm{arith}}=\underset{q}{\operatorname{arg\,min}}\,(1-\alpha)\operatorname{KL}(p\Vert q)+\alpha\operatorname{KL}(p^\star\Vert q).

This is a mixture-of-experts interpolation: the proposal combines the two conditionals additively. Such a path is natural when the objective is coverage. For example, since

qαarith(v∣c)≥(1−α)p(v),q_\alpha^{\mathrm{arith}}(v\mid c)\ge(1-\alpha)p(v),

the importance ratio p(v)/qαarith(v∣c)p(v)/q_\alpha^{\mathrm{arith}}(v\mid c) is uniformly bounded by 1/(1−α)1/(1-\alpha). This property is useful in settings where the proposal must cover the student distribution, such as defensive importance sampling or off-policy evaluation.

Implication for OPD. The proposal-design problem in OPD is different. We seek a rollout policy that remains anchored to the current student while shifting probability toward teacher supported completions. This is formalized in the main text through a local KL⁡(q∥πt)\operatorname{KL}(q\Vert\pi_t) trust region and the relative teacher support score sts_t. Under this formulation, the geometric path is not merely an interpolation heuristic; it is the variational solution induced by the chosen reverse-KL geometry. In contrast, the arithmetic path corresponds to a different notion of closeness and a different proposal objective, one that emphasizes additive coverage of the student–teacher mixture.

This distinction is also consistent with the reachability-structured noise analysis. That analysis does not require a particular interpolation path; it states that a rollout policy is favorable when its induced occupancy reduces reachability-risk exposure without incurring too large a second-moment penalty. The geometric rollout policy is aligned with this analysis because it changes the student rollout distribution through multiplicative reweighting by relative teacher support. Whether this reweighting reduces reachability-risk exposure depends on the covariance condition discussed earlier, not on an unconditional dominance claim over other interpolation paths.

In summary, arithmetic interpolation is a principled choice for coverage-oriented objectives and bounded importance ratios. Geometric interpolation is the appropriate path for the formulation studied in this work: a student-anchored, reverse-KL-regularized proposal that tilts the current student toward tokens with larger relative teacher support.

More Case Studies

Multi-baseline case study table
Case A: first-order time constant. Prompt. For y˙+5y=10u\dot{y} + 5y = 10u, what is the time constant?Ground truth. 1/5\boxed{1/5}.
Takeaway. Ours identifies the system parameter and stops after a valid boxed answer. The four baselines often reach the same value but continue into long, unparseable tails.
MethodOutcomeLengthInterpretation and raw-output snippet
OursCorrect303 tokensRead. Maps the ODE to y˙+ay=bu\dot{y} + ay = bu, sets a=5a = 5, and emits the final answer. Snippet. a=5a = 5, so τ=1/a=1/5\tau= 1/a = 1/5 . . . Final Answer: 1/51/5
OPDParse fail16.4k tokensRead. Contains the correct value, then repeats the final-answer pattern until truncation. Snippet. Final Answer: 0.20.2 . . . Final Answer: 0.20.2 . . . Final Answer: open boxed answer
ExOPDParse fail16.4k tokensRead. Starts from the right dynamics but drifts into a 63.2 percent transient discussion. Snippet. steady-state value is 2 . . . t=−15ln⁡(0...)t = -\frac{1}{5}\ln(0 . . .)
EOPDParse fail16.4k tokensRead. Continues after solving and loses the final answer in a self-contradictory transient calculation. Snippet. time constant is not the same . . . solution is y(t)=20(1−e−5t)y(t) = 20(1 - e^{-5t}) . . .
VETOParse fail16.4k tokensRead. Also over-extends the transient interpretation and ends without a parseable answer. Snippet. 2(1−e−5τ)=0.7362(1 - e^{-5\tau}) = 0.736 . . . τ=−ln⁡(0.632)/5\tau= -\ln(0.632)/5 . . .

Table 9.

Figure 7: Multi-baseline case study on Minerva Math id 128. The qualitative advantage is not just shorter length: Ours preserves answer extractability after the correct reasoning step, while all four baselines shown lose the parseable final answer.

Figure 7.

Multi-baseline case study table for tritium down quarks

Figure 8. Multi-baseline case study on Minerva Math id 141. Ours preserves the simple invariant: proton uuduud contributes one down quark and each neutron uddudd contributes two. The four baselines drift to 33.

MethodOutcomeLengthInterpretation and raw-output snippet
OursCorrect309 tokensRead. Uses one proton and two neutrons, then counts only down quarks. Snippet. 1 proton →\rightarrow 1 down; 2 neutrons →\rightarrow 2 + 2 down; total = 5 ... 5
OPDAnswer drift16.4k tokensRead. Repeats an incorrect final count after visiting the right topic. Snippet. Final Answer: 3 quarks ... Final Answer: 3 quarks ...
ExOPDAnswer drift16.4k tokensRead. Falls into the same wrong final count and continues repeating it. Snippet. Yes. Final answer is: 3 down quarks ... Final Answer: 3
EOPDAnswer drift16.4k tokensRead. Ends in repeated confirmation of the wrong boxed answer. Snippet. Final Answer: 3 ... (Yes.) Final ... (End.) ...
VETOAnswer drift1.9k tokensRead. Correctly names tritium as one proton and two neutrons, then swaps quark content. Snippet. 1 proton: 1 u and 2 d; 2 neutrons: each has 1 d and 2 u ... total = 3

caption: Case B: tritium down quarks. Prompt. How many down quarks does a tritium (H3H^{3}) nucleus contain? Ground truth. 5. Takeaway. The problem is short and easy to audit. Ours uses the nuclear composition directly; the four baselines drift to an internally inconsistent quark count.

:::

Case C: logarithmic equation. Prompt. Determine all real xx for which 2log⁡2(x−1)=1−log⁡2(x+2)2\log_{2}(x-1)=1-\log_{2}(x+2). Ground truth. 3\sqrt{3}.

Takeaway. Ours reduces the equation to a cubic with a domain check. The four baselines get trapped in numerical checking or contradiction loops and do not provide a parsed final answer.

MethodOutcomeLengthInterpretation and raw-output snippet
OursCorrect785 tokensRead. Combines logs, solves (x−1)2(x+2)=2(x-1)^2(x+2)=2, and applies x>1x>1. Snippet. (x−1)2(x+2)=2⇒x3−3x=0(x-1)^2(x+2)=2\Rightarrow x^3-3x=0; domain x>1⇒x=3x>1\Rightarrow x=\sqrt{3}; 3\sqrt{3}
OPDParse fail16.4k tokensRead. Derives the right transformed equation, then rejects its own solution. Snippet. (x−1)2(x+2)=2(x-1)^2(x+2)=2 ... only at x=3x=3, which is not a solution to ...
ExOPDParse fail16.4k tokensRead. Switches from symbolic solving to unstable decimal checking. Snippet. log⁡2(0.732)≈−0.448\log_{2}(0.732)\approx-0.448 ... f(1.732)≈1.035f(1.732)\approx1.035 ...
EOPDParse fail16.4k tokensRead. Keeps recomputing approximations and never returns a parseable final set. Snippet. ln⁡(0.732)/ln⁡(2)≈−0.455\ln(0.732)/\ln(2)\approx-0.455 ... log⁡2(1.09)≈...\log_{2}(1.09)\approx...
VETOParse fail16.4k tokensRead. Continues comparing sides numerically instead of closing the algebraic solution. Snippet. At x=2x=2: 0>−10>-1 ... LHS >> RHS ... only if LHS = RHS ...
Multi-baseline case study on OlympiadBench id 2548

Figure 9. Multi-baseline case study on OlympiadBench id 2548. Ours follows a short symbolic path and keeps the domain constraint; every one of the four baseline rows shown fails to produce a parsed answer after much longer output.

Case D: easy logarithm. Prompt. Let T=16T = 16. Compute xx such that log⁡4T=log⁡2x\log_{4} T = \log_{2} x. Ground truth. 44.

Takeaway. This case isolates the length advantage: several OPD-family baselines eventually get the same answer but hit the 16.4k-token budget doing so, while Ours solves and stops.

MethodOutcomeLengthInterpretation and raw-output snippet
OursCorrect239 tokensRead. Substitutes T=16T = 16, computes log⁡416=2\log_{4} 16 = 2, and solves x=22x = 2^{2}. Snippet. log⁡416=2\log_{4} 16 = 2; log⁡2x=2\log_{2} x = 2; x=22=4x = 2^{2} = 4; Final Answer: 44
OPDOverlong16.4k tokensRead. Correct answer, but output continues into repeated confirmation tokens. Snippet. ... 44 ... [repeated confirmation tokens continue]
ExOPDOverlong16.4k tokensRead. Correct answer, but continues long after the final answer. Snippet. ... Final Answer: 44 ... [repeated confirmation tokens continue]
EOPDOverlong16.4k tokensRead. Same overgeneration pattern despite solving the elementary equation. Snippet. ... Final Answer: 44 ... [repeated confirmation tokens continue]
VETOCorrect482 tokensRead. Correct and comparatively short, but still longer and less direct than Ours. Snippet. log⁡2x=2⇒x=22=4\log_{2} x = 2 \Rightarrow x = 2^{2} = 4 ... 44 ... check matches
Multi-baseline case study on OlympiadBench id 2783

Figure 10. Multi-baseline case study on OlympiadBench id 2783. Ours gives the shortest correct response among the compared methods; OPD, ExOPD, and EOPD solve the problem but exhaust a 16.4k-token generation budget after the answer.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2306.13649
  2. [2]Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereasoning: Advancing data distillation for competitive coding. arXiv preprint arXiv:2504.01943, 2025.arxiv.org/abs/2504.01943
  3. [3]Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Math-arena: Evaluating llms on uncontaminated math competitions. arXiv preprint arXiv:2505.23281, 2025.arxiv.org/abs/2505.23281
  4. [4]Akhilad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-menotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949, 2025.arxiv.org/abs/2505.00949
  5. [5]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  6. [6]DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026.arxiv.org/abs/2606.19348
  7. [7]Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849, 2025.arxiv.org/abs/2507.19849
  8. [8]Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes. arXiv preprint arXiv:2603.25562, 2026.arxiv.org/abs/2603.25562
  9. [9]Naibin Gu, Chenxu Yang, Qingyi Si, Chuanyu Qin, Dingyu Yao, Peng Fu, Zheng Lin, Weiping Wang, Nan Duan, and Jiaqi Wang. Co-evolving policy distillation. arXiv preprint arXiv:2604.27083, 2026.arxiv.org/abs/2604.27083
  10. [10]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The twelfth international conference on learning representations, 2024.
  11. [11]Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717, 2023.arxiv.org/abs/2305.15717
  12. [12]Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint arXiv:2506.04178, 2025.arxiv.org/abs/2506.04178
  13. [13]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shiromg Ma, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  14. [14]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850, 2024.
  15. [15]Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and Sanjeev Arora. Self-distillation zero: Self-revision turns binary rewards into dense supervision. arXiv preprint arXiv:2604.12002, 2026.arxiv.org/abs/2604.12002
  16. [16]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. stat, 1050:9, 2015.arxiv.org/abs/1503.02531
  17. [17]Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771–1800, 2002.pubmed.ncbi.nlm.nih.gov/12180402
  18. [18]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022.arxiv.org/abs/2106.09685
  19. [19]Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. m1: Unleash the potential of test-time scaling for medical reasoning with large language models. arXiv preprint arXiv:2504.00869, 2025.arxiv.org/abs/2504.00869
  20. [20]Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026.
  21. [21]Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation. arXiv preprint arXiv:2601.07155, 2026.arxiv.org/abs/2601.07155
  22. [22]Di Jin, Eileen Pan, Nassim Oufattolle, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
  23. [23]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2567–2577, 2019.arxiv.org/abs/1909.06146
  24. [24]Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026.arxiv.org/abs/2603.07079
  25. [25]Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016.arxiv.org/abs/1606.07947
  26. [26]Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation. arXiv preprint arXiv:2603.11137, 2026.arxiv.org/abs/2603.11137
  27. [27]Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898, 2024.arxiv.org/abs/2402.03898
  28. [28]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.arxiv.org/abs/2309.06180
  29. [29]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.arxiv.org/abs/2005.01643
  30. [30]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857, 2022.arxiv.org/abs/2206.14858
  31. [31]Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026.arxiv.org/abs/2604.13016
  32. [32]Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive knowledge distillation through imitation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6121–6133, 2020.arxiv.org/abs/2009.07253
  33. [33]Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Guilherme Penedo, Edward Beeching, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Openr1-math-220k. https://huggingface.co/datasets/open-r1/OpenR1-Math-220k, 2025.
  34. [34]Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025. https://thinkingmachines.ai/blog/on-policy-distillation.
  35. [35]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.arxiv.org/abs/2203.02155
  36. [36]Carlos Miguel Patiño, Kashif Rasul, Quentin Gallouédec, Ben Burtenshaw, Sergio Paniego, Vaibhav Srivastav, Thibaud Freire, Ed Beeching, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Unlocking on-policy distillation for any model family, 2025.
  37. [37]Yi Peng, Peiyu Wang, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, et al. Skywork r1v: Pioneering multimodal reasoning with chain-of-thought. arXiv preprint arXiv:2504.05599, 2025.arxiv.org/abs/2504.05599
  38. [38]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.arxiv.org/abs/2311.12022
  39. [39]Tao Ren, Jinyang Jiang, Hui Yang, Wan Tian, Minhao Zou, Guanghao Li, Zishi Zhang, Qinghao Wang, Shentao Qin, Yanjun Zhao, Rui Tao, Hui Shao, and Yijie Peng. Riskpo: Risk-based policy optimization via verifiable reward for llm post-training, 2025.arxiv.org/abs/2510.00911
  40. [40]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635. JMLR Workshop and Conference Proceedings, 2011.arxiv.org/abs/1011.0686
  41. [41]Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.arxiv.org/abs/1910.01108
  42. [42]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  43. [43]Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.DOI
  44. [44]Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html, 3(6):7, 2023.
  45. [45]Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026.
  46. [46]Qwen Team. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804, 2026.arxiv.org/abs/2604.15804
  47. [47]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024.arxiv.org/abs/2406.01574
  48. [48]Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. F-divergence minimization for sequence-level knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10817–10834, 2023.arxiv.org/abs/2307.15190
  49. [49]Yecheng Wu, Song Han, and Hai Cai. Lightning opd: Efficient post-training for large reasoning models with offline on-policy distillation. arXiv preprint arXiv:2604.13010, 2026.arxiv.org/abs/2604.13010
  50. [50]Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Guang Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026.arxiv.org/abs/2601.02780
  51. [51]LLM-Core-Team Xiaomi. Mimo: Unlocking the reasoning potential of language model – from pretraining to posttraining, 2025.arxiv.org/abs/2505.07608
  52. [52]Haobo Xu, Sirui Chen, Ruizhong Qiu, Yuchen Yan, Chen Luo, Monica Xiao Cheng, Jingrui He, and Hanghang Tong. Prune as you generate: Online rollout pruning for faster and better rlvr. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13876–13893, 2026.
  53. [53]Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. In The Thirteenth International Conference on Learning Representations.
  54. [54]Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026.arxiv.org/abs/2604.14084
  55. [55]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengchen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  56. [56]Wenkai Yang, Yankai Lin, Jie Zhou, and Ji-Rong Wen. Distilling rule-based knowledge into large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 913–932, 2025.
  57. [57]Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026.arxiv.org/abs/2602.12125
  58. [58]Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. Black-box on-policy distillation of large language models. arXiv preprint arXiv:2511.10643, 2025.arxiv.org/abs/2511.10643
  59. [59]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.arxiv.org/abs/2503.14476
  60. [60]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
  61. [61]Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xianggang Li. 1. 4 million open-source distilled reasoning dataset to empower large language model training. arXiv preprint arXiv:2503.19633, 2025.
  62. [62]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026.arxiv.org/abs/2601.18734
  63. [63]Yanjun Zhao, Ruizhong Qiu, Tianxin Wei, Yuanchen Bei, Zhining Liu, Lingjie Chen, Ismini Lourentzou, Hanghang Tong, and Jingrui He. Recontext: Recursive evidence replay as llm harness for long-context reasoning, 2026.
  64. [64]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025.arxiv.org/abs/2507.18071
  65. [65]Jiaru Zou, Rui Pan, Ruizhong Qiu, Pan Lu, Shizhe Diao, Jindong Jiang, Hanghang Tong, Tong Zhang, Markus J Buehler, Jingrui He, et al. Recursive multi-agent systems. arXiv preprint arXiv:2604.25917, 2026.arxiv.org/abs/2604.25917
  66. [66]Jiaru Zou, Yunzhe Qi, Sirui Chen, Federico Bianchi, Shang Zhu, Pan Lu, Jingrui He, and James Zou. Hybrid reasoning agents. arXiv preprint arXiv:2610.18861, 2026.arxiv.org/abs/2610.18861
  67. [67]Jiaru Zou, Xiyuan Yang, Ruizhong Qiu, Gaotang Li, Katherine Tieu, Pan Lu, Ke Shen, Hanghang Tong, Yejin Choi, Jingrui He, et al. Latent collaboration in multi-agent systems. arXiv preprint arXiv:2511.20639, 2025.arxiv.org/abs/2511.20639
  68. [68]Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025.arxiv.org/abs/2501.18362

Paper details

vlnc:W_q526rc8s
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 19

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLMs

View statement
Looking for related papers…
Contents