betaFeedback

WorksComputer Science

Bernoulli On-Policy Distillation: Don't Ignore High-Confidence Student Tokens

Abstract

On-policy distillation (OPD) trains a student language model on its own trajectories using token-level supervision from a stronger teacher. We show that standard sampled reverse-KL OPD systematically suppresses updates to high-confidence tokens, such that their weak gradient contribution does not necessarily imply weak or uninformative teacher supervision. We introduce Bernoulli On-Policy Distillation (Bernoulli OPD), which directly matches the teacher and student probabilities for that token. This yields a bounded probability space correction that preserves teacher-student disagreement while avoiding the additional confidence-dependent suppression induced by vanilla OPD. Fixed-rollout analyses show that Bernoulli OPD substantially restores optimization signal to high-confidence tokens while remaining sensitive to teacher-student disagreement. We further analyze the resulting model behavior through confidence distributions, answer calibration, policy movement, and training dynamics, showing that Bernoulli OPD's gains cannot be explained simply by longer reasoning, larger optimization steps, or more uniform imitation of the teacher. Code is available at https://github.com/duterscmy/Bernoulli-OPD.

Introduction

On-policy distillation (OPD) trains a student on its own generated trajectories while using a stronger teacher to provide token-level supervision on the visited states ([1, 14] (Gu et al. 2026)). By learning directly on student-generated trajectories, OPD reduces the distribution mismatch inherent to offline distillation and has shown strong potential for improving language-model reasoning.

Despite its effectiveness, the optimization behavior of sampled-token OPD remains poorly understood. In the commonly used sampled reverse-KL objective, the gradient assigned to a sampled token depends not only on teacher–student disagreement, but also strongly on the student’s own confidence. As the student becomes more confident in a sampled token, its gradient is increasingly suppressed, even when the teacher still assigns a meaningfully different probability to that token. This introduces a confidence-dependent bias into how the objective allocates optimization signal.

Figure 1 illustrates this effect. High-confidence tokens constitute a large fraction of sampled tokens in student-generated trajectories, yet receive only a small fraction of the total gradient mass under vanilla sampled OPD. Importantly, weak gradient contribution does not necessarily imply that these tokens contain little useful teacher supervision. When we train exclusively on highly confident sampled tokens, the student still improves substantially throughout training, although it remains weaker than training on all sampled tokens. This suggests that the limited contribution of high-confidence tokens under vanilla OPD is at least partly induced by the geometry of the objective itself rather than by a lack of useful learning signal.

Figure 1. Token distribution and gradient allocation across student-confidence regions. In our primary setting, about 72% of sampled tokens are high-confidence with pS>0.9p_S > 0.9. However, this region receives only about 2.4% of the total gradient mass under vanilla OPD, compared with about 46.5% under Bernoulli OPD. The contrast between token frequency and gradient allocation highlights the strong confidence-dependent suppression induced by vanilla sampled reverse-KL OPD.

These observations motivate a simple question: can we effectively learn from teacher corrections even when the student is already confident in its prediction? To address this issue, we propose Bernoulli On-Policy Distillation (Bernoulli OPD). Instead of using the sampled reverse-KL log ratio objective, Bernoulli OPD treats each sampled token as a simple binary prediction and directly matches the teacher and student probabilities for that token. This gives a bounded update that follows the difference between the teacher and student probabilities, without further weakening the training signal on high-confidence tokens.

Our experiments support both the proposed mechanism and its practical utility. On identical student rollouts, Bernoulli OPD allocates substantially more optimization signal to high-confidence tokens while remaining sensitive to teacher–student disagreement. In our primary setting, Bernoulli OPD improves the five-benchmark average from 44.38 to 51.76 over vanilla OPD, and this advantage is reproduced across three independent training seeds (Appendix § 11). More broadly, Bernoulli OPD consistently improves over vanilla OPD across four teacher–student settings, covering both Qwen3 and DeepSeek-based students, different types of teacher models, and both non-thinking and thinking generation modes.

Our contributions are summarized as follows.

  • We show that sampled reverse-KL OPD gives very little optimization signal to high-confidence tokens, even though these tokens are common in student rollouts and can still provide useful learning signal.

  • We propose Bernoulli OPD, a simple objective that directly matches teacher and student token probabilities and avoids unnecessarily weakening updates on high-confidence tokens.

  • Across four teacher–student settings, Bernoulli OPD consistently improves reasoning performance over vanilla OPD. We further analyze its behavior through confidence distributions, answer-level calibration, policy movement, and training dynamics.

Related Work

Off-policy and on-policy distillation. Classical knowledge distillation learns from a fixed data distribution [6], while sequence-level distillation often uses teacher-generated responses [9]. On-policy distillation instead trains on student generated trajectories, reducing the mismatch between training and inference states [1] (Gu et al. 2026). Existing OPD methods can be broadly divided into full vocabulary [8], Top-kk [12], and sampled token variants (Gu et al., 2026). Full vocabulary methods use dense teacher distributions, Top-kk methods retain only a restricted support, and sampled-based methods use student sampled tokens as Monte Carlo estimates. The latter is particularly attractive for long horizon reasoning research because of its lower memory and communication cost. Our work also focuses on this sampled-token setting.

Sampled-based OPD. Recent work has studied sampled OPD from several perspectives, including teacher–student compatibility, rollout length, sampling, and token-level optimization. OPSA [4] observes that learning under sampled reverse KL is concentrated on low-probability tokens. Building on similar observation, SuRe [13] analyzes the underlying gradient structure and further strengthens updates to low-confidence tokens. PowerOPD [18] focuses on a different issue, replacing the unbounded log-ratio reward with a bounded transformation to improve training stability. Our work revisits the gradient allocation of sampled OPD from the opposite direction. We ask whether the weak updates assigned to high-confidence tokens are actually justified, and show that these tokens can still provide useful learning signal. This motivates Bernoulli OPD, which directly uses the bounded teacher–student probability difference as its training signal. As a result, high-confidence teacher corrections remain effective without explicitly favoring either low- or high-confidence tokens.

Binary objectives in computer vision distillation. Binary formulations have been explored in computer vision distillation. Decoupled Knowledge Distillation (DKD) [19] separates the target class from the remaining classes in image classification. Related binary formulations have also been used to distill class predictions in object detection [16]. Bernoulli OPD uses a related binary view in a different setting. For each student-sampled token, we group the vocabulary into the sampled token and its complement, and match the corresponding teacher and student Bernoulli probabilities. Our goal is not to separate class-level knowledge, but to preserve useful teacher corrections on high-confidence tokens that receive weak gradients under sampled reverse KL.

Background and Motivation

Let xx denote a prompt and y1:T∼pS(⋅∣x)y_{1:T} \sim p_S(\cdot\mid x) a trajectory sampled from the current student. At step tt, let ct=(x,y<t)c_t = (x, y_{<t}) denote the visited context and yty_t the sampled next token. We define the student and teacher probabilities of the sampled token as

pS=pS(yt∣ct),pT=pT(yt∣ct).(1)p_S = p_S(y_t \mid c_t), \qquad p_T = p_T(y_t \mid c_t). \tag*{(1)}

The teacher is frozen, and the sampled trajectory is treated as fixed within each optimization step.

Vanilla sampled reverse KL. Define the teacher–student log-probability gap as

At=log⁡pT−log⁡pS.(2)A_t = \log p_T - \log p_S. \tag*{(2)}

A common sampled surrogate for reverse-KL OPD is

LVanilla,t=−sg⁡(At)log⁡pS,(3)\mathcal{L}_{\mathrm{Vanilla},t} = -\operatorname{sg}(A_t) \log p_S, \tag*{(3)}

where sg⁡(⋅)\operatorname{sg}(\cdot) denotes stop-gradient. The commonly used K2 objective, 12(log⁡pT−log⁡pS)2\frac{1}{2}(\log p_T - \log p_S)^2, induces the same gradient with respect to the student parameters.

Let zyz_y denote the logit of the sampled token. Its gradient is

∂LVanilla,t∂zy=−At(1−pS),(4)\frac{\partial\mathcal{L}_{\mathrm{Vanilla},t}}{\partial z_y} = -A_t(1-p_S), \tag*{(4)}

while the ℓ1\ell_1 norm of the corresponding full-vocabulary logit gradient is

∥∇zLVanilla,t∥1=2∣At∣(1−pS).(5)\lVert\nabla_z \mathcal{L}_{\mathrm{Vanilla},t} \rVert_1 = 2|A_t|(1-p_S). \tag*{(5)}

These expressions show that vanilla sampled OPD does not depend only on teacher–student disagreement. Its gradient contains an additional factor 1−pS1-p_S from the softmax geometry. As the student becomes more confident in the sampled token, this factor approaches zero and weakens the corresponding optimization signal. Therefore, even when the teacher still disagrees with a confident student prediction, the token can receive only a small gradient.

High-confidence tokens dominate rollouts but receive weak gradients. This confidence-dependent suppression is substantial in practice. As shown in Figure 1, more than 70% of sampled tokens lie in the highest-confidence region pS∈[0.9,1.0)p_S \in[0.9, 1.0). Despite dominating the rollout distribution,

these tokens have much smaller per-token gradients and consequently account for only a limited share of the total gradient mass. Optimization under vanilla sampled OPD is therefore distributed very differently from the sampled-token distribution itself.

Importantly, this imbalance does not by itself imply that high-confidence tokens are uninformative. A small gradient can arise either because the teacher and student already agree, or because the objective suppresses the gradient when the student is highly confident. Distinguishing these two cases is important for understanding whether the gradient allocation of vanilla OPD is desirable.

High-confidence tokens can still provide useful training signal. To test whether the high-confidence region contains useful learning signal, we train Vanilla OPD using only sampled tokens with pS>0.9p_S > 0.9. Figure 2 shows that the student continues to improve substantially throughout training despite discarding all lower-confidence tokens. High-confidence tokens are therefore not devoid of useful teacher supervision, even though vanilla OPD assigns them relatively little gradient mass.

Figure 2. Training with high-confidence sampled tokens. This diagnostic experiment uses our primary Qwen3-4B-Instruct-2507 → Qwen3-1.7B setting. Vanilla OPD restricted to tokens with pS>0.9p_S > 0.9 still yields substantial performance gains despite the strong gradient suppression in this confidence region.

Together, these results reveal a mismatch between token informativeness and objective-induced gradient allocation. High-confidence tokens are both abundant and learnable, yet their updates are systematically attenuated by the confidence dependence of sampled reverse KL. This motivates an objective that preserves teacher–student disagreement as a learning signal without additionally suppressing updates merely because the student is already confident.

Methods

Vanilla sampled reverse-KL OPD combines teacher–student disagreement with a confidence-dependent factor 1−pS1-p_S, which strongly suppresses updates on high-confidence sampled tokens. We study this issue through two complementary objectives. GC-OPD removes only the explicit confidence-dependent suppression and serves as a diagnostic control. Our main method, Bernoulli OPD, replaces the sampled log-ratio objective with a Bernoulli cross-entropy objective whose sampled-token logit gradient is bounded. Figure 3 compares their gradient allocation with Vanilla OPD and SuRe on the same fixed student rollouts.

Figure 3. Gradient allocation under different OPD objectives. All methods are evaluated on the same fixed student rollouts. The top row shows the mean per-token gradient proxy across student-confidence bins, and the bottom row shows the corresponding share of total gradient mass. Compared with (a) Vanilla OPD, (b) SuRe further emphasizes low-confidence updates, whereas (c) GC-OPD and (d) Bernoulli OPD substantially restore gradient contribution in high-confidence regions. Top-row y-axis scales differ across objectives; the bottom row reports normalized gradient-mass shares.

Geometry-Corrected OPD

We first isolate the explicit 1−pS1-p_S factor in vanilla OPD while preserving the same teacher–student log-probability signal. Recall

At=log⁡pT−log⁡pS.(6)A_t=\log p_T-\log p_S. \tag*{(6)}

We replace the sampled-token log probability with its log odds:

LGC,t=−sg⁡(At)log⁡pS1−pS.(7)\mathcal{L}_{\mathrm{GC},t}=-\operatorname{sg}(A_t)\log\frac{p_S}{1-p_S}. \tag*{(7)}

Because the derivative of the sampled-token log odds with respect to its logit zyz_y is one,

∂LGC,t∂zy=−At.(8)\frac{\partial\mathcal{L}_{\mathrm{GC},t}}{\partial z_y}=-A_t. \tag*{(8)}

GC-OPD therefore removes the explicit 1−pS1-p_S suppression of vanilla OPD while retaining the same log-ratio discrepancy. As shown in Figure 3(c), this substantially restores gradient contribution from high-confidence tokens. We use GC-OPD primarily as a diagnostic control to test whether the confidence-dependent softmax geometry itself limits useful teacher corrections.

Bernoulli OPD

GC-OPD removes the explicit confidence suppression but still relies on the unbounded log-probability gap. We therefore introduce Bernoulli OPD, which instead measures teacher–student disagreement directly in probability space.

For each sampled token yty_t, consider the binary event

Bt=1[Y=yt].(9)B_t=\mathbb{1}[Y=y_t]. \tag*{(9)}

The categorical teacher and student distributions induce Bernoulli marginals with parameters pTp_T and pSp_S. We distill these marginals using binary cross-entropy:

LBern,t=−pTlog⁡pS−(1−pT)log⁡(1−pS).(10)\mathcal{L}_{\mathrm{Bern},t}=-p_T\log p_S-(1-p_T)\log(1-p_S). \tag*{(10)}

This objective is a sampled token-versus-complement coarse-graining evaluated at student-visited states, rather than a full-vocabulary forward KL. Its derivative with respect to the sampled-token logit is

∂LBern,t∂zy=pS−pT.(11)\frac{\partial\mathcal{L}_{\mathrm{Bern},t}}{\partial z_y}=p_S-p_T. \tag*{(11)}

Bernoulli OPD therefore uses the direct teacher–student probability gap. Unlike vanilla OPD, the correction is not multiplied by an additional vanishing 1−pS1-p_S factor. The probability gap is also bounded in [−1,1][-1,1], avoiding the large scale that can arise from log-probability ratios. When teacher and student agree, the update naturally vanishes; when they disagree, the correction remains visible even at high student confidence.

This change produces a markedly different gradient allocation. Figure 3(d) shows that Bernoulli OPD shifts substantial optimization mass toward the high-confidence region that is nearly ignored by vanilla OPD, while still assigning larger per-token updates where teacher–student disagreement is stronger.

Summary. Table 1 summarizes the sampled-token gradients of our two objectives together with vanilla OPD. GC-OPD keeps the original teacher–student log-probability gap while removing the explicit 1−pS1-p_S factor, allowing us to isolate the effect of confidence-dependent gradient suppression. Bernoulli OPD further replaces the log-ratio signal with the bounded teacher–student probability difference, and serves as our main objective.

MethodSampled-token gradient ∂L/∂zy\partial L/\partial z_yRemove log gap?Remove 1−pS1-p_S factor?
Vanilla OPD−A(1−pS)-A(1-p_S)NoNo
SuRe−wtA(1−pS)-w_tA(1-p_S)NoNo
PowerOPD−(pTα−pSα)(1−pS)-(p_T^\alpha-p_S^\alpha)(1-p_S)YesNo
GC-OPD (Ours)−A-ANoYes
Bernoulli OPD (Ours)pS−pTp_S-p_TYesYes

Table 1. Table 1. Structural comparison of sampled OPD objectives. A=log⁡pT−log⁡pSA=\log p_T-\log p_S. We compare the sampled-token gradient together with whether each method removes the log-probability gap and the explicit 1−pS1-p_S factor. GC-OPD serves as a diagnostic variant, while Bernoulli OPD is our main proposed objective.

For a clearer comparison with existing sampled OPD methods, we also include SuRe and PowerOPD in the same gradient view. SuRe changes the relative weighting of token updates, while PowerOPD uses a bounded power transformation of the teacher–student signal. Together, this comparison shows how our objectives differ from related methods in both the form of the training signal and its confidence-dependent gradient behavior. We further study confidence rebalancing and gradient scale in § 5.3.

Experiments

Experimental Setup

Models and training. We evaluate four teacher–student configurations spanning two student models. For Qwen3-1.7B, we use Qwen3-4B-Instruct-2507 and Qwen3-8B-DAPO-RL11 as teachers, with Qwen3-4B-Instruct-2507 as our primary setting (Yang et al. [15]). For DeepSeek-R1-Distill-Qwen-1.5B, we use DeepSeek-R1-Distill-Qwen-7B and Skywork-OR1-Math-7B as teachers. Following the native generation conventions of the corresponding model families, we evaluate the Qwen3 settings in non-thinking mode and the DeepSeek-based settings in thinking mode. Training prompts are drawn from DAPO-Math-17K (Yu et al. (2025)). Within each teacher–student setting, all methods share the same student initialization, rollout prompts, generation mode, and optimization configuration; only the distillation objective changes, unless otherwise specified. Full training and implementation details are provided in Appendix § 8.

Evaluation. We evaluate checkpoints on AIME24 (Jia, [7]) and select the best checkpoint using Avg@8. Final evaluation covers AIME24, AIME25 (math ai, 2025), HMMT25 (Balunović et al., [2]), AMC23 (Cao et al., [3]), and MATH500 (Lightman et al., [10]). The first four benchmarks are reported with Avg@32, while MATH500 uses Avg@4. Decoding parameters and evaluation details are given in Appendix § 8.

Baselines and variants. We compare against vanilla sampled reverse-KL OPD, SuRe (Shao et al., [13]), and PowerOPD (Zhao et al., [18]). We implement SuRe within our framework and evaluate it under the same unified training setup as our methods. For PowerOPD, we evaluate both the unified and official reference configurations and report the better result for each teacher–student setting to ensure a fair and conservative comparison. Implementation details and results for both configurations are provided in Appendix § 8.§ 4.

Among our methods, GC-OPD serves as a diagnostic control that removes the explicit 1−pS1-p_S suppression while retaining the log-ratio signal, whereas Bernoulli OPD is our primary proposed objective. We additionally study Balanced Bernoulli OPD and a scale-matched Bernoulli control in § 5.3 to examine confidence-level rebalancing and the effect of overall optimization magnitude.

Main Results

Table 2 summarizes the main results across four teacher–student settings. Bernoulli OPD improves aggregate reasoning performance over vanilla OPD in all four settings, with gains ranging from 0.98 to 7.38 points. The largest improvement occurs in our primary Qwen3-4B-Instruct-2507 →\rightarrow Qwen3-1.7B setting, where Bernoulli OPD increases the five-benchmark average from 44.38 to 51.76. The improvement also persists with the Qwen3-8B-DAPO-RL teacher and across both settings using the DeepSeek-R1-Distill-Qwen-1.5B student, indicating that the benefit is not specific to a single student or teacher family.

MethodAIME24AIME25HMMT25AMC23MATH500Avg.Δ vs. Vanilla
Student: Qwen3-1.7B12.089.795.1047.5070.6529.02–
Teacher: Qwen3-4B-Instruct-2507
Vanilla OPD33.9621.7715.0068.3682.8044.38–
SuRe35.1023.0214.4869.8482.3044.95+0.57
PowerOPD38.6526.4618.2371.0983.9047.67+3.29
GC-OPD38.0227.6018.9675.3985.6049.11+4.73
Bernoulli OPD39.5833.0219.9078.6787.6551.76+7.38
Teacher: Qwen3-8B-DAPO-RL
Vanilla OPD50.2139.7925.4282.1989.2057.36–
SuRe48.7539.5825.4282.6688.8057.04-0.32
PowerOPD52.5039.1724.3885.3189.4558.16+0.80
GC-OPD52.5040.0024.3885.9486.6057.88+0.52
Bernoulli OPD51.0441.2525.4282.9791.2058.38+1.02
Student: DS-R1-Distill-Qwen-1.5B30.2124.5813.9669.2281.4543.88–
Teacher: DS-R1-Distill-Qwen-7B
Vanilla OPD28.4423.0215.1070.3182.7543.92–
SuRe29.0623.0214.1770.3182.9043.89-0.03
PowerOPD30.6323.9613.7570.1682.6044.22+0.30
GC-OPD31.1523.3314.2770.1682.7544.33+0.41
Bernoulli OPD30.1023.8514.3872.6683.5044.90+0.98
Teacher: Skywork-OR1-Math-7B
Vanilla OPD35.0027.6016.4675.7885.4048.05–
SuRe33.3327.1916.7773.5983.8546.95-1.10
PowerOPD34.9027.4016.4675.9484.8047.90-0.15
GC-OPD38.1327.1917.5073.9185.4548.44+0.39
Bernoulli OPD40.4226.8817.7179.2286.4050.13+2.08

Table 2. Main results across four teacher settings. The first four benchmarks use Avg@32 and MATH500 uses Avg@4. GC-OPD is a diagnostic variant, while Bernoulli OPD is our main proposed objective.

The existing baselines show different strengths across settings. SuRe modifies the confidence preference of sampled OPD, while PowerOPD uses a bounded transformation of the log-ratio reward and provides a strong competitive baseline. Bernoulli OPD achieves the highest aggregate performance across all four settings. GC-OPD also improves over vanilla OPD in every setting, providing evidence that reducing the confidence-dependent suppression of sampled reverse KL can be beneficial. Bernoulli OPD further improves over GC-OPD across all four settings. One possible explanation is that Bernoulli OPD not only removes the additional suppression of high-confidence tokens, but also expresses teacher–student disagreement directly through a Bernoulli cross-entropy objective with a bounded sampled-token logit gradient. This combination may provide a more effective training signal across different teacher–student settings.

The magnitude of the Bernoulli OPD improvement nevertheless varies substantially across teachers. This suggests that correcting gradient allocation determines which teacher corrections can act strongly on the student, but does not by itself determine how useful those corrections are. We further analyze this teacher-dependent variation in § 5.4.

Additional Variants for Bernoulli OPD

We further examine two questions: (1) does making the gradient allocation more uniform across confidence regions improve performance? (2) are the gains of Bernoulli OPD simply due to a larger overall optimization step?

Balanced Bernoulli OPD. Bernoulli OPD substantially restores optimization signal to high-confidence tokens, but it does not explicitly attempt to equalize gradient mass across confidence regions. Its gradient magnitude remains determined by the underlying teacher–student probability discrepancy, ∣pS−pT∣|p_S-p_T|, which can itself vary substantially with student confidence. This raises a natural question: once the severe suppression of high-confidence tokens is removed, would making the gradient allocation even more balanced across confidence regions lead to further gains?

To examine this question, we introduce Balanced Bernoulli OPD as a diagnostic extension that explicitly reweights Bernoulli updates according to the aggregate gradient mass in each confidence region. Specifically, we partition valid response tokens into BB bins according to their student probability pSp_S. For each bin bb, we compute its aggregate Bernoulli gradient mass

Mb=∑t∈b∣pS(t)−pT(t)∣.(12)M_b = \sum_{t \in b} \lvert p_S(t) - p_T(t) \rvert. \tag*{(12)}

Let Mˉ\bar{M} denote the mean mass across valid bins. We assign tokens in bin bb a detached weight

w~b=(MˉMb+ϵ)β,(13)\tilde{w}_b = \left(\frac{\bar{M}}{M_b + \epsilon}\right)^\beta, \tag*{(13)}

followed by normalization over valid tokens and clipping to limit extreme weights. The resulting objective is

LBalanced,t=wb(t)LBern,t,(14)\mathcal{L}_{\mathrm{Balanced},t} = w_b(t)\mathcal{L}_{\mathrm{Bern},t}, \tag*{(14)}

with sampled-token gradient

∂LBalanced,t∂zy=wb(t)(pS−pT),(15)\frac{\partial\mathcal{L}_{\mathrm{Balanced},t}}{\partial z_y} = w_b(t)(p_S - p_T), \tag*{(15)}

since wbw_b is detached. We use B=10B = 10, β=0.5\beta= 0.5, and clip the normalized weights to [1/5,5][1/5, 5].

By construction, Balanced Bernoulli pushes the gradient allocation toward a more even distribution across student-confidence regions than standard Bernoulli OPD. If the remaining imbalance under Bernoulli OPD were itself an important limitation, this stronger rebalancing should provide additional downstream benefit. Table 3, however, shows no such consistent improvement. Balanced Bernoulli yields only small gains of 0.11, 0.27, and 0.06 points in three settings, while decreasing the five-benchmark average by 0.92 points with the Skywork-OR1-Math-7B teacher.

TeacherBernoulliBalancedΔ\Delta
Qwen3-4B-Instruct51.7651.87+0.11
Qwen3-8B-DAPO-RL58.3858.65+0.27
DS-R1-Distill-Qwen-7B44.9044.96+0.06
Skywork-OR1-Math-7B50.1349.21-0.92

Table 3. Effect of stronger confidence-level rebalancing. We report the five-benchmark average for standard and Balanced Bernoulli OPD across the four teacher–student settings. Δ\Delta denotes Balanced Bernoulli minus Bernoulli OPD.

These results clarify that the goal is not to make gradient allocation uniform across confidence regions. Rather, the main benefit comes from removing the strong objective-induced suppression of high-confidence tokens while allowing the remaining gradient distribution to reflect the natural

teacher–student probability discrepancies present in the data. The gradient allocation induced directly by Bernoulli OPD therefore appears sufficient: explicitly flattening it further provides little additional benefit and can even be detrimental. We consequently use standard Bernoulli OPD as our primary objective and treat Balanced Bernoulli OPD only as a mechanism control.

Gradient-scale control. Figure 3 shows that Bernoulli OPD produces a substantially different gradient profile from vanilla OPD, raising the possibility that its gain results simply from a larger overall update. To isolate this effect, we construct a scale-matched Bernoulli variant whose loss is rescaled at initialization to match the full-model gradient norm of vanilla OPD.

Table 4 summarizes the result. Scale matching makes both the initial gradient norm and one-step policy movement nearly identical to vanilla OPD while preserving Bernoulli OPD’s distinct confidence-dependent gradient allocation. The downstream advantage also remains: scale-matched Bernoulli reaches an average accuracy of 52.58, compared with 44.38 for vanilla OPD and 51.76 for standard Bernoulli OPD. In this diagnostic setting, the gain therefore cannot be explained solely by a larger global update. We use scale matching only as a diagnostic control because it requires calibration against vanilla OPD. Full calibration details are provided in Appendix § 10.

Method∥g∥2\lVert g \rVert_2Policy moveHC massAvg.
Vanilla OPD8.1390.008911.44%44.38
Bernoulli OPD9.9930.0090839.42%51.76
Scale-Matched8.1380.0089239.42%52.58

Table 4. Gradient-scale control. Policy move is the mean absolute sampled-token log-probability change after one optimizer step; HC mass denotes the gradient-proxy mass assigned to pS∈[0.9,1.0]p_S \in[0.9, 1.0]; Avg. is the downstream five-benchmark average.

Teacher-Dependent Variation

The benefit of Bernoulli OPD varies substantially across teacher–student settings. In our primary Qwen3-4B-Instruct-2507 →\rightarrow Qwen3-1.7B setting, Bernoulli OPD improves the five-benchmark average over vanilla OPD by 7.38 points, whereas the gain with Qwen3-8B-DAPO-RL is only 1.02 points. The two DeepSeek-based settings also show different gains, ranging from 0.98 to 2.08 points. These results suggest that teacher strength or student–teacher performance headroom alone does not determine how much Bernoulli OPD benefits from the teacher signal.

To investigate this difference, we score exactly the same Qwen3-1.7B student rollouts with the 4B-Instruct and 8B-DAPO teachers. Both exhibit nearly the same confidence-dependent imbalance under vanilla OPD: more than 90% of sampled tokens lie in the highest-confidence region, yet this region receives only about 3% of the vanilla gradient-proxy mass. Bernoulli OPD restores this contribution to approximately 47–49% for both teachers. Thus, the existence of high-confidence suppression alone is also insufficient to predict the downstream gain.

Our additional analysis suggests that the quality of the recovered correction may matter in addition to its magnitude. The 4B-Instruct teacher provides slightly more outcome-aligned Bernoulli signals on the fixed rollouts, although the statistical difference is not conclusive. One possible factor is teacher provenance: Qwen3-4B-Instruct-2507 is a newer instruction-tuned checkpoint and may differ substantially in supervised post-training, whereas Qwen3-8B-DAPO-RL is obtained by further RL training a model from the same Qwen3 base generation as the student. Such differences may affect the compatibility and teachability of teacher supervision on student-generated trajectories. We provide the detailed signal analysis in Appendix § 9.

Model Behaviours

Confidence redistribution. We next examine how different OPD objectives affect the confidence distribution of the trained student. Figure 4 reports the change in token share

Figure 4. Confidence redistribution after OPD training. Changes in token share across confidence bins relative to the initial student under (a) fixed-rollout rescoring and (b) on-policy generation. Bernoulli OPD shows the strongest redistribution away from the highest-confidence region.

across confidence bins relative to the initial student.

Under fixed-rollout rescoring, where all models are evaluated on the same prefixes and target tokens, Bernoulli OPD produces the largest redistribution away from the highest-confidence region. GC-OPD shows a similar but weaker trend, while Vanilla OPD, SuRe, and PowerOPD induce substantially smaller changes. Because the evaluated states and tokens are fixed, this effect cannot be explained solely by differences in generated trajectories.

The same pattern becomes even more pronounced under on-policy generation. Bernoulli OPD substantially reduces the concentration of tokens in the pS∈[0.9,1.0]p_S \in[0.9, 1.0] bin and redistributes probability mass toward lower confidence regions. Together, these results suggest that the different gradient allocation of Bernoulli OPD leads to a measurable change in the confidence behavior of the trained model. Importantly, restoring gradients to high-confidence tokens does not imply increasing their confidence: teacher corrections can instead reduce excessively confident predictions, consistent with the redistribution observed here.

Answer-level calibration. We next examine whether the changes in confidence induced by different objectives translate into meaningful uncertainty at the answer level. On MATH500, we sample 8 responses per problem and define the confidence of the predicted answer as the fraction of samples belonging to its majority answer cluster. Figure 5 compares each method with Vanilla OPD in terms of majority-answer accuracy and Expected Calibration Error (ECE). Bernoulli OPD exhibits the most favorable point-estimate trade-off, suggesting that Bernoulli OPD’s confidence redistribution does not come at the expense of answer-level calibration; instead, it is accompanied by both higher answer consistency accuracy and lower calibration error.

Figure 5. Answer-level accuracy–calibration trade-off on MATH500. Changes in majority-answer accuracy and ECE relative to Vanilla OPD with eight samples per problem in MATH500.

Policy movement. We further examine whether the improvements from Bernoulli OPD can be explained by globally imitating the teacher’s next-token distribution. Figure 6 measures policy movement on fixed Base rollouts across two teacher–student settings. We compare the drift from the initial student policy with the corresponding change in Top-20 vocabulary overlap with the teacher. Across both settings, Bernoulli OPD substantially changes the student policy without a comparable increase in global teacher overlap. In contrast, Vanilla OPD and SuRe exhibit stronger increases in teacher overlap despite achieving weaker downstream performance. These results suggest that Bernoulli OPD’s gains do not require a corresponding increase in Top-20 vocabulary overlap with the teacher.

Figure 6. Policy movement after OPD training. On fixed student rollouts, we measure policy drift from the initial student and the corresponding change in Top-20 teacher vocabulary overlap. Bernoulli OPD induces substantial policy movement with little or no increase in global teacher overlap. For compactness, Qwen3-4B denotes Qwen3-4B-Instruct-2507, while DS-7B and DS-1.5B denote DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Qwen-1.5B, respectively.

Training dynamics. We finally examine how accuracy and generation length evolve during training in our primary Qwen3-4B-Instruct-2507 →\to Qwen3-1.7B setting. Figure 7 compares Vanilla OPD, GC-OPD, and Bernoulli OPD on AIME24 using the same evaluation protocol. All three methods exhibit a sharp but transient increase in response length early in training, accompanied by a substantial rise in truncation rate. Response length and truncation then steadily decrease as training continues.

Figure 7. Training dynamics of different OPD objectives. We report (a) AIME24 Avg@8, (b) mean response length, and (c) the fraction of responses truncated at the 32,768-token generation limit across training checkpoints.

Importantly, accuracy does not simply track response length. Bernoulli OPD maintains higher AIME24 validation accuracy over most later checkpoints even after its response length and truncation rate have fallen substantially. This suggests that its performance gain cannot be explained solely by generating longer responses, but reflects a different learning trajectory induced by the objective.

Conclusion

We show that sampled reverse-KL OPD systematically suppresses updates to high-confidence tokens, even though these tokens can still provide useful learning signal. We introduce Bernoulli OPD, which replaces the log-ratio surrogate with a Bernoulli cross-entropy objective whose sampled-token logit gradient directly reflects the bounded teacher–student probability difference. Across four teacher–student settings, Bernoulli OPD consistently improves over vanilla OPD. Our controls further show that the gain is not explained simply by a larger update or by enforcing more uniform gradients across confidence regions. Overall, our results highlight the importance of separating token informativeness from the gradient geometry of the distillation objective.

Impact Statement

This work improves the efficiency of transferring mathematical reasoning to smaller language models. While effective distillation can reduce deployment costs, it may also propagate teacher errors or undesirable behavior. Our analysis helps interpret such transfer, and all experiments use public mathematical datasets without sensitive data.

Additional Implementation Details

Training Setup

Unless otherwise specified, the following training settings apply to our unified experiments. All experiments are conducted on four NVIDIA GH200 GPUs. Two GPUs are used for actor training, while the remaining two are used for rollout generation and teacher inference.

Training prompts are sampled from DAPO-Math-17K. For the Qwen3-1.7B settings, generation is performed in non-thinking mode. For the DeepSeek-R1-Distill-Qwen-1.5B settings, we retain the default thinking behavior of the reasoning models. For each prompt, we generate one student response at temperature 1.0 with a maximum generation length of 12,000 tokens. Training uses a global batch size of 64 and Adam with a constant learning rate of 10−610^{-6}.

Within each teacher–student setting, all compared methods share the same student initialization, rollout prompts, generation configuration, and response masking rules unless otherwise specified. Each method generates its own on-policy trajectories and obtains teacher scores on the corresponding student-generated tokens. The unified experiments additionally share the same loss normalization, optimizer, and training schedule. PowerOPD results using its reference training recipe are described separately in Appendix § 8.§ 4.

Evaluation and Checkpoint Selection

We evaluate checkpoints every 20 training updates on AIME24. Checkpoint selection uses Avg@8 with temperature 0.7, top-p 0.8, top-k 20, and a maximum response length of 32,768 tokens. The selected checkpoint is the one with the highest AIME24 Avg@8.

Final evaluation uses the same decoding parameters. We report Avg@32 on AIME24, AIME25, HMMT25, and AMC23, and Avg@4 on MATH500. For the Qwen3-8B-DAPO-RL teacher setting, the maximum response length is increased to 38,912 tokens because the resulting student generations are substantially longer. Final-answer accuracy is computed using the same mathematical answer extraction and grading pipeline across all methods.

Balanced Bernoulli OPD

For Balanced Bernoulli OPD, valid response tokens are partitioned into 10 student-confidence bins. We use β=0.5\beta= 0.5 and clip the normalized detached bin weights to [1/5,5][1/5, 5]. The weighting coefficients are computed from aggregate Bernoulli gradient mass within each confidence bin, as defined in § 5.3. Balanced Bernoulli OPD is included as an exploratory mechanism control for studying explicit confidence-level rebalancing rather than as a fully tuned primary method.

PowerOPD Reference Recipe and Controlled Comparison

We independently implement PowerOPD ([18]) within our training framework. We first evaluate it under the same unified training setup used by the other objectives, so that the distillation objective is the main factor being changed. We then repeat the experiments using the reference PowerOPD training recipe.

Table 5 summarizes the main configuration differences. We report only settings that differ materially between the two recipes; both configurations use Adam, gradient clipping of 1.0, one response per prompt, DAPO-Math-17K, and α=5\alpha= 5. The same four-GH200 hardware allocation is used for both configurations.

SettingUnifiedReference
Optimizer updates2001,500
Learning rate1×10−61 \times10^{-6}5×10−75 \times10^{-7}
Warmup05%
Adam (β1,β2)(\beta_1, \beta_2)(0.9, 0.98)(0.9, 0.999)
Weight decay0.10
Global batch size6432
Max. rollout length12,0001,024
Sampling (T,p,k)(T, p, k)(1.0, 1.0, −1)(1.0, 0.95, 20)

Table 5. Key training differences between the unified and PowerOPD reference recipes. Only settings that differ materially between the two configurations are shown.

Table 6 compares the resulting downstream performance. The reference recipe outperforms the unified configuration in three of the four teacher–student settings, including our primary Qwen3-4B-Instruct-2507 setting, while the unified configuration performs better with Skywork-OR1-Math-7B. To ensure a competitive and conservative baseline comparison, we report the higher five-benchmark average from the two configurations for each teacher–student setting in Table 2. This avoids underestimating PowerOPD due to the choice of training configuration, while the unified results also provide a controlled comparison under the same optimization setup as our other methods.

TeacherUnifiedReferenceΔ\Delta
Qwen3-4B-Instruct-250745.4247.67+2.25
Qwen3-8B-DAPO-RL57.8058.16+0.36
DS-R1-Distill-Qwen-7B44.0944.22+0.13
Skywork-OR1-Math-7B47.9044.24−3.66

Table 6. PowerOPD under the unified and reference training recipes. We report the five-benchmark average in each teacher–student setting. Δ\Delta denotes Reference minus Unified. Bold values indicate the results used in the main comparison.

For completeness, the PowerOPD objective used in both configurations is

rtPower=sg⁡(pTα−pSα),(16)r_t^{\mathrm{Power}} = \operatorname{sg}\left(p_T^{\alpha} - p_S^{\alpha}\right), \tag*{(16)}

with

LPower,t=−rtPowerlog⁡pS,(17)\mathcal{L}_{\mathrm{Power},t} = -r_t^{\mathrm{Power}} \log p_S, \tag*{(17)}

where α=5\alpha= 5 in both configurations.

Answer-Level Calibration Details

For the answer-level calibration analysis in § 5.5, we sample eight responses per MATH500 problem. We extract and normalize the final mathematical answer from each response and cluster answers by mathematical equivalence using exact matching together with the MathD and SymPy graders. Responses for which no valid answer can be extracted are assigned to a single INVALID cluster. The largest answer cluster is treated as the model prediction, and its relative frequency among the eight samples defines the answer-level confidence. If multiple clusters are tied for the largest size, we select the cluster containing the earliest generated response.

We compute Expected Calibration Error (ECE) using 10 equal-width bins over [0,1][0,1]:

ECE=∑bnbN∣conf⁡(b)−acc⁡(b)∣,(18)\mathrm{ECE} = \sum_b \frac{n_b}{N} \left|\operatorname{conf}(b) - \operatorname{acc}(b)\right|, \tag*{(18)}

where nbn_b is the number of problems in bin bb, and conf⁡(b)\operatorname{conf}(b) and acc⁡(b)\operatorname{acc}(b) denote its mean answer-level confidence and empirical accuracy, respectively. Because each problem has eight samples, the confidence values are restricted to multiples of 1/81/8. This quantity should be interpreted as sampling-based answer consistency rather than the model’s exact probability assigned to the correct answer.

Numerical stability.

We compute the Bernoulli objective in FP32 and clamp student and teacher probabilities to [ϵ,1−ϵ][\epsilon, 1-\epsilon], where ϵ=10−6\epsilon= 10^{-6}. To avoid numerical instability near pS=1p_S = 1, we evaluate the complementary log-probability using log⁡1p(−p~S)\log1p(-\tilde{p}_S), where p~S=clip⁡(pS,ϵ,1−ϵ)\tilde{p}_S = \operatorname{clip}(p_S,\epsilon,1-\epsilon). The resulting implementation is

LBern=−p~Tlog⁡pS−(1−p~T)log⁡1p(−p~S),\mathcal{L}_{\mathrm{Bern}} = -\tilde{p}_T \log p_S - (1-\tilde{p}_T)\log1p(-\tilde{p}_S),

where p~T\tilde{p}_T is the clipped, detached teacher probability. Balanced Bernoulli OPD uses the same stable objective with an additional detached balancing weight.

Additional Analysis of Teacher-Dependent Gains

Controlled Comparison on Fixed Student Rollouts

To isolate teacher-dependent effects from differences in student trajectories, we compare Qwen3-4B-Instruct-2507 and Qwen3-8B-DAPO-RL on exactly the same Qwen3-1.7B student rollouts, comprising 585,751 sampled tokens. Because the visited states and sampled tokens are fixed, this comparison removes rollout variation and changes only the teacher supervision. Table 7 summarizes the resulting token-and sequence-level statistics.

TeacherMean ∣pT−pS∣\lvert p_T-p_S\rvertHC Vanilla MassHC Bernoulli MassBernoulli AUROC
Qwen3-4B-Instruct-25070.05972.95%48.60%0.704
Qwen3-8B-DAPO-RL0.05352.75%46.60%0.690

Table 7. Teacher signals on identical student rollouts. HC denotes the highest student-confidence region (pS∈[0.9,1.0]p_S \in[0.9,1.0]). Bernoulli AUROC measures how well the sequence-level Bernoulli teacher signal distinguishes correct from incorrect rollouts.

Outcome Alignment of Teacher Corrections

We next ask whether the teacher corrections emphasized by Bernoulli OPD are related to rollout quality. We evaluate both teachers on the same 120 fixed Base rollouts from AIME24, containing 17 correct and 103 incorrect responses. For each response ii, we define a sequence-level Bernoulli correction score as

si(T)=1∣Mi∣∑t∈Mi[pT(yi,t∣ci,t)−pS(yi,t∣ci,t)],(19)s_i^{(T)} = \frac{1}{|\mathcal{M}_i|}\sum_{t\in\mathcal{M}_i}\left[p_T(y_{i,t}\mid c_{i,t}) - p_S(y_{i,t}\mid c_{i,t})\right], \tag*{(19)}

where Mi\mathcal{M}_i contains valid response-token positions after applying the loss mask and excluding special tokens. Both teachers are evaluated on the same student-generated tokens using teacher-forced scoring. The score averages signed probability differences over valid tokens, including the observed tokens of truncated responses.

We measure how well this score distinguishes correct from incorrect responses using AUROC, with rollout correctness as the positive label. The resulting AUROC is 0.704 for Qwen3-4B-Instruct-2507 and 0.690 for Qwen3-8B-DAPO-RL. Although the difference is small, the numerically higher AUROC for Qwen3-4B-Instruct-2507 is consistent with its larger downstream gain. We do not draw a strong conclusion about teacher differences from this diagnostic alone.

Interpretation

These results point to two separate factors. The loss determines how much of the available teacher signal can affect the student, while the teacher determines how useful that signal is. Bernoulli OPD reduces the suppression of high-confidence corrections, but the final gain still depends on the quality of the teacher feedback on the states visited by the student.

One possible explanation lies in the different model origins and training histories of the teachers. Although Qwen3-8B-

DAPO-RL undergoes additional reinforcement learning, its base model, Qwen3-8B, belongs to the same model generation as the Qwen3-1.7B student. In contrast, Qwen3-4B-Instruct-2507 comes from a different Qwen3 generation with a potentially different pretraining and post-training history. These differences may affect the nature and usefulness of teacher corrections on student-generated trajectories. A full explanation of this teacher-dependent effect is left for future work.

Additional Details on Gradient-Scale Control

The scale-matched experiment controls for the possibility that Bernoulli OPD improves simply because it induces a larger overall optimization step. Here we provide the construction and calibration details for the control summarized in Table 4.

Scale-Matched Objective

To calibrate the scale-matched control, we evaluate Vanilla and Bernoulli OPD on a shared frozen batch with identical student initialization, rollouts, teacher scores, and response masks, using the same optimizer configuration.

On a shared frozen batch, the full-parameter gradient ℓ2\ell_2 norms of vanilla and Bernoulli OPD are 8.1393 and 9.9926, respectively. We therefore define

c=∥gVanilla∥2∥gBern∥2=0.8145,(20)c = \frac{\lVert g_{\mathrm{Vanilla}}\rVert_2}{\lVert g_{\mathrm{Bern}}\rVert_2}=0.8145, \tag*{(20)}

and optimize

LMatched=cLBern.(21)\mathcal{L}_{\mathrm{Matched}} = c\mathcal{L}_{\mathrm{Bern}}. \tag*{(21)}

The coefficient is calibrated once at initialization and remains fixed throughout training. Because this is a positive global rescaling, it changes the overall gradient magnitude while leaving the relative allocation of gradients across tokens unchanged.

One-Step Optimization Calibration

Matching the raw gradient norm does not necessarily imply a matched optimizer update, particularly under Adam. We therefore additionally measure the one-step change in the student policy after applying the optimizer update.

Specifically, starting from the same student parameters, we apply one optimizer step and measure the mean absolute change in sampled-token log probability,

M=Et[∣log⁡pθ+(yt∣ct)−log⁡pθ(yt∣ct)∣],(22)M = \mathbb{E}_t\left[\left\lvert\log p_{\theta^+}(y_t \mid c_t) - \log p_{\theta}(y_t \mid c_t)\right\rvert\right], \tag*{(22)}

where θ\theta and θ+\theta^+ denote the parameters before and after the update.

As reported in Table 4, scale matching makes both the full-model gradient norm and the resulting one-step policy movement nearly identical to vanilla OPD. At the same time, the high-confidence region retains the substantially larger share of gradient mass induced by the Bernoulli objective. Thus, the control matches global optimization magnitude without removing the confidence-dependent redistribution that distinguishes Bernoulli OPD.

Training the Scale-Matched Variant

We train the scale-matched variant in the primary Qwen3-4B-Instruct-2507 →\to Qwen3-1.7B setting. The coefficient c=0.8145c=0.8145 is fixed throughout training, with no per-step normalization or other change to the optimization configuration. Checkpoint selection follows the same AIME24 Avg@8 protocol as the main experiments.

Table 4 shows that matching the global update scale does not remove the downstream advantage of Bernoulli OPD. This rules out a larger initial gradient norm or larger one-step policy movement as the sole explanation for the observed improvement.

Interpretation

The scale-controlled experiment separates the overall magnitude of optimization from its allocation across tokens. After matching vanilla OPD’s global update scale, Bernoulli OPD preserves both its distinct confidence-dependent gradient allocation and its downstream advantage. This supports the interpretation that Bernoulli OPD changes where optimization is allocated rather than merely increasing how much optimization is applied.

Scale-matched Bernoulli slightly outperforms standard Bernoulli OPD in this run. We do not interpret this difference as evidence that scale matching is systematically beneficial, since the experiment uses a single training seed and requires an additional calibration step. We therefore treat scale matching strictly as a mechanism control rather than as a proposed training method.

Robustness Across Training Seeds

Experimental Protocol

We additionally evaluate the robustness of our primary comparison to training randomness. In the Qwen3-4B-Instruct-2507 →\to Qwen3-1.7B setting, we repeat both Vanilla OPD and Bernoulli OPD with three independent training seeds. For reference, we also evaluate the initial Qwen3-1.7B student three times using independent stochastic decoding runs.

For every trained model, checkpoint selection is performed independently using the same AIME24 Avg@8 protocol as in the main experiments. The first four final-evaluation benchmarks use Avg@32, while MATH500 uses Avg@4.

Results

Table 8 shows that both Vanilla OPD and Bernoulli OPD exhibit small run-to-run variation. The five-benchmark average of Vanilla OPD is 44.44±0.1644.44 \pm0.16, whereas Bernoulli OPD achieves 51.62±0.1851.62 \pm0.18. The approximately seven-point improvement of Bernoulli OPD is therefore substantially larger than the observed training-seed variation and is reproduced consistently across all three independent training runs.

MethodSeedStepAIME24AIME25HMMT25AMC23MATH500Avg.
Initial student: independent evaluation runs
Qwen3-1.7B Init.42012.089.795.1047.5070.6529.02
Qwen3-1.7B Init.43013.968.546.6744.5370.6028.86
Qwen3-1.7B Init.44014.279.585.6343.2870.8528.72
Mean––13.449.305.8045.1070.7028.87
Std.––1.190.670.802.170.130.15
Vanilla OPD: independent training seeds
Vanilla OPD123411933.9621.7715.0068.3682.8044.38
Vanilla OPD56787935.4220.9615.1467.5582.6044.33
Vanilla OPD1357911934.9222.0815.7668.0682.3044.62
Mean––34.7721.6015.3067.9982.5744.44
Std.––0.740.580.400.410.250.16
Bernoulli OPD: independent training seeds
Bernoulli OPD12343939.5833.0219.9078.6787.6551.76
Bernoulli OPD56783940.7333.3318.5478.9186.9551.69
Bernoulli OPD135793940.2134.4818.9676.4886.9551.42
Mean––40.1733.6119.1378.0287.1851.62
Std.––0.580.770.701.340.400.18

Table 8. Robustness across training and evaluation seeds in the primary Qwen3-4B-Instruct-2507 → Qwen3-1.7B setting. For Vanilla OPD and Bernoulli OPD, each row corresponds to an independent training seed and its independently selected checkpoint. For the initial student, the three rows correspond to independent evaluation runs of the same model. Mean and standard deviation are computed across the three runs.

The individual benchmark results are also broadly consistent across training seeds. In addition, final evaluation averages multiple stochastic generations per problem, which reduces

variance from stochastic decoding. Given the computational cost of repeating every combination of method and teacher, and the small training-seed variability observed in this controlled primary comparison, we report a single training run for the remaining experiments unless otherwise stated.

References

  1. [1]Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos, S., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes, 2024. URL https://arxiv.org/abs/2306.13649.
  2. [2]Balunović, M., Dekoninck, J., Petrov, I., Jovanović, N., and Vechev, M. Matharena: Evaluating llms on uncontaminated math competitions, 2026. URL https://arxiv.org/abs/2505.23281.
  3. [3]Cao, L., Zou, Y., Peng, C., Chen, R., Ning, W., and Li, Y. Step guided reasoning: Improving mathematical reasoning using guidance generation and step reasoning, 2025. URL https://arxiv.org/abs/2410.19817.
  4. [4]Ding, Y. and Zhang, R. Does on-policy distillation really distill? from noisy teacher to self-improvement, 2026. URL https://arxiv.org/abs/2608.31046.
  5. [6]Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network, 2015. URL https://arxiv.org/abs/1503.02531.
  6. [7]Jia, M. Aime problem set 2024, 2024. URL https://huggingface.co/datasets/Maxwell-Jia/AIME_2024.
  7. [8]Jin, W., Min, T., Yang, Y., Wei, D., Zhou, Y., Kadhe, S. R., Baracaldo, N., and Lee, K. Entropy-aware on-policy distillation of language models, 2026. URL https://arxiv.org/abs/2603.07079.
  8. [9]Kim, Y. and Rush, A. M. Sequence-level knowledge distillation, 2016. URL https://arxiv.org/abs/1606.07947.
  9. [10]Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step, 2023. URL https://arxiv.org/abs/2305.20050.
  10. [12]Meng, L., Gan, S., Li, Y., Wang, X., Ding, Z., Gou, D., Wu, Y., and Zhao, Z. Unbiased top-k estimation for on-policy distillation, 2026. URL https://arxiv.org/abs/2609.34447.
  11. [13]Shao, B., Zhang, J., Ma, L., Shen, Y., Jin, S., Guo, X., Yang, Y., Chai, M., Xi, Z., Liu, B., Shang, J., Wang, P., Zhang, Q., Gui, T., Zhang, Q., and Huang, X. A token-level analysis of sampled-token reverse-kl on-policy distillation, 2026. URL https://arxiv.org/abs/2608.25643.
  12. [14]Song, M. and Zheng, M. A survey of on-policy distillation for large language models, 2026. URL https://arxiv.org/abs/2604.00626.
  13. [15]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q., Men, R., Gao, R., Liu, S., Luo, S., Li, T., Tang, T., Yin, W., Ren, X., Wang, X., Zhang, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Zhang, Y., Wan, Y., Liu, Y., Wang, Z., Cui, Z., Zhang, Z., Zhou, Z., and Qiu, Z. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  14. [16]Yang, L., Zhou, X., Li, X., Qiao, L., Li, Z., Yang, Z., Wang, G., and Li, X. Bridging cross-task protocol inconsistency for distillation in dense object detection, 2024. URL https://arxiv.org/abs/2308.14286.
  15. [18]Zhao, A., Tong, J., Fan, Y., Nie, P., Li, W., and Shen, X. Powerpod: Stabilizing on-policy distillation with bounded power transformation, 2026. URL https://arxiv.org/abs/2606.17199.
  16. [19]Zhao, B., Cui, Q., Song, R., Qiu, Y., and Liang, J. Decoupled knowledge distillation, 2022. URL https://arxiv.org/abs/2203.08679.

Paper details

vlnc:W_dd8k8sad
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents