WorksComputer Science
OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
Abstract
Reverse and forward Kullback–Leibler (KL) divergences provide complementary supervision in on-policy distillation: reverse KL refines the student's active token rankings, and forward KL allocates mass to teacher-supported candidates. At student-generated prefixes, we form a four-observation matrix by treating the student and teacher as both candidate sources and evaluators. The diagonal self-observations measure uncertainty, while source-centered cross-observations recover the two KL objectives. We establish an exact identity equating the Fisher inverse-metric coupling of their logit gradients with the sum of these contrasts, yielding a common local descent direction for distinct positive distributions. Observation-coordinated on-policy distillation (OC-OPD) uses observations to set preferences, closed-form bargaining to coordinate gradients, and distributional discrepancy to scale the update. Across three heterogeneous student settings (non-thinking generation, thinking generation, and base-model initialization), OC-OPD achieves strong gains in mathematical reasoning and transfers effectively to out-of-distribution scientific question answering and code generation. Our code is publicly available at: https://github.com/RuoranXu/OC-OPD.
Introduction
Earlier knowledge distillation methods use KL divergences to transfer predictive structure from a teacher to a student [1, 2]. Adaptive KL (AKL) adjusts the early-training balance between forward and reverse KL according to distributional gaps within teacher-defined high-probability head and long-tail regions [3].
On-policy distillation (OPD) trains a student at its own generated prefixes, where a frozen teacher provides supervision for the same next-token decision [4, 5]. At each prefix, the aligned student distribution and teacher distribution define two complementary objectives. Reverse KL (RKL), , is associated with mode-seeking refinement of student-supported candidates, while forward KL (FKL), , provides mode-covering supervision toward teacher-supported alternatives [5, 6]. As the student changes, so do the visited prefixes and the conditional predictions of both models. How should these corrections be coordinated within one update at the current teacher–student state?
Recent OPD methods adapt supervision using uncertainty, token importance, teacher–student mismatch, and divergence routing [7, 8, 9, 10, 11]. At a visited prefix, two aspects of the predictive state inform this adaptation: each model’s self-uncertainty and the disagreement between their predictions. Self-uncertainty describes the dispersion of a model’s candidate probabilities; disagreement describes the mismatch requiring correction. Connecting these quantities to the two KL gradients provides a state-dependent basis for their relative influence and joint improvement.
We place these signals in a common source–evaluator system: the student and teacher each supply candidate mass, and either model can evaluate it. The resulting four-observation matrix separates self-uncertainty, represented by its diagonal entropies, from directed disagreement, recovered by centering its cross-entropies on the corresponding source entropies. These contrasts yield the refinement and coverage objectives at the same student-visited state.
This representation also links observations to correction geometry. For positive , , the reverse- and forward-KL logit gradients satisfy , where is the categorical Fisher matrix. For , this positive cross-observation excess establishes a common local descent direction for both KL objectives in student-logit space.
We develop observation-coordinated on-policy distillation (OC-OPD) by connecting the observed teaching state to three decisions: preference, direction, and strength. Self-entropies set preferences between the two corrections, a closed-form bargaining solution coordinates their normalized gradients within the common-descent region, and symmetric distributional discrepancy scales supervision strength. These components connect self-uncertainty, correction geometry, and disagreement in a single bounded logit update, applied with one student backward pass. Figure 1 summarizes the observation contrast coordination strength pipeline.
Figure 1. Training pipeline of OC-OPD. (a) Student rollouts provide prefixes at which the student and frozen teacher produce aligned next-token distributions. (b) Four self- and cross-observations at the shared prefix yield source-centered KL gradients for ranking/refinement and coverage. (c) Self-observation values set uncertainty-based priorities for closed-form gradient bargaining, while distributional discrepancy controls supervision strength. (d) The resulting bounded logit signal updates the student through a single backward pass. The updated student then generates new rollouts.
We evaluate OC-OPD on mathematical reasoning in three Qwen3 student settings [12], all trained with fresh student rollouts: non-thinking students at 1.7B and 4B scales, thinking-generation students, and a weaker base-model initialization. In the main non-thinking setting, the 1.7B student exceeds the strongest compared baseline by +6.88 Avg@32 and +11.11 Pass@32 points on average across AIME24–26; the 4B student exceeds EOPD [8] by +2.74 and +3.33 points, respectively (Table 1). Gains persist in the other two settings, and GPQA-Diamond [13] and MBPP+ [14] assess transfer to scientific question answering and code generation.
| Methods | AIME 2024 Avg@32 | AIME 2024 Pass@32 | AIME 2025 Avg@32 | AIME 2025 Pass@32 | AIME 2026 Avg@32 | AIME 2026 Pass@32 | MATH500 Avg@8 | MATH500 Pass@8 | OlympiadBench Avg@32 | OlympiadBench Pass@32 | GPQA Avg@32 | MBPP+ |
| Teacher: Qwen3-30B-A3B-Instruct-2507 | ||||||||||||
| Qwen3-1.7B | 14.16 | 43.33 | 9.90 | 30.00 | 7.71 | 23.33 | 71.88 | 52.67 | 29.59 | 56.03 | ||
| + GRPO | 34.17 | 70.00 | 25.52 | 46.67 | 26.04 | 50.00 | 82.21 | 62.57 | — | — | ||
| + OPD | 31.98 | 73.33 | 20.73 | 50.00 | 23.54 | 46.67 | 85.84 | 60.77 | — | — | ||
| + OPSD | 33.33 | 73.33 | 22.60 | 53.33 | 21.35 | 56.67 | 86.20 | 64.84 | — | — | ||
| + OC-OPD | 41.67 | 76.67 | 34.38 | 63.33 | 30.31 | 76.67 | 87.40 | 67.40 | 34.01 | 59.01 | ||
| Qwen3-4B | 23.85 | 56.67 | 22.40 | 56.67 | 15.10 | 53.33 | 79.20 | 69.73 | 39.60 | 65.55 | ||
| + OPD (RKL) | 61.04 | 86.67 | 51.04 | 76.67 | 52.19 | 80.00 | 87.40 | 73.04 | 43.43 | 64.29 | ||
| + OPD (FKL) | 55.42 | 83.33 | 53.75 | 73.33 | 54.17 | 83.33 | 85.20 | 75.65 | 40.91 | 65.55 | ||
| + EOPD | 60.10 | 86.67 | 53.13 | 83.33 | 54.69 | 83.33 | 92.80 | 78.93 | 45.45 | 64.81 | ||
| + OC-OPD | 60.94 | 93.33 | 56.67 | 83.33 | 58.54 | 86.67 | 94.43 | 80.86 | 46.80 | 66.01 | ||
| Δ vs. EOPD | +0.84 | +6.66 | +3.54 | +0.00 | +3.85 | +3.34 | +1.63 | +1.93 | +1.35 | +1.20 |
Table 1. Results for non-thinking students (accuracy %). Shaded rows denote OC-OPD. Dashes indicate scores not present in the supplied results. Δ reports the Qwen3-4B gain over EOPD in percentage points, computed from the displayed scores.
Our contributions are threefold. (1) We unifiedly describe token-level OPD through a source–evaluator system whose self- and cross-observations characterize the teaching state and whose source-centered contrasts recover the two KL corrections. (2) We prove a Fisher inverse-metric identity linking those contrasts to their logit gradients and establish common local descent for every distinct positive teacher–student pair. (3) We develop OC-OPD to determine the preference, direction, and strength of token-level supervision, and demonstrate its effectiveness across heterogeneous student settings and out-of-domain evaluations.
Related Work
KL directions in on-policy distillation. Earlier knowledge distillation methods transfer predictive structure from a teacher to a student [1, 2]. AKL adaptively combines forward and reverse KL using distributional gaps within teacher-defined head and tail regions, motivated by their different fitting priorities in early training [3]. A larger head gap increases the forward-KL weight, while a larger tail gap increases the reverse-KL weight. OPD evaluates teacher guidance at student-generated prefixes [4, 5], where both models produce aligned next-token distributions. GKD studies flexible divergence objectives at these states [5], and MiniLLM develops reverse-KL distillation under student sampling [6]. Related methods explore skewed divergences and adaptive sampling [15], contrastive teacher–student responses [16], and interleaved generation [17]. Reverse KL refines relative probabilities among student-supported candidates, while forward KL transfers mass toward teacher-supported candidates. EOPD conditions forward-KL supervision on teacher entropy [8], and SEAD adjusts the emphasis of the two KL directions across tokens [9]. We study how the current teacher–student state relates to these corrections: source-centering separates self-uncertainty from directed disagreement, and an exact observation–gradient identity establishes their joint local compatibility.
Observation-based adaptive supervision. At a shared prefix, either model can supply candidate mass and either model can evaluate it. These choices produce four observations: two self-entropies describe the predictive states of the student and teacher, and two cross-entropies combine source uncertainty with directed cross-model disagreement. Related methods use local information to determine how supervision is assigned. REPOPLD uses entropy-based token selection and relaxed reward shaping [7]; SMOPD selects informative positions for self-distillation [18]; TIP combines student uncertainty with divergence to assess token importance [10], while TIDE characterizes student-excess and student-deficit mismatch [11]. Region-dependent objectives, adaptive targets, bounded transformations, and trust-region control further shape local supervision [19, 20, 21, 22]. Our source–evaluator matrix connects these signals through the entropy decomposition of KL: source-centering isolates the two directed disagreements, whose gradients supply the refinement and coverage corrections. Self-entropies set their relative preferences, gradient geometry coordinates their direction, and distributional discrepancy sets supervision strength. Appendix § 14 summarizes representative methods in this notation.
Gradient coordination. Multi-objective optimization derives shared update directions via minimum-norm descent, gradient projection, conflict-averse optimization, and bargaining. Our KL gradients stem from aligned student–teacher distributions and source-centered observation contrasts, whose local coupling follows an exact Fisher-geometric identity. We leverage this identity to coordinate candidate-level refinement and coverage, adopting bargaining to obtain an observation-conditioned closed-form update direction. Scalarization and normalized gradient mixing serve as baselines for comparison.
Preliminaries and Motivation
This section formalizes shared-context OPD, organizes the aligned student and teacher distributions into four observations, and shows how their source-centered contrasts recover the ranking and coverage signals of reverse and forward KL.
Shared-context OPD as observation system
OPD evaluates supervision along student-generated responses by querying the teacher at the same prefixes [5, 6]. We distinguish this choice of training states from the information available at each visited state. In the shared-context setting, both models condition on the same prompt and student prefix, yielding an aligned pair of next-token predictive distributions. Our aim is to organize the information in this pair into effective supervision.
For a prompt , let and denote the student and frozen teacher. A representative reverse-KL OPD objective is
Student-sampled log ratios estimate this objective. At iteration , write and collect valid response positions from fresh student rollouts. A token-local surrogate assigns each sampled token a teacher-to-student log-ratio credit:
where and sg denotes stop-gradient. Prefixes and credits are fixed during differentiation. Full-vocabulary implementations evaluate the corresponding expectation over candidates at each collected prefix [5]. We use this objective as a reference and derive the four observations directly from the aligned distributions, making student-sourced and teacher-sourced supervision explicit.
Four observations from two distributions
At a collected prefix , write and over an aligned vocabulary. The fixed pair and supplies two candidate sources—student and teacher—and two evaluators—student and teacher. Pairing each source with each evaluator yields four expected observations. For and , define
Rows index evaluators and columns index candidate sources, both ordered as student, teacher. Accordingly, records the student evaluating student-sourced mass, the student evaluating teacher-sourced mass, the teacher evaluating student-sourced mass, and the teacher evaluating teacher-sourced mass. The diagonal self-observations are the student and teacher entropies and quantify distributional dispersion. The off-diagonal cross-observations measure how well one model accommodates candidates emphasized by the other.
The matrix provides a relational reparameterization of the same pair by separating the distribution supplying candidate mass from the distribution scoring it. Teacher-sourced expectations use the teacher distribution at the same student-visited prefix. The matrix records state-level expectations, while the underlying and retain the candidate-level resolution needed for an update. It gathers common adaptive OPD signals in one object: student entropy is , teacher entropy is , and teacher–student mismatch appears as the off-diagonal excess over the corresponding self-observation. Table 6 in appendix § 14 compares how different methods are described in this framework.
| Method | M₁₁ | M₁₂ | M₂₁ | M₂₂ |
| REOPOLD | • | – | – | – |
| TIP | • | – | – | – |
| EOPD | – | – | – | • |
| SEAD | • | – | – | • |
| OC-OPD (Ours) | • | • | • | • |
Table 6. Comparison of entropy and token-selection-based OPD methods under our four-observation framework. • denotes an observation directly used for token selection or divergence routing, while ◦ denotes an observation involved only through a derived KL contrast.
Source-centered contrasts reveal ranking and coverage
Centering each column on its candidate source’s self-observation isolates the difference between the two evaluators:
Thus reverse and forward KL arise as centered cross-observations from the student-sourced and teacher-sourced columns, respectively.
Student-sourced contrast: rank and refine active support. For reverse KL, the token-level credit compares teacher support with the student’s current allocation. Since exactly when , this credit ranks student-supported candidates by relative underallocation. The student-sourced contrast thereby refines the alternatives the student already explores. Its logit gradient is weighted by , concentrating the correction on candidates that carry student probability.
Teacher-sourced contrast: recover underrepresented support. Forward KL evaluates teacher-sourced candidates under the student. Its logit gradient directly reallocates mass toward teacher-supported alternatives, including those with very small student probability. This coverage correction allocates student capacity across the teacher-supported set [6]. In our full-vocabulary construction, the teacher-sourced gradient is evaluated directly at the visited prefix, yielding the exact candidate-level correction from the aligned distributions.
From complementary contrasts to a coordination problem. The matrix thus provides both statistics describing the current state and contrasts specifying candidate-level corrections. Existing entropy gates, token filters, and mismatch criteria can be interpreted as selecting or weighting particular regions of this object [7, 8, 9, 10, 11]. Together, the source-centered contrasts invite a geometric account of when their ranking/refinement and coverage gradients share a descent direction. § 4 establishes this compatibility; § 5 uses self-observation, compatibility, and discrepancy to determine preference, direction, and magnitude.
Theory: From Observations to Compatible Gradients
The matrix view yields two source-centered corrections whose candidate-level gradients can be studied together. Their ranking/refinement and coverage directions admit common descent, with the observation matrix providing an explicit certificate of compatibility.
From observation entries to candidate-level gradients
For current-state coefficients , define the local matrix loss
Writing the coefficient matrix as , its columns preserve each candidate source’s self-observation as a baseline. The resulting contrasts are and ; Appendix § 8.§ 1 derives these identities from the observations.
Differentiating the matrix with respect to the current student logits yields
The student self-observation and both cross-observations supply direct derivatives. The frozen teacher’s self-observation has and conditions the coefficients through its value. Thus all four entries contribute to the update. § 5 specifies the coefficients; Appendix § 8.§ 1 gives the entrywise derivatives and parameter update.
These derivatives make the complementary roles in § 3.3 explicit. For a fixed positive teacher and , their coordinates satisfy and . The teacher-sourced contrast therefore retains a direct correction for candidates with very small student probability. Coordinating the two directions refines the student’s current allocation while preserving a correction toward teacher-favored alternatives.
Compatibility certified by the observation contrast
Let be the categorical Fisher matrix[23, 24], with Moore–Penrose pseudoinverse . The following identity relates the source-centered gradients through the observation matrix.
Theorem 1 (Observation–gradient identity). For positive , the logit gradients in Eq. (6) satisfy
If , both gradients have positive norm and their Euclidean cosine is strictly greater than . A unit direction therefore exists with and .
The total observation excess certifies a common descent direction for both acute and obtuse Euclidean gradient configurations. The identity applies to every aligned pair in § 3.2. Appendix § 9 gives the proof, and Appendix § 10 extends compatibility to a finite family of differentiable strictly convex -divergences. We use observation values to select a direction within this feasible set and determine the update’s magnitude. The appendix H.2 provides empirical analysis of the theory.
Method: Observation-Coordinated Updates
At each student-visited prefix, OC-OPD follows an observation contrast coordination strength pipeline. The aligned distributions first determine , and column-centering yields the ranking/refinement and coverage contrasts together with their candidate-level gradients. The diagonal self-observations specify preferences between these roles; the compatibility geometry then selects a direction that improves both normalized contrasts. Finally, a symmetric discrepancy between and scales the resulting update.
Algorithm 1 Observation-Coordinated On-Policy Distillation
Require: Frozen teacher , student
Require: Entropy scale , preference floor
Require: Discrepancy scale , learning rate
Require: Training dataset , prompt batch size
for each training iteration do
Set and ; sample prompts .
for each prompt do
Sample .
for to do
, , .
, , .
.
if then
, .
Compute from using .
Compute closed-form bargaining direction from .
, .
Store in .
end for; collect valid prefixes from .
.
Update student to with one backward pass on at learning rate .
Observation-conditioned gradient coordination
For , the diagonal observations determine normalized entropies and positive preferences:
When the teacher is diffuse, this rule gives coverage greater influence; when the teacher is concentrated and the student diffuse, it gives ranking/refinement greater influence. Both sources retain positive preference (Appendix § 8.§ 2).
For , set , , and . Gradient bargaining (Navon et al., [25]) selects a direction by maximizing weighted relative improvement:
Theorem 1 guarantees that the feasible set is nonempty. Its unique solution is
The self-observations thus set priorities, while the current gradient geometry converts them into compatible gains. Equal preferences yield . Appendix § 11 proves the closed form and supplies its numerically stable evaluation; § 6.6 defines comparisons between the adaptive and symmetric directions.
Position strength and realized observation coefficients
Let and for . Expanding the direction and position strength gives the complete logit signal:
These coefficients instantiate in Eq. (5). The derivatives supply the candidate directions, the self-observation values condition their relative influence, and controls the contribution of the position. The coefficients remain fixed during differentiation.
Update properties. The signal obeys . For , its correction satisfies , retaining a positive limiting bound as for a fixed interior teacher. The coordinated direction has positive inner product with both source-centered gradients. Near agreement, discrepancy gating gives cubic attenuation in ; with aligned gradients, the update reduces to (Appendices § 12, § 13, and § 11.§ 1). The component comparisons examine these three aspects of the update: candidate support, direction, and magnitude.
We accumulate the signed surrogate over valid response positions and differentiate it with one student backward pass. Appendix § 15 gives the explicit batch objective, parameter gradient, and numerical implementation.
Experiments
Experimental settings
Models and training data. Students are Qwen3-1.7B and Qwen3-4B, with Qwen3-4B-Base used for the base-initialization study (Qwen Team, [12]). We use Qwen3-30B-A3B-Instruct-2507 and Qwen3-30B-A3B-Thinking-2507 as teachers (Qwen Team, [26]). The main setting pairs the Instruct teacher with non-thinking students; auxiliary settings use the Thinking teacher with thinking students and the Instruct teacher with Qwen3-4B-Base. All methods train on DAPO-Math-17k (Yu et al., [28]) with fresh student rollouts; distillation methods query a frozen teacher at the same student prefixes.
Evaluation benchmarks and metrics. We evaluate AIME24–26, MATH500 (Lightman et al., [29]) and OlympiadBench(math,english) (He et al., [30]) for mathematical reasoning and GPQA-Diamond (Rein et al., [13]) and MBPP+ (Liu et al., [14]) for out-of-distribution transfer. Avg@32 is the mean accuracy over 32 responses per problem and Pass@32 the fraction of problems solved by at least one; OOD columns report Avg@32 in Table 1 and Pass@32 in Table 3. Additional mathematical benchmarks are reported with Avg@8 or Pass@8 as indicated. Variance and statistical-significance analyses for the AIME evaluations are provided in Appendix § 17. We compare with GRPO (Shao et al., [31]), OPD (Agarwal et al., [32]), OPSD (Zhao et al., [33]), EOPD (Jin et al., [8]), and KD (Hinton et al., [1]; Kim and Rush, [2]), as applicable. The 4B non-thinking setting also reports pure RKL and FKL objectives. Within each comparison, methods share initialization, prompts, rollout budget, decoding, and evaluation protocol, and teacher-based methods share the same teacher.
| Methods | Math (I.D.) | O.O.D. | ||||||||
| AIME24 avg@32 | AIME24 pass@32 | AIME25 avg@32 | AIME25 pass@32 | AIME26 avg@32 | AIME26 pass@32 | OlympiadBench avg@8 | OlympiadBench pass@8 | GPQA_D pass@32 | MBPP+ pass@32 | |
| 1.7B | 50.94 | 83.33 | 37.19 | 73.33 | 38.65 | 70.00 | 59.33 | 79.38 | 77.78 | 84.66 |
| + OPD | 48.54 | 86.67 | 40.21 | 73.33 | 42.08 | 73.33 | 58.91 | 79.08 | 79.29 | 83.60 |
| + OC-OPD | 52.81 | 86.67 | 46.46 | 76.66 | 48.85 | 76.67 | 59.70 | 80.12 | 81.82 | 85.45 |
| 4B | 72.71 | 86.67 | 65.52 | 83.33 | 61.04 | 83.33 | 69.08 | 84.91 | 84.34 | 87.04 |
| + OPD | 78.13 | 83.33 | 65.94 | 80.00 | 64.17 | 83.33 | 69.88 | 85.72 | 81.31 | 87.57 |
| + OC-OPD | 80.21 | 93.33 | 66.98 | 86.67 | 66.77 | 90.00 | 71.90 | 88.09 | 85.35 | 88.36 |
Table 3. Thinking-model results (accuracy %). Shaded rows denote OC-OPD.
Non-thinking student results
Performance on mathematical reasoning. Table 1 shows consistent gains for both non-thinking student scales. For Qwen3-1.7B, OC-OPD outperforms GRPO, OPD, and OPSD on all six AIME metrics, beating the strongest baseline by +6.88 Avg@32 and +11.11 Pass@32 points on average. For Qwen3-4B, it exceeds EOPD by +2.74 Avg@32 and +3.33 Pass@32 points across AIME24–26. OC-OPD is best or tied-best in five of six columns, with RKL higher only on AIME24 Avg@32 by 0.10 points. Thus, the gains cover both average accuracy and multi-sample success.
Combining the two KL directions. Neither KL direction dominates at 4B: RKL leads FKL by 5.62 Avg@32 points on AIME24, whereas FKL is stronger on AIME25–26 and keeps MBPP+ at initialization. OC-OPD retains both strengths: it is within 0.10 points of RKL on AIME24, exceeds the stronger one by +2.92 and +4.37 Avg@32 points on AIME25–26, and surpasses both OOD. It also beats the adaptive linear mixture on all six AIME metrics (Table 4), so the gain comes from coordination rather than mixing alone.
| Method | AIME 2024 Avg@32 | AIME 2024 Pass@32 | AIME 2025 Avg@32 | AIME 2025 Pass@32 | AIME 2026 Avg@32 | AIME 2026 Pass@32 |
| OC-OPD | 60.94 | 93.33 | 56.67 | 83.33 | 58.54 | 86.67 |
| Forward KL (FKL) | 55.42 | 83.33 | 53.75 | 73.33 | 54.17 | 83.33 |
| Reverse KL (RKL) | 61.04 | 86.67 | 51.04 | 76.67 | 52.19 | 80.00 |
| Adaptive linear mixture | 60.21 | 86.67 | 53.02 | 80.00 | 53.96 | 80.00 |
| Equal-preference | 58.65 | 86.67 | 53.54 | 80.00 | 56.67 | 83.33 |
| w/o position gate | 60.63 | 90.00 | 54.06 | 80.00 | 56.67 | 83.33 |
Table 4. Ablation study. The student is Qwen3-4B with non-thinking generation and the teacher is Qwen3-30B-A3B-Instruct-2507. Bold denotes the best value in each column.
Out-of-distribution evaluation
Generalization beyond mathematical reasoning. Although trained only on DAPO-Math-17k, OC-OPD also improves out-of-distribution performance (Table 1, last two columns). Relative to its initialization, the 1.7B student gains +4.42 points on GPQA-Diamond and +2.98 points on MBPP+. At 4B, it reaches 46.80 and 66.01, exceeding EOPD by +1.35 and +1.20 points. On MBPP+, OC-OPD stays above initialization, whereas EOPD and OPD (RKL) drop by 0.74 and 1.26 points.
Base-model initialization
Performance from base-model initialization. We further test whether the gains depend on instruction-tuned initialization by distilling into Qwen3-4B-Base. Table 2 shows that OC-OPD reaches 40.00 Pass@8 on both AIME24 and AIME25, improving the base model by +16.67 points and EOPD by +6.67 points on each benchmark. Across sampling budgets (Figure 2, Figure 3), OC-OPD exceeds baseline at four of five tested values of and ties at , so the advantage persists from a substantially weaker initialization.
| Methods | AIME 2024 Avg@8 | AIME 2024 Pass@8 | AIME 2025 Avg@8 | AIME 2025 Pass@8 | OlympiadBench Pass@8 | MATH500 Avg@8 | MATH500 Pass@8 |
| Qwen3-4B-Base | 12.19 | 23.33 | 7.29 | 23.33 | 52.52 | 48.60 | 85.20 |
| + KD | 12.50 | 26.67 | 12.08 | 23.33 | 62.43 | 74.73 | 92.00 |
| + OPD | 18.91 | 33.33 | 12.50 | 30.00 | 58.80 | 78.40 | 90.40 |
| + EOPD | 17.92 | 33.33 | 17.50 | 33.33 | 62.61 | 80.05 | 92.60 |
| + OC-OPD | 21.67 | 40.00 | 18.75 | 40.00 | 63.20 | 81.75 | 93.20 |
Table 2. Avg@8 and Pass@8 performance on AIME24–25, MATH500, and OlympiadBench with the Qwen3-4B-Base student. Shaded rows denote OC-OPD. Bold denotes the best value in each column; dashes indicate unavailable scores.
Figure 2. Pass@k performance comparison between OPD, EOPD and OC-OPD on AIME24 with the Qwen3-4B-Base student.
Figure 3. Pass@k performance comparison between OPD, EOPD and OC-OPD on AIME25 with the Qwen3-4B-Base student.
Thinking teacher and thinking students
Performance with thinking generation. Table 3 shows that the advantage of OC-OPD persists with thinking generation. Relative to OPD, the 1.7B student improves by +5.76 Avg@32 and +2.22 Pass@32 points on average across AIME24–26, while the 4B student gains +1.91 and +7.78 points, respectively. OC-OPD also exceeds OPD on both GPQA-Diamond and MBPP+ at both model scales. These results show that the proposed update is not specific to non-thinking decoding and remains effective when the student already produces explicit reasoning traces.
Ablation study
Effect of individual design choices. Table 4 shows that no ablated objective matches the full OC-OPD across AIME24–26. RKL is only 0.10 points higher on AIME24 Avg@32, but is clearly worse on AIME25–26 and on Pass@32, while FKL shows a similar coverage gap. Adaptive linear mixing and equal-preference weighting recover part of the performance but remain below the full method. Removing the position gate is the strongest ablation, yet it reduces Pass@32 by about 3.33 points on all three benchmarks. These comparisons show that direction-aware preferences and position-sensitive weighting provide complementary benefits. Detailed definitions are given in Appendix § 16.§ 4; the corresponding AIME results and statistical-analysis protocol are summarized in Appendix § 17.
Conclusion
OC-OPD organizes teacher and student predictions at each student-generated prefix into four self- and cross-observations, yielding a unified supervision state. Source-centered contrasts recover the complementary ranking and coverage corrections of reverse and forward KL. The observation–gradient identity establishes their local compatibility, while observation-conditioned coordination combines them in a bounded update that retains a correction for teacher-supported candidates with low student probability. Experiments across non-thinking, thinking, and base-initialized students show consistent gains in mathematical reasoning. Scientific question answering and code generation further assess transfer beyond the mathematical training domain.
AI use statement
Generative AI tools were used solely for limited assistance with language polishing, formatting, routine coding, and debugging. The research, analysis, interpretation of results, and preparation of the final content were carried out by the human authors. All results and citations were independently verified by the authors, who take full responsibility for the work.
Ethics Statement
This work studies on-policy distillation using publicly available datasets and pre-trained open-source models, and all experiments follow the licenses of the corresponding resources. Because distillation can transfer characteristics of the teacher into the student, downstream use should include task-appropriate evaluation of factuality, bias, and safety together with standard deployment safeguards.
Reproducibility statement
Full proofs appear in Appendices. Appendix § 15 describes the gradient implementation, and Appendix § 16 provides training and sampling hyperparameters, evaluation protocols, and controlled comparisons.
Notation, assumptions, and the observation matrix
| Symbol | Meaning |
| A prefix visited by the current student behavior policy | |
| Frozen teacher and current student distributions with positive support | |
| Student logits, with | |
| Self-/cross-entropy observation matrix in Eq. (3) | |
| Student-sourced and teacher-sourced observation contrasts | |
| Their Euclidean gradients with respect to current logits | |
| , with kernel | |
| Gradient norm and normalized gradient | |
| Euclidean cosine | |
| Positive bargaining preferences summing to one | |
| Unit-norm bargaining direction, used with a minus sign in descent | |
| Squared Hellinger discrepancy and position weight | |
| Final logit signal | |
| Stop-gradient operator evaluated at the current snapshot | |
| Parameter-to-logit Jacobian and block coupling |
Table 5. Notation used throughout the paper. All local quantities refer to the same current prefix.
Interior support. The theoretical domain is . Theorems 1 and 5 hold throughout this open simplex, including arbitrarily small positive coordinates. The expansion around agreement and the coordinate limit use a fixed interior teacher. Boundary limits are taken through interior distributions, with divergence values interpreted in the extended-real sense at boundary points.
Freezing conventions. The teacher probabilities, prefix samples, preferences, gate, and bargaining solution are detached during the local optimization step. The student probabilities inside in Eq. (5), or inside in Eq. (18), remain differentiable. The differentiated objective is conditional on the current prefix distribution. Fresh student sampling supplies the state distribution for each subsequent update.
What each observation contributes. supplies student dispersion and the student-source baseline. supplies teacher dispersion and the teacher-source baseline. The cross-observations and evaluate each source under the other model. Their within-source differences define centered supervision signals, and their total excess supplies the geometric certificate. Frozen teacher entropy influences the adaptive coefficients through its numerical value.
Organized observations and candidate resolution. The matrix provides a canonical organization of four expected scores. The aligned distributions provide candidate-level probabilities, likelihood ratios, and gradient geometry. OC-OPD uses this information jointly: determines position strength and allocates the learning signal. Every candidate remains represented in the gradient construction.
From the four observations to a parameter gradient
The convention averages the evaluator’s negative log probability under candidate source . Each column fixes that source while changing the evaluator. The centered contrasts in Eq. (4) therefore admit the representations
These identities express the source-centered observation costs in standard divergence notation.
Entrywise derivatives. At a fixed prefix, let and hold fixed. Direct differentiation of the four entries gives
For the first identity, and multiplication by subtracts the -mean. For the third, , so the same centering gives the stated expression. The second is the usual cross-entropy logit derivative, and the last follows because the teacher is frozen.
Substituting these derivatives into the matrix loss yields
The student self-observation contributes the centering term through its derivative. Combining it with the teacher’s evaluation gives the student-sourced direction . For the teacher-sourced contrast, the constant establishes the zero-at-agreement baseline, and supplies the derivative .
Values condition the update; derivatives provide their direction. All four entries are evaluated at the current snapshot. The diagonal values condition the preference rule in Eq. (15); the two excesses and their derivatives define the candidate-level objectives and geometry. Teacher entropy influences the numerical coefficients through the conditioning path, with a zero direct derivative. Alignment and gate saturation can make some coefficient sensitivities vanish at particular pairs, while the observation-to-gradient map retains all four inputs.
The coefficient structure pairs each source’s self- and cross-observation through one realized coefficient. The frozen-coefficient convention differentiates with held at their current values. In the product-rule expansion, the derivative of these fixed coefficients is zero, yielding exactly . Recomputing the coefficients after each student update refreshes their dependence on the current observations.
Exact batch gradient. Let contain the valid response positions in a fresh student batch, and evaluate the controller at . With , define
For plain SGD, . Other optimizers consume the same gradient with their own parameter-space transformation. The coefficients are deterministic functions of current distributions. Each update refreshes the matrix and coefficient computation, with the complete autoregressive prefix serving as the conditioning state.
Complete current-state coefficient rule
Self-observations to preferences. Define
Here . The are unnormalized preferences; they are distinct from the realized coefficients below. Increasing shifts preference toward the teacher-sourced signal; increasing at fixed shifts it toward the student-sourced signal. A concentrated teacher paired with a diffuse student therefore emphasizes the teacher’s assessment of student-distributed candidates. A diffuse teacher gives greater priority to representing its distributed support. The positive floor retains both views when both distributions are concentrated, including when their favored candidates differ. This continuous entropy-aware parameterization instantiates the preference layer of the controller. The compatibility and bargaining results hold for every positive preference pair, separating the structural guarantee from the specific preference map.
Cross-observation excesses to direction. Compute from Eq. (6), , , and . Let be the closed-form quantities in Eq. (25). Then . The ratio is an algebraic solution of the two-objective bargaining problem; it is distinct from the iteration index .
Discrepancy to position strength. With , set
This bounded discrepancy is computed directly from the aligned distributions and controls position strength. It complements the matrix’s organization of expected scores with a full-distribution measure of agreement. Hellinger-based supervision also appears in sampled excess-token reward shaping [11]; here it determines the position-level coefficient. Appendix § 12 proves its unique scalar optimum.
Realized loss coefficients. Expanding in the source-centered observation gradients gives
Preferences, realized influence, and total strength have complementary roles: values relative gains, the gradient angle determines the compromise, and sets its norm. Near agreement, the small discrepancy smoothly attenuates the final signal at any preference. The coefficients above make Eq. (5) an exact current-point realization of the controller.
A convenient gradient surrogate. A numerically direct realization computes the vector and uses
Since , its current logit gradient is exactly ; Appendix § 15 expands the derivative. The matrix loss and signed surrogate have identical current gradients. Their scalar values follow their respective definitions.
Complete proof of the compatibility identity
Lemma 2 (Logit differentiation and the tangent space). Let . Then , , and is positive definite on . For any ,
Proof. The softmax derivative is . For any , . Since every , this variance is zero exactly when is constant. Thus the kernel is and restriction to is positive definite.
For , define . Then
The vector is the unique zero-sum solution of this equation; any two solutions differ by a constant vector. Hence for some scalar . Multiplication by , with , proves Eq. (19). This equality identifies the bilinear forms restricted to the tangent space .
Lemma 3 (KL gradients and their zero sets). With the teacher frozen, Eq. (6) holds. Each gradient is zero if and only if .
Proof. Let . Differentiating with respect to gives . Therefore
For , the derivative of gives . Each gradient sums to zero. If , equality of distributions is immediate. If , positivity of implies for every , so . Normalization forces , and . The converse follows by substitution.
Proof of Theorem 1. By Lemmas 2 and 3,
The -multiple disappears because the distributions have equal total mass. Substituting their definitions in Eq. (3) proves the matrix contrast.
If , strict nonnegativity of KL implies , and both gradients are nonzero. Suppose their Euclidean cosine were . Then for some , and
contradicting the identity. Thus . The vector
has unit norm and satisfies . This is an explicit common-descent witness. The case is permitted and simply gives .
Metric-dependent gradient geometry. Equation (7) weights coordinate products by , giving small-student-probability coordinates substantial influence. The Euclidean inner product weights these coordinates uniformly. The identity therefore certifies compatibility in the Fisher inverse metric across both acute and obtuse Euclidean configurations. At each interior pair with , the Euclidean cosine satisfies ; the resulting angular margin is pair-dependent.
Extension to strictly convex divergences
The following extension identifies a monotonicity principle underlying the compatibility identity. It establishes a shared descent direction for a finite family of smooth, strictly convex distributional contrasts.
Theorem 4 (Monotone divergence compatibility). Let , , and let be continuously differentiable strictly convex functions with . Define
and similarly define . Then
For any finite collection satisfying these assumptions, there is a single direction such that for every .
Proof. The derivative of with respect to is . Multiplying by the softmax Jacobian gives Eq. (23). Applying Lemma 2,
For arbitrary sequences , expansion of the double sum gives
The derivative of a differentiable strictly convex function on an interval is strictly increasing: equality of its derivative at two distinct points would force it to be constant between them and the function to be affine on that interval. Hence every summand in Eq. (24) is nonnegative. Since , at least two ratios differ, so at least one summand is strictly positive. This proves the pairing claim.
For a finite family, set . For every ,
In particular , so is a unit common-descent direction.
Generators and assumptions. The generators and recover the two source-centered observation contrast definitions. Adding an affine function to a generator preserves its logit gradient for normalized distributions. Differentiability supplies the gradient representation, and strict convexity supplies the strictly increasing derivatives used in the positivity argument. For general differentiable convex generators, the covariance is nonnegative; strict positivity holds when at least one positively weighted pair contributes a strictly positive product.
Compatibility and the bounded controller. The witness establishes feasibility for the finite family. Its norm reflects the inverse metric and can grow near the simplex boundary. OC-OPD uses a unit Euclidean budget and the explicit two-direction solution to implement bounded local updates. The extension explains the geometric principle, while the source-centered observation construction specifies the candidate populations and statistical meaning of the implemented directions.
Existence, uniqueness, and the closed-form bargaining solution
We prove Theorem 5 and two useful consequences. Let , , and throughout this section.
Theorem 5 (Unique closed-form compromise). Under Theorem 1 with and , Eq. (9) has a unique solution. Set
Then , and both are strictly positive.
Existence. Theorem 1 gives a feasible point . On the closed unit ball both utilities lie in . For a maximizing sequence with objective at least that of , both utilities stay bounded away from zero: approaching zero would send a positive-weight logarithm to , while the other logarithm is at most zero. A convergent subsequence of this sequence therefore has a limit in the strictly positive-utility domain. Continuity proves that the maximum is attained.
The norm constraint is active. If a feasible has , then is feasible and increases the objective by . Every optimum consequently has unit norm.
Uniqueness. Suppose distinct optima exist. Their midpoint is feasible and, by concavity of the log utilities, has objective at least the optimal value. Strict convexity of the Euclidean ball implies for distinct unit vectors. Radially enlarging the midpoint strictly increases the objective, a contradiction. This proof covers collinear gradients with as well as linearly independent gradients.
First-order conditions. The objective is differentiable at the optimum. Its gradient is
Writing the multiplier for as , the KKT condition is . Taking the inner product with the unit vector gives . Hence
Here are coefficients of the unit direction in normalized gradients; the state-indexed in Eq. (17) multiply the unnormalized gradients in the final update. It follows that
For , eliminating yields
The constant term is negative and the leading coefficient positive, so exactly one root is positive. This is Eq. (25). Since the actual unique optimum has this positive ratio, normalizing reconstructs it. Conversely, this vector satisfies the first-order condition and positive-utility constraints and is therefore optimal by concavity. This completes the proof of Theorem 5.
Corollary 6 (Scale invariance and proportional fairness). The direction is invariant to positive rescaling of either objective at fixed preferences. For every feasible with ,
Proof. For , normalization gives , so each utility and its maximizing direction are invariant. By the KKT equation,
Subtracting proves Eq. (28). The inequality characterizes the weighted relative local gains of the two source-centered contrasts. ▫
Equal weights and the simplest strong baseline. If , then for every . Thus . This direction is already a feasible common descent direction, independently of uncertainty. The appropriate incremental question is whether observation-dependent preferences yield more useful learning than this symmetric baseline, at a common norm, gate, learning rate search budget, and teacher cost. For , the direct normalized-gradient mixture has unnormalized local gains
These expressions characterize how unequal mixing preferences interact with the current gradient geometry.
Stable root evaluation. Let and . An algebraically equivalent computation is
The second branch avoids subtracting nearly equal positive quantities. Computing with a hypot routine further reduces overflow risk. The reference implementation corrects roundoff by clipping the computed cosine to and normalizes the resulting vector directly. Theorem 1 gives for each distinct interior pair.
Aligned-gradient and binary reductions
Proposition 7 (Exact reduction under aligned gradients). If and , then for every positive preference pair.
Proof. When , Eq. (9) is maximized by their common unit vector, independently of the preferences. A binary logit tangent space is one-dimensional; Theorem 1 rules out opposite directions, so the normalized gradients coincide.
Distinct normalized candidate directions create room for preference-dependent coordination. A binary vocabulary yields the exact reduction . The directions also align to first order near agreement, as established in Appendix § 13. Mismatch and gradient angle together characterize the positions at which coordination changes the signal.
Strength, local descent, and candidate-deficit guarantees
Proposition 8 (Local descent, boundedness, and teacher-deficit signal). For , sufficiently small isolated logit steps strictly decrease both and . Moreover,
For a fixed positive teacher with , as , Eq. (15) gives
The coordinate lower bound concerns a logit correction. The softmax Jacobian translates this correction into probability changes, whose scale depends on the current student probabilities.
Position-weight optimum. The objective has second derivative , so its constrained maximum is unique. The unconstrained stationary point is ; projection onto gives Eq. (16). Because for distinct distributions, whenever . If , the selected weight is zero.
Norm bound. For distributions ,
since each sum of squares is at most one and the cross term is nonnegative. Since and , Eq. (30) follows. The universal constant is a supremum over pairs approaching distinct point masses. With a bounded parameter-to-logit Jacobian, the corresponding parameter gradient is bounded by the product of its operator norm and this logit bound.
An explicit local descent condition. Fix with , and let be a Lipschitz constant of on a ball of radius around , for . Such finite constants exist on every compact logit neighborhood for a fixed positive teacher. Set and . The descent lemma gives
Thus both losses strictly decrease whenever
with a term interpreted as if its Lipschitz constant is zero. The restriction keeps the segment inside the ball. This condition relates the admissible local step to the positive gain margin and the smoothness of each source-centered contrast.
Lower bound for a teacher-supported deficit. From Eq. (26), , because . For , . Also and , so . The coefficients are positive; hence
This argument uses the signs of the individual deficit coordinates and the bargaining KKT equation. The vocabulary construction evaluates every deficit coordinate and supplies the deterministic correction in Eq. (30).
Uniformly positive source preferences. Because ,
Both preferences therefore obey .
The limit as student support vanishes. Fix . Cauchy–Schwarz on the remaining coordinates implies
Consequently,
Combining this bound, continuity and monotonicity of , the preference floor, and gives Eq. (31). This proves Proposition 8.
Proposition 9 (Finite virtual-step probability-ratio control). Let , for any . Then
In particular, both and are at most .
Proof. Write . Then . The logarithm of the weighted exponential average lies between and . The absolute difference is therefore at most . Taking expectations under either distribution gives the KL bounds. The bound describes an isolated logit step.
Near-agreement asymptotics and position strength
Proposition 10 (Cubic attenuation). Fix and . Let , , and . Then
In particular, along any nonzero fixed perturbation direction.
Proof. Because is fixed in the interior, Taylor expansions are uniform in a sufficiently small neighborhood. For each coordinate,
Squaring and summing gives
For small enough , , so the gate is unsaturated and . Also, and . Equation Equation (6) therefore gives
Writing for nonzero , it follows that and . The preferences are continuous near and uniformly positive. The positive root is consequently bounded above and away from zero there. Normalization of gives , and hence
Multiplying by Eq. (38) divided by yields
as claimed. Positive definiteness of supplies the matching upper and lower cubic orders along any fixed nonzero direction. □
Interpretation. Near agreement, the two source-centered gradients agree to first order. Preference-dependent changes enter at higher order, and the gate controls the leading attenuation. Equation Equation (37) quantifies the resulting cubic allocation of supervision to small discrepancies, including its slower fine-matching scale. A positive gate floor would restore a first-order term. The unit-gate ablation isolates the contribution of discrepancy-dependent attenuation.
Behavior at agreement and concentrated predictions. For , the exact update is zero at every entropy level, preserving the matching distribution. Concentrated predictions with different favored candidates can have large Hellinger discrepancy and receive strong supervision. Position strength thus responds to cross-model mismatch, while self-observations condition the relative candidate-source influence.
Comparison related works
Implementation details
Algorithm 1 gives the main training loop. The following details specify its sampling, masking, and numerical realization.
Training semantics and numerical implementation
Current-policy sampling. The mathematical formulation samples from the full current student distribution at temperature one. The sampled batch is fixed for one student update, after which fresh rollouts are collected.
Alignment and masking. Teacher and student distributions share vocabulary semantics and next-token positions. Both models evaluate the exact student continuation. The loss sums over valid response positions, including the response end token. The signed token-vocabulary surrogate is summed over these positions and divided by their global count . This preserves the position strength in Eq. (11).
Degeneracies and stable normalization. Away from agreement, the two gradient norms are nonzero by Theorem 1. The stable quadratic-root expression in Appendix § 11 supports accurate evaluation under unequal preferences. Direct normalization of the combined vector gives the final gradient at the prescribed magnitude.
Loss realization. Because is zero-sum,
Thus the signed surrogate produces the desired current logit gradient with one student backward pass. The coefficients and controller computations are detached; student log probabilities retain their computation graph. The teacher remains frozen. The frozen-coefficient convention realizes the matrix differential in Eq. (6) at the current snapshot.
Real-update diagnostics
Purpose. The geometric analysis in the main text is expressed in terms of a change in student logits, while optimization acts on shared neural-network parameters. We therefore measure whether the logit-space quantities predict the change produced by a real optimizer step. This diagnostic uses a Qwen3-1.7B non-thinking student and the Qwen3-30B-A3B-Instruct-2507 teacher. Training starts from the same zero-LoRA initialization used by the corresponding non-thinking experiments.
Quantities measured. For a fixed token prefix, let be the frozen teacher distribution and let denote the student distribution immediately before the optimizer update. We define
After the production backward pass and AdamW update, we evaluate the same prefix again to obtain , set , and compare the predictions
with the measured changes
All quantities are computed over the vocabulary. A negative predicted or measured change denotes descent for the corresponding objective. We report the Pearson correlation between predicted and measured changes, the fraction of token positions for which their signs agree, and whether the prediction correctly identifies simultaneous descent of both objectives.
Probe construction. Measurements are taken every ten successful optimizer updates, always using the same teacher target before and after the update. We use three probe types. The on-policy probe samples positions from the current rollout. The fixed in-domain probe reuses fixed prompt–continuation pairs drawn from the training domain. The HMMT25 OOD probe contains four held-out prompts with fixed short continuations and is used only to diagnose distribution shift.
Table 7 and Figure 4 show that the local approximation remains accurate after an update to real network parameters. Pearson correlations range from 0.9751 to 0.9993, while individual-objective sign agreement ranges from 98.00% to 100.00%. Joint sign accuracy is at least 99.00% for the two fixed probes and 99.57% on on-policy rollouts. The few sign errors occur close to zero, where the measured change is small. These results empirically validate the local predictive role of under the observed optimizer steps.
| Probe | Updates | Tokens | Sign | Sign | Both | ||
| On-policy rollout | 19 | 464 | 0.9977 | 99.57 | 0.9989 | 99.57 | 99.57 |
| Fixed in-domain | 25 | 300 | 0.9751 | 99.67 | 0.9992 | 98.00 | 99.00 |
| HMMT25 OOD | 25 | 400 | 0.9987 | 99.00 | 0.9993 | 100.00 | 99.25 |
Table 7. Agreement between first-order predictions and measured KL changes. Correlations and sign agreement are computed over token positions. “Both” is the accuracy of predicting whether RKL and FKL decrease simultaneously.
Figure 4. RKL (top) and FKL (bottom) are shown separately for on-policy, fixed in-domain, and HMMT25 OOD probes. Each point is a token position evaluated immediately before and after the same real optimizer step. The dashed line denotes and red crosses indicate sign disagreement.
Interpretation and scope. The diagnostic supports the specific bridge used by our analysis: for the optimizer steps observed during training, the logit-space terms remain accurate predictors of the KL changes induced by a shared-parameter neural-network update. Agreement on current rollouts, fixed in-domain examples, and held-out HMMT25 prompts indicates that this fidelity is not confined to one probe distribution.
Experimental details
Models and experimental organization
The teacher checkpoints are Qwen3-30B-A3B-Thinking-2507 and Qwen3-30B-A3B-Instruct-2507. The main experiment pairs the Instruct teacher with non-thinking Qwen3-1.7B and Qwen3-4B students. The first auxiliary experiment pairs the Thinking teacher with thinking students at both sizes; the second pairs the Instruct teacher with Qwen3-4B-Base. Within each comparison block, methods use the same student initialization, teacher, prompt source, training budget, generation/evaluation configuration, and metric implementation; only the learning objective or controller component being compared is changed.
Each iteration generates fresh student continuations and aligns teacher and student distributions on the collected prefixes. The four observations, gradient geometry, and coefficients are recomputed at every valid response position. The teacher remains frozen, and the backward pass updates the student. The selected student regime determines the generation configuration of its training and evaluation continuations, and all methods in that regime use the same prompt wrapper and decoding protocol.
Training and sampling hyperparameters
Table 8: Hyperparameters for OC-OPD. Prompt and response lengths are measured in tokens. A dash denotes the fresh-rollout setting, with one optimization step per rollout batch.
| Hyperparameter | OC-OPD |
| General | |
| Max prompt length | 8192 |
| GPUs | NVIDIA A800 80GB |
| Training precision | BF16 |
| Optimization | |
| Global batch size | 8 |
| Mini-batch size | 8 |
| Mini-batches | 1 |
| Group size | 1 |
| Off-policy epoch | — |
| Optimizer | AdamW |
| Learning rate | |
| LR schedule | Constant |
| Warmup ratio | 0 |
| Weight decay | 0 |
| Adam betas | (0.9, 0.999) |
| Gradient clipping | Global L2 norm 1.0 |
| Sampling | |
| Rollout max response length | 32768 |
| Rollout temperature | 1.0 |
| Rollout top- | 1.0 |
| Validation max response length | 32768 |
| Validation temperature | 1.0 |
| Validation top- | 1.0 |
| Method-specific | |
| 1.0 | |
| 0.05 | |
| 0.01 |
Table 8. Hyperparameters for OC-OPD. Prompt and response lengths are measured in tokens. A dash denotes the fresh-rollout setting, with one optimization step per rollout batch.
Prompt templates
The prompt wrapper is fixed by the student generation regime and is shared by all compared methods in that regime. The placeholder {problem} is replaced by the benchmark problem text. We use the following templates.
Non-thinking.
{problem}\n\nPlease solve the following problem and put your final answer within\n\n\boxed{}.
Thinking.
{problem}\n\nPlease reason step by step, and put your final answer within\n\n\boxed{}.
Base.
Problem:\n{problem}
Controlled objectives and ablations
The mathematical ablations use Qwen3-30B-A3B-Instruct-2507 with the Qwen3-4B non-thinking student on DAPO-Math-17k, matching the 4B setting in Table 1. All ablations reuse the same initialization, prompts, optimization and sampling setup, and evaluation budget. Table 9 reports the key controls. The comparison design separates the two information paths in Eq. (5): derivatives of the observations form the candidate-level gradients, while observation values condition their mixture.
| Method | AIME 2024 | AIME 2025 | AIME 2026 | |||
| Avg@32 | Pass@32 | Avg@32 | Pass@32 | Avg@32 | Pass@32 | |
| OC-OPD | 60.94 | 93.33 | 56.67 | 83.33 | 58.54 | 86.67 |
| Forward KL (FKL) | 55.42 | 83.33 | 53.75 | 73.33 | 54.17 | 83.33 |
| Reverse KL (RKL) | 61.04 | 86.67 | 51.04 | 76.67 | 52.19 | 80.00 |
| Adaptive linear mixture | 60.21 | 86.67 | 53.02 | 80.00 | 53.96 | 80.00 |
| Equal-preference | 58.65 | 86.67 | 53.54 | 80.00 | 56.67 | 83.33 |
| w/o position gate | 60.63 | 90.00 | 54.06 | 80.00 | 56.67 | 83.33 |
Table 9. Ablation study on mathematical reasoning. The student is Qwen3-4B with non-thinking generation and the teacher is Qwen3-30B-A3B-Instruct-2507. Scores are percentages; bold denotes the best value in each column.
Full OC-OPD attains the highest Pass@32 on all three AIME editions, while RKL is slightly higher on AIME24 Avg@32. The single-KL controls, adaptive linear mixture, equal-preference variant, and no-gate variant each lose ground on at least one of the coverage metrics. Together, these comparisons support the use of both source-centered directions, observation-conditioned preferences, bargaining-based coordination, and discrepancy-dependent strength.
KL-only controls. Forward KL (FKL) optimizes the teacher-sourced contrast , while reverse KL (RKL) optimizes the student-sourced contrast . These rows isolate the two candidate-source objectives without the observation-conditioned coordination used by OC-OPD.
Adaptive linear mixture This variant uses the same observation-conditioned preferences , source-centered normalized gradients, and discrepancy gate as OC-OPD, but replaces the bargaining solution with the normalized linear direction . The final magnitude remains , matching the full method.
Equal preferences. Equal-preference bargaining variant fixes while retaining the source-centered gradients, bargaining rule, and discrepancy gate. It therefore removes observation-conditioned preference adaptation without changing the two candidate sources or the update-strength mechanism.
No position gate. This variant sets , preserving the original preference rule and coordinated direction while removing discrepancy-dependent attenuation of the update magnitude. It directly tests the position-strength component analyzed in Appendix § 13.
Variance and Statistical Significance
Figure 5. Additional AIME results to demonstrate variance and statistical significance. (accuracy, %). We retain the mode for all the results across six runs, and figures report the variance.
References
- [1]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/abs/1503.02531.
- [2]Yoon Kim and Alexander M. Rush. Sequence-Level Knowledge Distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327. Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1139. URL https://aclanthology.org/D16-1139/.DOI
- [3]Taiqiang Wu, Chaofan Tao, Jiahao Wang, Running Yang, Zhe Zhao, and Ngai Wong. Rethinking Kullback-Leibler divergence in knowledge distillation for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5737–5755, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.383/.arxiv.org/abs/2404.02657
- [4]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In International Conference on Artificial Intelligence and Statistics, pages 627–635, 2011. URL https://proceedings.mlr.press/v15/ross11a.html.arxiv.org/abs/1011.0686
- [5]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations, 2024a. URL https://arxiv.org/abs/2306.13649.
- [6]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge Distillation of Large Language Models. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2306.08543v4.
- [7]Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling Reasoning Efficiently via Relaxed On-Policy Distillation. arXiv preprint arXiv:2603.11137, 2026. URL https://arxiv.org/abs/2603.11137.
- [8]Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-Aware On-Policy Distillation of Language Models. arXiv preprint arXiv:2603.07079, 2026. URL https://arxiv.org/abs/2603.07079.
- [9]Chia-Hsuan Lee, Zelei Cheng, Yu Wang, Renkun Ni, Sambit Sahu, Shi-Xiong Zhang, and William Campbell. SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision. arXiv preprint arXiv:2606.28562, 2026. URL https://arxiv.org/abs/2606.28562.
- [10]Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. TIP: Token Importance in On-Policy Distillation. arXiv preprint arXiv:2604.14084, 2026. URL https://arxiv.org/abs/2604.14084.
- [11]Zichao Yu, Chengzhi Yu, Shengze Xu, Yujin Han, Bingqing Jiang, Xu Wang, and Difan Zou. Mismatch Matters: On-Policy Distillation Beyond Token Agreement. arXiv preprint arXiv:2608.09836, 2026. URL https://arxiv.org/abs/2608.09836.
- [12]Qwen Team. Qwen3: Think Deeper, Act Faster. Official model release, 2025a. URL https://qwenlm.github.io/blog/qwen3/.
- [13]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanze Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.arxiv.org/abs/2311.12022
- [14]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.01210
- [15]Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. DistLLM: Towards Streamlined Distillation for Large Language Models. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 24872–24895. PMLR, 2024. URL https://proceedings.mlr.press/v235/ko24c.html.arxiv.org/abs/2402.03898
- [16]Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. DistLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267, pages 31044–31062. PMLR, 2025. URL https://proceedings.mlr.press/v267/ko25a.html.arxiv.org/abs/2503.07067
- [17]Wenda Xu, Rujun Han, Zifeng Wang, Long T. Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2410.11325v3.
- [18]Chenyang Jiang and Changhan Huang. SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillation. arXiv preprint arXiv:2608.14647, 2026. URL https://arxiv.org/abs/2608.14647.
- [19]Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuiliang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai, and Zequn Sun. Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level. arXiv preprint arXiv:2605.06387, 2026. URL https://arxiv.org/abs/2605.06387v4.
- [20]Ijun Jang, Jewon Yeom, Juan Yeo, Hyungyu Lim, and Taesup Kim. Stable On-Policy Distillation through Adaptive Target Reformulation. arXiv preprint arXiv:2601.07155, 2026. URL https://arxiv.org/abs/2601.07155.
- [21]Anhao Zhao, Junlong Tong, Yingqi Fan, Ping Nie, Wenjie Li, and Xiaoyu Shen. PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation. arXiv preprint arXiv:2606.17199, 2026a. URL https://arxiv.org/abs/2606.17199.
- [22]Xingrun Xing, Haoqing Wang, Boyan Gao, Ziheng Li, and Yehui Tang. Trust Region On-Policy Distillation. arXiv preprint arXiv:2606.01249, 2026. URL https://arxiv.org/abs/2606.01249.
- [23]Shun-ichi Amari. Natural Gradient Works Efficiently in Learning. Neural Computation, 10(2):251–276, 1998. doi: 10.1162/089976698300017746. URL https://doi.org/10.1162/089976698300017746.
- [24]Frank Nielsen. An Elementary Introduction to Information Geometry. Entropy, 22(10):1100, 2020. doi: 10.3390/e22101100. URL https://doi.org/10.3390/e22101100.
- [26]Qwen Team. Qwen3-30B-A3B-Instruct-2507. Official model card, 2025b. URL https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507.
- [28]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503.14476, 2025. URL https://arxiv.org/abs/2503.14476.
- [29]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2305.20050.
- [30]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.211. URL https://aclanthology.org/2024.acl-long.211/.DOI
- [31]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [32]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations (ICLR). OpenReview.net, 2024b.arxiv.org/abs/2306.13649
- [33]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026b.arxiv.org/abs/2601.18734




