betaFeedback

WorksComputer Science

OC-OPD: Observation-coordinated Supervision for On-Policy Distillation

Abstract

Reverse and forward Kullback–Leibler (KL) divergences provide complementary supervision in on-policy distillation: reverse KL refines the student's active token rankings, and forward KL allocates mass to teacher-supported candidates. At student-generated prefixes, we form a four-observation matrix by treating the student and teacher as both candidate sources and evaluators. The diagonal self-observations measure uncertainty, while source-centered cross-observations recover the two KL objectives. We establish an exact identity equating the Fisher inverse-metric coupling of their logit gradients with the sum of these contrasts, yielding a common local descent direction for distinct positive distributions. Observation-coordinated on-policy distillation (OC-OPD) uses observations to set preferences, closed-form bargaining to coordinate gradients, and distributional discrepancy to scale the update. Across three heterogeneous student settings (non-thinking generation, thinking generation, and base-model initialization), OC-OPD achieves strong gains in mathematical reasoning and transfers effectively to out-of-distribution scientific question answering and code generation. Our code is publicly available at: https://github.com/RuoranXu/OC-OPD.

Introduction

Earlier knowledge distillation methods use KL divergences to transfer predictive structure from a teacher to a student [1, 2]. Adaptive KL (AKL) adjusts the early-training balance between forward and reverse KL according to distributional gaps within teacher-defined high-probability head and long-tail regions [3].

On-policy distillation (OPD) trains a student at its own generated prefixes, where a frozen teacher provides supervision for the same next-token decision [4, 5]. At each prefix, the aligned student distribution qq and teacher distribution pp define two complementary objectives. Reverse KL (RKL), DKL(q∥p)D_{\mathrm{KL}}(q\Vert p), is associated with mode-seeking refinement of student-supported candidates, while forward KL (FKL), DKL(p∥q)D_{\mathrm{KL}}(p\|q), provides mode-covering supervision toward teacher-supported alternatives [5, 6]. As the student changes, so do the visited prefixes and the conditional predictions of both models. How should these corrections be coordinated within one update at the current teacher–student state?

Recent OPD methods adapt supervision using uncertainty, token importance, teacher–student mismatch, and divergence routing [7, 8, 9, 10, 11]. At a visited prefix, two aspects of the predictive state inform this adaptation: each model’s self-uncertainty and the disagreement between their predictions. Self-uncertainty describes the dispersion of a model’s candidate probabilities; disagreement describes the mismatch requiring correction. Connecting these quantities to the two KL gradients provides a state-dependent basis for their relative influence and joint improvement.

We place these signals in a common source–evaluator system: the student and teacher each supply candidate mass, and either model can evaluate it. The resulting four-observation matrix separates self-uncertainty, represented by its diagonal entropies, from directed disagreement, recovered by centering its cross-entropies on the corresponding source entropies. These contrasts yield the refinement and coverage objectives at the same student-visited state.

This representation also links observations to correction geometry. For positive pp, qq, the reverse- and forward-KL logit gradients satisfy gRJ(q)+gF=DKL(q∥p)+DKL(p∥q)g_R J(q) + g_F = D_{\mathrm{KL}}(q\|p) + D_{\mathrm{KL}}(p\|q), where J(q)J(q) is the categorical Fisher matrix. For p≠qp \ne q, this positive cross-observation excess establishes a common local descent direction for both KL objectives in student-logit space.

We develop observation-coordinated on-policy distillation (OC-OPD) by connecting the observed teaching state to three decisions: preference, direction, and strength. Self-entropies set preferences between the two corrections, a closed-form bargaining solution coordinates their normalized gradients within the common-descent region, and symmetric distributional discrepancy scales supervision strength. These components connect self-uncertainty, correction geometry, and disagreement in a single bounded logit update, applied with one student backward pass. Figure 1 summarizes the observation →\to contrast →\to coordination →\to strength pipeline.

Figure 1. Training pipeline of OC-OPD. (a) Student rollouts provide prefixes at which the student and frozen teacher produce aligned next-token distributions. (b) Four self- and cross-observations at the shared prefix yield source-centered KL gradients for ranking/refinement and coverage. (c) Self-observation values set uncertainty-based priorities for closed-form gradient bargaining, while distributional discrepancy controls supervision strength. (d) The resulting bounded logit signal updates the student through a single backward pass. The updated student then generates new rollouts.

We evaluate OC-OPD on mathematical reasoning in three Qwen3 student settings [12], all trained with fresh student rollouts: non-thinking students at 1.7B and 4B scales, thinking-generation students, and a weaker base-model initialization. In the main non-thinking setting, the 1.7B student exceeds the strongest compared baseline by +6.88 Avg@32 and +11.11 Pass@32 points on average across AIME24–26; the 4B student exceeds EOPD [8] by +2.74 and +3.33 points, respectively (Table 1). Gains persist in the other two settings, and GPQA-Diamond [13] and MBPP+ [14] assess transfer to scientific question answering and code generation.

MethodsAIME 2024 Avg@32AIME 2024 Pass@32AIME 2025 Avg@32AIME 2025 Pass@32AIME 2026 Avg@32AIME 2026 Pass@32MATH500 Avg@8MATH500 Pass@8OlympiadBench Avg@32OlympiadBench Pass@32GPQAD_D Avg@32MBPP+
Teacher: Qwen3-30B-A3B-Instruct-2507
Qwen3-1.7B14.1643.339.9030.007.7123.3371.8852.6729.5956.03
+ GRPO34.1770.0025.5246.6726.0450.0082.2162.57——
+ OPD31.9873.3320.7350.0023.5446.6785.8460.77——
+ OPSD33.3373.3322.6053.3321.3556.6786.2064.84——
+ OC-OPD41.6776.6734.3863.3330.3176.6787.4067.4034.0159.01
Qwen3-4B23.8556.6722.4056.6715.1053.3379.2069.7339.6065.55
+ OPD (RKL)61.0486.6751.0476.6752.1980.0087.4073.0443.4364.29
+ OPD (FKL)55.4283.3353.7573.3354.1783.3385.2075.6540.9165.55
+ EOPD60.1086.6753.1383.3354.6983.3392.8078.9345.4564.81
+ OC-OPD60.9493.3356.6783.3358.5486.6794.4380.8646.8066.01
Δ vs. EOPD+0.84+6.66+3.54+0.00+3.85+3.34+1.63+1.93+1.35+1.20

Table 1. Results for non-thinking students (accuracy %). Shaded rows denote OC-OPD. Dashes indicate scores not present in the supplied results. Δ reports the Qwen3-4B gain over EOPD in percentage points, computed from the displayed scores.

Our contributions are threefold. (1) We unifiedly describe token-level OPD through a source–evaluator system whose self- and cross-observations characterize the teaching state and whose source-centered contrasts recover the two KL corrections. (2) We prove a Fisher inverse-metric identity linking those contrasts to their logit gradients and establish common local descent for every distinct positive teacher–student pair. (3) We develop OC-OPD to determine the preference, direction, and strength of token-level supervision, and demonstrate its effectiveness across heterogeneous student settings and out-of-domain evaluations.

Related Work

KL directions in on-policy distillation. Earlier knowledge distillation methods transfer predictive structure from a teacher to a student [1, 2]. AKL adaptively combines forward and reverse KL using distributional gaps within teacher-defined head and tail regions, motivated by their different fitting priorities in early training [3]. A larger head gap increases the forward-KL weight, while a larger tail gap increases the reverse-KL weight. OPD evaluates teacher guidance at student-generated prefixes [4, 5], where both models produce aligned next-token distributions. GKD studies flexible divergence objectives at these states [5], and MiniLLM develops reverse-KL distillation under student sampling [6]. Related methods explore skewed divergences and adaptive sampling [15], contrastive teacher–student responses [16], and interleaved generation [17]. Reverse KL refines relative probabilities among student-supported candidates, while forward KL transfers mass toward teacher-supported candidates. EOPD conditions forward-KL supervision on teacher entropy [8], and SEAD adjusts the emphasis of the two KL directions across tokens [9]. We study how the current teacher–student state relates to these corrections: source-centering separates self-uncertainty from directed disagreement, and an exact observation–gradient identity establishes their joint local compatibility.

Observation-based adaptive supervision. At a shared prefix, either model can supply candidate mass and either model can evaluate it. These choices produce four observations: two self-entropies describe the predictive states of the student and teacher, and two cross-entropies combine source uncertainty with directed cross-model disagreement. Related methods use local information to determine how supervision is assigned. REPOPLD uses entropy-based token selection and relaxed reward shaping [7]; SMOPD selects informative positions for self-distillation [18]; TIP combines student uncertainty with divergence to assess token importance [10], while TIDE characterizes student-excess and student-deficit mismatch [11]. Region-dependent objectives, adaptive targets, bounded transformations, and trust-region control further shape local supervision [19, 20, 21, 22]. Our source–evaluator matrix connects these signals through the entropy decomposition of KL: source-centering isolates the two directed disagreements, whose gradients supply the refinement and coverage corrections. Self-entropies set their relative preferences, gradient geometry coordinates their direction, and distributional discrepancy sets supervision strength. Appendix § 14 summarizes representative methods in this notation.

Gradient coordination. Multi-objective optimization derives shared update directions via minimum-norm descent, gradient projection, conflict-averse optimization, and bargaining. Our KL gradients stem from aligned student–teacher distributions and source-centered observation contrasts, whose local coupling follows an exact Fisher-geometric identity. We leverage this identity to coordinate candidate-level refinement and coverage, adopting bargaining to obtain an observation-conditioned closed-form update direction. Scalarization and normalized gradient mixing serve as baselines for comparison.

Preliminaries and Motivation

This section formalizes shared-context OPD, organizes the aligned student and teacher distributions into four observations, and shows how their source-centered contrasts recover the ranking and coverage signals of reverse and forward KL.

Shared-context OPD as observation system

OPD evaluates supervision along student-generated responses by querying the teacher at the same prefixes [5, 6]. We distinguish this choice of training states from the information available at each visited state. In the shared-context setting, both models condition on the same prompt and student prefix, yielding an aligned pair of next-token predictive distributions. Our aim is to organize the information in this pair into effective supervision.

For a prompt xx, let qθq_\theta and pTp_T denote the student and frozen teacher. A representative reverse-KL OPD objective is

JOPD(θ;x)=DKL(qθ(⋅∣x)∥pT(⋅∣x))=Ey∼qθ(⋅∣x)[∑t=1∣y∣log⁡qθ(yt∣st)pT(yt∣st)],st=(x,y<t).(1)\mathcal{J}_{\mathrm{OPD}}(\theta; x) = D_{\mathrm{KL}}\left(q_\theta(\cdot\mid x)\mathrel{\|}p_T(\cdot\mid x)\right) = \mathbb{E}_{y\sim q_\theta(\cdot\mid x)} \left[ \sum_{t=1}^{|y|} \log\frac{q_\theta(y_t \mid s_t)}{p_T(y_t \mid s_t)} \right], \qquad s_t = (x,y_{<t}). \tag*{(1)}

Student-sampled log ratios estimate this objective. At iteration kk, write qk=qθkq_k = q_{\theta_k} and collect valid response positions SkS_k from fresh student rollouts. A token-local surrogate assigns each sampled token a teacher-to-student log-ratio credit:

L^OPD,k(θ)=−1N∑t∈Sksg⁡[Ak,t]log⁡qθ(yt∣st),Ak,t=log⁡pT(yt∣st)qk(yt∣st),(2)\widehat{\mathcal{L}}_{\mathrm{OPD},k}(\theta) = -\frac{1}{N}\sum_{t\in S_k}\operatorname{sg}[A_{k,t}]\log q_\theta(y_t\mid s_t), \qquad A_{k,t}=\log\frac{p_T(y_t\mid s_t)}{q_k(y_t\mid s_t)}, \tag*{(2)}

where N=∣Sk∣N=|S_k| and sg denotes stop-gradient. Prefixes and credits are fixed during differentiation. Full-vocabulary implementations evaluate the corresponding expectation over candidates at each collected prefix [5]. We use this objective as a reference and derive the four observations directly from the aligned distributions, making student-sourced and teacher-sourced supervision explicit.

Four observations from two distributions

At a collected prefix s=(x,y<t)s=(x,y_{<t}), write q=qθk(⋅∣s)q=q_{\theta_k}(\cdot\mid s) and p=pT(⋅∣s)p=p_T(\cdot\mid s) over an aligned vocabulary. The fixed pair pp and qq supplies two candidate sources—student and teacher—and two evaluators—student and teacher. Pairing each source with each evaluator yields four expected observations. For H(a,b)=−∑iailog⁡biH(a,b)=-\sum_i a_i\log b_i and H(a)=H(a,a)H(a)=H(a,a), define

M(s)=(H(q)H(p,q)H(q,p)H(p))=−∑i∈V(qilog⁡qipilog⁡qiqilog⁡pipilog⁡pi).(3)M(s)= \begin{pmatrix} H(q) & H(p,q)\\ H(q,p) & H(p) \end{pmatrix} = -\sum_{i\in\mathcal{V}} \begin{pmatrix} q_i\log q_i & p_i\log q_i\\ q_i\log p_i & p_i\log p_i \end{pmatrix}. \tag*{(3)}

Rows index evaluators and columns index candidate sources, both ordered as student, teacher. Accordingly, M11M_{11} records the student evaluating student-sourced mass, M12M_{12} the student evaluating teacher-sourced mass, M21M_{21} the teacher evaluating student-sourced mass, and M22M_{22} the teacher evaluating teacher-sourced mass. The diagonal self-observations are the student and teacher entropies and quantify distributional dispersion. The off-diagonal cross-observations measure how well one model accommodates candidates emphasized by the other.

The matrix provides a relational reparameterization of the same p,qp, q pair by separating the distribution supplying candidate mass from the distribution scoring it. Teacher-sourced expectations use the teacher distribution at the same student-visited prefix. The matrix records state-level expectations, while the underlying pp and qq retain the candidate-level resolution needed for an update. It gathers common adaptive OPD signals in one object: student entropy is M11M_{11}, teacher entropy is M22M_{22}, and teacher–student mismatch appears as the off-diagonal excess over the corresponding self-observation. Table 6 in appendix § 14 compares how different methods are described in this framework.

MethodM₁₁ H(q)H(q)M₁₂ H(p,q)H(p,q)M₂₁ H(q,p)H(q,p)M₂₂ H(p)H(p)
REOPOLD•–––
TIP•–––
EOPD–––•
SEAD•––•
OC-OPD (Ours)••••

Table 6. Comparison of entropy and token-selection-based OPD methods under our four-observation framework. • denotes an observation directly used for token selection or divergence routing, while ◦ denotes an observation involved only through a derived KL contrast.

Source-centered contrasts reveal ranking and coverage

Centering each column on its candidate source’s self-observation isolates the difference between the two evaluators:

R=M21−M11=H(q,p)−H(q)=DKL(q∥p),R = M_{21} - M_{11} = H(q,p) - H(q) = D_{\mathrm{KL}}(q\mathbin{\|}p),
F=M12−M22=H(p,q)−H(p)=DKL(p∥q).(4)F = M_{12} - M_{22} = H(p,q) - H(p) = D_{\mathrm{KL}}(p\mathbin{\|}q). \tag*{(4)}

Thus reverse and forward KL arise as centered cross-observations from the student-sourced and teacher-sourced columns, respectively.

Student-sourced contrast: rank and refine active support. For reverse KL, the token-level credit Ai=log⁡(pi/qi)A_i = \log(p_i/q_i) compares teacher support with the student’s current allocation. Since Ai>AjA_i > A_j exactly when pi/qi>pj/qjp_i/q_i > p_j/q_j, this credit ranks student-supported candidates by relative underallocation. The student-sourced contrast thereby refines the alternatives the student already explores. Its logit gradient is weighted by qiq_i, concentrating the correction on candidates that carry student probability.

Teacher-sourced contrast: recover underrepresented support. Forward KL evaluates teacher-sourced candidates under the student. Its logit gradient q−pq - p directly reallocates mass toward teacher-supported alternatives, including those with very small student probability. This coverage correction allocates student capacity across the teacher-supported set [6]. In our full-vocabulary construction, the teacher-sourced gradient is evaluated directly at the visited prefix, yielding the exact candidate-level correction from the aligned distributions.

From complementary contrasts to a coordination problem. The matrix thus provides both statistics describing the current state and contrasts specifying candidate-level corrections. Existing entropy gates, token filters, and mismatch criteria can be interpreted as selecting or weighting particular regions of this object [7, 8, 9, 10, 11]. Together, the source-centered contrasts invite a geometric account of when their ranking/refinement and coverage gradients share a descent direction. § 4 establishes this compatibility; § 5 uses self-observation, compatibility, and discrepancy to determine preference, direction, and magnitude.

Theory: From Observations to Compatible Gradients

The matrix view yields two source-centered corrections whose candidate-level gradients can be studied together. Their ranking/refinement and coverage directions admit common descent, with the observation matrix providing an explicit certificate of compatibility.

From observation entries to candidate-level gradients

For current-state coefficients as,bsa_s,b_s, define the local matrix loss

ℓs(θ)=⟨sg⁡[(−asbsas−bs)],Mθ(s)⟩F=sg⁡[as][H(qθ,p)−H(qθ)]+sg⁡[bs][H(p,qθ)−H(p)].(5)\begin{aligned} \ell_s(\theta) = \left\langle\operatorname{sg}\left[\begin{pmatrix} -a_s & b_s \\ a_s & -b_s \end{pmatrix}\right], M_\theta(s) \right\rangle_F \\ &= \operatorname{sg}[a_s]\left[H(q_\theta,p) - H(q_\theta)\right] + \operatorname{sg}[b_s]\left[H(p,q_\theta) - H(p)\right]. \tag*{(5)} \end{aligned}

Writing the coefficient matrix as WsW_s, its columns preserve each candidate source’s self-observation as a baseline. The resulting contrasts are R=DKL(q∥p)R = D_{\mathrm{KL}}(q\mathbin{\|}p) and F=DKL(p∥q)F = D_{\mathrm{KL}}(p\mathbin{\|}q); Appendix § 8.§ 1 derives these identities from the observations.

Differentiating the matrix with respect to the current student logits yields

gR=∇z(M21−M11)=q⊙(log⁡(q/p)−R1),g_R = \nabla_z(M_{21} - M_{11}) = q \odot\left(\log(q/p) - R\mathbf{1}\right),
gF=∇z(M12−M22)=q−p,g_F = \nabla_z(M_{12} - M_{22}) = q - p,
∇zℓs=−as∇zM11+bs∇zM12+as∇zM21−bs∇zM22=asgR+bsgF.(6)\nabla_z\ell_s = -a_s\nabla_zM_{11} + b_s\nabla_zM_{12} + a_s\nabla_zM_{21} - b_s\nabla_zM_{22} = a_sg_R + b_sg_F. \tag*{(6)}

The student self-observation and both cross-observations supply direct derivatives. The frozen teacher’s self-observation has ∇zM22=0\nabla_zM_{22} = 0 and conditions the coefficients through its value. Thus all four entries contribute to the update. § 5 specifies the coefficients; Appendix § 8.§ 1 gives the entrywise derivatives and parameter update.

These derivatives make the complementary roles in § 3.3 explicit. For a fixed positive teacher and qj→0q_j \to0, their coordinates satisfy gR,j→0g_{R,j} \to0 and gF,j→−pjg_{F,j} \to-p_j. The teacher-sourced contrast therefore retains a direct correction for candidates with very small student probability. Coordinating the two directions refines the student’s current allocation while preserving a correction toward teacher-favored alternatives.

Compatibility certified by the observation contrast

Let J(q)=diag⁡(q)−qqTJ(q)=\operatorname{diag}(q)-q q^{\mathsf{T}} be the categorical Fisher matrix[23, 24], with Moore–Penrose pseudoinverse J(q)+J(q)^+. The following identity relates the source-centered gradients through the observation matrix.

Theorem 1 (Observation–gradient identity). For positive p,qp, q, the logit gradients in Eq. (6) satisfy

gRTJ(q)+gF=∑i(qi−pi)log⁡qipi=M12+M21−M11−M22=R+F.(7)g_R^{\mathsf{T}}J(q)^+g_F=\sum_i(q_i-p_i)\log\frac{q_i}{p_i}=M_{12}+M_{21}-M_{11}-M_{22}=R+F. \tag*{(7)}

If p≠qp \ne q, both gradients have positive norm and their Euclidean cosine is strictly greater than −1-1. A unit direction dd therefore exists with gRTd>0g_R^{\mathsf{T}}d>0 and gFTd>0g_F^{\mathsf{T}}d>0.

The total observation excess certifies a common descent direction for both acute and obtuse Euclidean gradient configurations. The identity applies to every aligned pair in § 3.2. Appendix § 9 gives the proof, and Appendix § 10 extends compatibility to a finite family of differentiable strictly convex ff-divergences. We use observation values to select a direction within this feasible set and determine the update’s magnitude. The appendix H.2 provides empirical analysis of the theory.

Method: Observation-Coordinated Updates

At each student-visited prefix, OC-OPD follows an observation →\to contrast →\to coordination →\to strength pipeline. The aligned distributions first determine M(s)M(s), and column-centering yields the ranking/refinement and coverage contrasts together with their candidate-level gradients. The diagonal self-observations specify preferences between these roles; the compatibility geometry then selects a direction that improves both normalized contrasts. Finally, a symmetric discrepancy between pp and qq scales the resulting update.

Algorithm 1 Observation-Coordinated On-Policy Distillation

Require: Frozen teacher pTp_T, student qθq_\theta

Require: Entropy scale h0h_0, preference floor ϵ\epsilon

Require: Discrepancy scale τ\tau, learning rate η\eta

Require: Training dataset D\mathcal{D}, prompt batch size BB

  1. for each training iteration kk do

  2. Set qk←qθkq_k \gets q_{\theta_k} and R←∅\mathcal{R} \gets\varnothing; sample BB prompts B⊂D\mathcal{B} \subset\mathcal{D}.

  3. for each prompt x∈Bx \in\mathcal{B} do

  4. Sample y=(y1,…,y∣y∣)∼qk(⋅∣x)y=(y_1,\ldots,y_{|y|}) \sim q_k(\cdot\mid x).

  5. for t=1t=1 to ∣y∣|y| do

  6. st←(x,y<t)s_t \gets(x,y_{<t}), q←qk(⋅∣st)q \gets q_k(\cdot\mid s_t), p←pT(⋅∣st)p \gets p_T(\cdot\mid s_t).

  7. M←[H(q)H(q,p)H(q,p)H(p)]M \gets\begin{bmatrix}H(q)&H(q,p)\\H(q,p)&H(p)\end{bmatrix}, R←M21−M11R \gets M_{21}-M_{11}, F←M12−M22F \gets M_{12}-M_{22}.

  8. gst∗←0g_{s_t}^* \gets0.

  9. if p≠qp \ne q then

  10. gR←q⊙(log⁡(q/p)−R1)g_R \gets q\odot(\log(q/p)-R\mathbf{1}), gF←q−pg_F \gets q-p.

  11. Compute (ωR,ωF)(\omega_R,\omega_F) from (M11,M22)(M_{11},M_{22}) using h0,ϵh_0,\epsilon.

  12. Compute closed-form bargaining direction d∗d^* from gR,gF,ωR,ωFg_R,g_F,\omega_R,\omega_F.

  13. w←min⁡{12τ∑i(pi−qi)2,1}w \gets\min\left\{\frac{1}{2\tau}\sum_i(\sqrt{p_i}-\sqrt{q_i})^2,1\right\}, gst∗←w∥gF∥2d∗g_{s_t}^* \gets w\lVert g_F\rVert^{2d^*}.

  14. Store (x,y,{gst∗}t=1∣y∣)(x,y,\{g_{s_t}^*\}_{t=1}^{|y|}) in R\mathcal{R}.

  15. end for; collect valid prefixes from R\mathcal{R}.

  16. Lk(θ)←1∣Sk∣∑s∈Sk∑gsg⁡[gs,j∗]log⁡qθ(i∣s)L_k(\theta)\gets\frac{1}{|S_k|}\sum_{s\in S_k}\sum_{g} \operatorname{sg}[g_{s,j}^*]\log q_\theta(i\mid s).

  17. Update student to θk+1\theta_{k+1} with one backward pass on LkL_k at learning rate η\eta.

Observation-conditioned gradient coordination

For h0,ϵ>0h_0,\epsilon>0, the diagonal observations determine normalized entropies and positive preferences:

hS=M11M11+h0,hT=M22M22+h0,h_S=\frac{M_{11}}{M_{11}+h_0},\qquad h_T=\frac{M_{22}}{M_{22}+h_0},
νR=ϵ+(1−hT)hS,νF=ϵ+hT,ωj=νjνR+νF,j∈{R,F}.(8)\nu_R=\epsilon+(1-h_T)h_S,\qquad\nu_F=\epsilon+h_T,\qquad\omega_j=\frac{\nu_j}{\nu_R+\nu_F},\qquad j\in\{R,F\}. \tag*{(8)}

When the teacher is diffuse, this rule gives coverage greater influence; when the teacher is concentrated and the student diffuse, it gives ranking/refinement greater influence. Both sources retain positive preference (Appendix § 8.§ 2).

For p≠qp \ne q, set ηj=∥gj∥2\eta_j = \lVert g_j \rVert_2, ej=gj/ηje_j = g_j/\eta_j, and c=eRTeFc = e_R^{\mathsf{T}}e_F. Gradient bargaining (Navon et al., [25]) selects a direction by maximizing weighted relative improvement:

d∗=arg⁡max⁡∥d∥2≤1eRTd>0, eFTd>0{ωRlog⁡(eRTd)+ωFlog⁡(eFTd)}.(9)d^* = \underset{\substack{\lVert d\rVert_2\leq1\\ e_R^{\mathsf{T}}d>0,\ e_F^{\mathsf{T}}d>0}}{\arg\max}\left\{\omega_R\log(e_R^{\mathsf{T}}d)+\omega_F\log(e_F^{\mathsf{T}}d)\right\}. \tag*{(9)}

Theorem 1 guarantees that the feasible set is nonempty. Its unique solution is

κ=c(ωR−ωF)+c2(ωR−ωF)2+4ωRωF2ωF,\kappa= \frac{c(\omega_R-\omega_F)+\sqrt{c^2(\omega_R-\omega_F)^2+4\omega_R\omega_F}}{2\omega_F},
d∗=κeR+eFD,D=κ2+2κc+1.(10)d^*=\frac{\kappa e_R+e_F}{D},\qquad D=\sqrt{\kappa^2+2\kappa c+1}. \tag*{(10)}

The self-observations thus set priorities, while the current gradient geometry converts them into compatible gains. Equal preferences yield (eR+eF)/∥eR+eF∥2(e_R+e_F)/\lVert e_R+e_F\rVert_2. Appendix § 11 proves the closed form and supplies its numerically stable evaluation; § 6.6 defines comparisons between the adaptive and symmetric directions.

Position strength and realized observation coefficients

Let B=12∑i(pi−qi)2B=\frac{1}{2}\sum_i(\sqrt{p_i}-\sqrt{q_i})^2 and w=min⁡(B/τ,1)w=\min(B/\tau,1) for τ>0\tau>0. Expanding the direction and position strength gives the complete logit signal:

gs∗=wηFd∗=min⁡{12τ∑i(pi−qi)2,1}κ2+2κc+1[ηFκηRgR+gF],(11)g_s^*=w\eta_Fd^*=\frac{\min\left\{\frac{1}{2\tau}\sum_i(\sqrt{p_i}-\sqrt{q_i})^2,1\right\}}{\sqrt{\kappa^2+2\kappa c+1}}\left[\frac{\eta_F\kappa}{\eta_R}g_R+g_F\right], \tag*{(11)}
gs∗=asgR+bsgF,as=wηFκDηR,bs=wD.g_s^*=a_sg_R+b_sg_F,\qquad a_s=\frac{w\eta_F\kappa}{D\eta_R},\qquad b_s=\frac{w}{D}.

These coefficients instantiate WsW_s in Eq. (5). The derivatives supply the candidate directions, the self-observation values condition their relative influence, and ww controls the contribution of the position. The coefficients remain fixed during differentiation.

Update properties. The signal obeys ∥gs∗∥2≤2\lVert g_s^*\rVert_2\leq\sqrt{2}. For qj<pjq_j<p_j, its correction satisfies −gs,j∗≥wωF(pj−qj)-g_{s,j}^*\geq w\omega_F(p_j-q_j), retaining a positive limiting bound as qj→0q_j\to0 for a fixed interior teacher. The coordinated direction has positive inner product with both source-centered gradients. Near agreement, discrepancy gating gives cubic attenuation in ∥q−p∥2\lVert q-p\rVert_2; with aligned gradients, the update reduces to w(q−p)w(q-p) (Appendices § 12, § 13, and § 11.§ 1). The component comparisons examine these three aspects of the update: candidate support, direction, and magnitude.

We accumulate the signed surrogate over valid response positions and differentiate it with one student backward pass. Appendix § 15 gives the explicit batch objective, parameter gradient, and numerical implementation.

Experiments

Experimental settings

Models and training data. Students are Qwen3-1.7B and Qwen3-4B, with Qwen3-4B-Base used for the base-initialization study (Qwen Team, [12]). We use Qwen3-30B-A3B-Instruct-2507 and Qwen3-30B-A3B-Thinking-2507 as teachers (Qwen Team, [26]). The main setting pairs the Instruct teacher with non-thinking students; auxiliary settings use the Thinking teacher with thinking students and the Instruct teacher with Qwen3-4B-Base. All methods train on DAPO-Math-17k (Yu et al., [28]) with fresh student rollouts; distillation methods query a frozen teacher at the same student prefixes.

Evaluation benchmarks and metrics. We evaluate AIME24–26, MATH500 (Lightman et al., [29]) and OlympiadBench(math,english) (He et al., [30]) for mathematical reasoning and GPQA-Diamond (Rein et al., [13]) and MBPP+ (Liu et al., [14]) for out-of-distribution transfer. Avg@32 is the mean accuracy over 32 responses per problem and Pass@32 the fraction of problems solved by at least one; OOD columns report Avg@32 in Table 1 and Pass@32 in Table 3. Additional mathematical benchmarks are reported with Avg@8 or Pass@8 as indicated. Variance and statistical-significance analyses for the AIME evaluations are provided in Appendix § 17. We compare with GRPO (Shao et al., [31]), OPD (Agarwal et al., [32]), OPSD (Zhao et al., [33]), EOPD (Jin et al., [8]), and KD (Hinton et al., [1]; Kim and Rush, [2]), as applicable. The 4B non-thinking setting also reports pure RKL and FKL objectives. Within each comparison, methods share initialization, prompts, rollout budget, decoding, and evaluation protocol, and teacher-based methods share the same teacher.

MethodsMath (I.D.)O.O.D.
AIME24 avg@32AIME24 pass@32AIME25 avg@32AIME25 pass@32AIME26 avg@32AIME26 pass@32OlympiadBench avg@8OlympiadBench pass@8GPQA_D pass@32MBPP+ pass@32
1.7B50.9483.3337.1973.3338.6570.0059.3379.3877.7884.66
+ OPD48.5486.6740.2173.3342.0873.3358.9179.0879.2983.60
+ OC-OPD52.8186.6746.4676.6648.8576.6759.7080.1281.8285.45
4B72.7186.6765.5283.3361.0483.3369.0884.9184.3487.04
+ OPD78.1383.3365.9480.0064.1783.3369.8885.7281.3187.57
+ OC-OPD80.2193.3366.9886.6766.7790.0071.9088.0985.3588.36

Table 3. Thinking-model results (accuracy %). Shaded rows denote OC-OPD.

Non-thinking student results

Performance on mathematical reasoning. Table 1 shows consistent gains for both non-thinking student scales. For Qwen3-1.7B, OC-OPD outperforms GRPO, OPD, and OPSD on all six AIME metrics, beating the strongest baseline by +6.88 Avg@32 and +11.11 Pass@32 points on average. For Qwen3-4B, it exceeds EOPD by +2.74 Avg@32 and +3.33 Pass@32 points across AIME24–26. OC-OPD is best or tied-best in five of six columns, with RKL higher only on AIME24 Avg@32 by 0.10 points. Thus, the gains cover both average accuracy and multi-sample success.

Combining the two KL directions. Neither KL direction dominates at 4B: RKL leads FKL by 5.62 Avg@32 points on AIME24, whereas FKL is stronger on AIME25–26 and keeps MBPP+ at initialization. OC-OPD retains both strengths: it is within 0.10 points of RKL on AIME24, exceeds the stronger one by +2.92 and +4.37 Avg@32 points on AIME25–26, and surpasses both OOD. It also beats the adaptive linear mixture on all six AIME metrics (Table 4), so the gain comes from coordination rather than mixing alone.

MethodAIME 2024 Avg@32AIME 2024 Pass@32AIME 2025 Avg@32AIME 2025 Pass@32AIME 2026 Avg@32AIME 2026 Pass@32
OC-OPD60.9493.3356.6783.3358.5486.67
Forward KL (FKL)55.4283.3353.7573.3354.1783.33
Reverse KL (RKL)61.0486.6751.0476.6752.1980.00
Adaptive linear mixture60.2186.6753.0280.0053.9680.00
Equal-preference58.6586.6753.5480.0056.6783.33
w/o position gate60.6390.0054.0680.0056.6783.33

Table 4. Ablation study. The student is Qwen3-4B with non-thinking generation and the teacher is Qwen3-30B-A3B-Instruct-2507. Bold denotes the best value in each column.

Out-of-distribution evaluation

Generalization beyond mathematical reasoning. Although trained only on DAPO-Math-17k, OC-OPD also improves out-of-distribution performance (Table 1, last two columns). Relative to its initialization, the 1.7B student gains +4.42 points on GPQA-Diamond and +2.98 points on MBPP+. At 4B, it reaches 46.80 and 66.01, exceeding EOPD by +1.35 and +1.20 points. On MBPP+, OC-OPD stays above initialization, whereas EOPD and OPD (RKL) drop by 0.74 and 1.26 points.

Base-model initialization

Performance from base-model initialization. We further test whether the gains depend on instruction-tuned initialization by distilling into Qwen3-4B-Base. Table 2 shows that OC-OPD reaches 40.00 Pass@8 on both AIME24 and AIME25, improving the base model by +16.67 points and EOPD by +6.67 points on each benchmark. Across sampling budgets (Figure 2, Figure 3), OC-OPD exceeds baseline at four of five tested values of kk and ties at k=32k = 32, so the advantage persists from a substantially weaker initialization.

MethodsAIME 2024 Avg@8AIME 2024 Pass@8AIME 2025 Avg@8AIME 2025 Pass@8OlympiadBench Pass@8MATH500 Avg@8MATH500 Pass@8
Qwen3-4B-Base12.1923.337.2923.3352.5248.6085.20
+ KD12.5026.6712.0823.3362.4374.7392.00
+ OPD18.9133.3312.5030.0058.8078.4090.40
+ EOPD17.9233.3317.5033.3362.6180.0592.60
+ OC-OPD21.6740.0018.7540.0063.2081.7593.20

Table 2. Avg@8 and Pass@8 performance on AIME24–25, MATH500, and OlympiadBench with the Qwen3-4B-Base student. Shaded rows denote OC-OPD. Bold denotes the best value in each column; dashes indicate unavailable scores.

Figure 2. Pass@k performance comparison between OPD, EOPD and OC-OPD on AIME24 with the Qwen3-4B-Base student.

Figure 3. Pass@k performance comparison between OPD, EOPD and OC-OPD on AIME25 with the Qwen3-4B-Base student.

Thinking teacher and thinking students

Performance with thinking generation. Table 3 shows that the advantage of OC-OPD persists with thinking generation. Relative to OPD, the 1.7B student improves by +5.76 Avg@32 and +2.22 Pass@32 points on average across AIME24–26, while the 4B student gains +1.91 and +7.78 points, respectively. OC-OPD also exceeds OPD on both GPQA-Diamond and MBPP+ at both model scales. These results show that the proposed update is not specific to non-thinking decoding and remains effective when the student already produces explicit reasoning traces.

Ablation study

Effect of individual design choices. Table 4 shows that no ablated objective matches the full OC-OPD across AIME24–26. RKL is only 0.10 points higher on AIME24 Avg@32, but is clearly worse on AIME25–26 and on Pass@32, while FKL shows a similar coverage gap. Adaptive linear mixing and equal-preference weighting recover part of the performance but remain below the full method. Removing the position gate is the strongest ablation, yet it reduces Pass@32 by about 3.33 points on all three benchmarks. These comparisons show that direction-aware preferences and position-sensitive weighting provide complementary benefits. Detailed definitions are given in Appendix § 16.§ 4; the corresponding AIME results and statistical-analysis protocol are summarized in Appendix § 17.

Conclusion

OC-OPD organizes teacher and student predictions at each student-generated prefix into four self- and cross-observations, yielding a unified supervision state. Source-centered contrasts recover the complementary ranking and coverage corrections of reverse and forward KL. The observation–gradient identity establishes their local compatibility, while observation-conditioned coordination combines them in a bounded update that retains a correction for teacher-supported candidates with low student probability. Experiments across non-thinking, thinking, and base-initialized students show consistent gains in mathematical reasoning. Scientific question answering and code generation further assess transfer beyond the mathematical training domain.

AI use statement

Generative AI tools were used solely for limited assistance with language polishing, formatting, routine coding, and debugging. The research, analysis, interpretation of results, and preparation of the final content were carried out by the human authors. All results and citations were independently verified by the authors, who take full responsibility for the work.

Ethics Statement

This work studies on-policy distillation using publicly available datasets and pre-trained open-source models, and all experiments follow the licenses of the corresponding resources. Because distillation can transfer characteristics of the teacher into the student, downstream use should include task-appropriate evaluation of factuality, bias, and safety together with standard deployment safeguards.

Reproducibility statement

Full proofs appear in Appendices. Appendix § 15 describes the gradient implementation, and Appendix § 16 provides training and sampling hyperparameters, evaluation protocols, and controlled comparisons.

Notation, assumptions, and the observation matrix

SymbolMeaning
s=(x,y<t)s=(x,y_{<t})A prefix visited by the current student behavior policy
p,q∈ΔV∘p,q \in\Delta_{\mathsf{V}}^{\circ}Frozen teacher and current student distributions with positive support
z∈RVz \in\mathbb{R}^{\mathsf{V}}Student logits, with q=softmax⁡(z)q=\operatorname{softmax}(z)
MMSelf-/cross-entropy observation matrix in Eq. (3)
R,FR,FStudent-sourced and teacher-sourced observation contrasts
gR,gFg_R,g_FTheir Euclidean gradients with respect to current logits
J(q)J(q)diag⁡(q)−qqT\operatorname{diag}(q)-qq^{\mathsf{T}}, with kernel span⁡{1}\operatorname{span}\{1\}
nj,ejn_j,e_jGradient norm ∥gj∥2\lVert g_j\rVert_2 and normalized gradient gj/njg_j/n_j
ccEuclidean cosine eReFe_R e_F
ωR,ωF\omega_R,\omega_FPositive bargaining preferences summing to one
d∗d^*Unit-norm bargaining direction, used with a minus sign in descent
B,wB,wSquared Hellinger discrepancy and position weight
g∗g^*Final logit signal wnFd∗w_nF d^*
sg⁡[⋅]\operatorname{sg}[\cdot]Stop-gradient operator evaluated at the current snapshot
As,KstA_s,K_{st}Parameter-to-logit Jacobian and block coupling AsAtTA_sA_t^{\mathsf{T}}

Table 5. Notation used throughout the paper. All local quantities refer to the same current prefix.

Interior support. The theoretical domain is ΔV∘={q∈RV:qi>0, ∑iqi=1}\Delta_{\mathsf{V}}^{\circ}=\{q\in\mathbb{R}^{\mathsf{V}}:q_i>0,\ \sum_i q_i=1\}. Theorems 1 and 5 hold throughout this open simplex, including arbitrarily small positive coordinates. The expansion around agreement and the coordinate limit gR,j→0g_{R,j}\to0 use a fixed interior teacher. Boundary limits are taken through interior distributions, with divergence values interpreted in the extended-real sense at boundary points.

Freezing conventions. The teacher probabilities, prefix samples, preferences, gate, and bargaining solution are detached during the local optimization step. The student probabilities inside R,FR,F in Eq. (5), or inside log⁡qθ\log q_\theta in Eq. (18), remain differentiable. The differentiated objective is conditional on the current prefix distribution. Fresh student sampling supplies the state distribution for each subsequent update.

What each observation contributes. M11M_{11} supplies student dispersion and the student-source baseline. M22M_{22} supplies teacher dispersion and the teacher-source baseline. The cross-observations M21M_{21} and M12M_{12} evaluate each source under the other model. Their within-source differences define centered supervision signals, and their total excess supplies the geometric certificate. Frozen teacher entropy influences the adaptive coefficients through its numerical value.

Organized observations and candidate resolution. The matrix provides a canonical organization of four expected scores. The aligned distributions p,qp,q provide candidate-level probabilities, likelihood ratios, and gradient geometry. OC-OPD uses this information jointly: ww determines position strength and g∗g^* allocates the learning signal. Every candidate remains represented in the gradient construction.

From the four observations to a parameter gradient

The convention H(a,b)=−∑iailog⁡biH(a,b)=-\sum_i a_i\log b_i averages the evaluator’s negative log probability under candidate source aa. Each column fixes that source while changing the evaluator. The centered contrasts in Eq. (4) therefore admit the representations

R=H(q,p)−H(q)=∑iqilog⁡(qipi)=DKL(q∥p),R=H(q,p)-H(q)=\sum_i q_i\log\left(\frac{q_i}{p_i}\right)=D_{\mathrm{KL}}(q\mathbin{\|}p),
F=H(p,q)−H(p)=∑ipilog⁡(piqi)=DKL(p∥q).F=H(p,q)-H(p)=\sum_i p_i\log\left(\frac{p_i}{q_i}\right)=D_{\mathrm{KL}}(p\mathbin{\|}q).

These identities express the source-centered observation costs in standard divergence notation.

Entrywise derivatives. At a fixed prefix, let q=qθ=softmax⁡(z)q=q_\theta=\operatorname{softmax}(z) and hold pp fixed. Direct differentiation of the four entries gives

∇zM11=−q⊙(log⁡q+H(q)1),∇zM12=q−p,∇zM21=−q⊙(log⁡p+H(q,p)1),∇zM22=0.(12)\begin{aligned} \nabla_z M_{11}&=-q\odot\left(\log q+H(q)\mathbf{1}\right),\\ \nabla_z M_{12}&=q-p,\\ \nabla_z M_{21}&=-q\odot\left(\log p+H(q,p)\mathbf{1}\right),\\ \nabla_z M_{22}&=0. \tag*{(12)} \end{aligned}

For the first identity, ∇qH(q)=−(log⁡q+1)\nabla_q H(q)=-(\log q+1) and multiplication by J(q)J(q) subtracts the qq-mean. For the third, ∇qH(q,p)=−log⁡p\nabla_q H(q,p)=-\log p, so the same centering gives the stated expression. The second is the usual cross-entropy logit derivative, and the last follows because the teacher is frozen.

Substituting these derivatives into the matrix loss yields

∇zℓs=−as∇zM11+bs∇zM12+as∇zM21−bs∇zM22=asq⊙(log⁡(q/p)−R1)+bs(q−p).(13)\begin{aligned} \nabla_z\ell_s &= -a_s\nabla_zM_{11}+b_s\nabla_zM_{12}+a_s\nabla_zM_{21}-b_s\nabla_zM_{22}\\ &=a_sq\odot\left(\log(q/p)-R1\right)+b_s(q-p). \tag*{(13)} \end{aligned}

The student self-observation contributes the centering term through its derivative. Combining it with the teacher’s evaluation gives the student-sourced direction gRg_R. For the teacher-sourced contrast, the constant H(p)H(p) establishes the zero-at-agreement baseline, and H(p,q)H(p,q) supplies the derivative gFg_F.

Values condition the update; derivatives provide their direction. All four entries are evaluated at the current snapshot. The diagonal values condition the preference rule in Eq. (15); the two excesses and their derivatives define the candidate-level objectives and geometry. Teacher entropy influences the numerical coefficients through the conditioning path, with a zero direct derivative. Alignment and gate saturation can make some coefficient sensitivities vanish at particular pairs, while the observation-to-gradient map retains all four inputs.

The coefficient structure WsW_s pairs each source’s self- and cross-observation through one realized coefficient. The frozen-coefficient convention differentiates MθM_\theta with as,bsa_s,b_s held at their current values. In the product-rule expansion, the derivative of these fixed coefficients is zero, yielding exactly as∇θR+bs∇θFa_s\nabla_\theta R+b_s\nabla_\theta F. Recomputing the coefficients after each student update refreshes their dependence on the current observations.

Exact batch gradient. Let SkS_k contain the NkN_k valid response positions in a fresh student batch, and evaluate the controller at θk\theta_k. With As=∂zs/∂θA_s=\partial z_s/\partial\theta, define

Lk(θ)=1Nk∑s∈Sk⟨sg⁡[Ws(θk)],Mθ(s)⟩F,L_k(\theta)=\frac{1}{N_k}\sum_{s\in S_k}\left\langle\operatorname{sg}[W_s(\theta_k)],M_\theta(s)\right\rangle_F,
∇θLk(θ)∣θk=1Nk∑s∈SkAsT(asgR,s+bsgF,s)=1Nk∑s∈SkAsTgs∗.(14)\left.\nabla_\theta L_k(\theta)\right|_{\theta_k} =\frac{1}{N_k}\sum_{s\in S_k}A_s^{\mathsf T}(a_sg_{R,s}+b_sg_{F,s}) =\frac{1}{N_k}\sum_{s\in S_k}A_s^{\mathsf T}g_s^*. \tag*{(14)}

For plain SGD, θk+1=θk−ηk∇θLk(θk)\theta_{k+1}=\theta_k-\eta_k\nabla_\theta L_k(\theta_k). Other optimizers consume the same gradient with their own parameter-space transformation. The coefficients are deterministic functions of current distributions. Each update refreshes the matrix and coefficient computation, with the complete autoregressive prefix serving as the conditioning state.

Complete current-state coefficient rule

Self-observations to preferences. Define

hS=M11M11+h0,hT=M22M22+h0,h_S=\frac{M_{11}}{M_{11}+h_0},\qquad h_T=\frac{M_{22}}{M_{22}+h_0},
νF=ϵ+hT,νR=ϵ+(1−hT)hS,ωj=νjνR+νF,j∈{R,F}.(15)\nu_F=\epsilon+h_T,\qquad\nu_R=\epsilon+(1-h_T)h_S,\qquad\omega_j=\frac{\nu_j}{\nu_R+\nu_F},\qquad j\in\{R,F\}. \tag*{(15)}

Here h0,ϵ>0h_0,\epsilon>0. The νj\nu_j are unnormalized preferences; they are distinct from the realized coefficients as,bsa_s,b_s below. Increasing hTh_T shifts preference toward the teacher-sourced signal; increasing hSh_S at fixed hTh_T shifts it toward the student-sourced signal. A concentrated teacher paired with a diffuse student therefore emphasizes the teacher’s assessment of student-distributed candidates. A diffuse teacher gives greater priority to representing its distributed support. The positive floor retains both views when both distributions are concentrated, including when their favored candidates differ. This continuous entropy-aware parameterization instantiates the preference layer of the controller. The compatibility and bargaining results hold for every positive preference pair, separating the structural guarantee from the specific preference map.

Cross-observation excesses to direction. Compute gR,gFg_R,g_F from Eq. (6), nj=∥gj∥2n_j=\lVert g_j\rVert_2, ej=gj/nje_j=g_j/n_j, and c=eRTeFc=e_R^{\mathsf T}e_F. Let κ,D\kappa,D be the closed-form quantities in Eq. (25). Then d∗=(κeR+eF)/Dd^*=(\kappa e_R+e_F)/D. The ratio κ\kappa is an algebraic solution of the two-objective bargaining problem; it is distinct from the iteration index kk.

Discrepancy to position strength. With τ>0\tau>0, set

B=12∑i(pi−qi)2∈[0,1],(16)B=\frac{1}{2}\sum_i\left(\sqrt{p_i}-\sqrt{q_i}\right)^2\in[0,1], \tag*{(16)}
w=arg max⁡0≤u≤1{uB−τ2u2}=clip⁡(B/τ,0,1).w=\underset{0\le u\le1}{\operatorname{arg\,max}}\left\{uB-\frac{\tau}{2}u^2\right\}=\operatorname{clip}(B/\tau,0,1).

This bounded discrepancy is computed directly from the aligned distributions and controls position strength. It complements the matrix’s organization of expected scores with a full-distribution measure of agreement. Hellinger-based supervision also appears in sampled excess-token reward shaping [11]; here it determines the position-level coefficient. Appendix § 12 proves its unique scalar optimum.

Realized loss coefficients. Expanding gs∗=wnFd∗g_s^* = w n_F d^* in the source-centered observation gradients gives

as=wnFκDnR,bs=wD,gs∗=asgR+bsgF.(17)a_s = \frac{w n_F \kappa}{D n_R}, \qquad b_s = \frac{w}{D}, \qquad g_s^* = a_s g_R + b_s g_F. \tag*{(17)}

Preferences, realized influence, and total strength have complementary roles: ωj\omega_j values relative gains, the gradient angle determines the compromise, and wnFw n_F sets its norm. Near agreement, the small discrepancy smoothly attenuates the final signal at any preference. The coefficients above make Eq. (5) an exact current-point realization of the controller.

A convenient gradient surrogate. A numerically direct realization computes the vector gs∗g_s^* and uses

L~s(θ)=∑isg⁡[gs,i∗]log⁡qθ(i∣s).(18)\widetilde{\mathcal{L}}_s(\theta) = \sum_i \operatorname{sg}[g_{s,i}^*]\log q_\theta(i \mid s). \tag*{(18)}

Since 1Tgs∗=0\mathbf{1}^{\mathsf{T}}g_s^* = 0, its current logit gradient is exactly gs∗g_s^*; Appendix § 15 expands the derivative. The matrix loss and signed surrogate have identical current gradients. Their scalar values follow their respective definitions.

Complete proof of the compatibility identity

Lemma 2 (Logit differentiation and the tangent space). Let q=softmax⁡(z)q = \operatorname{softmax}(z). Then ∂q/∂z=J(q)\partial q/\partial z = J(q), J(q)1=0J(q)\mathbf{1} = 0, and J(q)J(q) is positive definite on T={v:1Tv=0}T = \{v : \mathbf{1}^{\mathsf{T}}v = 0\}. For any u,v∈Tu,v \in T,

uTJ(q)+v=uTdiag⁡(q)−1v.(19)u^{\mathsf{T}}J(q)^{+}v = u^{\mathsf{T}}\operatorname{diag}(q)^{-1}v. \tag*{(19)}

Proof. The softmax derivative is ∂qi/∂zj=qi(1i=j−qj)\partial q_i/\partial z_j = q_i(\mathbb{1}_{i=j}-q_j). For any xx, xTJx=∑iqixi2−(∑iqixi)2=Var⁡q(xi)x^{\mathsf{T}}Jx = \sum_i q_i x_i^2 - (\sum_i q_i x_i)^2 = \operatorname{Var}_q(x_i). Since every qi>0q_i > 0, this variance is zero exactly when xx is constant. Thus the kernel is span⁡{1}\operatorname{span}\{\mathbf{1}\} and restriction to TT is positive definite.

For v∈Tv \in T, define x=diag⁡(q)−1vx = \operatorname{diag}(q)^{-1}v. Then

Jx=v−q(qTdiag⁡(q)−1v)=v−q1Tv=v.Jx = v - q(q^{\mathsf{T}}\operatorname{diag}(q)^{-1}v) = v - q\mathbf{1}^{\mathsf{T}}v = v.

The vector J+vJ^{+}v is the unique zero-sum solution of this equation; any two solutions differ by a constant vector. Hence J+v=x+α1J^{+}v = x + \alpha\mathbf{1} for some scalar α\alpha. Multiplication by uTu^{\mathsf{T}}, with u∈Tu \in T, proves Eq. (19). This equality identifies the bilinear forms restricted to the tangent space TT.

Lemma 3 (KL gradients and their zero sets). With the teacher frozen, Eq. (6) holds. Each gradient is zero if and only if p=qp = q.

Proof. Let ℓi=log⁡(qi/pi)\ell_i = \log(q_i/p_i). Differentiating R=∑iqiℓiR = \sum_i q_i\ell_i with respect to qq gives ℓ+1\ell+ \mathbf{1}. Therefore

∇zR=J(ℓ+1)=q⊙ℓ−q(qTℓ)=q⊙(ℓ−R1).\nabla_z R = J(\ell+ \mathbf{1}) = q \odot\ell- q(q^{\mathsf{T}}\ell) = q \odot(\ell- R\mathbf{1}).

For F=∑ipilog⁡pi−∑ipilog⁡qiF = \sum_i p_i\log p_i - \sum_i p_i\log q_i, the derivative of log⁡qi\log q_i gives ∇zF=q−p\nabla_z F = q - p. Each gradient sums to zero. If gF=0g_F = 0, equality of distributions is immediate. If gR=0g_R = 0, positivity of qiq_i implies ℓi=R\ell_i = R for every ii, so qi=eRpiq_i = e^R p_i. Normalization forces eR=1e^R = 1, and q=pq = p. The converse follows by substitution.

Proof of Theorem 1. By Lemmas 2 and 3,

gRTJ+gF=∑i(qi−pi)(ℓi−R)(20)g_R^{\mathsf{T}}J^{+}g_F = \sum_i(q_i-p_i)(\ell_i-R) \tag*{(20)}
=∑iqilog⁡(qi/pi)+∑ipilog⁡(pi/qi)(21)= \sum_i q_i\log(q_i/p_i) + \sum_i p_i\log(p_i/q_i) \tag*{(21)}
=R+F.(22)= R + F. \tag*{(22)}

The RR-multiple disappears because the distributions have equal total mass. Substituting their definitions in Eq. (3) proves the matrix contrast.

If p≠qp \ne q, strict nonnegativity of KL implies R+F>0R+F>0, and both gradients are nonzero. Suppose their Euclidean cosine were −1-1. Then gR=−αgFg_R=-\alpha g_F for some α>0\alpha>0, and

gRTJ+gF=−αgFTJ+gF<0,g_R^{\mathsf T}J^+g_F=-\alpha g_F^{\mathsf T}J^+g_F<0,

contradicting the identity. Thus c>−1c>-1. The vector

d0=eR+eF2+2cd_0=\frac{e_R+e_F}{\sqrt{2+2c}}

has unit norm and satisfies eRTd0=eFTd0=(1+c)/2+2c>0e_R^{\mathsf T}d_0=e_F^{\mathsf T}d_0=(1+c)/\sqrt{2+2c}>0. This is an explicit common-descent witness. The case c=1c=1 is permitted and simply gives d0=eR=eFd_0=e_R=e_F.

Metric-dependent gradient geometry. Equation (7) weights coordinate products by 1/qi1/q_i, giving small-student-probability coordinates substantial influence. The Euclidean inner product weights these coordinates uniformly. The identity therefore certifies compatibility in the Fisher inverse metric across both acute and obtuse Euclidean configurations. At each interior pair with p≠qp\ne q, the Euclidean cosine satisfies c>−1c>-1; the resulting angular margin 1+c1+c is pair-dependent.

Extension to strictly convex divergences

The following extension identifies a monotonicity principle underlying the compatibility identity. It establishes a shared descent direction for a finite family of smooth, strictly convex distributional contrasts.

Theorem 4 (Monotone divergence compatibility). Let p,q∈Δ∘p,q\in\Delta^\circ, p≠qp\ne q, and let f,h:(0,∞)→Rf,h:(0,\infty)\to\mathbb{R} be continuously differentiable strictly convex functions with f(1)=h(1)=0f(1)=h(1)=0. Define

Df(q∥p)=∑ipif(ri),ri=qipi,gf=∇zDf(q∥p),D_f(q\|p)=\sum_i p_i f(r_i),\qquad r_i=\frac{q_i}{p_i},\qquad g_f=\nabla_zD_f(q\|p),

and similarly define ghg_h. Then

gf=q⊙(f′(r)−Eq[f′(r)]1),(23)g_f=q\odot\left(f'(r)-\mathbb{E}_q[f'(r)]\mathbf{1}\right), \tag*{(23)}
gfTJ+gh=12∑i,jqiqj(f′(ri)−f′(rj))(h′(ri)−h′(rj))>0.(24)g_f^{\mathsf T}J^+g_h=\frac{1}{2}\sum_{i,j}q_iq_j\left(f'(r_i)-f'(r_j)\right)\left(h'(r_i)-h'(r_j)\right)>0. \tag*{(24)}

For any finite collection f1,…,fmf_1,\ldots,f_m satisfying these assumptions, there is a single direction dd such that gfjTd>0g_{f_j}^{\mathsf T}d>0 for every jj.

Proof. The derivative of DfD_f with respect to qiq_i is f′(ri)f'(r_i). Multiplying by the softmax Jacobian gives Eq. (23). Applying Lemma 2,

gfTJ+gh=∑iqi(f′(ri)−Eqf′)(h′(ri)−Eqh′)=Cov⁡q(f′(r),h′(r)).g_f^{\mathsf T}J^+g_h=\sum_iq_i\left(f'(r_i)-\mathbb{E}_qf'\right)\left(h'(r_i)-\mathbb{E}_qh'\right)=\operatorname{Cov}_q(f'(r),h'(r)).

For arbitrary sequences ai,bia_i,b_i, expansion of the double sum gives

Cov⁡q(a,b)=12∑i,jqiqj(ai−aj)(bi−bj).\operatorname{Cov}_q(a,b)=\frac{1}{2}\sum_{i,j}q_iq_j(a_i-a_j)(b_i-b_j).

The derivative of a differentiable strictly convex function on an interval is strictly increasing: equality of its derivative at two distinct points would force it to be constant between them and the function to be affine on that interval. Hence every summand in Eq. (24) is nonnegative. Since p≠qp\ne q, at least two ratios rir_i differ, so at least one summand is strictly positive. This proves the pairing claim.

For a finite family, set v=J+∑j=1mgfjv=J^+\sum_{j=1}^{m}g_{f_j}. For every kk,

gfkTv=∑jgfkTJ+gfj>0.g_{f_k}^{\mathsf T}v=\sum_jg_{f_k}^{\mathsf T}J^+g_{f_j}>0.

In particular v≠0v\ne0, so d=v/∥v∥2d=v/\lVert v\rVert_2 is a unit common-descent direction.

Generators and assumptions. The generators f(r)=rlog⁡rf(r)=r\log r and h(r)=−log⁡rh(r)=-\log r recover the two source-centered observation contrast definitions. Adding an affine function to a generator preserves its logit gradient for normalized distributions. Differentiability supplies the gradient representation, and strict convexity supplies the strictly increasing derivatives used in the positivity argument. For general differentiable convex generators, the covariance is nonnegative; strict positivity holds when at least one positively weighted pair contributes a strictly positive product.

Compatibility and the bounded controller. The witness J+∑jgjfjJ^{+}\sum_j g_j f_j establishes feasibility for the finite family. Its norm reflects the inverse metric and can grow near the simplex boundary. OC-OPD uses a unit Euclidean budget and the explicit two-direction solution to implement bounded local updates. The extension explains the geometric principle, while the source-centered observation construction specifies the candidate populations and statistical meaning of the implemented directions.

Existence, uniqueness, and the closed-form bargaining solution

We prove Theorem 5 and two useful consequences. Let uj(d)=ejTdu_j(d)=e_j^{\mathsf T}d, ωR+ωF=1\omega_R+\omega_F=1, and ωj>0\omega_j>0 throughout this section.

Theorem 5 (Unique closed-form compromise). Under Theorem 1 with p≠qp\ne q and ωR,ωF>0\omega_R,\omega_F>0, Eq. (9) has a unique solution. Set

κ=c(ωR−ωF)+c2(ωR−ωF)2+4ωRωF2ωF,D=κ2+1+2κc.(25)\kappa=\frac{c(\omega_R-\omega_F)+\sqrt{c^2(\omega_R-\omega_F)^2+4\omega_R\omega_F}}{2\omega_F}, \qquad D=\sqrt{\kappa^2+1+2\kappa c}. \tag*{(25)}

Then d∗=(κeR+eF)/Dd^*=(\kappa e_R+e_F)/D, and both gjTd∗g_j^{\mathsf T}d^* are strictly positive.

Existence. Theorem 1 gives a feasible point d0d_0. On the closed unit ball both utilities lie in [−1,1][-1,1]. For a maximizing sequence with objective at least that of d0d_0, both utilities stay bounded away from zero: approaching zero would send a positive-weight logarithm to −∞-\infty, while the other logarithm is at most zero. A convergent subsequence of this sequence therefore has a limit in the strictly positive-utility domain. Continuity proves that the maximum is attained.

The norm constraint is active. If a feasible dd has ∥d∥2<1\lVert d\rVert_2<1, then d/∥d∥2d/\lVert d\rVert_2 is feasible and increases the objective by −log⁡∥d∥2>0-\log\lVert d\rVert_2>0. Every optimum consequently has unit norm.

Uniqueness. Suppose distinct optima d1,d2d_1,d_2 exist. Their midpoint is feasible and, by concavity of the log utilities, has objective at least the optimal value. Strict convexity of the Euclidean ball implies ∥(d1+d2)/2∥2<1\lVert(d_1+d_2)/2\rVert_2<1 for distinct unit vectors. Radially enlarging the midpoint strictly increases the objective, a contradiction. This proof covers collinear gradients with c=1c=1 as well as linearly independent gradients.

First-order conditions. The objective is differentiable at the optimum. Its gradient is

∇dΦ(d)=ωReRuR(d)+ωFeFuF(d).\nabla_d\Phi(d)=\frac{\omega_R e_R}{u_R(d)}+\frac{\omega_F e_F}{u_F(d)}.

Writing the multiplier for (∥d∥2−1)/2≤0(\lVert d\rVert^2-1)/2\le0 as λ\lambda, the KKT condition is ∇Φ(d)=λd\nabla\Phi(d)=\lambda d. Taking the inner product with the unit vector dd gives λ=ωR+ωF=1\lambda=\omega_R+\omega_F=1. Hence

d=aeR+beF,a=ωRuR(d)>0,b=ωFuF(d)>0.(26)d=a e_R+b e_F,\qquad a=\frac{\omega_R}{u_R(d)}>0,\quad b=\frac{\omega_F}{u_F(d)}>0. \tag*{(26)}

Here a,ba,b are coefficients of the unit direction in normalized gradients; the state-indexed as,bsa_s,b_s in Eq. (17) multiply the unnormalized gradients in the final update. It follows that

a2+cab=ωR,b2+cab=ωF.(27)a^2+cab=\omega_R,\qquad b^2+cab=\omega_F. \tag*{(27)}

For κ=a/b>0\kappa=a/b>0, eliminating bb yields

ωFκ2+c(ωF−ωR)κ−ωR=0.\omega_F\kappa^2+c(\omega_F-\omega_R)\kappa-\omega_R=0.

The constant term is negative and the leading coefficient positive, so exactly one root is positive. This is Eq. (25). Since the actual unique optimum has this positive ratio, normalizing κeR+eF\kappa e_R+e_F reconstructs it. Conversely, this vector satisfies the first-order condition and positive-utility constraints and is therefore optimal by concavity. This completes the proof of Theorem 5.

Corollary 6 (Scale invariance and proportional fairness). The direction d∗d^* is invariant to positive rescaling of either objective at fixed preferences. For every feasible vv with ∥v∥2≤1\lVert v\rVert_2\le1,

ωRuR(v)−uR(d∗)uR(d∗)+ωFuF(v)−uF(d∗)uF(d∗)≤0.(28)\omega_R\frac{u_R(v)-u_R(d^*)}{u_R(d^*)}+\omega_F\frac{u_F(v)-u_F(d^*)}{u_F(d^*)}\le0. \tag*{(28)}

Proof. For αj>0\alpha_j>0, normalization gives αjgj/∥αjgj∥2=ej\alpha_jg_j/\lVert\alpha_jg_j\rVert_2=e_j, so each utility and its maximizing direction are invariant. By the KKT equation,

∑jωjuj(v)uj(d∗)=d∗Tv≤1.\sum_j\omega_j\frac{u_j(v)}{u_j(d^*)}=d^{*\mathsf T}v\le1.

Subtracting ∑jωj=1\sum_j\omega_j=1 proves Eq. (28). The inequality characterizes the weighted relative local gains of the two source-centered contrasts. ▫

Equal weights and the simplest strong baseline. If ωR=ωF=1/2\omega_R = \omega_F = 1/2, then κ=1\kappa= 1 for every c>−1c > -1. Thus d∗=(eR+eF)/∥eR+eF∥2d^* = (e_R + e_F) / \lVert e_R + e_F \rVert_2. This direction is already a feasible common descent direction, independently of uncertainty. The appropriate incremental question is whether observation-dependent preferences yield more useful learning than this symmetric baseline, at a common norm, gate, learning rate search budget, and teacher cost. For c=eR⊤eFc = e_R^\top e_F, the direct normalized-gradient mixture v=αeR+(1−α)eFv = \alpha e_R + (1-\alpha)e_F has unnormalized local gains

eR⊤v=α+(1−α)c,eF⊤v=αc+(1−α).e_R^\top v = \alpha+ (1-\alpha)c,\qquad e_F^\top v = \alpha c + (1-\alpha).

These expressions characterize how unequal mixing preferences interact with the current gradient geometry.

Stable root evaluation. Let A=c(ωR−ωF)A = c(\omega_R-\omega_F) and Q=A2+4ωRωFQ = \sqrt{A^2 + 4\omega_R\omega_F}. An algebraically equivalent computation is

κ={(A+Q)/(2ωF),A≥0,2ωR/(Q−A),A<0.(29)\kappa= \begin{cases} (A+Q)/(2\omega_F), & A \ge0,\\ 2\omega_R/(Q-A), & A < 0. \end{cases} \tag*{(29)}

The second branch avoids subtracting nearly equal positive quantities. Computing QQ with a hypot routine further reduces overflow risk. The reference implementation corrects roundoff by clipping the computed cosine to [−1,1][-1,1] and normalizes the resulting vector directly. Theorem 1 gives c>−1c>-1 for each distinct interior pair.

Aligned-gradient and binary reductions

Proposition 7 (Exact reduction under aligned gradients). If p≠qp \ne q and c=1c=1, then g∗=w(q−p)g^* = w(q-p) for every positive preference pair.

Proof. When eR=eFe_R=e_F, Eq. (9) is maximized by their common unit vector, independently of the preferences. A binary logit tangent space is one-dimensional; Theorem 1 rules out opposite directions, so the normalized gradients coincide.

Distinct normalized candidate directions create room for preference-dependent coordination. A binary vocabulary yields the exact reduction g∗=w(q−p)g^* = w(q-p). The directions also align to first order near agreement, as established in Appendix § 13. Mismatch and gradient angle together characterize the positions at which coordination changes the signal.

Strength, local descent, and candidate-deficit guarantees

Proposition 8 (Local descent, boundedness, and teacher-deficit signal). For p≠qp \ne q, sufficiently small isolated logit steps z+=z−ηg∗z^+ = z-\eta g^* strictly decrease both RR and FF. Moreover,

∥g∗∥2=w∥q−p∥2≤2,−gj∗≥wωF(pj−qj)whenever qj<pj.(30)\lVert g^* \rVert_2 = w\lVert q-p\rVert_2 \le\sqrt{2},\qquad-g_j^* \ge w\omega_F(p_j-q_j)\quad\text{whenever }q_j<p_j. \tag*{(30)}

For a fixed positive teacher with pj=ρ>0p_j=\rho>0, as qj→0q_j\to0, Eq. (15) gives

lim inf⁡qj→0(−gj∗)≥ρϵ1+2ϵmin⁡{1−1−ρτ,1}>0.(31)\liminf_{q_j\to0}(-g_j^*) \ge\rho\frac{\epsilon}{1+2\epsilon}\min\left\{\frac{1-\sqrt{1-\rho}}{\tau},1\right\}>0. \tag*{(31)}

The coordinate lower bound concerns a logit correction. The softmax Jacobian translates this correction into probability changes, whose scale depends on the current student probabilities.

Position-weight optimum. The objective μB−τu2/2\mu B-\tau u^2/2 has second derivative −τ<0-\tau<0, so its constrained maximum is unique. The unconstrained stationary point is u=B/τu=B/\tau; projection onto [0,1][0,1] gives Eq. (16). Because B>0B>0 for distinct distributions, w>0w>0 whenever p≠qp\ne q. If p=qp=q, the selected weight is zero.

Norm bound. For distributions p,qp,q,

∥p−q∥22=∑ipi2+∑iqi2−2∑ipiqi≤2.\lVert p-q\rVert_2^2=\sum_i p_i^2+\sum_i q_i^2-2\sum_i p_iq_i\le2.

since each sum of squares is at most one and the cross term is nonnegative. Since 0≤w≤10\le w\le1 and ∥d∗∥2=1\lVert d^*\rVert_2=1, Eq. (30) follows. The universal constant is a supremum over pairs approaching distinct point masses. With a bounded parameter-to-logit Jacobian, the corresponding parameter gradient is bounded by the product of its operator norm and this logit bound.

An explicit local descent condition. Fix zz with p≠qp \ne q, and let LjL_j be a Lipschitz constant of ∇zℓj\nabla_z \ell_j on a ball of radius r>0r > 0 around zz, for j∈{R,F}j \in\{R,F\}. Such finite constants exist on every compact logit neighborhood for a fixed positive teacher. Set mj=gj∗⊤d∗>0m_j = g_j^{*\top}d^* > 0 and a=wnF>0a = w n_F > 0. The descent lemma gives

ℓj(z−ηad∗)≤ℓj(z)−ηamj+Lj2η2a2.(32)\ell_j(z-\eta a d^*) \le\ell_j(z)-\eta a m_j+\frac{L_j}{2}\eta^2a^2. \tag*{(32)}

Thus both losses strictly decrease whenever

0<η<min⁡{ra,2mRLRa,2mFLFa}.(33)0 < \eta< \min\left\{\frac{r}{a},\frac{2m_R}{L_Ra},\frac{2m_F}{L_Fa}\right\}. \tag*{(33)}

with a term interpreted as +∞+\infty if its Lipschitz constant is zero. The restriction ηa<r\eta a < r keeps the segment inside the ball. This condition relates the admissible local step to the positive gain margin and the smoothness of each source-centered contrast.

Lower bound for a teacher-supported deficit. From Eq. (26), b=ωF/(eF⊤d∗)≥ωFb = \omega_F/(e_F^\top d^*) \ge\omega_F, because 0<eF⊤d∗≤10 < e_F^\top d^* \le1. For qj<pjq_j < p_j, gF,j=qj−pj<0g_{F,j} = q_j-p_j < 0. Also log⁡(qj/pj)<0\log(q_j/p_j) < 0 and R≥0R \ge0, so gR,j<0g_{R,j} < 0. The coefficients a,ba,b are positive; hence

−gj∗=−wnF(aeR,j+beF,j)(34)-g_j^*=-w n_F(a e_{R,j}+b e_{F,j}) \tag*{(34)}
≥−wnFbeF,j=wb(pj−qj)≥wωF(pj−qj).(35)\ge-w n_F b e_{F,j}=w b(p_j-q_j)\ge w\omega_F(p_j-q_j). \tag*{(35)}

This argument uses the signs of the individual deficit coordinates and the bargaining KKT equation. The vocabulary construction evaluates every deficit coordinate and supplies the deterministic correction in Eq. (30).

Uniformly positive source preferences. Because 0≤hS,hT<10 \le h_S,h_T < 1,

νR+νF=2ϵ+hT+(1−hT)hS≤1+2ϵ,νR,νF≥ϵ.\nu_R+\nu_F=2\epsilon+h_T+(1-h_T)h_S\le1+2\epsilon,\qquad\nu_R,\nu_F\ge\epsilon.

Both preferences therefore obey ωj≥ϵ/(1+2ϵ)>0\omega_j \ge\epsilon/(1+2\epsilon)>0.

The limit as student support vanishes. Fix pj=ρ∈(0,1)p_j=\rho\in(0,1). Cauchy–Schwarz on the remaining coordinates implies

∑ipiqi≤ρqj+(1−ρ)(1−qj).\sum_i\sqrt{p_iq_i}\le\sqrt{\rho q_j}+\sqrt{(1-\rho)(1-q_j)}.

Consequently,

lim inf⁡qj→0B≥1−1−ρ>0.\liminf_{q_j\to0}B\ge1-\sqrt{1-\rho}>0.

Combining this bound, continuity and monotonicity of min⁡(B/τ,1)\min(B/\tau,1), the preference floor, and pj−qj→ρp_j-q_j\to\rho gives Eq. (31). This proves Proposition 8.

Proposition 9 (Finite virtual-step probability-ratio control). Let q′=softmax⁡(z−ηg∗)q'=\operatorname{softmax}(z-\eta g^*), for any η≥0\eta\ge0. Then

∣log⁡qi′qi∣≤2η∥g∗∥∞≤22ηfor every i.(36)\left|\log\frac{q_i'}{q_i}\right|\le2\eta\lVert g^*\rVert_\infty\le2\sqrt{2}\eta\quad\text{for every }i. \tag*{(36)}

In particular, both DKL(q′∥q)D_{\mathrm{KL}}(q'\mathbin\Vert q) and DKL(q∥q′)D_{\mathrm{KL}}(q\mathbin\Vert q') are at most 2η∥g∗∥∞2\eta\lVert g^*\rVert_\infty.

Proof. Write v=−ηg∗v=-\eta g^*. Then log⁡(qi′/qi)=vi−log⁡∑jqjevj\log(q_i'/q_i)=v_i-\log\sum_jq_je^{v_j}. The logarithm of the weighted exponential average lies between min⁡jvj\min_jv_j and max⁡jvj\max_jv_j. The absolute difference is therefore at most max⁡jvj−min⁡jvj≤2∥v∥∞\max_jv_j-\min_jv_j\le2\lVert v\rVert_\infty. Taking expectations under either distribution gives the KL bounds. The bound describes an isolated logit step.

Near-agreement asymptotics and position strength

Proposition 10 (Cubic attenuation). Fix p∈ΔV∘p\in\Delta_V^\circ and h0,ϵ,τ>0h_0,\epsilon,\tau>0. Let q=p+δ∈ΔV∘q=p+\delta\in\Delta_V^\circ, 1⊤δ=0\mathbf{1}^\top\delta=0, and ∥δ∥2→0\lVert\delta\rVert_2\to0. Then

g∗=18τ(∑iδi2pi)δ+O(∥δ∥24).(37)g^*=\frac{1}{8\tau}\left(\sum_i\frac{\delta_i^2}{p_i}\right)\delta+O(\lVert\delta\rVert_2^4). \tag*{(37)}

In particular, ∥g∗∥2=Θ(∥δ∥23)\lVert g^*\rVert_2=\Theta(\lVert\delta\rVert_2^3) along any nonzero fixed perturbation direction.

Proof. Because pp is fixed in the interior, Taylor expansions are uniform in a sufficiently small neighborhood. For each coordinate,

pi+δi−pi=δi2pi+O(δi2).\sqrt{p_i+\delta_i}-\sqrt{p_i}=\frac{\delta_i}{2\sqrt{p_i}}+O(\delta_i^2).

Squaring and summing gives

B=18∑iδi2pi+O(∥δ∥23).(38)B=\frac{1}{8}\sum_i\frac{\delta_i^2}{p_i}+O(\lVert\delta\rVert_2^3). \tag*{(38)}

For small enough δ\delta, B<τB<\tau, so the gate is unsaturated and w=B/τw=B/\tau. Also, log⁡(1+δi/pi)=δi/pi+O(δi2)\log(1+\delta_i/p_i)=\delta_i/p_i+O(\delta_i^2) and R=O(∥δ∥22)R=O(\lVert\delta\rVert_2^2). Equation Equation (6) therefore gives

gF=δ,gR=δ+O(∥δ∥22).g_F=\delta,\qquad g_R=\delta+O(\lVert\delta\rVert_2^2).

Writing v=δ/∥δ∥2v=\delta/\lVert\delta\rVert_2 for nonzero δ\delta, it follows that eF=ve_F=v and eR=v+O(∥δ∥2)e_R=v+O(\lVert\delta\rVert_2). The preferences are continuous near pp and uniformly positive. The positive root κ\kappa is consequently bounded above and away from zero there. Normalization of κeR+eF\kappa e_R+e_F gives d∗=v+O(∥δ∥2)d^*=v+O(\lVert\delta\rVert_2), and hence

nFd∗=δ+O(∥δ∥22).n_Fd^*=\delta+O(\lVert\delta\rVert_2^2).

Multiplying by Eq. (38) divided by τ\tau yields

q∗=18τ(∑iδi2pi)δ+O(∥δ∥24),q^*=\frac{1}{8\tau}\left(\sum_i\frac{\delta_i^2}{p_i}\right)\delta+O(\lVert\delta\rVert_2^4),

as claimed. Positive definiteness of diag⁡(p)−1\operatorname{diag}(p)^{-1} supplies the matching upper and lower cubic orders along any fixed nonzero direction. □

Interpretation. Near agreement, the two source-centered gradients agree to first order. Preference-dependent changes enter at higher order, and the gate controls the leading attenuation. Equation Equation (37) quantifies the resulting cubic allocation of supervision to small discrepancies, including its slower fine-matching scale. A positive gate floor would restore a first-order term. The unit-gate ablation isolates the contribution of discrepancy-dependent attenuation.

Behavior at agreement and concentrated predictions. For p=qp=q, the exact update is zero at every entropy level, preserving the matching distribution. Concentrated predictions with different favored candidates can have large Hellinger discrepancy and receive strong supervision. Position strength thus responds to cross-model mismatch, while self-observations condition the relative candidate-source influence.

Comparison related works

Implementation details

Algorithm 1 gives the main training loop. The following details specify its sampling, masking, and numerical realization.

Training semantics and numerical implementation

Current-policy sampling. The mathematical formulation samples from the full current student distribution qq at temperature one. The sampled batch is fixed for one student update, after which fresh rollouts are collected.

Alignment and masking. Teacher and student distributions share vocabulary semantics and next-token positions. Both models evaluate the exact student continuation. The loss sums over valid response positions, including the response end token. The signed token-vocabulary surrogate is summed over these positions and divided by their global count NN. This preserves the position strength in Eq. (11).

Degeneracies and stable normalization. Away from agreement, the two gradient norms are nonzero by Theorem 1. The stable quadratic-root expression in Appendix § 11 supports accurate evaluation under unequal preferences. Direct normalization of the combined vector gives the final gradient at the prescribed magnitude.

Loss realization. Because g∗g^* is zero-sum,

∂∂zj∑isg⁡[gi∗]log⁡qi=gj∗−qj∑igi∗=gj∗.\frac{\partial}{\partial z_j}\sum_i \operatorname{sg}[g_i^*]\log q_i = g_j^* - q_j\sum_i g_i^* = g_j^*.

Thus the signed surrogate produces the desired current logit gradient with one student backward pass. The coefficients and controller computations are detached; student log probabilities retain their computation graph. The teacher remains frozen. The frozen-coefficient convention realizes the matrix differential in Eq. (6) at the current snapshot.

Real-update diagnostics

Purpose. The geometric analysis in the main text is expressed in terms of a change in student logits, while optimization acts on shared neural-network parameters. We therefore measure whether the logit-space quantities predict the change produced by a real optimizer step. This diagnostic uses a Qwen3-1.7B non-thinking student and the Qwen3-30B-A3B-Instruct-2507 teacher. Training starts from the same zero-LoRA initialization used by the corresponding non-thinking experiments.

Quantities measured. For a fixed token prefix, let p=softmax⁡(zT)p = \operatorname{softmax}(z_T) be the frozen teacher distribution and let q0=softmax⁡(z0)q_0 = \operatorname{softmax}(z_0) denote the student distribution immediately before the optimizer update. We define

R(z0)=KL(q0∥p),F(z0)=KL(p∥q0),(39)R(z_0) = \mathrm{KL}(q_0 \mathbin{\|} p), \qquad F(z_0) = \mathrm{KL}(p \mathbin{\|} q_0), \tag*{(39)}
gR=q0⊙(log⁡q0−log⁡p−R(z0)),gF=q0−p.(40)g_R = q_0 \odot\left(\log q_0 - \log p - R(z_0)\right), \qquad g_F = q_0 - p. \tag*{(40)}

After the production backward pass and AdamW update, we evaluate the same prefix again to obtain z1z_1, set Δz=z1−z0\Delta z = z_1 - z_0, and compare the predictions

ΔR^=gRTΔz,ΔF^=gFTΔz(41)\widehat{\Delta R} = g_R^{\mathsf{T}}\Delta z, \qquad\widehat{\Delta F} = g_F^{\mathsf{T}}\Delta z \tag*{(41)}

with the measured changes

ΔR=R(z1)−R(z0),ΔF=F(z1)−F(z0).(42)\Delta R = R(z_1) - R(z_0), \qquad\Delta F = F(z_1) - F(z_0). \tag*{(42)}

All quantities are computed over the vocabulary. A negative predicted or measured change denotes descent for the corresponding objective. We report the Pearson correlation between predicted and measured changes, the fraction of token positions for which their signs agree, and whether the prediction correctly identifies simultaneous descent of both objectives.

Probe construction. Measurements are taken every ten successful optimizer updates, always using the same teacher target before and after the update. We use three probe types. The on-policy probe samples positions from the current rollout. The fixed in-domain probe reuses fixed prompt–continuation pairs drawn from the training domain. The HMMT25 OOD probe contains four held-out prompts with fixed short continuations and is used only to diagnose distribution shift.

Table 7 and Figure 4 show that the local approximation remains accurate after an update to real network parameters. Pearson correlations range from 0.9751 to 0.9993, while individual-objective sign agreement ranges from 98.00% to 100.00%. Joint sign accuracy is at least 99.00% for the two fixed probes and 99.57% on on-policy rollouts. The few sign errors occur close to zero, where the measured change is small. These results empirically validate the local predictive role of gTΔzg^{\mathsf{T}}\Delta z under the observed optimizer steps.

ProbeUpdatesTokensrRr_RSignR_RrFr_FSignF_FBoth
On-policy rollout194640.997799.570.998999.5799.57
Fixed in-domain253000.975199.670.999298.0099.00
HMMT25 OOD254000.998799.000.9993100.0099.25

Table 7. Agreement between first-order predictions and measured KL changes. Correlations and sign agreement are computed over token positions. “Both” is the accuracy of predicting whether RKL and FKL decrease simultaneously.

Figure 4. RKL (top) and FKL (bottom) are shown separately for on-policy, fixed in-domain, and HMMT25 OOD probes. Each point is a token position evaluated immediately before and after the same real optimizer step. The dashed line denotes y=xy = x and red crosses indicate sign disagreement.

Interpretation and scope. The diagnostic supports the specific bridge used by our analysis: for the optimizer steps observed during training, the logit-space terms remain accurate predictors of the KL changes induced by a shared-parameter neural-network update. Agreement on current rollouts, fixed in-domain examples, and held-out HMMT25 prompts indicates that this fidelity is not confined to one probe distribution.

Experimental details

Models and experimental organization

The teacher checkpoints are Qwen3-30B-A3B-Thinking-2507 and Qwen3-30B-A3B-Instruct-2507. The main experiment pairs the Instruct teacher with non-thinking Qwen3-1.7B and Qwen3-4B students. The first auxiliary experiment pairs the Thinking teacher with thinking students at both sizes; the second pairs the Instruct teacher with Qwen3-4B-Base. Within each comparison block, methods use the same student initialization, teacher, prompt source, training budget, generation/evaluation configuration, and metric implementation; only the learning objective or controller component being compared is changed.

Each iteration generates fresh student continuations and aligns teacher and student distributions on the collected prefixes. The four observations, gradient geometry, and coefficients are recomputed at every valid response position. The teacher remains frozen, and the backward pass updates the student. The selected student regime determines the generation configuration of its training and evaluation continuations, and all methods in that regime use the same prompt wrapper and decoding protocol.

Training and sampling hyperparameters

Table 8: Hyperparameters for OC-OPD. Prompt and response lengths are measured in tokens. A dash denotes the fresh-rollout setting, with one optimization step per rollout batch.

HyperparameterOC-OPD
General
Max prompt length8192
GPUs4×4 \times NVIDIA A800 80GB
Training precisionBF16
Optimization
Global batch size8
Mini-batch size8
Mini-batches1
Group size1
Off-policy epoch—
OptimizerAdamW
Learning rate1×10−51 \times10^{-5}
LR scheduleConstant
Warmup ratio0
Weight decay0
Adam betas(0.9, 0.999)
Gradient clippingGlobal L2 norm 1.0
Sampling
Rollout max response length32768
Rollout temperature1.0
Rollout top-pp1.0
Validation max response length32768
Validation temperature1.0
Validation top-pp1.0
Method-specific
h0h_01.0
ϵ\epsilon0.05
τ\tau0.01

Table 8. Hyperparameters for OC-OPD. Prompt and response lengths are measured in tokens. A dash denotes the fresh-rollout setting, with one optimization step per rollout batch.

Prompt templates

The prompt wrapper is fixed by the student generation regime and is shared by all compared methods in that regime. The placeholder {problem} is replaced by the benchmark problem text. We use the following templates.

Non-thinking.

{problem}\n\nPlease solve the following problem and put your final answer within\n\n\boxed{}.

Thinking.

{problem}\n\nPlease reason step by step, and put your final answer within\n\n\boxed{}.

Base.

Problem:\n{problem}

Controlled objectives and ablations

The mathematical ablations use Qwen3-30B-A3B-Instruct-2507 with the Qwen3-4B non-thinking student on DAPO-Math-17k, matching the 4B setting in Table 1. All ablations reuse the same initialization, prompts, optimization and sampling setup, and evaluation budget. Table 9 reports the key controls. The comparison design separates the two information paths in Eq. (5): derivatives of the observations form the candidate-level gradients, while observation values condition their mixture.

MethodAIME 2024AIME 2025AIME 2026
Avg@32Pass@32Avg@32Pass@32Avg@32Pass@32
OC-OPD60.9493.3356.6783.3358.5486.67
Forward KL (FKL)55.4283.3353.7573.3354.1783.33
Reverse KL (RKL)61.0486.6751.0476.6752.1980.00
Adaptive linear mixture60.2186.6753.0280.0053.9680.00
Equal-preference58.6586.6753.5480.0056.6783.33
w/o position gate60.6390.0054.0680.0056.6783.33

Table 9. Ablation study on mathematical reasoning. The student is Qwen3-4B with non-thinking generation and the teacher is Qwen3-30B-A3B-Instruct-2507. Scores are percentages; bold denotes the best value in each column.

Full OC-OPD attains the highest Pass@32 on all three AIME editions, while RKL is slightly higher on AIME24 Avg@32. The single-KL controls, adaptive linear mixture, equal-preference variant, and no-gate variant each lose ground on at least one of the coverage metrics. Together, these comparisons support the use of both source-centered directions, observation-conditioned preferences, bargaining-based coordination, and discrepancy-dependent strength.

KL-only controls. Forward KL (FKL) optimizes the teacher-sourced contrast F=DKL(p∥q)F = D_{\mathrm{KL}}(p\mathbin\Vert q), while reverse KL (RKL) optimizes the student-sourced contrast R=DKL(q∥p)R = D_{\mathrm{KL}}(q\mathbin\Vert p). These rows isolate the two candidate-source objectives without the observation-conditioned coordination used by OC-OPD.

Adaptive linear mixture This variant uses the same observation-conditioned preferences ωR,ωF\omega_R,\omega_F, source-centered normalized gradients, and discrepancy gate as OC-OPD, but replaces the bargaining solution with the normalized linear direction dlin=(ωReR+ωFeF)/∥ωReR+ωFeF∥2d_{\mathrm{lin}} = (\omega_R e_R + \omega_F e_F)/\lVert\omega_R e_R + \omega_F e_F\rVert_2. The final magnitude remains wnFw n_F, matching the full method.

Equal preferences. Equal-preference bargaining variant fixes ωR=ωF=1/2\omega_R = \omega_F = 1/2 while retaining the source-centered gradients, bargaining rule, and discrepancy gate. It therefore removes observation-conditioned preference adaptation without changing the two candidate sources or the update-strength mechanism.

No position gate. This variant sets w=1w = 1, preserving the original preference rule and coordinated direction while removing discrepancy-dependent attenuation of the update magnitude. It directly tests the position-strength component analyzed in Appendix § 13.

Variance and Statistical Significance

Figure 5. Additional AIME results to demonstrate variance and statistical significance. (accuracy, %). We retain the mode for all the results across six runs, and figures report the variance.

References

  1. [1]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/abs/1503.02531.
  2. [2]Yoon Kim and Alexander M. Rush. Sequence-Level Knowledge Distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327. Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1139. URL https://aclanthology.org/D16-1139/.DOI
  3. [3]Taiqiang Wu, Chaofan Tao, Jiahao Wang, Running Yang, Zhe Zhao, and Ngai Wong. Rethinking Kullback-Leibler divergence in knowledge distillation for large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5737–5755, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.383/.arxiv.org/abs/2404.02657
  4. [4]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In International Conference on Artificial Intelligence and Statistics, pages 627–635, 2011. URL https://proceedings.mlr.press/v15/ross11a.html.arxiv.org/abs/1011.0686
  5. [5]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations, 2024a. URL https://arxiv.org/abs/2306.13649.
  6. [6]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge Distillation of Large Language Models. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2306.08543v4.
  7. [7]Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling Reasoning Efficiently via Relaxed On-Policy Distillation. arXiv preprint arXiv:2603.11137, 2026. URL https://arxiv.org/abs/2603.11137.
  8. [8]Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-Aware On-Policy Distillation of Language Models. arXiv preprint arXiv:2603.07079, 2026. URL https://arxiv.org/abs/2603.07079.
  9. [9]Chia-Hsuan Lee, Zelei Cheng, Yu Wang, Renkun Ni, Sambit Sahu, Shi-Xiong Zhang, and William Campbell. SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision. arXiv preprint arXiv:2606.28562, 2026. URL https://arxiv.org/abs/2606.28562.
  10. [10]Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. TIP: Token Importance in On-Policy Distillation. arXiv preprint arXiv:2604.14084, 2026. URL https://arxiv.org/abs/2604.14084.
  11. [11]Zichao Yu, Chengzhi Yu, Shengze Xu, Yujin Han, Bingqing Jiang, Xu Wang, and Difan Zou. Mismatch Matters: On-Policy Distillation Beyond Token Agreement. arXiv preprint arXiv:2608.09836, 2026. URL https://arxiv.org/abs/2608.09836.
  12. [12]Qwen Team. Qwen3: Think Deeper, Act Faster. Official model release, 2025a. URL https://qwenlm.github.io/blog/qwen3/.
  13. [13]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanze Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.arxiv.org/abs/2311.12022
  14. [14]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.01210
  15. [15]Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. DistLLM: Towards Streamlined Distillation for Large Language Models. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 24872–24895. PMLR, 2024. URL https://proceedings.mlr.press/v235/ko24c.html.arxiv.org/abs/2402.03898
  16. [16]Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. DistLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267, pages 31044–31062. PMLR, 2025. URL https://proceedings.mlr.press/v267/ko25a.html.arxiv.org/abs/2503.07067
  17. [17]Wenda Xu, Rujun Han, Zifeng Wang, Long T. Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2410.11325v3.
  18. [18]Chenyang Jiang and Changhan Huang. SMOPD: Selective Token-Entropy Masking for Dirty-History Multi-Turn On-Policy Self-Distillation. arXiv preprint arXiv:2608.14647, 2026. URL https://arxiv.org/abs/2608.14647.
  19. [19]Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuiliang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai, and Zequn Sun. Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level. arXiv preprint arXiv:2605.06387, 2026. URL https://arxiv.org/abs/2605.06387v4.
  20. [20]Ijun Jang, Jewon Yeom, Juan Yeo, Hyungyu Lim, and Taesup Kim. Stable On-Policy Distillation through Adaptive Target Reformulation. arXiv preprint arXiv:2601.07155, 2026. URL https://arxiv.org/abs/2601.07155.
  21. [21]Anhao Zhao, Junlong Tong, Yingqi Fan, Ping Nie, Wenjie Li, and Xiaoyu Shen. PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation. arXiv preprint arXiv:2606.17199, 2026a. URL https://arxiv.org/abs/2606.17199.
  22. [22]Xingrun Xing, Haoqing Wang, Boyan Gao, Ziheng Li, and Yehui Tang. Trust Region On-Policy Distillation. arXiv preprint arXiv:2606.01249, 2026. URL https://arxiv.org/abs/2606.01249.
  23. [23]Shun-ichi Amari. Natural Gradient Works Efficiently in Learning. Neural Computation, 10(2):251–276, 1998. doi: 10.1162/089976698300017746. URL https://doi.org/10.1162/089976698300017746.
  24. [24]Frank Nielsen. An Elementary Introduction to Information Geometry. Entropy, 22(10):1100, 2020. doi: 10.3390/e22101100. URL https://doi.org/10.3390/e22101100.
  25. [25]Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-Task Learning as a Bargaining Game. In International Conference on Machine Learning, 2022. URL https://proceedings.mlr.press/v162/navon22a.html.arxiv.org/abs/2202.01017
  26. [26]Qwen Team. Qwen3-30B-A3B-Instruct-2507. Official model card, 2025b. URL https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507.
  27. [28]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503.14476, 2025. URL https://arxiv.org/abs/2503.14476.
  28. [29]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2305.20050.
  29. [30]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.211. URL https://aclanthology.org/2024.acl-long.211/.DOI
  30. [31]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  31. [32]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations (ICLR). OpenReview.net, 2024b.arxiv.org/abs/2306.13649
  32. [33]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026b.arxiv.org/abs/2601.18734

Paper details

vlnc:W_gug8ajbn
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Generative AI tools

View statement
Looking for related papers…
Contents