betaFeedback

Reading v1View latest version

WorksComputer Science

Reinforcing What the Student Has Learned: Teacher-Guided On-Policy Distillation

Abstract

On-policy distillation (OPD) improves language-model reasoning through dense teacher feedback on student-generated responses.We first observe an amplitude disparity in this feedback: on math reasoning responses, style-related positions exhibit 3.5 times the mean absolute teacher–student log-probability difference of mathematical positions.Scaling OPD updates by these discrepancies thus directs much of the update strength toward matching the teacher's expression. We instead take the student's own pretrained base as a reference: the student–base difference records the token preferences acquired during post-training and provides supervision that does not depend on the student's distance from the teacher. Building on this reference, we propose Student-Delta On-Policy Distillation (SD-OPD), which uses centered student–base differences as update coefficients, keeps each coefficient only when its sign agrees with centered teacher–student feedback, and recomputes the student signal every training round. Across 5 teacher–student configurations with Qwen3 students of 1.7B to 8B parameters, SD-OPD improves six-task mathematics average pass@1 (avg@16) over OPD by 7.76 to 10.37 percentage points. Code is available at https://github.com/aim-uofa/sd-opd.

Introduction

On-policy distillation (OPD) is a post-training approach in which a language model learns from a teacher on responses generated by its own current policy [1] (Gu et al. 2023). The teacher scores every token at prefixes the student visits, so supervision is dense and lies

on the student’s own trajectories. As the student changes, new responses are sampled and scored, and the training distribution follows the student’s current behavior, as in imitation learning on learner-induced states [20]. This combination of on-policy sampling and token-level feedback has supported both capability transfer from larger teachers to smaller students [27, 28] and the consolidation of specialized teachers into a single model [26].

However, the effectiveness of OPD depends on how teacher feedback is constructed and weighted [24] (Fu et al. 2026). In sampled-token OPD, the magnitude of teacher–student log-probability differences determines the strength of token-level supervision [28] (Gu et al. 2023). Post-training also shifts a model’s preferences for wordings and formatting [18, 36], so teacher–student disagreement reflects differences in expression as well as gaps in reasoning. Weighting updates by this disagreement thus spends update strength on the teacher’s expression as well as its reasoning. This raises two questions: how is update strength distributed across tokens, and can it come from a source other than teacher disagreement?

For the first question, on DAPO-Math responses, style-related positions have 3.5 times the mean absolute teacher–student log-probability difference of mathematical positions (Figure 2). For the second, we compare the student with its own pretrained base [18], which records the token preferences the student has already acquired, independently of its distance from the teacher. These preferences include reasoning expressions such as suppose, try, and strategy, but their amplitude is also concentrated at style-related positions, with a style-to-mathematics ratio of 4.1, so they need selection. We select them by teacher sign agreement, keeping a student preference only where the teacher favors the same direction. This test retains very different shares of preferences with similar strength (Figure 3) and lowers the style-to-mathematics ratio to 3.1, whereas the same test with teacher–student magnitudes still leaves 3.7 (§ 3.2).

Figure 2. Feedback on the same student responses: raw (a) teacher–student rtr_t and (b) student–base dtd_t, and both after the same-sign mask mt=1[rtdt>0]m_t = 1[r_td_t > 0], (c) mtrtm_tr_t and (d) mtdtm_td_t. Panels (c) and (d) retain identical positions and directions and differ only in magnitude source. All panels share the scale, categories, and averaging procedure in Equation (2); filtered-out positions contribute zero and remain in the denominator.

Figure 3. Student–base preferences and teacher selection. (a, b) Fifty representative strings per sign of dtd_t, with larger, darker text indicating greater averaged amplitude. A piecewise-linear size scale separates ten high-amplitude examples from smaller context strings. (c) Six representative strings: parts of dtd_t with dt>0d_t > 0 (right of zero) and dt<0d_t < 0 (left), split into the part retained by 1[rtdt>0]1[r_td_t > 0] and the part removed, with the retained share of absolute amplitude. Appendix § 12.§ 4 describes aggregation and string selection.

These findings lead to Student-Delta On-Policy Distillation (SD-OPD; Figure 1). Centered student–base logprobability differences supply the coefficients, and a coefficient is kept only when its sign agrees with centered teacher–student feedback. The teacher thus decides which student preferences to develop, but its disagreement magnitude does not scale the updates. The teacher and base stay frozen, while the student signal is recomputed after each training round, so it covers both the initial post-training changes and those acquired during distillation.

Figure 1. SD-OPD overview. Each round scores student-generated responses on shared prefixes. The student–base difference supplies the coefficient, and teacher–student feedback selects its admissible direction. Here, AT=AOPDA_T = A_{\mathrm{OPD}}, Adirect=ASD-OPDA_{\mathrm{direct}} = A_{\mathrm{SD\text{-}OPD}}, and ρt=wt\rho_t = w_t. The diagram shows full-vocabulary centering; training uses the top-1,024 approximation in Appendix § 8.§ 2. Advantages remain fixed within each round and are recomputed after refreshing the student. Teacher TT and base SBS_B remain frozen.

We evaluate SD-OPD in 5 teacher–student configurations with three teachers and Qwen3 students of 1.7B, 4B, and 8B parameters, using six mathematics benchmarks and matched rollout and response budgets. SD-OPD outperforms OPD on every benchmark in every configuration and achieves the highest six-task student average in all 5 configurations (Table 1). Average gains range from 7.76 to 10.37 percentage points over OPD and from 3.45 to 14.89 over ExOPD.

MethodAIME24AIME25AIME26HMMT25AMC23MATH500Average
Qwen3-4B-Instruct-2507 → Qwen3-1.7B
Initial student13.128.547.716.6744.6969.4025.02
OPD32.0826.2524.3816.8873.7583.9042.87
ExOPD40.4228.1228.7520.0079.0686.1147.08
SD-OPD45.4233.3340.6221.0486.7288.7252.64
Δ\Delta vs. OPD+13.33+7.08+16.25+4.17+12.97+4.83+9.77
Teacher58.7544.3853.1229.1793.4493.3362.03
Qwen3-30B-A3B-Instruct-2507 → Qwen3-1.7B
Initial student13.128.547.716.6744.6969.4025.02
OPD31.8727.0826.6716.6772.0384.6043.15
ExOPD35.0029.3830.8317.7171.0985.8444.97
SD-OPD44.7937.0840.2118.3388.9189.5353.14
Δ\Delta vs. OPD+12.92+10.00+13.54+1.67+16.88+4.93+9.99
Teacher77.0860.0072.2942.5095.3195.7473.82
MiMo-7B-RL-0530 → Qwen3-1.7B
Initial student13.128.547.716.6744.6969.4025.02
OPD43.5436.6734.5821.2579.8488.0150.65
ExOPD35.4230.0027.5020.6267.8185.0244.40
SD-OPD54.3743.3345.2131.0490.9490.8459.29
Δ\Delta vs. OPD+10.83+6.67+10.63+9.79+11.09+2.83+8.64
Teacher75.2163.1263.7542.9286.5689.6170.20
Qwen3-30B-A3B-Instruct-2507 → Qwen3-4B
Initial student21.4618.5414.3711.6768.9181.4936.07
OPD60.6246.2554.7933.5489.2292.3462.79
ExOPD66.6755.2160.0035.4292.0393.2767.10
SD-OPD69.7962.2960.6242.0895.7892.7470.55
Δ\Delta vs. OPD+9.17+16.04+5.83+8.54+6.56+0.40+7.76
Teacher77.0860.0072.2942.5095.3195.7473.82
Qwen3-30B-A3B-Instruct-2507 → Qwen3-8B
Initial student26.4620.4215.4210.8366.7282.3537.03
OPD67.7154.3755.0032.7191.5693.1165.74
ExOPD68.9651.2561.4632.9293.9193.7567.04
SD-OPD80.4268.9667.5048.7597.3493.7076.11
Δ\Delta vs. OPD+12.71+14.58+12.50+16.04+5.78+0.59+10.37
Teacher77.0860.0072.2942.5095.3195.7473.82

Table 1. Pass@1 (%) across 5 teacher–student settings (teacher →\to student), averaged over 16 sampled responses per problem. Average is the six-task mean; averages and gains use unrounded scores. Gray rows at the bottom of each block give teacher reference scores; boldface marks the highest student scores within each setting, and green rows report SD-OPD gains over OPD in percentage points.

Our contributions are as follows:

  • A diagnosis of how OPD weights its updates. We show that teacher–student disagreement is not a neutral measure of what the student should learn: its mean absolute magnitude is 3.5 times larger at style-related than at mathematical positions, the gap persists with a larger teacher and on GSM8K, and sign-agreement filtering does not remove it while teacher–student differences set the magnitude.

  • The student’s own post-training changes as a source of supervision. We show that the difference between a post-trained student and its pretrained base can set update strength, with the teacher deciding only which directions to reinforce. SD-OPD implements this division of roles and recomputes the student signal every round, so update strength follows what the current student prefers rather than how far it is from the teacher. It improves over OPD in every evaluated configuration and lets an 8B student exceed the average accuracy of its Qwen3-30B-A3B teacher.

Related Work

On-policy knowledge distillation. Knowledge distillation transfers teacher predictions to a student ([11]), while sequence-level distillation trains on teacher-generated targets ([13]). Teacher-generated reasoning data extend this approach to mathematical and code reasoning ([8, 7]). GKD and MiniLLM obtain supervision on student-generated sequences through distribution matching and reverse-KL policy optimization ([1] (Gu et al. 2023)). DistiLLM combines skew-divergence objectives with reuse of student outputs, while DistiLLM-2 contrasts teacher- and student-generated responses ((Ko et al., 2024; 2025)). Black-box OPD learns feedback from a discriminator trained on teacher and student text ((Ye et al., 2025)). Recent work also studies local distribution matching and token-specific supervision weights ([24] (Fu et al. 2026)). In these methods, teacher outputs determine the strength of supervision; SD-OPD instead takes update strength from the student’s own post-training changes and uses the teacher only to select their direction.

Self-distillation and contextual supervision. Context-conditioned self-teaching obtains additional information from reference solutions in Self-Distilled Reasoner ([34]), textual feedback in SDPO ([12]), and demonstrations in SDFT ([22]).

On-Policy Context Distillation also learns from a context-conditioned teacher, including configurations with different teacher and student sizes (Ye et al., 2026). In these approaches, the additional supervision comes from privileged information placed in the teacher’s context; in SD-OPD, it comes from the student’s own post-training history, measured against a fixed base model, with an independent teacher constraining its direction.

Post-training differences and delta distillation. URIAL analyzes changes between aligned and base token distributions, including changes in expression ([18]); we turn this comparison into a training signal. Policy–reference log-ratios connect preference optimization to implicit rewards ([19]); implicit process reward models and PRIME extend this connection to dense reasoning feedback learned from outcome labels ([33, 3]). G-OPD introduces a reference policy and reward scaling; its ExOPD variant studies extrapolation with both an initial-student reference and a teacher pre-RL reference ([28]). OPD2 also uses centering and a same-sign condition, but its primary signal is a teacher–base difference ([10]). SD-OPD instead uses the current student’s changes relative to the student’s own base, recomputed every round, and lets teacher feedback decide which directions to reinforce.

Observations

We compare teacher–student disagreement with the student’s own post-training changes, using the initial student S0S_0 (Qwen3-1.7B), its pretrained base SBS_B (Qwen3-1.7B-Base), and teacher TT (Qwen3-4B-Instruct-2507). All analyses use all eight responses to each of the same 1,024 questions sampled without replacement from DAPO-Math-17K (Yu et al., 2026). This gives 8,192 responses with about 11.3 million response tokens. All three models then score the same token IDs at the same causal prefixes, using full-vocabulary log-softmax normalization. Appendix § 12 gives generation and scoring details, and Appendix § 13 repeats the category-level analyses with another teacher and on GSM8K.

Teacher–Student Disagreement Varies across Token Categories

To compare teacher–student disagreement across token categories, we use the sampled-token log-ratio feedback studied in OPD ([28] (Gu et al. 2023)). For response token yty_t at prefix ht=(x,y<t)h_t = (x, y_{<t}), this feedback is

rt=log⁡pT(yt∣ht)−log⁡pS0(yt∣ht).(1)r_t = \log p_T(y_t \mid h_t) - \log p_{S_0}(y_t \mid h_t). \tag*{(1)}

We categorize token positions using fixed textual rules. Style-related positions include discourse wording, answer labels, and Markdown formatting, while mathematical positions include numbers, operators, formula variables, a fixed mathematical lexicon, and logical expressions. Markup, whitespace, and other tokens form separate groups; Appendix § 12 details the classification.

Because the categories differ in frequency (about 0.6 million style-related and 4.7 million mathematical tokens, both present in every response), we compare per-token means rather than totals. For positions Ti,j,cT_{i,j,c} in category cc of response jj to question ii, we compute the per-token mean

Mi,j,c(r)=1∣Ti,j,c∣∑t∈Ti,j,c∣ri,j,t∣.(2)M_{i,j,c}(r) = \frac{1}{|T_{i,j,c}|}\sum_{t \in T_{i,j,c}} |r_{i,j,t}|. \tag*{(2)}

We average these values equally over the eight responses to each question and then equally over questions, preventing longer responses from receiving extra weight.

Style-related positions have larger feedback amplitudes. Figure 2(a) shows mean absolute feedback of 1.226 at style-related positions and 0.347 at mathematical positions, a 3.5-fold difference. OPD uses this log-ratio as a coefficient in token-level policy gradients [28] (Gu et al. 2023), so for the initial student, style-related positions receive larger coefficients on average. For example, differences in wording or formatting receive large coefficients even when the student expresses the same mathematical idea.

Student Preferences and Teacher Selection

Since teacher disagreement mixes expression with reasoning, we look for a reference that measures update strength from what the student has already learned. The student’s own pretrained base provides one: comparing student and base predictions at shared prefixes reveals preferences acquired during post-training, including how the student expresses mathematical reasoning [18]. On the same responses and prefixes, we define

dt=log⁡pS0(yt∣ht)−log⁡pSB(yt∣ht).(3)d_t = \log p_{S_0}(y_t \mid h_t) - \log p_{S_B}(y_t \mid h_t). \tag*{(3)}

The sign of dtd_t indicates whether the observed token has become more probable (dt>0d_t > 0) or less probable (dt<0d_t < 0) relative to the base at that prefix. Since the reference is the pretrained base, this difference summarizes cumulative changes across post-training.

Acquired preferences favor exploration and planning. Figure 3(a–b) shows the strings whose probability post-training raised or lowered most. The raised strings are reasoning moves rather than formatting: they form hypotheses and explore alternatives (suppose, maybe, try), plan and organize a solution (strategy, idea, structure), and test candidates (plug, examine). The lowered strings instead narrate routine computation (calculations, distribute, rewrite) or assert that a step is easy or correct (straightforward, ensuring), and their shifts are much smaller. Post-training has thus mainly added ways of deciding how to approach a problem, rather than removing old habits of describing computations.

These changes reflect how this particular student learned to reason, and a teacher trained differently need not share them. The student–base difference therefore records reasoning behavior the student has already acquired, which makes it a natural source of supervision for further distillation. Not every shift is desirable, however: the checking step verify is also lowered, so these changes still need selection.

Teacher agreement selects which acquired preferences to reinforce. These changes also include stylistic preferences, with a style-to-mathematics ratio of 4.1 (Figure 2(b)). We therefore keep dtd_t only where the teacher agrees in sign, using the mask mt=1[rtdt>0]m_t = 1[r_td_t > 0].

This test separates preferences of similar strength (Figure 3(c)): try and strategy keep only 19.3% and 13.8% of their absolute amplitude, whereas idea keeps 71.4%. For straightforward, it keeps the decrease and removes occurrences where the student raised its probability. Across categories, it removes 69.1% of the style-related but 58.7% of the mathematical amplitude, lowering the ratio to 3.1 (Figure 2(d)).

The magnitude source sets the gated balance. The mask mtm_t decides which positions and directions are retained, while the magnitude source decides their relative weight. To isolate the magnitude source, we apply the same mask to teacher–student feedback and compare mtrtm_tr_t with mtdtm_td_t (Figure 2(c–d)). Gated teacher–student coefficients are smaller than raw feedback in every category, yet style-related positions still receive 0.380 versus 0.102 at mathematical positions, a ratio of 3.7 compared with 3.5 before filtering. Student–base magnitudes lower the style-related amplitude by 40.3% and the mathematical amplitude from 0.102 to 0.074, giving a ratio of 3.1 instead. Because both signals retain identical positions and a uniform rescaling of OPD would leave the ratio unchanged, this difference comes from

Algorithm 1 SD-OPD
Require: $S_0$; frozen $T$, $S_B$; $D$, $K$, $R$, $c_{\mathrm{OPD}}$
1: $S_\theta \leftarrow S_0$
2: for $r = 1, \ldots, R$ do
3: $p \leftarrow p_{S_\theta}$, $\mathcal{R} \leftarrow \operatorname{Rollout}(D,p)$
4: for $(x,y) \in \mathcal{R}$, $t = 1,\ldots,|y|$ do
5: $h_t \leftarrow (x,y_{<t})$
6: $\widetilde{A}_{S,t} \leftarrow C_{p,K}[\log(p/p_{S_B})](y_t \mid h_t)$
7: $\widetilde{A}_{\mathrm{OPD},t} \leftarrow C_{p,K}[\log(p_T/p)](y_t \mid h_t)$
8: $g_t \leftarrow \mathbf{1}[\widetilde{A}_{S,t}\widetilde{A}_{\mathrm{OPD},t} > 0]$
9: $\widehat{A}_t \leftarrow \operatorname{sg}(c_{\mathrm{OPD}}g_t\widetilde{A}_{S,t})$
10: end for
11: for each minibatch $B$ from $\mathcal{R}$ do
12: $w_t \leftarrow p_{S_\theta}(y_t \mid h_t)/p(y_t \mid h_t)$
13: $\mathcal{L}_B \leftarrow \frac{\sum_{(x,y)\in B}\sum_{t=1}^{|y|}\ell_t^{\mathrm{dual}}}{\sum_{(x,y)\in B}|y|}$
14: $\theta \leftarrow \operatorname{AdamStep}(\theta,\nabla_\theta\mathcal{L}_B)$
15: end for
16: end for
17: return $S_\theta$

the magnitude source rather than from filtering. These patterns also hold with another teacher and on GSM8K (Appendix D). These observations motivate combining student-derived strength with teacher-guided selection.

Method

Since teacher–student disagreement is larger at style-related than at mathematical positions (Section 3), SD-OPD takes update strength from the student’s own changes and uses the teacher to decide which directions to retain (Figure 1).

Preliminaries: On-Policy Distillation

For prompt xx, write qθ=pSθq_\theta= p_{S_\theta} and ht=(x,y<t)h_t = (x,y_{<t}). OPD samples student responses and minimizes reverse KL to a fixed teacher TT (Agarwal et al., [1]; Gu et al., 2023):

LOPD(θ)=Ey∼qθ[log⁡qθ(y∣x)pT(y∣x)](4)\mathcal{L}_{\mathrm{OPD}}(\theta) = \mathbb{E}_{y\sim q_\theta}\left[\log\frac{q_\theta(y\mid x)}{p_T(y\mid x)}\right] \tag*{(4)}
−∇θLOPD=Ey∼qθ[∑tGt∇θlog⁡qθ(yt∣ht)].-\nabla_\theta\mathcal{L}_{\mathrm{OPD}} = \mathbb{E}_{y\sim q_\theta}\left[\sum_t G_t\nabla_\theta\log q_\theta(y_t\mid h_t)\right].

Here Gt=∑u=t∣y∣[log⁡pT(yu∣hu)−log⁡qθ(yu∣hu)]G_t = \sum_{u=t}^{|y|}[\log p_T(y_u\mid h_u)-\log q_\theta(y_u\mid h_u)] sums log-ratios from token tt onward. The score-function identity removes zero-mean terms, and KL descent reverses the log-ratio. The zero-discount approximation keeps only the current log-ratio (Yang et al., [28]), yielding the uncentered token advantage at rollout snapshot p=pSoldp = p_{S_{\mathrm{old}}}:

AOPD,traw=log⁡pT(yt∣ht)−log⁡p(yt∣ht).(5)A^{\mathrm{raw}}_{\mathrm{OPD},t} = \log p_T(y_t\mid h_t)-\log p(y_t\mid h_t). \tag*{(5)}

Each round fixes pp and advantages during clipped updates, then refreshes both. Variants share this sampling and update pipeline by replacing the advantage: ExOPD adds a scaled teacher–reference log-ratio (Appendix A.5); Our Method uses student-derived strength and teacher-selected direction.

A Student-Derived Distillation Signal

We initialize the student from a post-trained model S0S_0 and freeze its pretrained base SBS_B, whose comparison with the student identifies preferences acquired through post-training (Section 3.2). To measure each token’s preference relative to the average shift at its prefix, we center log-ratios under pp, with prefix conditioning implicit (Sutton et al., [23]; Heo et al., [10]):

Cp[f](a)=f(a)−Ev∼p[f(v)].(6)C_p[f](a)=f(a)-\mathbb{E}_{v\sim p}[f(v)]. \tag*{(6)}

The student-derived coefficient is

AS=Cp[log⁡p−log⁡pSB].(7)A_S=C_p[\log p-\log p_{S_B}]. \tag*{(7)}

Positive and negative coefficients identify student–base shifts above and below the current-policy average, respectively. Because the numerator uses the current student, each refresh incorporates preferences acquired during distillation alongside the initial post-training changes.

Teacher-Guided Reinforcement

Because these changes also contain stylistic preferences (Section 3.2), we reinforce a student-derived direction only when the teacher supports it. Centering the raw advantage in (5) gives

AOPD=Cp[log⁡pT−log⁡p].(8)A_{\mathrm{OPD}}=C_p[\log p_T-\log p]. \tag*{(8)}

The distillation coefficient is

ASD-OPD=1[ASAOPD>0]⏟gAS.(9)A_{\mathrm{SD\text{-}OPD}}=\underbrace{\mathbf{1}[A_SA_{\mathrm{OPD}}>0]}_{g}A_S. \tag*{(9)}

Positive agreement reinforces the sampled token, negative agreement suppresses it, and opposite signs or exact zeros yield no update.

For sampled token yty_t, let A~SD-OPD,t\widetilde{A}_{\mathrm{SD\text{-}OPD},t} denote (9) evaluated with approximate centering. We use it as a fixed advantage:

A^t=sg⁡(cOPDA~SD-OPD,t),(10)\widehat{A}_t=\operatorname{sg}(c_{\mathrm{OPD}}\widetilde{A}_{\mathrm{SD\text{-}OPD},t}), \tag*{(10)}

where cOPDc_{\mathrm{OPD}} is the distillation scale and sg stops gradients through the scores and gate. We optimize SθS_\theta with the dual-clipped policy loss (Appendix A.3), using probability ratios relative to pp (Schulman et al., [21]) and averaging over all response tokens, including zero-gated positions. Algorithm 1 summarizes the complete procedure.

Implementation. Centering uses the student’s top K=1,024K = 1{,}024 candidates, including the sampled token, with full-vocabulary probabilities (Appendix A.2).

Experiments

Experimental Setup

Models. We evaluate 5 teacher–student settings with Qwen3 students (Yang et al., [27]) of 1.7B, 4B, and 8B parameters (Table 1). The teachers are Qwen3-4B-Instruct-2507, Qwen3-30B-A3B-Instruct-2507, and MiMo-7B-RL-0530; each table block identifies the teacher and student for that setting. SD-OPD uses each student’s corresponding pretrained base checkpoint as the frozen reference. Implementation details for the MiMo teacher are provided in Appendix A.4. We compare SD-OPD with the initial student, OPD, and ExOPD (Yang et al., [28]), whose formulation is detailed in Appendix A.5.

Training. All trained models use DAPO-Math-17K prompts (Yu et al., 2026) for 272 rollout rounds, corresponding to one full pass over the training dataset (one epoch). Each round samples 64 prompts and four responses per prompt. The learning rate is 10−610^{-6} and the response batch size is 64. We match rollout and response budgets across methods and evaluate their final checkpoints. Appendices A.1 and A.3 detail the shared training hyperparameters and method-specific clipping settings, respectively.

Evaluation. We evaluate on six mathematics benchmarks: AIME24 and AIME25 (zhuzilin, 2025a;b), AIME26 and HMMT25 (February) from MathArena (Dekoninck et al., [4]), AMC23 (math-ai, 2025), and MATH500 (Hendrycks et al., [9]; Lightman et al., [17]). We sample 16 responses per problem in non-thinking mode and report pass@1 avaveraged over these samples (avg@16) on each benchmark. The evaluation protocol uses temperature 0.7, top-pp 0.8, top-kk 20, and a maximum response length of 38,912 tokens. We retain completed and truncated responses and check extracted final answers for normalized or symbolic equivalence.

Main Results

Gains over OPD on every benchmark and setting. Table Table 1 shows that SD-OPD improves six-task average pass@1 over OPD by 7.76–10.37 percentage points across all 5 teacher–student configurations. The improvements hold on every benchmark in every configuration, covering all 30 comparisons under matched rollout and response budgets, and SD-OPD achieves the highest six-task student average in every configuration.

Gains across teachers. The gains do not depend on a particular teacher family. For Qwen3-1.7B, average gains over OPD are 9.77, 9.99, and 8.64 percentage points for Qwen3-4B-Instruct-2507, Qwen3-30B-A3B-Instruct-2507, and MiMo-7B-RL-0530 teachers, respectively. ExOPD improves the average with both Qwen3 teachers, but falls below OPD with MiMo: 44.40% versus 50.65%, a 6.25-point decrease. In the same MiMo setting, SD-OPD reaches 59.29%, exceeding ExOPD by 14.89 points under the shared-vocabulary protocol in Appendix § 8.§ 4.

Gains across student sizes. Using Qwen3-30B-A3B-Instruct-2507 as the teacher, average gains over OPD are 9.99, 7.76, and 10.37 percentage points for the 1.7B, 4B, and 8B students, respectively. The margin over ExOPD is smallest for the 4B student (3.45 points) and widens to 9.07 points for the 8B student. Across all 5 configurations, SD-OPD also exceeds ExOPD on 28 of the 30 benchmark comparisons.

Students can surpass their teachers. Distilled from Qwen3-30B-A3B-Instruct-2507, the 8B student reaches 76.11% average pass@1, exceeding the teacher’s 73.82% by 2.29 points, while OPD and ExOPD remain below it; this student also exceeds the teacher on AIME24, AIME25, HMMT25, and AMC23. On individual benchmarks, the 4B student also exceeds this teacher on AIME25 and AMC23, and the 1.7B student exceeds MiMo on AMC23 and MATH500.

Ablation Study

The two ablations test the two design choices of § 4 in the 4B-to-1.7B setting: UNGATED asks whether student-derived strength needs teacher selection, and FROZEN asks whether the student signal must be refreshed.

Teacher guidance. UNGATED uses ASA_S directly, with the same candidate approximation, replacing gASg_{A_S} and removing teacher scoring. Its pass@1 is zero on all six tasks (Table 2). All 10,560 responses complete within 5–13 tokens, including the termination token, with a mean of 6.05 tokens. Appendix § 11.§ 2 traces how these short, low-entropy responses emerge and explains them with a local gradient analysis: without the gate, the student–base signal reinforces deviations from the base.

VariantAIME24AIME25AIME26HMMT25AMC23MATH500Average
SD-OPD45.4233.3340.6221.0486.7288.7252.64
Frozen23.7517.0813.1211.2563.1278.4634.47
Ungated0.000.000.000.000.000.000.00

Table 2. Teacher guidance and source refresh ablations for the 1.7B student, using matched training and evaluation settings. FROZEN substitutes the initial student in the student–base difference. UNGATED removes the teacher sign gate and teacher scoring.

Student-source refresh. FROZEN replaces the current student in the numerator of the student–base ratio with the fixed initial student:

AS(0)=cp[log⁡pS0−log⁡pSB].(11)A_S^{(0)} = c_p[\log p_{S_0} - \log p_{S_B}]. \tag*{(11)}
AFrozen=1[AS(0)AOPD>0]AS(0).(12)A_{\mathrm{Frozen}} = \mathbf{1}\left[A_S^{(0)} A_{\mathrm{OPD}} > 0\right] A_S^{(0)}. \tag*{(12)}

FROZEN uses the candidate approximation in Equation (13); only the student–base source is fixed at S0S_0, while candidate selection, centering weights, rollout generation, and teacher feedback follow the current policy pp.

FROZEN attains 34.47% average pass@1, 18.18 points below SD-OPD and also below OPD (42.87%). Without refresh, both the coefficient magnitudes and the gate decisions follow the preferences of S0S_0, so supervision cannot build on what the student acquires during distillation.

Training Dynamics

To see how the choice of coefficient shapes the policy during training, Figure 4 tracks response length and policy entropy for OPD, ExOPD, and SD-OPD in the 30B-to-1.7B setting. We focus on the first 50 rollout steps, where the main differences emerge.

Figure 4. Early training dynamics with Qwen3-30B-A3B-Instruct-2507 teaching Qwen3-1.7B. Each step is one rollout round with four optimizer updates. Length is the rollout mean; entropy averages the four logged update means. Curves are unsmoothed. Appendix § 11.§ 1 shows all 272 training steps.

All three methods rapidly lengthen responses. Mean response length rises from approximately 1.7k tokens to around 6k within the first 20 steps. Over steps 41–50, OPD, ExOPD, and SD-OPD average 5.76k, 6.28k, and 6.45k tokens, respectively. ExOPD and SD-OPD reach similar lengths in this phase, although they build their coefficients differently.

SD-OPD develops a more concentrated policy. The entropy curves distinguish the methods more clearly. After an initial rise, SD-OPD entropy declines to an average of 0.158 nats over steps 41–50, compared with 0.314 for OPD and 0.308 for ExOPD. SD-OPD therefore keeps responses long while concentrating next-token predictions at the prefixes it visits. This pattern is consistent with reinforcing selected student preferences. Beyond step 50, the separation persists, with further gradual entropy reduction in SD-OPD and continuing length fluctuations (Appendix § 11.§ 1).

Conclusion

We showed that a student’s existing post-training changes can drive continued on-policy distillation. Since teacher–student disagreement is several times larger at style-related than at mathematical positions, SD-OPD instead takes update strength from student–base differences, constrains their direction with teacher feedback, and refreshes the student source throughout training. Across 5 teacher–student configurations, SD-OPD improves all six mathematics benchmarks over OPD and achieves higher six-task averages than both OPD and ExOPD. Average gains over OPD range from 7.76 to 10.37 percentage points when either the teacher or the student size changes.

The ablations show that both components are needed: reinforcing student preferences without teacher feedback collapses all six benchmarks to zero, and fixing the student source falls below OPD. Further work can examine which of the student’s own post-training changes provide useful supervision, how their value evolves during training, and how they can support continued learning across tasks.

Limitations

Our evaluation covers three teacher checkpoints from two families (Qwen3 and MiMo) and Qwen3 students of 1.7B, 4B, and 8B parameters on mathematical reasoning; extending SD-OPD to larger students and other domains is a natural next step. The comparisons match rollout and response budgets (clipping settings in Appendix § 8.§ 3), and the token analysis uses fixed rule-based categories that apply unchanged across models and datasets. SD-OPD uses the student’s pretrained base, released with open model families such as Qwen3, and adds one base scoring pass per round. Top-KK centering (K=1,024K = 1{,}024) evaluates the centering sums over a shared candidate set at each prefix, using full-vocabulary probabilities.

Detailed Training Settings

Training Hyperparameters

Table Table A1 lists the training hyperparameters shared by OPD and SD-OPD across the reported student sizes. Each round samples 64 prompts and four responses per prompt, yielding 256 responses for four optimizer updates with a global batch size of 64. Training comprises 272 rounds and 1,088 updates, and the rollout policy is synchronized after every round. We use the final checkpoint for evaluation.

HyperparameterValue
Rollout rounds272
Prompts per round64
Responses per prompt4
Response batch / microbatch64 / 1
Updates per round / total4 / 1,088
Maximum prompt tokens1,024
Maximum response tokens7,168
Maximum context tokens8,192
Rollout temperature1.0
Rollout top-pp1.0
Rollout top-kkDisabled
Learning rate10−610^{-6}
Learning-rate scheduleConstant
Warmup updates0
OptimizerAdam
Weight decay0.1
Adam (β1,β2\beta_1,\beta_2)(0.9, 0.98)
Adam ϵ\epsilon10−810^{-8}
Gradient norm clipping1.0
PrecisionBF16
PPO lower / upper clipping0.2 / 0.2
Distillation coefficient1.0
Advantage normalizationDisabled
Entropy coefficient0
Auxiliary reference KLDisabled
Training / rollout seed11 / 11

Table A1. Shared training hyperparameters for OPD and SD-OPD. The same settings are used for the FROZEN and UNGATED ablations.

Both methods use the non-thinking template and optimize the policy loss over response tokens, with no task reward or auxiliary reference-KL penalty. For Qwen3 teachers, OPD obtains sampled-token teacher scores through SGLang [35], whereas SD-OPD obtains teacher and base scores over the shared candidate set defined in Appendix § 8.§ 2. The candidate count K=1,024K = 1{,}024 controls the centering approximation and is independent of rollout top-kk sampling. ExOPD uses the corresponding OPD training configuration; Appendix § 8.§ 5 specifies its update rule and extrapolation scale.

Candidate-Set Centering

We approximate the expectation in Equation (6) using a shared candidate set VK(h,yt)V_K(h,y_t) at each prefix hh. The set contains the current rollout policy’s top K=1,024K = 1{,}024 tokens; if the sampled token yty_t is absent, it replaces the lowest-probability candidate. All three models score this same set using full-vocabulary probabilities, without renormalization within the set:

C~p,K[f](yt∣h)=f(yt∣h)−∑v∈VKp(v∣h)f(v∣h).(13)\widetilde{C}_{p,K}[f](y_t \mid h) = f(y_t \mid h) - \sum_{v \in V_K} p(v \mid h)f(v \mid h). \tag*{(13)}

Applying this operator to the student–base and teacher–student log-ratios gives A~S,t\widetilde{A}_{S,t} and A~OPD,t\widetilde{A}_{\mathrm{OPD},t}, respectively. Their signs determine the gate, and their gated student-derived coefficient supplies the fixed advantage in Equation (10). The candidate set, centering weights, and coefficients are recomputed from each new rollout-policy snapshot and remain fixed during its associated updates.

Dual Clipping

We use the PPO clipped objective with ϵ=0.2\epsilon= 0.2 [21]. For a response token yty_t at prefix hth_t, let wt=pSθ(yt∣ht)/p(yt∣ht)w_t = p_{\mathrm{S}\theta}(y_t \mid h_t)/p(y_t \mid h_t) be the policy ratio relative to the rollout policy. Using the fixed advantage A^t\widehat{A}_t from Equation (10), the token loss is

ℓtPPO=max⁡{−wtA^t, −clip⁡(wt,1−ϵ,1+ϵ)A^t}.(14)\ell_t^{\mathrm{PPO}} = \max\left\{-w_t\widehat{A}_t,\, -\operatorname{clip}(w_t,1-\epsilon,1+\epsilon)\widehat{A}_t\right\}. \tag*{(14)}

For a negative advantage, the first branch grows as wt∣A^t∣w_t\lvert\widehat{A}_t\rvert when wtw_t increases. Ordinary PPO clipping therefore does not bound the loss from large probability ratios on negatively weighted tokens. SD-OPD is exposed to this case because teacher-supported negative student–base coefficients actively suppress tokens, and the optimized policy can drift from the fixed rollout policy across minibatch updates; a few large ratios can then dominate the update.

Dual clipping adds an upper bound to this negative-advantage branch (Ye et al., 2020):

ℓtdual={min⁡{ℓtPPO,−cdualA^t},A^t<0,ℓtPPO,A^t≥0.(15)\ell_t^{\mathrm{dual}} = \begin{cases} \min\{\ell_t^{\mathrm{PPO}},-c_{\mathrm{dual}}\widehat{A}_t\}, & \widehat{A}_t < 0,\\ \ell_t^{\mathrm{PPO}}, & \widehat{A}_t \ge0. \end{cases} \tag*{(15)}

We set cdual=3.0c_{\mathrm{dual}} = 3.0 for SD-OPD, FROZEN, and UNGATED; the OPD and ExOPD baselines with Qwen3 teachers use ordinary PPO clipping. The additional bound caps the negative-advantage loss at 3∣A^t∣3\lvert\widehat{A}_t\rvert, while positive advantages retain the standard PPO objective. The loss is averaged over all response-token positions, including those with zero gated advantage. Dual clipping only limits extreme ratio contributions; which student-derived directions receive supervision is still decided by the teacher gate.

Compatibility with the MiMo Teacher

When distilling MiMo-7B-RL-0530 into Qwen3-1.7B, we align the response vocabulary and the prompt text while keeping each model’s own prompt tokenization.

Shared response vocabulary. The MiMo and Qwen3 output heads have 151,680 and 151,936 rows, respectively. Token IDs 0–151,664 are identical in the two tokenizers: they cover the 151,643 BPE tokens and the 22 special tokens from <|endoftext|> to <|file_sep|>, with the same normalization, pre-tokenization, and decoding. The vocabularies differ only beyond this range. Qwen3 adds four special tokens that MiMo lacks, <tool_response>, </tool_response>, <think>, and </think> (IDs 151,665–151,668), and the remaining rows of both heads (MiMo IDs 151,665–151,679 and Qwen3 IDs 151,669–151,935) are padding with no assigned token. We therefore restrict responses to the 151,665 shared IDs: a custom SGLang logit processor sets all other logits to −∞-\infty before sampling, and the old and updated student probabilities, policy entropy, and teacher and base scores are all normalized over this shared vocabulary. Excluding <think> and </think> from responses matches the non-thinking template, in which these tags appear only in the prompt. Model weights and output heads keep their original dimensions.

Prompt alignment. The common ChatML text appends /no_think to the user message and retains Qwen3’s empty thinking block in the assistant prefix. The student and base receive the original Qwen3 prompt IDs, including the prompt-only thinking tags. For MiMo, we decode the entire prompt while preserving special tokens and whitespace, then re-encode it with the teacher tokenizer. We append the sampled response IDs unchanged and align causal scores using the teacher’s re-encoded prompt length. This preserves the prompt text and response targets even when the two tokenizers produce different prompt lengths. After conversion, we check that the prompt and response fit within the teacher’s context length.

Scoring and optimization. We run the frozen MiMo teacher with its own Transformers model code and score the student responses by teacher forcing. OPD uses the sampled teacher–student log-ratio without centering or a base model. ExOPD uses the same teacher scorer, an extrapolation scale of λ=1.25\lambda= 1.25, and the frozen initial Qwen3-1.7B student as its reference, with all probabilities normalized over the shared vocabulary. For SD-OPD, the current student’s top K=1,024K = 1{,}024 candidates include the sampled token, and centering uses probabilities normalized over the entire shared vocabulary, with no renormalization within the candidate set. The student–base coefficient and teacher gate otherwise follow § 4. In the MiMo setting, OPD, ExOPD, and SD-OPD share the vocabulary mask and prompt protocol, and all three use cdual=3.0c_{\mathrm{dual}} = 3.0.

ExOPD Formulation and Hyperparameters

ExOPD is the reward-extrapolation variant of generalized on-policy distillation Yang et al. [28]. Our baseline uses the frozen initial post-trained student S0S_0 as its reference, while pp denotes the current rollout policy as in § 4.1. In the zero-discount approximation, its token-level advantage is

AExOPD,t=λlog⁡pTpS0−log⁡ppS0=AOPD,traw+(λ−1)log⁡pTpS0,(16)A_{\mathrm{ExOPD},t} = \lambda\log\frac{p_T}{p_{S_0}} - \log\frac{p}{p_{S_0}} = A_{\mathrm{OPD},t}^{\mathrm{raw}} + (\lambda- 1) \log\frac{p_T}{p_{S_0}}, \tag*{(16)}

where every probability is evaluated at the sampled token yty_t and its prefix hth_t. The teacher–reference log-ratio supplies an implicit reward, and the student–reference log-ratio is the sampled contribution to KL regularization toward S0S_0. The scale λ\lambda controls the reward’s weight relative to this regularization. Setting λ=1\lambda= 1 recovers the OPD advantage in Equation (5); λ>1\lambda> 1 strengthens the teacher–reference reward, yielding ExOPD. In the generalized formulation, 0<λ<10 < \lambda< 1 instead corresponds to reward interpolation.

Settings used in this paper. We fix λ=1.25\lambda= 1.25 for every ExOPD configuration, the value used by Yang et al. [28]. The coefficient therefore adds 0.25log⁡(pT/pS0)0.25 \log(p_T/p_{S_0}) to standard OPD while keeping the student–reference term’s weight fixed. The reference S0S_0 remains frozen throughout training and differs from the pretrained base SBS_B used by SD-OPD. We use A^t=sg⁡(clip⁡OPDAExOPD,t)\hat{A}_t = \operatorname{sg}(\operatorname{clip}_{\mathrm{OPD}} A_{\mathrm{ExOPD},t}) with cOPD=1.0c_{\mathrm{OPD}} = 1.0 in the clipped policy loss, without advantage normalization, task rewards, or an additional reference-KL loss. Shared training hyperparameters are listed in Table A1; Qwen3 and MiMo runs follow the clipping and scoring settings in Appendices § 8.§ 3 and § 8.§ 4, respectively.

Supplementary Analysis

Full Training Dynamics

Figure B1 extends the comparison in § 5.4 beyond the first 50 rollout steps. All three methods use Qwen3-30B-A3B-Instruct-2507 as teacher and Qwen3-1.7B as the initial student. Each run covers all 272 rollout steps, or 1,088 optimizer updates.

Shared protocol and aggregation. All runs use seed 11, DAPO-Math-17K prompts, 64 prompts and four sampled responses per rollout step, and four optimizer updates per step. The learning rate, sampling settings, and response budget are shared; method-specific clipping follows Appendix § 8.§ 3. Response length averages all 256 generated responses, including those reaching the 7,168-token cap. Policy entropy is the arithmetic mean of the four logged update-level entropy means, measured on that method’s own response tokens. The two panels therefore compare generation behavior and predictive concentration under the same rollout schedule.

Figure B1. Full training trajectories in the 30B-to-1.7B setting: 272 rollout steps for each method. The dotted line marks step 50, the endpoint of Figure 4. Each step contains four optimizer updates. Length is the rollout mean; entropy averages the four update means. All recorded points are shown without smoothing.

The early separation persists later in training. The first 50 steps capture the rapid initial increase in response length and the emergence of a lower-entropy SD-OPD policy. Later length trajectories continue to fluctuate, while the entropy difference remains clear. Over the final 20 steps (253–272), mean lengths are 4.73k, 5.25k, and 6.52k tokens for OPD, ExOPD, and SD-OPD, respectively; the corresponding entropies are 0.323, 0.322, and 0.128 nats. For SD-OPD, the decrease from 0.158 nats over steps 41–50 to 0.128 over the final 20 steps is gradual relative to its initial decline. The longer-response, lower-entropy profile of SD-OPD thus persists beyond the main-text window, although none of the curves has fully plateaued.

These statistics characterize each policy on the prefixes it generates. Lower token entropy indicates more concentrated next-token predictions at those prefixes. Under SD-OPD, responses stay long while these predictions become increasingly concentrated.

Analysis of Ungated Degeneration

Observed training dynamics. In the 4B-to-1.7B ablation setting, the ungated run gradually collapses into short, low-entropy responses. Table B1 compares the training trajectories of UNGATED and SD-OPD. For UNGATED, mean response length first falls below 20 tokens at round 105 and reaches six tokens by round 180. The logged entropy initially rises, reaching 2.1405 at round 60, before falling to 0.0074 at round 120 and 0.0004 at round 272. The run thus passes through an early phase of growing uncertainty before collapsing onto a few tokens. SD-OPD retains a final training-rollout mean length of 6,176.3 tokens and logged entropy of 0.1243 under the same rollout budget and clipping configuration.

RoundUNGATED LengthUNGATED EntropySD-OPD LengthSD-OPD Entropy
11752.50.25361663.90.2628
201793.80.33925669.30.1966
40280.62.03285880.60.1636
601519.92.14055777.30.1469
10030.10.54004906.20.1170
12016.00.00746894.20.1532
1806.00.00095665.60.1195
2726.00.00046176.30.1243

Table B1. Training dynamics at selected rounds. Length is the mean number of response tokens generated at the start of the round. Entropy is the arithmetic mean of the four logged per-update response-token entropy means. Each method generates its own responses.

The final evaluation confirms this collapse: none of the 10,560 UNGATED responses is correct, and all are complete responses of only 5–13 tokens. Of these, 10,249 contain exactly six tokens.

Case study: final UNGATED outputs. The first AIME24 question asks how many Aimeville residents own all four listed items, with a gold answer of 73. The first three responses sampled from the final UNGATED checkpoint are identical:

ShanesPid !_ bureauacr
newIndex<|im_end|>

Each response contains six tokens, including the termination token, and receives zero reward. The repeated fragment contains no reasoning about the counting problem and no numerical answer, making the collapse visible in the generated text.

Self-reinforcement without a quality constraint. A local policy-gradient analysis relates the student–base signal to divergence from the base. Fix a prefix and let θold\theta_{\mathrm{old}} denote the behavior-policy parameters, with p=pS-oldp=p_{\mathrm{S\text{-}old}}. Assume exact full-vocabulary centering and consider the policy gradient at the behavior policy before clipping becomes active. The base distribution pSBp_{\mathrm{S_B}} is fixed, and the sampled coefficient ASA_{\mathrm{S}} is treated as constant during the update. Suppressing the fixed prefix in the notation gives

Gh=Ea∼p[AS(a)∇θlog⁡pθ(a)]∣θ=θold=∇θDKL(pSθ∥pSB)∣θ=θold.(17)G_h = \mathbb{E}_{a\sim p}\left[A_{\mathrm{S}}(a)\nabla_\theta\log p_\theta(a)\right]\bigg|_{\theta=\theta_{\mathrm{old}}} = \nabla_\theta D_{\mathrm{KL}}\left(p_{\mathrm{S}\theta}\mathrel{\Vert}p_{\mathrm{S_B}}\right)\bigg|_{\theta=\theta_{\mathrm{old}}}. \tag*{(17)}

To see this, substitute AS(a)=log⁡[p(a)/pSB(a)]−DKL(p∥pSB)A_{\mathrm{S}}(a) = \log[p(a)/p_{\mathrm{S}_{\mathrm{B}}}(a)] - D_{\mathrm{KL}}(p\mathbin{\|}p_{\mathrm{S}_{\mathrm{B}}}) and use the zero-mean score identity Ea∼p[∇θlog⁡pSθ(a)]=0\mathbb{E}_{a\sim p}[\nabla_{\theta}\log p_{\mathrm{S}_{\theta}}(a)] = 0 at the behavior policy, as in policy-gradient baseline arguments [23]. The centering term and the constant term arising when differentiating KL both cancel. Consequently, at a nonstationary point, a sufficiently small step along this gradient increases divergence from the base.

The signal reinforces tokens whose student–base log-ratio exceeds its current-policy average and suppresses those below it, regardless of answer quality. Because the numerator is refreshed after each round, newly amplified deviations become part of the next round’s supervision. This feedback can reinforce accidental preferences as well as useful post-training changes. The identity DKL(p∥pSB)=−H(p)−Ep[log⁡pSB]D_{\mathrm{KL}}(p\mathbin{\|}p_{\mathrm{S}_{\mathrm{B}}})=-H(p)-\mathbb{E}_{p}[\log p_{\mathrm{S}_{\mathrm{B}}}] also shows why increasing divergence need not preserve entropy: it can favor concentration or movement toward tokens assigned low probability by the base. Neither direction is intrinsically tied to solving the question.

The role of teacher guidance. The teacher gate supplies a direction test that is absent from UNGATED: it blocks student coefficients whose signs disagree with centered teacher feedback. Both methods use the same dual-clipping constant, while task reward, entropy regularization, and auxiliary reference-KL penalties are disabled. Clipping controls the surrogate incentive for local probability-ratio changes, while the teacher gate selects which student-derived directions receive reinforcement.

Together, the local analysis and the observed trajectories point to a self-reinforcing mechanism: without the gate, deviations amplified in one round become supervision in the next, and unguided preferences grow into degenerate behavior. The teacher’s direction test breaks this loop by admitting only the student changes the teacher also supports.

Token Categorization and Preference Statistics

Observation Protocol

We sample 1,024 training questions and analyze all eight student responses per question, without filtering by correctness or length. Generation uses the non-thinking template, temperature 1.0, top-pp 1.0, no top-kk truncation, and at most 7,168 response tokens. The maximum context length is 8,275 tokens, and the per-question generation seed is 11+8×11+8\times dataset index. We use vLLM for generation [16] and Transformers for offline scoring [25]. The teacher, student, and base score identical token IDs at the same causal prefixes using untempered logits and full-vocabulary log-softmax. Log probabilities are computed in FP32 and differenced in FP64; all log probabilities and entropies use natural logarithms.

Token Categories

We assign the five categories in Figure 2 using fixed English and Chinese phrase lists and textual rules, independently of model scores.

Style-related tokens. This category covers discourse wording, answer labels, and Markdown formatting used to organize or present a response. Examples include let’s, next, and finally; line-initial Answer or Solution labels; and Markdown markers such as # and **.

Mathematical tokens. This category covers numbers, operators, formula variables, mathematical vocabulary, and logical expressions used in derivations. Examples include numerals such as 3 and 12, operators such as + and =, formula variables such as xx, and commands such as \frac and \sqrt. It also includes terms such as integer and triangle, and logical expressions such as if, therefore, and suppose.

Markup tokens. This category covers formula delimiters, structural punctuation within formulas, and LaTeX commands outside the mathematical and logical command lists. Examples include $, [, and ] as delimiters, braces and subscript markers (_) within formulas, and formatting commands such as \text and \left.

Whitespace tokens. This category contains tokens composed entirely of whitespace. Examples include a single space, consecutive spaces, a tab, or one or more line breaks. Leading spaces attached to a word do not make that token a whitespace token.

Other tokens. This category contains unassigned text and punctuation, special tokens, and tokens crossing incompatible category boundaries. Examples include ordinary prose words such as the and and, commas or periods in prose, and special tokens such as <|im_end|>.

Token boundaries. We label character spans and map them to the original tokens through UTF-8 byte spans, without retokenization. Ignoring whitespace, tokens spanning compatible mathematical or stylistic subtypes keep that category; incompatible mixtures are assigned to Other. Logical expressions are matched separately and merged into the mathematical group after this boundary handling.

Category-Level Aggregation

The five displayed categories are disjoint and account for all response tokens in the 8,192 responses. For each category, we compute the mean absolute log-ratio within each response, average the eight response means within each question with equal weight, and then average over the 1,024 questions with equal weight. Every displayed group occurs in every response, so all eight responses contribute to every question’s category mean. For the mathematical group, we combine mathematical-content and logical-expression tokens within each response before computing its mean. This procedure gives responses equal weight regardless of length. The student–base and same-sign-filtered panels use exactly the same category positions and averaging order. For filtering, we multiply each student–base difference by 1[rtdt>0]1[r_t d_t > 0]; opposite signs and exact zeros yield zero. Zeroed positions remain in the original category denominator, so the filtered means measure retained amplitude per original token.

Expression-Level Preference Statistics

We consider alphabetic English strings with at least 50 occurrences across at least 20 questions, merging decoded tokens after lowercasing and stripping leading and trailing whitespace; subword fragments therefore remain separate strings. The word clouds in Figure 3(a–Figure 3b) show 50 representative strings for each sign: ten high-amplitude examples and forty lower-amplitude context strings. Within each panel, a monotone piecewise-linear size scale expands variation among the context strings while preserving larger sizes for the high-amplitude examples; all strings are horizontal, and color darkens with size. Figure 3(c) uses six representative strings chosen by lexical function and observed sign: suppose, try, strategy, and idea express hypothesis formation, exploration, and planning, while straightforward and simplicity describe procedural ease. They occur in 280, 554, 112, 166, 47, and 110 questions, respectively, and we keep all their occurrences across the eight responses per question.

For each string, we average its occurrences within each response, then the responses containing it within each question, and finally the questions containing it. We apply this same aggregation to dtd_t, its positive and negative parts, and those parts multiplied by 1[rtdt>0]1[r_t d_t > 0]. Rejected positions contribute zero and remain in the original denominators. The retained amplitude percentage divides the aggregated retained absolute amplitude by the aggregated original absolute amplitude.

SettingStyle-to-math ratio Before OPDStyle-to-math ratio Before S–BStyle-to-math ratio After OPDStyle-to-math ratio After S–BRemoved (%) StyleRemoved (%) Math
Original3.5304.0973.7173.06469.158.7
30B-A3B teacher3.2624.0973.7393.52063.357.2
GSM8K data†5.6465.2463.8112.31881.057.1

Table D1. Category-level feedback patterns for Qwen3-1.7B across teachers and datasets. OPD uses teacher–student log-probability differences; student–base uses the student’s differences from its pretrained base. Ratios compare mean absolute amplitudes at style-related and mathematical positions, aggregated as in Equation (2). Before/After refer to sign gating, which retains positions where the two signals agree; S–B denotes student–base, and Removed gives the share of student–base amplitude removed by gating. †For GSM8K, averages use responses and questions containing the category: style-related tokens occur in 8,105 responses across 1,021 questions; mathematical tokens occur in all responses.

Observations across Teachers and Datasets

For the Qwen3-1.7B student, we examine the category-level amplitude disparity and the effect of teacher selection from § 3 when the teacher or dataset changes.

Controlled settings. The original setting uses Qwen3-4B-Instruct-2507 as teacher, Qwen3-1.7B as student, Qwen3-1.7B-Base as base, and DAPO-Math-17K. The 30B-A3B teacher setting replaces only the teacher with Qwen3-30B-A3B-Instruct-2507; it reuses the original responses and student/base scores and recomputes teacher scores on exactly the same prefixes and token IDs. The GSM8K data setting retains the original models and samples 1,024 questions without replacement from the GSM8K training split (Cobbe et al., [2]), using the same surrounding prompt instructions.

Each setting includes all eight responses to 1,024 questions, without correctness or length filtering. Sampling, scoring, and category definitions follow Appendix § 12. We compare uncentered log-probability differences on the initial student’s responses, and teacher-agreement filtering follows § 3.2, with rejected positions contributing zero to the original averages.

Larger OPD amplitudes at style-related positions. Table D1 shows that style-related positions receive larger mean absolute teacher–student log-ratios in both additional settings, with style-to-mathematics amplitude ratios of 3.262 for the 30B-A3B teacher and 5.646 on GSM8K. Changing the teacher alters the source of feedback on fixed responses, whereas changing the dataset also changes the trajectories being scored. Both comparisons reproduce the disparity identified in § 3.1: teacher–student disagreement assigns larger per-token coefficients to differences in expression.

Teacher agreement reduces the relative style amplitude. Across the two additional settings, gating removes 63.3–81.0% of the student–base style amplitude, compared with 57.1–57.2% of the mathematical amplitude. The style-to-mathematics ratio consequently falls in each setting, with the largest reduction on GSM8K, from 5.246 to 2.318. Because filtered positions remain in the original denominators, these reductions measure how much of each category’s acquired amplitude survives the teacher’s direction test. Both settings reproduce the observation in § 3.2: teacher selection changes the relative allocation of student-derived feedback, retaining proportionally less amplitude at style-related positions.

The magnitude source changes the gated balance. With the 30B-A3B teacher, gated student–base weighting gives a style-to-mathematics ratio of 3.520, compared with 3.739 for gated OPD. On GSM8K, the corresponding ratios are 2.318 and 3.811. Each comparison holds the sign mask fixed, so these differences come from the source of update magnitudes. Both comparisons reproduce the lower ratio from student-derived magnitudes observed for Qwen3-1.7B in § 3.2.

References

  1. [1]Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos Garea, S., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pp. 21246–21263, 2024.arxiv.org/abs/2306.13649
  2. [2]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plapper, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.arxiv.org/abs/2110.14168
  3. [3]Cui, G., Yuan, L., Wang, Z., Wang, H., Zhang, Y., Chen, J., Li, W., He, B., Fan, Y., Yu, T., et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.arxiv.org/abs/2502.01456
  4. [4]Dekoninck, J., Jovanović, N., Gehrunger, T., Røgnvaldsson, K., Petrov, I., Sun, C., and Vechev, M. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms. arXiv preprint arXiv:2605.00674, 2026.arxiv.org/abs/2605.00674
  5. [7]Guha, E., Marten, R., Keh, S., Raoof, N., Smyrnis, G., Bansal, H., Nezhurina, M., Mercat, J., Vu, T., Sprague, Z., et al. Openthoughts: Data recipes for reasoning models. In International Conference on Learning Representations, volume 2026, pp. 108059–108130, 2026. URL https://huggingface.co/datasets/math-ai/amc23.
  6. [8]Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  7. [9]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
  8. [10]Heo, B., Hwang, J., Yun, S., and Han, D. On-policy delta distillation. arXiv preprint arXiv:2607.15161, 2026.arxiv.org/abs/2607.15161
  9. [11]Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1502.03150, 2015.
  10. [12]Hübotter, J., Lübeck, F., Behric, L., Baumann, A., Bagatella, M., Marta, D., Hakimi, I., Shenfeld, I., Bueneng, T. K., Guestrin, C., et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026.
  11. [13]Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp. 1317–1327, 2016.arxiv.org/abs/1606.07947
  12. [16]Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611–626, 2023.arxiv.org/abs/2309.06180
  13. [17]Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pp. 39578–39601, 2024.arxiv.org/abs/2305.20050
  14. [18]Lin, B. Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K., Bhagavatula, C., and Choi, Y. The unlocking spell on base llms: Rethinking alignment via in-context learning. In International Conference on Learning Representations, volume 2024, pp. 24907–24933, 2024.arxiv.org/abs/2312.01552
  15. [19]Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 53728–53741, 2023.arxiv.org/abs/2305.18290
  16. [20]Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627–635. JMLR Workshop and Conference Proceedings, 2011.
  17. [21]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  18. [22]Shenfeld, I., Damani, M., Hübotter, J., and Agrawal, P. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026.arxiv.org/abs/2601.19897
  19. [23]Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
  20. [24]Wang, Z., Wang, T., Zhang, L., and Hu, Y. Dr. opd: Learning what to follow for optimal on-policy distillation of large language models. arXiv preprint arXiv:2609.38025, 2026.
  21. [25]Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45, 2020.DOI
  22. [26]Xiao, B., Xia, B., Yang, B., Gao, B., Shen, B., Zhang, C., He, C., Lou, C., Luo, F., Wang, G., et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026.arxiv.org/abs/2601.02780
  23. [27]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  24. [28]Yang, W., Liu, W., Xie, R., Yang, K., Yang, S., and Lin, Y. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125, 2026.arxiv.org/abs/2602.12125
  25. [33]Yuan, L., Li, W., Chen, H., Cui, G., Ding, N., Zhang, K., Zhou, B., Liu, Z., and Peng, H. Free process rewards without process labels. arXiv preprint arXiv:2412.01981, 2024.arxiv.org/abs/2412.01981
  26. [34]Zhao, S., Xie, Z., Liu, M., Huang, J., Pang, G., Chen, F., and Grover, A. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026.arxiv.org/abs/2601.18734
  27. [35]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyraki, C., Stoica, I., Gonzalez, J. E., et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024.arxiv.org/abs/2312.07104
  28. [36]Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023.arxiv.org/abs/2305.11206

Paper details

vlnc:W_nwpa6u2q
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 18

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents