$D^2$-OPD: Separating Direction and Distance in Extrapolative On-Policy Distillation

Introduction

On-policy distillation (OPD) trains a student language model on its own responses: the student generates a response and, at each prefix it visits, learns from the teacher’s next-token distribution [2, 7, 17]. Training on student-generated prefixes reduces the mismatch between the responses seen in training and those the student generates at inference [24, 16]. Prior work has studied several aspects of on-policy distillation, including the efficient use of student-generated responses [14, 13], teacher entropy [11], teacher–student disagreement [33], and training stability [32]. These studies show that the way teacher supervision is constructed and transferred matters beyond the choice of teacher alone.

However, the teacher’s distribution may not be the best target: prior work shows that a more capable teacher does not necessarily yield a better student [20, 15]. Extrapolative OPD aims to construct a better target beyond the teacher by amplifying the teacher’s log-probability shift from a reference policy. If the teacher was obtained from the reference by KL-regularized post-training, this shift equals its regularized $Q$-function up to scale and a prefix-dependent constant [6, 22], so amplifying it moves probability toward tokens with higher teacher $Q$-values. Constructing an extrapolative target involves two decisions: the direction in which the target moves away from the teacher, and the distance it moves. In existing methods, the distance is tied to the direction. For example, ExOPD [36] fixes the direction through its reference, but under a fixed coefficient its distance varies with the scale of the teacher–reference gap across prefixes; in Direct-OPD [5], changing the batch-shared adaptive coefficient generally changes both the direction and the distance. In both cases, the distance follows from the coefficient and the log-ratios at each prefix, rather than from how far the target can be trusted beyond the teacher. We show that, under fixed-coefficient extrapolation, the target can place substantial probability on tokens the teacher rarely predicts, even when the error in the extrapolation signal is small on average under the teacher.

To address this coupling, we propose $D^2$-OPD, which decouples distance from direction. First, a per-prefix hard KL constraint bounds the target’s distance from the teacher, independently of the scale of the extrapolation signal; this also bounds the probability the target can assign to tokens the teacher rarely predicts. Second, we construct the direction by combining the teacher’s post-training shift and the initial student–teacher gap. The weights adapt at each prefix, giving more weight to the nearer signal. Finally, forward KL transfers the target while maintaining non-vanishing gradients for tokens rarely generated by the student.

Vector (PDF)

We evaluate D²-OPD on three Qwen3 teacher–student pairs covering a setting where standard OPD performs well, a setting for which prior work recommends a supervised cold start [15], and a setting without a separate teacher reference. D²-OPD achieves the highest average score across AIME24, AIME25, and AMC23 in all three settings. In Qwen3-8B → Qwen3-4B-Base distillation, it outperforms OPD by 19.62 points and the strongest baseline by 9.45 points. In the cold-start setting, it exceeds the released SFT + OPD recipe without supervised fine-tuning. Ablations confirm the contributions of the distance constraint, the combined direction, and forward-KL distillation. Response-length and entropy analyses indicate that the gains are not merely due to longer answers or more concentrated student policies.

Related work

Distillation objectives. Knowledge distillation trains students using teacher predictions, with applications to policy learning and reasoning [9, 12, 25, 30, 26, 10, 29]. On-policy imitation and distillation reduce the mismatch between training and inference by training on the learner’s own trajectories [24, 16, 2]. Related work studies divergence choices and response reuse for transferring teacher supervision [31, 7, 14, 13].

Teacher–student gaps. A larger or more capable teacher does not necessarily produce a better student [20, 28, 1, 3, 34] (Zhong et al. 2024). EOPD [11] adds forward KL when teacher entropy is high, while TrOPD [33] applies forward KL where teacher–student disagreement is large. Li et al. [15] link training failure to low initial overlap between the teacher’s and student’s high-probability token sets.

Extrapolation and distance constraints. Under KL-regularized optimization, the log-ratio between the optimized policy and its reference encodes the regularized $Q$-function [21, 6, 23, 22]. Extrapolative methods use this relation to construct new distillation targets. ExOPD uses a fixed coefficient to scale a direction set by its reference [36]. Direct-OPD transfers the teacher’s post-training shift through an objective anchored at the initial student, with an adaptive regularization coefficient applied uniformly across the batch [5]. Changing this coefficient generally changes both the target’s direction and its distance relative to the teacher. TOP-D constructs a proximal teacher and uses trust-region updates [32]; ERPD constrains distillation relative to the rollout policy [4].

Preliminaries

Let $\mathcal{D}$ be the training prompt distribution and $\pi_\theta$ the current student with parameters $\theta$. For a prompt $x \sim\mathcal{D}$, we sample $G$ responses $y_1,\ldots,y_G$ from the student. The index $i$ identifies a response, $t$ a token position, and $\lvert y_i\rvert$ its response length. Writing $y_{i,<t}$ for the tokens before position $t$, we define the prefix $h_{i,t} = (x,y_{i,<t})$. We use $y$, $h$, and $a$ for a generic response, prefix, and candidate next token. We denote the initial student by $\pi_0$, the post-trained teacher by $q^+$, and the checkpoint from which the teacher was post-trained by $t_{\mathrm{ref}}$. For any policy $\pi$, $\pi(a \mid h)$ denotes a token probability and $\pi(\cdot\mid h)$ the next-token distribution, while $\pi(y \mid x)$ denotes a response probability. Let $K$ be the number of candidate tokens and $S(h)$ the top-$K$ tokens under the teacher distribution $q^+(\cdot\mid h)$. For any token set $B$, write $\pi(B \mid h) = \sum_{a \in B} \pi(a \mid h)$. The distribution renormalized on $S(h)$ is $\pi_S(a \mid h) = \pi(a \mid h)/\pi(S(h) \mid h)$ for $a \in S(h)$, and zero otherwise. In particular, $q^+_S(\cdot\mid h)$ and $\pi_{\theta,S}(\cdot\mid h)$ denote the renormalized teacher and student distributions, respectively, renormalized over the same top-$K$ token set $S(h)$.

On-policy distillation

Reverse-KL OPD [7, 17] minimizes the loss $\mathcal{L}_{\mathrm{OPD}}(\theta)$,

At a student-generated prefix, the token reward is $r_\theta(h,a) = \log(q^+(a \mid h)/\pi_\theta(a \mid h))$. Its expectation under the student is the negative reverse KL:

Maximizing the expected sum of token rewards is equivalent to minimizing Eq. (1).

Extrapolative targets

For a reference $b$, OPD is KL-regularized RL with reward $\log(q^+(y \mid x)/b(y \mid x))$ and regularizer anchored at $b(\cdot\mid x)$. Scaling that reward by $\lambda> 0$ gives the ExOPD target $\pi^*_\lambda$ [36],

It recovers the teacher at $\lambda= 1$ and extrapolates along $\log(q^+(y \mid x)/b(y \mid x))$ for $\lambda> 1$.

The teacher reference gives this direction a policy-improvement interpretation. Let $\Delta_T(h,a) = \log(q^+(a \mid h)/t_{\mathrm{ref}}(a \mid h))$ be the teacher’s post-training shift. If $q^+$ solves a KL-regularized objective relative to $t_{\mathrm{ref}}$ with coefficient $\beta_T > 0$, its regularized $Q$-function $Q^+(h,a)$ satisfies the following identity for an action-independent term $c(h)$ [6, 22]:

$\Delta_T(h,a)$ and $Q^+(h,a)$ rank tokens identically. The teacher’s post-training shift can therefore serve as a signal for further policy improvement. Direct-OPD uses $\Delta_T(h,a)$ as its token reward and anchors its regularizer at $\pi_0$, with regularization coefficient $\alpha> 0$. Its optimal target $\pi^*$ is [5]

This target need not lie on the teacher-reference ray in Eq. (2). In both objectives, distance from $q^+(\cdot\mid x)$ depends on the log-ratio scale as well as the coefficient. Appendix § 15 gives the full objectives.

Method

The objectives in § 3.2 tie two decisions together: the direction in which the target leaves the teacher, and how far it travels. We construct the extrapolation direction and separately bound the target’s distance from the teacher at each prefix.

For a prefix $h$ visited by a student rollout, we build a target distribution $q(\cdot\mid h)$ along a direction derived from two reference signals, at a distance we bound explicitly, and train the student toward it with forward KL, using the loss $\mathcal{L}(\theta)$:

We next explain the rationales behind each of our design choices. § 4.1 bounds how far $q(\cdot\mid h)$ may move from the teacher, through a hard radius $\epsilon\ge0$ and a soft coefficient $0 < \kappa\le1$. § 4.2 determines the combined direction $d_\eta(h,a)$, where $\eta(h) \in[0,1]$ is derived to weight the student signal by minimizing relative deviation from both signals. § 4.3 explains why the divergence in Eq. (5) is forward rather than reverse KL.

Vector (PDF)

Limiting the extrapolation distance

As discussed in § 3.2, existing extrapolation coefficients control the distance from $q^{+}(\cdot\mid h)$ only indirectly: a coefficient multiplies a log-ratio whose scale varies across prefixes, so one value can hold the target close to $q^{+}(\cdot\mid h)$ at one prefix and carry it much farther at another. The distance is worth controlling on its own, because the teacher is a policy we can observe and evaluate, whereas a target farther from $q^{+}(\cdot\mid h)$ rests more heavily on extrapolation. For a candidate target distribution $q(\cdot\mid h)$ supported on $S(h)$, let $K(q;h)=D_{\mathrm{KL}}(q(\cdot\mid h)\|q_{S}^{+}(\cdot\mid h))$. We impose $K(q;h)\leq\epsilon$ at every prefix, which bounds how far the target may move. To express token preference in the target distribution, let $d(h,a)$ be a per-token score function on $S(h)$, whose construction is introduced in § 4.2. We choose $\tilde{q}(\cdot\mid h)$ to increase the expected value of $d(h,a)$ while penalizing its KL distance from the renormalized teacher $q_{S}^{+}(\cdot\mid h)$. The coefficient $\kappa$ controls this trade-off, and the constraint limits the distance to $\epsilon$. Therefore, the target distribution $\tilde{q}(\cdot\mid h)$ is defined as:

subject to $K(q;h)\leq\epsilon$. {#eq:6}

Solving the constrained problem. Let $\lambda(h)\geq0$ be the Lagrange multiplier on the KL constraint and $\beta(h)$ the extrapolation coefficient. The optimum of Eq. (6) has the form

The target is normalized over $S(h)$ and zero elsewhere. The unconstrained solution yields $\lambda(h)=0$ and coefficient $\beta_{\max}=(1-\kappa)/\kappa$. At $\kappa=1$, the target reduces to $q_{S}^{+}(\cdot\mid h)$ while smaller values allow stronger extrapolation.

For $0<\kappa<1$, we retain the unconstrained target if it satisfies the bound; otherwise, we perform 24 bisection iterations on $[0,\beta_{\max}]$, where $D_{\mathrm{KL}}(\tilde{q}(\cdot\mid h)\|q_{S}^{+}(\cdot\mid h))$ is nondecreasing in $\beta(h)$, to find the constraint boundary. We process prefixes in parallel on the teacher’s top-$K$ support. The search is computationally inexpensive.¹

ExoPD observes longer responses under stronger extrapolation and attributes this partly to potential length bias in the implicit reward [36]. We propose another possible mechanism: extrapolation may amplify overestimated scores for low-probability teacher tokens, potentially encouraging repetitive or prolonged reasoning. The KL bound limits the probability mass assigned to such tokens.

For fixed $\epsilon$, this bound tends to zero as the teacher probability $\rho(h)$ tends to zero. Appendix § 16 contains the proof and expands on the connection to response-length growth. It also shows that fixed-coefficient extrapolation can assign substantial probability to these tokens even when the score error is small in teacher-weighted mean square.

Determining the extrapolation direction

The teacher’s post-training shift and the initial student–teacher gap typically point in different directions. Although both provide useful guidance, constructing a target requires combining them into a single direction. As shown in § 3.2, Direct-OPD’s regularization coefficient $\alpha$ controls both direction and distance, without a separate criterion for combining the signals. We therefore introduce an explicit criterion that balances the direction’s relative deviation from the two signals.

We use the teacher-reference signal $\Delta_T(h,a)=\log(q_S^+(a\mid h)/t_{\mathrm{ref}}(a\mid h))$ and the student signal $\Delta_{\pi_0}(h,a)=\log(q_S^+(a\mid h)/\pi_0(a\mid h))$. For convenience, we center both signals so that their variances equal their squared norms under $q_S^+(\cdot\mid h)$. At each prefix, for $j\in\{T,\pi_0\}$, we define $\Delta_j^c(h,a)=\Delta_j(h,a)-\mathbb{E}_{b\sim q_S^+(\cdot\mid h)}[\Delta_j(h,b)]$.

Let $\sigma_T^2(h)$ and $\sigma_{\pi}^2(h)$ denote the variances of the teacher-reference and student signals, respectively, under $q_S^+(\cdot\mid h)$. In centered log-probability space weighted by $q_S^+(\cdot\mid h)$, each signal’s variance is the squared distance from the teacher to its reference. We refer to $\sigma_T(h)$ and $\sigma_\pi(h)$ as reference distances and normalize each squared deviation by its corresponding squared reference distance. When both reference distances are positive, let $R_T(d;h)=\mathbb{E}_{a\sim q_S^+(\cdot\mid h)}[(d(h,a)-\Delta_T^c(h,a))^2]/\sigma_T^2(h)$ and define $R_\pi(d;h)$ analogously using $\Delta_{\pi_0}^c(h,a)$ and $\sigma_\pi^2(h)$. We define $d_\eta(h,\cdot)$ as:

subject to $\mathbb{E}_{a\sim q_S^+(\cdot\mid h)}[d(h,a)]=0.

Proposition 2 When both reference distances $\sigma_T(h)$ and $\sigma_\pi(h)$ are positive, the objective above has the unique minimizer

The proof is given in Appendix § 16. The weight $\eta(h)$ is determined by the two signals at each prefix, giving more weight to the nearer reference.

For the geometric interpretation, let $u(h,a)=\Delta_T^c(h,a)/\sigma_T(h)$ and $v(h,a)=\Delta_{\pi_0}^c(h,a)/\sigma_\pi(h)$ be the unit directions. The solution satisfies

Figure 2 illustrates how this combination places the direction closer to that of the nearer reference. In our implementation, we compute the closed-form solution directly without forming $u(h,a)$ and $v(h,a)$, avoiding division by either reference distance. We use Eq. (8) when both reference distances exceed $10^{-3}$. If only one is at most $10^{-3}$, we omit that signal and use the other centered signal. If both are at most $10^{-3}$, we set $d_\eta(h,a)=0$, so the target remains $q_S^+(\cdot\mid h)$.

We set $d(h,a)=d_\eta(h,a)$ in Eq. (6). Following ExOPD, we can equivalently express this direction using a constructed reference $t_\eta(\cdot\mid h)$ on $S(h)$:

After normalization over $S(h)$, $\log(q_S^+(a\mid h)/t_\eta(a\mid h))$ equals $d_\eta(h,a)$ up to a token-independent constant. This constructed reference requires no additional model.

Transferring the target to the student

The constructed target can place probability to tokens that are rare under the student. We use forward KL because its gradient remains non-vanishing in this case. The forward-KL gradient with respect to the student logit is $\pi_\theta(a\mid h)-\tilde{q}(a\mid h)$. For tokens with positive target probability, it approaches $-\tilde{q}(a\mid h)$ as $\pi_\theta(a\mid h)\to0$, whereas the reverse-KL gradient vanishes.

In practice, $S(h)$ contains the top-$K$ tokens under the teacher distribution $q^+(\cdot\mid h)$. We construct and renormalize $\tilde{q}(\cdot\mid h)$ on $S(h)$, but keep the student distribution over the full vocabulary. Estimating forward KL from student-sampled tokens can have high variance. The importance weights $\tilde{q}(a\mid h)/\pi_\theta(a\mid h)$ can be large when the target assigns substantial probability to tokens the student rarely samples. We therefore compute the exact sum over all tokens in $S(h)$ at each visited prefix. With support mass $\pi_\theta(S(h)\mid h)$ and renormalized student $\pi_{\theta,S}(\cdot\mid h)$ defined in Section 3, we have

The first term shifts student probability mass onto $S(h)$, while the second matches its relative distribution within it. Renormalizing the student removes the first term and its mass-transfer effect. Unlike Eq. (11), reverse KL has no such mass-transfer term and only adjusts relative probabilities within $S(h)$. We examine the importance of this forward-KL choice in the ablation experiments in § 5.

Experiments

Experimental setup

Models. We evaluate three teacher–student pairs covering three complementary settings. All models are from the Qwen3 family Yang et al., [35]. (1) Setting A distills Qwen3-8B into Qwen3-4B-Base, with $t_{\mathrm{ref}}$ = Qwen3-8B-Base. Under this setting, standard OPD performs well and a genuine teacher reference is available. (2) Setting B distills Qwen3-4B (non-thinking) into Qwen3-1.7B-Base, with $t_{\mathrm{ref}}$ = Qwen3-4B-Base. Li et al. [15] report that OPD leads to poor performance under this setting and recommend a supervised finetuning as a cold start, while a genuine teacher reference is available. (3) Setting C distills Qwen3-30B-A3B-Instruct-2507 into Qwen3-8B-Base. Standard OPD performs well in this setting, but no teacher reference checkpoint is available, so we set $t_{\mathrm{ref}} = \pi_0$. There is then only one extrapolative direction, leaving $\epsilon$ and $\kappa$ as the active controls.

Training data. Settings A and C train on the full DAPO-Math-17K dataset Yu et al., 2025. Setting B trains on approximately 30K prompts from the math-domain subset of OpenThoughts3-1.2M Guha et al., [8]. Every run trains for one epoch.

Evaluation. We evaluate on AIME24, AIME25, and AMC23 Mathematical Association of America, n.d.a, n.d.b, reporting avg@16 with a shared maximum evaluation response length of 31,744 tokens. We report results from the final checkpoint of one training run per configuration.

Main results

Method	AIME 24	AIME 25	AMC 23	Avg	$\Delta$
Student $\pi_0$	0.83	1.04	12.66	4.84	−9.38
Teacher $q^+$	24.17	15.62	64.22	34.67	+20.45
OPD	6.25	5.62	30.78	14.22	—
SFT	7.71	4.58	40.00	17.43	+3.21
SFT + OPD	14.58	10.00	37.81	20.80	+6.58
EOPD	4.79	9.79	38.44	17.67	+3.45
Direct-OPD	8.12	6.67	40.00	18.26	+4.04
OPD w. forward KL	8.54	6.46	41.56	18.85	+4.63
TOP-D	8.54	7.08	41.25	18.96	+4.74
ExOPD	13.33	6.88	39.84	20.02	+5.80
D²-OPD	9.79	7.29	49.84	22.31	+8.09

Baselines. We compare against OPD (Lu and Thinking Machines Lab, [17]), EOPD (Jin et al., [11]), TOP-D (Xie et al., [32]), ExOPD (Yang et al., [36]), and Direct-OPD (Feng et al., [5]). We use the on-policy variant of TOP-D, the reward-correction variant of ExOPD, and the published adaptive $\alpha$ controller of Direct-OPD. For method-specific hyperparameters, we use the authors’ recommended values. Appendix § 10 gives implementation details and parameter-search results.

Hyperparameters. All methods share the default training setup of Li et al. [15], except for a maximum training response length of 4,096 tokens. Appendix § 9 provides training and evaluation details.

Method parameters. Unless otherwise stated, we use $\kappa= 0.05$, with $\epsilon= 1$ in Settings A and B and $\epsilon= 4$ in Setting C. Appendix § 11 describes parameter selection and sensitivity.

Main results

Tables 1–3 report the main results across the three settings. In these three tables, Avg is the mean score across the three benchmarks, and $\Delta$ denotes its difference from OPD. Bold and underline indicate the best and second-best results, respectively, among trained methods.

Setting A. OPD improves the initial student’s average score from 19.58 to 27.57. Our method reaches 47.19, improving OPD by 19.62 points and the strongest baseline, TOP-D, by 9.45 points. It leads on all three benchmarks.

Setting B. We also compare the SFT checkpoint and SFT + OPD recipe released by Li et al. [15].

Method	AIME 24	AIME 25	AMC 23	Avg	$\Delta$
Student $\pi_0$	19.76	11.67	50.62	27.35	$-11.56$
Teacher $q^{+}$	73.33	62.29	99.69	78.44	$+39.53$
ExOPD	28.33	25.21	47.81	33.78	$-5.13$
EOPD	33.54	19.79	60.94	38.09	$-0.82$
OPD	30.00	25.00	61.72	38.91	—
TOP-D	29.17	30.42	70.78	43.45	$+4.54$
Direct-OPD	35.42	25.62	72.50	44.51	$+5.60$
D²-OPD	35.42	27.92	75.47	46.27	$+7.36$

OPD w. forward KL uses § 4.3 without extrapolation or direction selection. Forward KL alone raises OPD’s average score from 14.22 to 18.85. The full method reaches 22.31, exceeding ExOPD by 2.29 points and SFT + OPD by 1.51 points, without a supervised cold start. SFT + OPD remains stronger on AIME24 and AIME25; our method leads on AMC23.

Setting C. We set $t_{\mathrm{ref}} = \pi_0$ in ExOPD, Direct-OPD, and our method. Their reference directions coincide, so this setting isolates control along a fixed direction. The constraint binds more often here; we use $\epsilon= 4$ (Appendix § 11). Our method reaches 46.27, improving OPD by 7.36 points and the strongest baseline, Direct-OPD, by 1.76 points. TOP-D, which constructs an intermediate teacher from the current student, also remains competitive.

Our method has the highest average score in all three settings, while the strongest competing baseline changes across settings.

Analysis

How restrictive is $\epsilon$? To start with, we give a simple interpretation of the scale of $\epsilon$. For a one-hot target $\delta_a(\cdot\mid h)$ that assigns all probability to a token $a \in S(h)$, $D_{\mathrm{KL}}(\delta_a(\cdot\mid h)\|q_S^{+}(\cdot\mid h)) = -\log q_S^{+}(a \mid h)$, which means that $\epsilon= 1$ permits the target to concentrate entirely on a token with $q_S^{+}(a \mid h) \ge e^{-1} \approx0.37$. Although this makes $\epsilon= 1$ fairly permissive, Figure 5 shows that it still binds a meaningful fraction of positions in practice. The constraint binds on more than 60% of positions early in training and has a median active fraction of 34% over the second half of training. As expected, increasing $\kappa$ makes the constraint less active, since the soft objective itself proposes less aggressive extrapolation. Even at $\kappa= 0.3$, the hard constraint still binds on about 11% of positions.

Are the gains explained by longer responses? ExOPD observes that extrapolation can increase response length (Yang et al., [36]). We observe the same behavior for ExOPD, but not for extrapolative methods in general. In Figure 3, D²-OPD and Direct-OPD remain among the shorter methods across all three settings. TOP-D has a similar length and also remains well below the generation limit. In contrast, ExOPD and EOPD generally produce much longer responses. It is the most obvious in Settings B and C, where several methods nearly exhaust the 31,744-token budget while D²-OPD does not.² D²-OPD achieves the highest average score in all three settings while generating shorter responses than several baselines.

Are the gains explained by lower entropy? A common intuition is that a lower-entropy policy is more concentrated and may therefore achieve higher single-response accuracy (pass@1). Figure 4 shows that this does not explain our results. Across all three settings, D²-OPD remains among the higher-entropy students while achieving the best average score. TOP-D and Direct-OPD, which are also among the stronger baselines, generally lie in the higher-entropy group as well. Appendix § 13.§ 2 examines the student–teacher entropy gap.

Computational cost. Table 4 reports time and throughput per training step excluding rollout generation. D$^{2}$-OPD has roughly comparable training cost to OPD, taking 15% more time per step. Appendix § 14 describes this experiment’s setup.

Method	Time/step (s)	Throughput (tokens/s)	Relative time
OPD	33.42	1,840	1.00×
Direct-OPD	43.26	1,422	1.29×
TOP-D	32.96	1,866	0.99×
D$^{2}$-OPD	38.48	1,598	1.15×

Ablations

We conduct ablations in Setting A while keeping all other training and evaluation settings fixed. Each

the block in Table 5 removes or replaces one component of the full method: distance control, direction construction, the distillation loss, or extrapolation. Disabling extrapolation causes the largest drop, reducing the average score from 47.19 to 34.22. With extrapolation retained, removing the hard distance constraint lowers the score to 43.30, while removing only the soft penalty gives 45.59. Thus, the hard bound contributes beyond the soft penalty.

Component	Variant	Avg	$\Delta$
full		47.19	—
constraint	$-\kappa$	45.59	−1.60
	$-\epsilon$	43.30	−3.89
	$-$both	42.73	−4.46
direction	$\pi_0$ alone	45.12	−2.07
	$t_{\mathrm{ref}}$ alone	44.27	−2.92
KL direction	reverse KL w. mass	45.36	−1.83
	reverse KL	44.36	−2.83
extrapolation	tilt off	34.22	−12.97

Using only the student reference or teacher reference gives 45.12 and 44.27, respectively, whereas combining the signals performs better. Replacing forward KL with reverse KL lowers the score to 44.36. Retaining the support-mass term improves this to 45.36, still below the full method. This suggests that forward KL contributes beyond moving probability onto the candidate support.

Conclusion

We presented D$^{2}$-OPD, decoupling extrapolation direction from distance in extrapolative on-policy distillation. The method combines two directional signals while enforcing a per-prefix KL bound on the target’s distance from the teacher. On three teacher–student settings, $D^2$-OPD outperforms OPD baselines on mathematical reasoning benchmarks. Ablations show that combined direction signals, the distance constraint, and forward-KL distillation each contribute to these gains.

Limitations

Our experiments cover three teacher–student pairs from the Qwen3 family and three mathematical-reasoning benchmarks. They do not establish whether the same gains hold for other model families or tasks. The direction construction uses a teacher reference checkpoint when one is available. Without that checkpoint, the two directions coincide, so the method cannot use the teacher’s post-training shift as a separate signal. The distance bound and soft extrapolation coefficient are hyperparameters; the method does not determine an optimal extrapolation distance. The parameter study also shows that the choice of distance bound matters across settings.

Ethical Considerations

Distillation can transfer undesirable behavior from the teacher, including biased or harmful outputs. Our mathematical-reasoning evaluations do not assess these behaviors or privacy risks. Training and evaluation also incur energy costs.

Training and evaluation details

Our experiments focus on mathematical reasoning, with evaluation on English-language competition problems. We do not assess performance across languages, specific linguistic phenomena, or demographic groups.

Training. All methods share the same default training setup of Li et al. [15], except that we set the maximum response length to 4,096 rather than 7,168 tokens. This lies between the 3K and 7K moderate lengths that they report as yielding the strongest results (Li et al. [15]). Table 6 lists the default training hyperparameters.

Item	Value	Item	Value
Training temperature	1.0	Learning rate	1e-6
Global batch size	64	Epoch	1
Mini batch size	64	Actor KL-loss coefficient	0.0
Rollout number	4	Optimizer	AdamW
LogProb top-$K$	16	LR schedule	constant
Top-$p$	1.0	Gradient clipping	1.0
Max prompt length	1024	Loss aggregation	token-mean
Max response length	4096	Model dtype	fp32

Evaluation. We report avg@16 using temperature 0.7, top-$p$ sampling with $p = 0.95$, and a maximum response length of 31,744 tokens. Responses are graded using the rule-based verifier of Sheng et al. [27]. Each setting is evaluated with the same template used during training.

Hardware and compute. All methods were trained on AWS nodes with eight NVIDIA A100 40GB GPUs per node. Settings A and B used one node, and Setting C used two nodes. Training for the reported experiments required approximately 9,000 GPU-hours across 50 runs: 33 in Setting A, eight in B, and nine in C. These runs cover the main comparisons, ablations, parameter sweeps, and constraint-activity analysis. The calculation uses approximate average training times of 20, 16, and 18 hours per run in Settings A, B, and C, respectively.

Baseline configurations

OPD follows the implementation released by Li et al. [15]. ExOPD’s reward-correction variant corresponds to the $\lambda> 1$ case of G-OPD with $b = t_{\mathrm{ref}}$. For baselines with method-specific hyperparameters, we use the authors’ recommended values and also search their recommended ranges in Setting A.

Table 7 reports the baseline configurations evaluated in Setting A. For each method, ^† marks the authors’ default configuration, which is used in Table 1; the remaining rows vary its method-specific parameters.

Method	AIME 24	AIME 25	AMC 23	Avg	$\Delta$
Student $\pi_0$	12.50	6.25	40.00	19.58	−7.99
Teacher $q^+$	75.00	64.17	94.22	77.80	+50.23
OPD	18.96	14.38	49.38	27.57	—
EOPD	20.00	21.67	54.22	31.96	+4.39
ExOPD	17.08	24.79	54.37	32.08	+4.51
Direct-OPD	23.75	20.62	57.97	34.11	+6.54
TOP-D	25.62	23.54	64.06	37.74	+10.17
D²-OPD	35.62	35.00	70.94	47.19	+19.62

Method	Configuration	AIME24	AIME25	AMC23	Avg
EOPD ([11])	$\tau= 0.8$, $\alpha= 1.0$†	20.00	21.67	54.22	31.96
EOPD ([11])	$\tau= 0.4$, $\alpha= 1.0$	22.29	19.38	54.06	31.91
EOPD ([11])	$\tau= 0.8$, $\alpha= 2.0$	22.92	19.38	51.88	31.39
EOPD ([11])	$\tau= 0.4$, $\alpha= 2.0$	22.29	15.83	50.94	29.69
TOP-D ([32])	$\alpha= 0.2$†	25.62	23.54	64.06	37.74
TOP-D ([32])	$\alpha= 0.1$	22.71	20.62	64.53	35.95
TOP-D ([32])	$\alpha= 0.5$	23.54	16.25	62.66	34.15
Direct-OPD ([5])	$\alpha= 1.0$ adaptive†	23.75	20.62	57.97	34.11
Direct-OPD ([5])	$\alpha= 0.01$ fixed	23.96	20.21	52.50	32.22
Direct-OPD ([5])	$\alpha= 0.1$ fixed	22.29	19.38	51.56	31.08
ExOPD ([36])	$\lambda= 1.25$†	17.08	24.79	54.37	32.08
ExOPD ([36])	$\lambda= 1.0$	15.83	17.71	44.69	26.08
ExOPD ([36])	$\lambda= 2.0$	17.50	13.75	38.75	23.33

For EOPD, $\tau$ is the teacher-entropy threshold and $\alpha$ weights the additional forward-KL loss ([11]). For TOP-D, $\alpha$ is the teacher’s weight in the proximal teacher–student mixture ([32]). For Direct-OPD, $\alpha$ is the regularization coefficient introduced in § 3.2; for ExOPD, $\lambda$ is the reward-scaling coefficient. The published configuration achieves the highest average score among the tested configurations for all four baselines.

Parameter sensitivity

Setting A. We evaluate $\epsilon\in\{1,4,8\}$ and $\kappa\in\{0.02,0.05,0.1\}$. Table 8 reports the average scores, which range from 45.00 to 47.83. Underlining denotes the configuration used in the main experiments.

	$\epsilon= 1$	$\epsilon= 4$	$\epsilon= 8$
$\kappa= 0.02$	46.53	45.00	46.55
$\kappa= 0.05$	47.19	47.83	45.45
$\kappa= 0.10$	45.38	45.99	45.66

Setting C. We select $\epsilon$ based on training-time constraint activity. With $\epsilon= 1$, the constraint binds at about 89% of positions early in training. Increasing $\epsilon$ to 4 reduces this to about 60%, similar to Setting A. We therefore use $\epsilon= 4$ in the main experiments and report larger radii in Table 9. The average score increases from 43.21 at $\epsilon= 1$ to 48.45 at $\epsilon= 16$ over the tested values.

	$\epsilon= 1$	$\epsilon= 4$	$\epsilon= 8$	$\epsilon= 16$
Avg	43.21	46.27	47.05	48.45

Per-benchmark ablations

Table 10 gives the AIME24, AIME25, and AMC23 scores for the ablations summarized in Table 5.

Component	Variant	AIME24	AIME25	AMC23	Avg	$\Delta$
full		35.62	35.00	70.94	47.19	—
constraint	$-\kappa$	34.38	30.21	72.19	45.59	−1.60
	$-\epsilon$	30.21	26.88	72.81	43.30	−3.89
	$-$both	30.00	26.46	71.72	42.73	−4.46
direction	$\pi_{0}$ alone	31.67	30.42	73.28	45.12	−2.07
	$t_{\mathrm{ref}}$ alone	29.38	30.00	73.44	44.27	−2.92
KL direction	reverse KL w. mass	32.08	33.54	70.47	45.36	−1.83
	reverse KL	32.71	28.96	71.41	44.36	−2.83
extrapolation	tilt off	25.00	18.75	58.91	34.22	−12.97

Constraint variants. The $-\kappa$ variant sets $\kappa= 0$ and retains $\epsilon= 1$. The $-\epsilon$ variant sets $\epsilon= 10^{6}$ to make the constraint slack and retains $\kappa= 0.05$. The $-$both variant uses $\kappa= 0$ and $\epsilon= 10^{6}$, with the extrapolation coefficient limited only by the numerical cap of $10^{4}$.

Direction variants. The $\pi_{0}$-alone and $t_{\mathrm{ref}}$-alone variants replace the combined score by $\Delta^{c}_{\pi_{0}}(h,a)$ and $\Delta^{c}_{t_{\mathrm{ref}}}(h,a)$, respectively, while retaining $\epsilon= 1$ and $\kappa= 0.05$. If the selected signal’s reference distance is at most $10^{-3}$, its score is set to zero.

KL and target variants. Reverse KL uses $D_{\mathrm{KL}}(\pi_{\theta,S}(\cdot\mid h)\Vert\widetilde{q}(\cdot\mid h))$, with the student renormalized on $S(h)$. The variant with mass adds $-\log\pi_{\theta}(S(h)\mid h)$, retaining the support-mass term in Eq. (11). Both variants retain the full method’s target construction. The tilt-off variant uses the renormalized teacher distribution $q^{+}_{S}(\cdot\mid h)$ as the target and retains forward-KL distillation.

Additional analysis

Response length

Figure 6 gives the full response-length comparison, including the initial students and teachers.

Entropy gap

Figure 7 shows the student–teacher entropy gap across the three settings. $D^{2}$-OPD has the largest mean student–teacher entropy gap $H(\pi_{\theta}(\cdot\mid h)) - H(q^{+}(\cdot\mid h))$ in all three settings. Across the tested methods, a smaller mean entropy gap does not consistently correspond to a higher average score.

Computational-cost experiment setup

We use the model checkpoints from Setting A on two NVIDIA A100 40GB GPUs on one node. All methods process the same saved batch of 16 prompts with four responses per prompt. The maximum prompt and response lengths are 1,024 and 4,096 tokens, respectively.

Timing a full training step would include differences in the responses generated by each method. In particular, response length affects both generation and training time. Reusing the same saved responses fixes the token sequences and their lengths, allowing us to compare the cost of computing supervision and updating the student.

We discard the first two training steps as warmup and report the mean time over the following three steps. The timed computation includes model scoring, target or reward construction, and student optimization. Throughput counts valid prompt and response tokens across both GPUs.

Extrapolative distillation objectives

This appendix gives the baseline objectives summarized in § 3.

ExOPD. For an arbitrary reference $b$, Yang et al. [36] rewrite reverse-KL OPD as

Scaling the reward by $\lambda> 0$ gives Eq. (2). The target interpolates between $b(\cdot\mid x)$ and $q^{+}(\cdot\mid x)$ for $\lambda< 1$, recovers $q^{+}(\cdot\mid x)$ at $\lambda= 1$, and continues past it along $\log(q^{+}(y\mid x)/b(y\mid x))$ for $\lambda> 1$.

Direct-OPD. Direct-OPD (Feng et al. [5]) transfers the teacher’s pre- to post-training log-ratio and regularizes the student toward its own initialization:

Its optimum is Eq. (4). Direct-OPD adapts a shared regularization coefficient using the batch’s mean teacher-shift reward. For the corresponding fixed-prefix objective, denote its target by $q_{\mathrm{Direct}}(\cdot\mid h)$.

It can be written relative to the teacher as

Changing $\alpha$ generally changes both direction and distance relative to $q^{+}(\cdot\mid h)$. ExOPD with a fixed reference keeps the direction fixed, but the realized distance still depends on the scale of its log-ratio at each prefix. Our method constructs the direction first, then selects a separate coefficient at each prefix to enforce $D_{\mathrm{KL}}(\widetilde{q}(\cdot\mid h)\mathbin\Vert q^{+}(\cdot\mid h))\leq\epsilon$.

Direction and distance in target construction

We provide proofs for Propositions 1 and 2, together with examples and further discussion of direction and distance in target construction. We fix a prefix $h$ and use the finite candidate support $S(h)$. The renormalized teacher $q^{+}_{S}(\cdot\mid h)$ is positive on this support, and candidate targets are zero outside it.

Distance

Proof of Proposition 1. Let $B$ and $\rho(h)\equiv q^{+}_{S}(B\mid h)$ be as in the proposition. Let $w(h)=q(B\mid h)$ and let $H_{\mathrm{bin}}(w)=-w\log w-(1-w)\log(1-w)$ denote binary entropy. Grouping tokens into $B$ and its complement gives

Combining this inequality with the KL constraint proves the bound. □

Fixed-coefficient counterexample. For every fixed extrapolation coefficient $\beta>0$ and target mass $m\in(0,1)$, there are centered score errors $e_{\rho}(h,a)$ with $\mathbb{E}_{a\sim q^{+}_{S}(\cdot\mid h)}[e_{\rho}(h,a)^{2}]\to0$ such that the normalized target $q_{\beta}(a\mid h)\propto q^{+}_{S}(a\mid h)\exp[\beta e_{\rho}(h,a)]$ assigns probability $m$ to $B$, even as $\rho(h)\to0$.

For the fixed-coefficient construction, let $\mathbf{1}_{B}(a)$ be the indicator of membership in $B$, and set $c_{\rho}(h)=\beta^{-1}\log(m(1-\rho(h))/((1-m)\rho(h))).$ Starting from a zero score, introduce the error $e_{\rho}(h,a)=c_{\rho}(h)(\mathbf{1}_{B}(a)-\rho(h))$. This error is centered, and

The score difference between $B$ and its complement is $c_{\rho}(h)$. Therefore,

This establishes the contrast. $\square$

Adaptation to score scale. For an extrapolation coefficient $\beta\ge0$, consider the normalized target $\widetilde{q}_{\beta}(a\mid h)\propto q_{S}^{+}(a\mid h)\exp[\beta d_{\eta}(h,a)]$. For a scale factor $c>0$, if the constraint binds, replacing $d_{\eta}(h,a)$ by $cd_{\eta}(h,a)$ leaves the boundary target unchanged whenever the coefficient cap permits reaching it. The selected coefficient becomes $\beta/c$, and

Thus, at active prefixes, a positive rescaling of the direction changes the selected coefficient but not the target.

Connection to response length. ExOPD reports increasing response length under stronger extrapolation and identifies amplification of biased log-ratios as a possible source of instability [36]. The fixed-coefficient counterexample above describes one mechanism that could contribute to this behavior. If teacher-rare tokens that prolong or repeat reasoning receive spuriously large scores, fixed-strength extrapolation can assign them substantial probability. Our KL constraint limits this amplification at each prefix.

Direction

Proof of Proposition 2. Write the objective in Eq. (9) as $F_h(d(h,\cdot))$. Completing the square gives

The solution in Eq. (8) is centered, so it is feasible. The coefficient is positive, establishing uniqueness. $\square$

This solution gives more weight to the nearer reference. Its direction is proportional to $\sigma_{\pi}(h)u(h,a)+\sigma_T(h)v(h,a)$, where $u(h,a)=\Delta_T^c(h,a)/\sigma_T(h)$ and $v(h,a)=\Delta^c_{\pi0}(h,a)/\sigma_{\pi}(h)$.

Conflicting signals. If the normalized signals point in opposite directions, $u(h,a)=-v(h,a)$ for all $a\in S(h)$, their equal-weight sum vanishes. Our combination instead satisfies

It remains nonzero when the reference distances differ and follows the signal from the nearer reference. When the reference distances are equal, the signals cancel and the target remains $q_{S}^{+}(\cdot\mid h)$.

For example, consider a two-token support with $\pi_0(\cdot\mid h)=(0.1,0.9)$, $t_{\mathrm{ref}}(\cdot\mid h)=(0.5,0.5)$, and $q^{+}(\cdot\mid h)=q_{S}^{+}(\cdot\mid h)=(0.3,0.7)$. The teacher’s post-training shift favors the second token, while the teacher–student gap favors the first. Their normalized directions are opposite. Our combination gives $d_{\eta}(h,\cdot)\approx(-0.16,0.07)$, so for an extrapolation coefficient $\beta\ge0$, the target $\widetilde{q}_{\beta}(\cdot\mid h)\propto q_{S}^{+}(\cdot\mid h)\exp[\beta d_{\eta}(h,\cdot)]$ moves probability toward the second token.

Suppose the desired distribution is $\pi^{\star}(\cdot\mid h)=(0.2,0.8)$, and assess a policy $\pi(\cdot\mid h)$ by $-D_{\mathrm{KL}}(\pi^{\star}(\cdot\mid h)\Vert\pi(\cdot\mid h))$. The teacher is better than the initial student: their divergences from $\pi^{\star}(\cdot\mid h)$ are approximately $0.0257$ and $0.0444$, respectively. Along the constructed direction, $\beta\approx2.38$ gives $\widetilde{q}_{\beta}(\cdot\mid h)\approx\pi^{\star}(\cdot\mid h)$.