Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation

Introduction

On-policy distillation (OPD) has become an effective approach for training smaller language models under the guidance of larger teachers (Agarwal et al., 2024; Gu et al., [6]; Qwen Team, [11]; Lu and Thinking Machines Lab, [9]), and is increasingly adopted for multi-turn agentic tasks (Wang et al., 2026b; Li et al., [8]; Zhou et al., 2026a; Zhong et al., 2026). By matching the teacher’s action distribution at each decision state along the student’s own trajectories, OPD reduces the train-test mismatch associated with fixed demonstrations (Ross et al., [13]). Since the teacher serves as the target of distillation (Hinton et al., [7]; Gou et al., [5]), recent approaches have sought to surpass its performance through stronger supervision or additional reinforcement learning (Wang et al., 2026a; Tan et al., 2026; Ye et al., 2026; Yu et al., 2026b).

In this work, we uncover a distillation overshoot phenomenon in agentic OPD: a smaller, initially much weaker student can substantially outperform its larger teacher during training, only to lose much of this advantage as distillation continues. For instance, on ALFWorld, a Gemma E4B student improves from an initial 16.5% success, compared with 68.9% for its 31B teacher, to a peak of 82.8%, before declining to 73.7% by the end of training (Figure 1(a)). Similar overshoot-then-regression patterns emerge across multiple model–benchmark settings and OPD variants (Figure 3). These findings challenge the view of teacher performance as a natural ceiling for the distillation process.

To understand this overshoot phenomenon, we examine how OPD affects individual student decisions and their impact on task success. Our analysis reveals two competing effects: repair of weak student decisions and erosion of successful student choices that the teacher disfavors (Figure 2). Crucially, these effects unfold at different rates. Early in training, repair outpaces erosion, allowing the student to combine teacher corrections with its own strengths and surpass the teacher. As distillation continues, however, erosion overtakes repair, driving regression. We substantiate this mechanism at both the decision and task levels. Successful actions strongly disfavored by the teacher lose about two-thirds of their probability within five updates and remain suppressed, while teacher-endorsed actions continue to strengthen (Figure 1(b), bottom right). At the task level, gains dominate losses before the peak, but this balance reverses afterward (Figure 1(b), bottom left).

This diagnosis suggests that preserving beyond-teacher gains requires protecting successful student choices while retaining useful teacher guidance. Existing approaches fall short: selecting an intermediate checkpoint halts further learning and cannot preserve gains that peak at different times across tasks, while retiring the teacher (Yu et al., 2026b) also removes potentially beneficial corrections. Recent agentic OPD methods adjust rollout horizons, teacher interventions, supervision selection, or loss weights (Wang et al., 2026b; Li et al., [8]; Zhou et al., 2026a; Zhong et al., 2026), but do not explicitly protect successful student choices from conflicting teacher supervision.

Motivated by this diagnosis, we propose Success-Preserving On-Policy Distillation (SPOD), a simple modification of OPD that preserves successful student choices while continuing to learn from the teacher elsewhere. Specifically, when an action belongs to a successful student trajectory and the teacher assigns it lower probability than the student, SPOD masks the corresponding distillation loss and applies a soft probability floor. These two components address complementary sources of erosion: masking removes the direct gradient that pulls the student toward the teacher, while the probability floor counteracts indirect suppression caused by parameter updates at other decisions. All remaining actions receive ordinary teacher supervision (Figure 1(c)). SPOD uses only binary episode success and model probabilities already available in OPD, requiring no additional reward model, replay buffer, or task-family-specific supervision.

Experiments over two model families (Qwen Team, [12] and Gemma Team, Google DeepMind, [4]) and three multi-turn agentic benchmarks (ALFWorld (Shridhar et al., 2021), WebShop (Yao et al., 2022), and ScienceWorld (Wang et al., 2022)) demonstrate that SPOD outperforms ordinary OPD, the teacher, and evaluated OPD variants, improving average final success over ordinary OPD by up to 12.4 points. Gains reach 15.2 points over ordinary OPD and 20.0 points over the teacher. Notably, SPOD’s final checkpoints surpass even ordinary OPD’s best intermediate checkpoints in every setting, demonstrating that it not only preserves but also extends the overshoot gains. Ablation studies further confirm the complementary roles of masking conflicting supervision and preserving successful action probabilities. To summarize, our contributions are:

• We reveal distillation overshoot phenomenon in agentic OPD: smaller, initially weaker students can substantially surpass their larger teachers mid-training before regressing.

• We trace this phenomenon to the competing effects of repairing weak student decisions and eroding successful choices that the teacher disfavors, explaining the beyond-teacher performance and its subsequent decline.

• We propose SPOD, a distillation objective that selectively protects successful student choices from conflicting teacher supervision while retaining useful guidance, surpassing the best intermediate checkpoints of ordinary OPD.

Related Work

On-policy distillation. OPD trains students on their own visited states, reducing the train–deployment mismatch of fixed demonstrations [13]. GKD applies this principle to language models [1], MiniLLM uses reverse KL [6], and OPD is widely used in model post-training [11, 9]. For multi-turn agents, TCOD schedules rollout horizons (Wang et al., 2026b), Guided-OPD introduces teacher turns [8], SAGE-OPD selects interventions using teacher judgments (Zhou et al., 2026a), and SOD and ATOD reweight supervision using student–teacher disagreement (Zhong et al., 2026; Tan et al., 2026). Less is More truncates rollouts to avoid degraded teacher guidance at later token positions (Zhou et al., 2026b). These methods adjust the collection or application of teacher guidance while SPOD selectively protects successful student choices that this guidance would erode.

Students surpassing teachers. Students can exceed their teachers through same-size distillation [3, 1], weak-to-strong learning [2] (Yu et al. 2026a), and multi-teacher supervision (Wang et al., 2026a). OPD studies likewise report students outperforming their teachers after training [10] (Wang et al. 2026b; Tan et al. 2026; Yu et al. 2026b). However, we show that final performance above the teacher can conceal a much stronger intermediate policy, even for smaller, initially weaker students in multi-turn agentic OPD. We further explain this rise and regression through competing repair and erosion, motivating SPOD.

Understanding Distillation Overshoot

Having established the overshoot pattern Figure 1 & Figure 3, we investigate how it occurs and why continued distillation erodes this advantage.

Analysis Setup

We study ordinary forward-KL OPD on ALFWorld using Qwen3.5-2B/27B and Gemma-4-4B/31B student–teacher pairs. Given an interaction history $s$, the student $p_{\theta}$ generates an action $a = (a_1,\ldots,a_n)$ and learns from a frozen teacher $q$ through the token-averaged OPD loss

Both distributions condition on the same history and student-generated prefix $(s,a_{<j})$, using each model’s native prompt template. Sampled histories and tokens are held fixed during differentiation. Our analysis links decision-level changes to task-level outcomes. Every five updates, we rescore fixed state–action pairs to track decision probabilities and evaluate success on 274 seen/unseen tasks, averaging success over three evaluation seeds. § 5 evaluates SPOD objective across all benchmarks and model families without re-tuning.

Overshoot Emerges as Repair Initially Outpaces Erosion

Repair initially outpaces erosion within trajectories. To understand why the student surpasses its teacher, we first examine individual decisions. In a successfully learned Qwen episode (“heat some apple and put it in sidetable”), the probability of “go to microwave” rises from $9.3 \times10^{-5}$ to $0.87$ by update 15, completing the failure-to-success flip, even though the already-correct “heat apple” falls only from $0.82$ to $0.58$ (teacher: $0.076$); a second example appears in Figure 1b (top). The pattern is general: $82.7\%$ of strongly teacher-disfavored actions from successful episodes at update 11 halve by update 15, yet task success still rises from $52.1\%$ to $86.1\%$. Thus early in training, repair often outpaces erosion.

Decision-level imbalance yields beyond-teacher task-level performance. Consistent with this mechanism, task-level gains initially far exceed losses. Summing increases and decreases in per-task success between adjacent checkpoints, Qwen gains 188.7 task-equivalents from update 5 to its peak at update 15 but loses only 7.3 (Figure 1b (bottom left)); gains exceed losses before the peak in all five ordinary-OPD runs across both model families. Moreover, at the peak both students solve $57.4\%$ of the ALFWorld tasks the teacher usually fails (teachers: $16.0\%$ and $4.7\%$), while still solving $93.2\%$ and $94.5\%$ of those it usually solves (Figure 4). Consequently, the peak student surpasses the teacher by retaining its reliable behavior while adding successes where it is unreliable.

Regression Emerges as Erosion Overtakes Repair

Continued imitation suppresses successful student choices. Despite repair’s initial lead, erosion still occurs, and its source is the distillation objective itself. Specifically, at a token position, forward KL has the exact logit derivative

Whenever the student assigns a token higher probability than the teacher, this gradient pushes its logit down, regardless of whether the episode succeeded. Ordinary OPD thus treats a successful departure from the teacher the same as a mistake, and this pressure accumulates on the student’s useful choices as training continues.

Successful choices erode as teacher-supported actions strengthen. We examine this pressure empirically by rescoring fixed state–action pairs from successful trajectories near the peak. In Qwen, teacher-supported actions rise in mean probability from 0.80 to 0.89, whereas strongly teacher-disfavored successful actions fall by 67%, from 0.36 to 0.12, within five updates and remain low (Figure 1b (bottom right)). Continued imitation thus strengthens teacher-supported behavior while eroding the student’s useful departures from it.

Decision-level erosion translates into task-level regression. Does this erosion explain why tasks are lost? For the 25 Qwen tasks solved at the peak but lost by the end, we track the action on each successful peak trajectory whose probability falls most. We find that these actions are almost always teacher-disfavored (92%) and erode quickly: 22 of 25 halve within five updates, at or before the checkpoint where the task is first lost in 96% of tasks (76% at exactly that checkpoint; Figure 5). Gemma shows the same pattern across 37 lost tasks. Tasks are thus lost when continued imitation erodes a decisive choice the student had gotten right.

In tasks that remain solved, erosion is milder (median 7-fold versus 41-fold) and rarely decisive, since an alternative route usually survives. Repair also continues after the peak (Qwen gains 85.7 task-equivalents) but is outpaced by erosion (100.7 lost), thus producing the regression.

A Stylized Model of Repair and Erosion

We formalize how simultaneous repair and erosion yield an above-teacher intermediate policy even as the imitation loss decreases. Consider an episode requiring two correct binary decisions, with student success probabilities $p_i = \sigma(z_i)$, teacher probabilities $q_i$, and episode success $J = p_1p_2$ versus $J_T = q_1q_2$. Under forward-KL gradient flow,

where $\lambda_i$ is each decision’s effective update rate. Let the student start with one strength, $p_1(0) > q_1$, and one weakness, $p_2(0) < q_2$, with $J(0) < J_T$.

When the weak decision is repaired faster than the strength erodes, the student briefly holds both; continued imitation then removes the strength. With $q_i = 0.83$ (teacher success 68.9%, as for Gemma on ALFWorld), $p(0) = (0.99, 0.10)$ and $\lambda= (0.025, 1)$, success rises from 9.9% to 82.0% before settling at the teacher’s level. Here $\lambda$ is chosen for illustration rather than fitted. We defer proofs to Appendix § 11.

Teacher Reliability Predicts Where Gains Are Fragile

To locate post-peak losses, we group tasks by how often the teacher solves them. Losses concentrate where the teacher is unreliable: on tasks it never solves, success falls from the peak to the final checkpoint by 27.4 points for Qwen and 22.1 for Gemma, while tasks it always solves stay near their peak (Figure 4); this holds in all five ordinary-OPD runs. Task families also peak at different times: for Gemma, on ALFWorld, “Look” peaks at update 15 and falls from 74.2% to 28.0%, “Clean” peaks at update 25 and falls to 50.0%, while “Cool” and “Heat” peak at update 30 and hold. Thus no single stopping time, and no family-level rule, retains every gain, motivating decision-level protection.

Success-Preserving On-Policy Distillation (SPOD)

The preceding analysis motivates protecting successful student choices from conflicting supervision while retaining useful teacher guidance. We therefore propose SPOD, a lightweight intervention which keeps ordinary OPD’s student rollouts and teacher scoring and modifies only the loss at each sampled decision.

The SPOD Objective

Identifying choices to protect. Let $R(\xi) \in\{0,1\}$ indicate whether a student trajectory $\xi$ succeeds, and for an action $a = (a_1,\ldots,a_n)$ at history $s$ define the sequence log-probabilities

SPOD protects an action when its trajectory succeeds and the teacher assigns it lower probability than the student:

Success marks behavior worth keeping and relative teacher support marks conflict, so the gate neither protects disagreements in failed episodes nor withholds guidance from entire successful trajectories. It is computed before each update and held fixed during differentiation.

Masking conflicting supervision and countering erosion. For gated actions, SPOD removes the distillation term and applies a soft probability floor,

giving the objective

The floor raises a protected action only while its probability is below $\tau_0$ and is inactive above it. We set $\tau_0 = 0.5$, the smallest value at which the protected action is guaranteed to remain the student’s most likely action, and $\lambda= 1$, unchanged across all settings. Masking removes the teacher’s pull at the decision itself; the floor counters declines caused by updates elsewhere. All ungated decisions receive ordinary forward-KL supervision.

Masking Makes the Floor Effective

A natural alternative is to add the floor to ordinary OPD without masking. However, this would pit the teacher’s pull against the floor’s push, settling the protected action below the floor. SPOD avoids this by removing the teacher term on protected actions, leaving the floor unopposed. A one-decision model makes this precise. Let action 1 succeed and action 0 fail, with student and teacher probabilities $p = \sigma(z)$ and $q$ for action 1; actions are sampled from the student, and the sampling distribution and gate are held fixed, as in SPOD.

For example, with $q = 0.16$ and $\tau_0 = 0.5$, the floor alone settles at a compromise with the teacher at $p = 0.4$, while masking lets the protected action reach the floor. This matches the ablation: the floor alone or masking alone underperforms their combination (subsection 5.4). Proofs and an analysis of shared-parameter effects are in Appendix § 11.

Implementation and cost. SPOD adds negligible cost: it reuses OPD’s student and teacher scoring passes, adding only sequence log-probability sums and a scalar gate. Action probabilities include the end-of-action token and are computed before temperature or action constraints. Losses are averaged over episodes, states, then tokens, so long actions do not dominate. Beyond ordinary OPD, SPOD needs only binary episode success: no stored policy, per-family anchor, extra rollout, or teacher judgment are required.

Experiments

Setup

Models and benchmarks. We evaluate two model families, Qwen3.5-2B distilled from Qwen3.5-27B and Gemma-4-E4B distilled from Gemma-4-31B ([12, 4]), on three multi-turn agentic benchmarks: ALFWorld (Shridhar et al., 2021), WebShop (Yao et al., 2022), and ScienceWorld (Wang et al., 2022) (we follow common practice and train on a subset of task types (Xi et al., 2025; Yang et al., 2026) chosen by a zero-shot evaluation before any training, requiring the teacher to outperform a non-zero student to ensure distillation is meaningful).

Benchmark	Domain	Task families	Train	Test
ALFWorld	Household tasks	6	3,553	274
WebShop	Online shopping	—	6,410	500
ScienceWorld	Science experiments	6/30	2,880	180

Comparison methods. We compare against three recent agentic OPD methods: TCOD varies the rollout horizon (Wang et al., 2026b), Guided-OPD introduces teacher interventions ([8]), and SAGE-OPD selects turns for supervision (Zhou et al., 2026a). Our comparisons focus on preserving gains within teacher-supervised OPD, with success-conditioned self-imitation and extensive component ablations testing alternative uses of the same outcome information.

Controls and ablations. We additionally report (1) the student before OPD, (2) ordinary OPD’s best checkpoint, (3) stopping at the first evaluation dip, (4) stopping at the first training-success dip, and (5) retiring at teacher level. On Qwen ALF-World, we also compare (6) a lower learning rate, (7) weight averaging, (8) reverse KL, (9) teacher-confidence gating, and (10) self-imitation, and ablate SPOD into (11) masking only, (12) floor only, and (13) episode-level filtering.

Training and evaluation. We follow the standard OPD training protocol with a cosine learning-rate schedule. Teachers are fixed off-the-shelf models. For ordinary OPD and variants, we use the peak learning rate that works best in each setting from $\{1 \times10^{-5}, 1 \times10^{-6}, 2 \times10^{-6}\}$, further lowered when training diverged. For SPOD (Ours), we use a fixed learning rate of $10^{-5}$, $\tau_0 = 0.5$, and $\lambda= 1$ in all settings. Additional training and evalu-

caption: Final success rates (%). ALFWorld is reported per task family; WebShop and ScienceWorld overall; Overall averages the three benchmarks. Subscripts indicate SPOD’s gain over ordinary OPD. *Uses evaluation results to choose the checkpoint (oracle). SPOD outperforms all compared methods on every benchmark.

:::

ation details, including random seeds, are provided in subsection E.1.

Main Results

SPOD improves final performance in every setting. As shown in Table 1, SPOD achieves the best final success rate in every setting. Over ordinary OPD, it gains 10.3, 12.3, and 14.6 points for Qwen and 15.2, 7.6, and 7.8 points for Gemma on ALFWorld, WebShop, and ScienceWorld, raising the overall average by 12.4 and 10.2 points. SPOD also surpasses the teacher everywhere, by up to 20.0 points, with a single untuned configuration.

SPOD surpasses even the best ordinary-OPD checkpoint. No ordinary-OPD checkpoint, even one selected on evaluation results, matches SPOD’s final model (e.g., 86.1% vs. 91.0% for Qwen and 82.8% vs. 88.9% for Gemma on ALFWorld). Practical stopping rules recover at most the peak and often stop early. SPOD thus not only retains the overshoot but extends it.

Gains concentrate where ordinary OPD regresses. The largest gains occur in families that ordinary OPD loses after its peak, typically where

the teacher is weak: Gemma Clean rises from 50.0% to 92.5% and Look from 28.0% to 57.0%; Qwen Cool rises from 69.6% to 90.6% and Clean from 71.3% to 89.7%.

Guidance-based baselines still regress heavily. TCOD, Guided-OPD, and SAGE-OPD change when or where the teacher supervises, but not what it pulls the student away from. They therefore overshoot and regress as well: Guided-OPD falls from 67.2% to 54.8% with Gemma and TCOD from 45.6% to 37.8% with Qwen on ScienceWorld. SPOD can complement other OPD methods. Adding SPOD to Guided-OPD on Gemma ScienceWorld raises final success to 66.5%, improving on Guided-OPD alone by 11.7 points and SPOD alone by 4.8 points (Table 4), suggesting that SPOD could serve as a drop-in addition to existing agentic OPD methods rather than an alternative to them.

Method	Final (%)
Guided-OPD	54.8
SPOD	61.7
Guided-OPD + SPOD	66.5

SPOD Preserves the Overshoot

Decision level. Figure 6 tracks the actions SPOD protects: successful choices the teacher disfavors, which erode under ordinary OPD (subsection 3.3). The student acquires them during training, with median probability rising from 0.002 before training to 0.88 when protected. At the final checkpoints, ordinary OPD assigns these same actions a median of 0.05, whereas SPOD keeps 0.71, above $\tau_0$, the empirical counterpart of Figure 2.

Task level. Preserving these choices preserves the tasks they solve. On ALFWorld tasks the teacher usually fails, where ordinary OPD’s losses concentrate (subsection 3.5), ordinary OPD falls from 57.4% at its peak to 30.9% (Qwen) and 38.8% (Gemma), whereas SPOD ends at 71.0% and 75.2%, above ordinary OPD’s peak (Figure 7). On tasks the teacher usually solves, SPOD matches or exceeds ordinary OPD (95.9% vs. 92.9%; 95.2% vs. 89.7%), so protection does not stop the student from learning elsewhere.

Ablation Study and Alternatives

Masking enables effective preservation. Table 5 shows that removing conflicting supervision provides most of SPOD’s improvement: masking alone gains 7.1 points on Qwen and 13.3 on Gemma. Step-level masking slightly outperforms episode-level masking while retaining teacher guidance on nonconflicting decisions. The floor alone provides inconsistent gains, yet adds a further 3.2 and 1.9 points when paired with masking. This matches subsection 4.2: with teacher supervision retained, the floor competes with a conflicting target; masking removes that opposition, allowing the floor to counter subsequent erosion.

Variant	Mask	Floor	Qwen	Gemma
Ordinary OPD	–	–	80.7	73.7
Floor only	–	✓	84.5	72.6
Episode-level mask	episode	–	87.5	86.3
Step-level mask	step	–	87.8	87.0
SPOD (ours)	step	✓	91.0	88.9

Simpler alternatives do not retain the gains. Lower learning rates, weight averaging, reverse

KL, and teacher-based gating or stopping all underperform SPOD (Table 3). Self-imitation adds a next-token loss on successful student trajectories while retaining teacher supervision, reaching 84.4%, but still considerably below ordinary OPD’s peak (86.1%) and SPOD (91.0%). Combined with the masking ablation, this also shows that simply reinforcing successful behavior is less effective than removing conflicting supervision and preserving those choices. SPOD also outperforms tuned GRPO trained on the same success signal by 5.5 points on average across the three Qwen benchmarks (Table 8 & subsection E.5).

Method	Final (%)
Ordinary OPD	80.7
best ckpt.*	86.1
Lower learning rate ($\times0.3$)	82.8
Weight averaging (EMA)	80.9
Reverse KL	82.5
Reverse KL, lower learning rate	77.0
Teacher-confidence gating	74.1
Retire at teacher level	79.8
OPD + NTP on successes	84.4
SPOD	91.0

Benchmark	GRPO	SPOD (ours)
ALFWorld	85.2	91.0 (+5.8)
WebShop	42.8	50.3 (+7.5)
ScienceWorld	52.8	55.9 (+3.1)
Average	60.3	65.7 (+5.5)

Conclusion

We characterize distillation overshoot in multi-turn agentic OPD: smaller, initially weaker students can substantially surpass their larger teachers mid-training before regressing. The teacher initially repairs weak decisions faster than it erodes successful student choices; regression follows as erosion takes over. Motivated by this diagnosis, SPOD withholds conflicting teacher supervision and adds a soft probability floor on successful actions the teacher disfavors. Across six model–benchmark settings, SPOD outperforms the teacher, ordinary OPD, and its best intermediate checkpoints, preserving and extending the overshoot gains. Results with Guided-OPD further demonstrate SPOD’s potential to complement existing OPD methods.

Limitations

Statistical support. SPOD, ordinary OPD, alternatives, ablations, and sensitivity experiments use three training seeds; TCOD, Guided-OPD, and SAGE-OPD use one each. Every checkpoint is evaluated with three evaluation seeds. Additional training seeds would strengthen comparisons with the single-seed baselines.

Experimental scope. We evaluate two teacher–student pairs and three text-based agentic benchmarks. Compute constraints limit our study to smaller students; larger-scale distillation remains a natural extension. Within each pair, student and teacher share a vocabulary, enabling full-vocabulary distillation. Extending SPOD across tokenizers and to real-world environments would broaden its applicability.

Replication and coverage. SPOD, ordinary OPD, alternatives, ablations, and sensitivity experiments use three training seeds, while TCOD, Guided-OPD, and SAGE-OPD use one each. All checkpoints are evaluated with three evaluation seeds. Additional training seeds would strengthen comparisons with these methods. Hyperparameter sensitivity and the alternatives comparison focus on Qwen ALFWorld, while composition with Guided-OPD is evaluated on Gemma ScienceWorld. The latter demonstrates a promising combination; broader evaluation would establish how widely this complementarity extends.

Mechanism analysis. Our detailed decision-level analysis focuses on ALFWorld. The action-tracing evidence is retrospective, while the two-decision model isolates repair and erosion using independent decisions and illustrative update rates. Extending these probes to other environments and testing interventions on individual decisions would further characterize the mechanism.

Outcome supervision. SPOD uses binary episode success to identify behavior worth preserving, without requiring action-level annotations. This signal does not distinguish each action’s contribution to success. The current gate assumes reliable episode outcomes; incorporating noisy or graded feedback could extend preservation to tasks with less structured evaluation.

Ethics Statement

Our experiments use existing models and benchmark environments, with no human participants or collection of personal data. We identify no ethical concerns specific to this study.

Reproducibility Statement

We will publicly release all training and evaluation code, experiment configurations, and trained student checkpoints to support reproduction of our results and further research on agentic distillation.

AI Use Statement

We used AI tools to assist with code development, figure generation, and manuscript polishing. The authors reviewed and verified the resulting code, figures, and text and take full responsibility for the paper’s content.

Proofs and Additional Theory

Proof of Proposition 3.1

Write $p_i=\sigma(z_i)$, let $a=p_1(0)$ and $b=p_2(0)$, and define $L_i=D_{\mathrm{KL}}(\operatorname{Ber}(q_i)\mathbin\Vert\operatorname{Ber}(p_i))$. Direct differentiation gives $\partial L_i/\partial z_i=p_i-q_i$. Consequently, under Eq. (3),

at every finite time for the specified nonstationary initialization. Each probability approaches its teacher target monotonically without crossing it. Put $u(t)=p_1(t)-q_1>0$ and $v(t)=q_2-p_2(t)>0$. Both tend to zero, and their local dynamics give

where $C_i>0$ and $\kappa_i=\lambda_iq_i(1-q_i)$. To obtain these asymptotics directly, divide each scalar equation by its distance from equilibrium. The resulting logarithmic derivative converges to $-\kappa_i$; its remaining term is integrable because the distance already decays exponentially in a neighborhood of equilibrium. Thus the rescaled distances converge to positive constants.

The performance difference is

If $\kappa_2>\kappa_1$, the positive first term dominates for sufficiently large $t$. Hence $J(t)>J_T$ eventually. Substitution of the scalar dynamics also gives

for all sufficiently large times. Because $J(0)<J_T$ and $J(t)\to J_T$, continuity ensures a finite crossing and a finite above-teacher maximum. This proves the proposition. The statement guarantees eventual regression; it does not require every point before the maximum to be increasing. If an initially increasing trajectory is desired, the additional condition is

which follows by evaluating $\dot{J}(0)/J(0)$.

An explicit finite-time lower bound. For $0<\epsilon<q_2-b$, define

Since $p_2$ increases on $[b,q_2]$, its deficit obeys $\dot{v}\le-\lambda_2m_2v$, so $p_2(t_\epsilon)\ge q_2-\epsilon$. Meanwhile

using $p_1(1-p_1)\le1/4$ and $p_1-q_1\le a-q_1$. Whenever the first factor is nonnegative,

With the parameters in subsection 3.4 and $\epsilon=0.05$, we have $m_2=0.09$, $t_\epsilon\approx29.79$, and a success lower bound of 0.7490, approximately 6.0 percentage points above the teacher. The integrated trajectory in Figure 8 peaks at 82.0%. Conversely, if both initial student probabilities are at most their teacher targets, the independent dynamics preserve that ordering and cannot overshoot. Local complementarity is essential in this model.

Successful Trajectories Enrich Useful Actions

Let $R$ denote binary eventual success under the rollout policy $\pi$. Define $Q^\pi(s,a)=\Pr(R=1\mid s,a)$, $V^\pi(s)=\Pr(R=1\mid s)$, and $A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s)$. For a visited state with $V^\pi(s)> 0, Bayes’ rule gives

Subtracting $\pi(a \mid s)$ yields $\pi(a \mid s)A^{\pi}(s,a)/V^{\pi}(s)$. Positive-advantage actions are therefore overrepresented among successes. This identity uses the actual rollout policy, including its sampling and action constraints. SPOD’s relative-support comparison instead uses raw model sequence probabilities. The roles are different: the outcome selects successful experiences, and the model scores determine where teacher supervision conflicts with student support.

Proof of Proposition 4.1

For a Bernoulli student, the forward-KL logit derivative is $p-q$. When $p>q$, the gate fires only if action 1 is sampled, an event of probability $p$. Below the floor, the derivative of its negative log probability is $-(1-p)$. Thus the four expected gradients are

Objective	Expected derivative w.r.t. $z$
OPD	$p-q$
Mask only	$(1-p)(p-q)$
Floor only	$p-q-p(1-p)\mathbf{1}_{p<\tau_0}$
SPOD	$(1-p)(p-q)-p(1-p)\mathbf{1}_{p<\tau_0}$

These expectations do not differentiate through the sampling distribution. They average the per-sample gradients produced by the algorithm. For the floor-only objective below $\tau_0$, the expression simplifies to $p^2-q$, negative below $\sqrt{q}$ and positive above it. Since $q<\tau_0^2$ and $q<\sqrt{q}$, the equilibrium is inside the active gated region and is stable under gradient descent.

For SPOD below $\tau_0$, the derivative is $-q(1-p)<0$; above $\tau_0$, it is $(1-p)(p-q)>0$. Thus the vector fields point toward the floor from both sides. The hinge is nondifferentiable exactly at the boundary, which is interpreted as an attracting switching threshold; a finite-step optimizer can fluctuate around it. This proves the proposition without claiming an exact stationary probability under arbitrary neural optimization.

Local Pressure and Shared-Parameter Coupling

For a frozen minibatch with nonnegative weights $w_j$, write $h_i=\nabla_{\theta}\log p_{\theta}(a_i\mid s_i)$ and $g_j=\nabla_{\theta}\ell_j$. A preconditioned step $\Delta\theta=-\eta M\sum_j w_jg_j$, for fixed positive-semidefinite $M$, changes the selected action to first order by

The first term is the action’s own loss contribution and the second is interference or transfer from other decisions. Masking removes the action’s own distillation contribution. For a gated action below the floor, $g_i=\lambda\nabla_{\theta}\ell_{\mathrm{floor},i}=-\lambda h_i/n_i$, so its own contribution becomes

Cross-state terms remain. This is the precise local preservation effect; no claim of a universal hard probability guarantee is needed. For an actual optimizer step, including Adam with momentum, the general first-order identity is $\Delta\log p_i=h_i^{\mathsf{T}}\Delta\theta+O(\lVert\Delta\theta\rVert^2)$.

A sequence comparison is not an exact gradient sign. Even for a single softmax token under unpreconditioned gradient flow,

For $p=(0.40,0.59,0.01)$ and $q=(0.39,0.31,0.30)$, the first token has $p_a>q_a$ yet its log probability increases, because the normalization term dominates. Multi-token actions introduce additional coupling. Equation (2) is exact at the logit level; Eq. (5) is a low-cost relative-support selector whose usefulness is tested by the fixed-state probes and ablations.

Additional Experiment Details

Additional Implementation Details

Task sets and metrics. ALFWorld is evaluated on 274 tasks (140 seen, 134 unseen) spanning six families: Pick (59), Look (31), Clean (58), Heat (39), Cool (46), and Pick2 (41). The reported average is the task-weighted success rate over all 274 tasks. WebShop is evaluated on 500 held-out shopping instructions, and the metric is full task success rather than the partial reward score. ScienceWorld is evaluated on six task families (inclined-plane angle determination, friction with named and with unnamed surfaces, melting-point measurement with known and with unknown substances, and finding a non-living thing), with 30 held-out instances each (180 in total).

Training protocol. Within our computational budget, we prioritize multi-seed replication for SPOD, ordinary OPD, and the controlled alternatives and ablations. Hence, we use three training seeds for SPOD and ordinary OPD in Table 1, all alternatives in Table 3, all ablation variants in Table 5, and all hyperparameter sensitivity experiments in Table 6, as well as the corresponding figures. TCOD, Guided-OPD, and SAGE-OPD are each trained with one seed. For teacher scoring, we format the student’s interaction history using the teacher’s native chat template. Within each model family, the student and teacher share the same vocabulary and token IDs, enabling full-vocabulary distillation.

	Final success (%)
$\tau_0$ ($\lambda=1$)	0.3	0.5	0.7
	89.4	91.0	93.8
$\lambda$ ($\tau_0=0.5$)	0.5	1	2
	84.4	91.0	89.4

Evaluation protocol. We evaluate every checkpoint with three evaluation seeds and report the mean performance across these evaluations.

Optimization. All runs use three warmup updates to the peak learning rate, followed by cosine decay to one tenth of the peak, with gradient norm clipped at 1.0. SPOD uses a single fixed peak learning rate of $10^{-5}$ across all settings, without learning-rate tuning. Learning-rate selection is applied only to ordinary OPD and the compared methods: we choose their best-performing peak learning rate from $\{10^{-5}, 2 \times10^{-6}, 10^{-6}\}$ (§ 5). Specifically, ordinary OPD on Qwen WebShop uses $2 \times10^{-6}$, while SAGE-OPD uses $10^{-6}$, matching its original paper.

SPOD training update

1. Roll out the current student on a batch of training tasks; record each episode’s binary success.

2. Sample up to eight decisions per episode and obtain teacher token distributions at those contexts.

3. Compute $\ell_p$, $\ell_q$, and the detached gate in Eq. (5).

4. Use the floor for gated decisions and forward KL for the others; average and update the student.

Sensitivity to $\tau_0$ and $\lambda$

SPOD has two hyperparameters: the floor level $\tau_0$ and the floor weight $\lambda$. All main results use fixed defaults of $\tau_0=0.5$ and $\lambda=1$, without task-specific tuning. We choose $\tau_0=0.5$ because an action at or above this probability is at least as likely as any alternative. Setting $\lambda=1$ gives the floor the same coefficient as the distillation term it replaces on protected decisions.

Table 6 varies one hyperparameter at a time on Qwen ALFWorld, holding everything else fixed. Every tested setting outperforms ordinary OPD (80.7%); all except $\lambda=0.5$ also exceed its best checkpoint (86.1%) and the mask-only variant (87.8%). The default $\lambda=1$ performs best among the tested weights, while increasing $\tau_0$ to 0.7 raises final success to 93.8%, 2.8 points above the main reported results. Thus, the main results use a shared default rather than the sweep’s best-performing configuration.

Definitions of controls, alternatives, and ablations

All variants use the ordinary-OPD rollout protocol, teacher-scoring procedure, and evaluation protocol described in § 3 and subsection E.1. Each changes only the component specified below.

Checkpoint selection and stopping (Table 1). Best checkpoint selects the checkpoint with the highest evaluation success; stop at eval dip selects the checkpoint immediately before the first decline. Both use evaluation results and are marked as oracles. Stop at train dip stops at the first decline in training-rollout success between consecutive five-update windows. Retire at teacher level stops once the student’s success over the last five training updates reaches the teacher’s training-rollout success rate, using a teacher-level criterion inspired by RetireOPD (Yu et al., 2026b). The latter two rules use only training-time information and report the checkpoint at the corresponding milestone.

Alternatives (Table 3). Lower learning rate multiplies the peak and final learning rates by 0.3, retaining the schedule shape. Weight averaging evaluates an exponential moving average of model weights with decay 0.9 per update. Reverse KL replaces token-level forward KL with $D_{\mathrm{KL}}(p_\theta\|q)$ over the full vocabulary; its lower-learning-rate variant also scales the rates by 0.3. Teacher-confidence gating removes distillation on actions whose mean teacher token entropy exceeds the previous update’s median, selecting supervision by teacher uncertainty. OPD + NTP on successes adds token-averaged negative log-likelihood of the student’s sampled actions from successful episodes to ordinary OPD. It retains teacher supervision everywhere and uses the same binary success information as SPOD, testing whether directly reinforcing successful behavior suffices to retain the gains.

Ablations (Table 5). Floor only retains forward KL everywhere and adds the soft floor in (6) on actions selected by the gate in (5), isolating preservation without removing conflicting supervision. Episode-level mask removes distillation from every action in a successful episode, with no floor. Step-level mask removes distillation only from successful actions the teacher disfavors, as selected by the same gate, with no floor. SPOD combines this selective mask with the floor. All floor terms use $\tau_0 = 0.5$ and $\lambda= 1$. These controls separate the effects of removing teacher supervision, selecting where to remove it, and countering subsequent erosion.

Distillation overshoot with reverse KL

To test whether overshoot is specific to forward KL, we evaluate reverse-KL OPD on Qwen AlFWorld (Table 7). Both tested configurations surpass the teacher (80.0%) during training and subsequently regress. The better-performing configuration peaks at 84.7% and finishes at 82.5%, still 8.5 points below SPOD. These results show that overshoot extends beyond forward KL and that changing the KL direction alone does not eliminate post-peak regression.

Objective	Peak (%)	Final (%)
Forward KL (ordinary OPD)	86.1	80.7
Reverse KL	84.7	82.5
Reverse KL, lower learning rate	83.5	77.0
SPOD (forward KL)	92.0	91.0

Comparison with outcome-based RL

We additionally compare with tuned GRPO on Qwen3.5-2B (Table 8). SPOD achieves higher final success on all three benchmarks, with an average gain of 5.5 points while using approximately one-quarter of GRPO’s training GPU-hours, extending its advantage beyond distillation baselines.