Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation

Introduction

On-policy distillation (OPD) transfers reasoning capabilities from a teacher model to a student through feedback on the student’s own rollouts. Training on student-generated prefixes brings the training distribution closer to the one visited at inference time and mitigates exposure bias ([2, 5, 1] (Ranzato et al. 2016; Song and Zheng 2026)). OPD also provides token-level teacher feedback throughout each response ([5, 1]). How to weight this supervision along a response is therefore an important question for OPD post-training.

In practical token-level OPD, supervision is determined by the sampled-token student–teacher log-ratio $r_t$ ([4]). A positive log-ratio penalizes a token that the student assigns higher probability to than the teacher. The current penalty value describes how strongly a sampled token is penalized, but does not describe how that penalty changes along the response. A high penalty can persist across neighboring segments or emerge as an abrupt increase. Comparing consecutive blocks of tokens provides a local reference within the same response, allowing us to examine these increases separately from sustained high penalties. We investigate whether local penalty changes carry information relevant to supervision allocation and whether using this information to reweight supervision improves training.

Our diagnostic analysis shows that local penalty changes reveal differences in teacher predictions that the current penalty value alone does not express. We summarize each block by its block-averaged penalty, the mean positive log-ratio within a block. We call an abrupt increase in the block-averaged penalty between consecutive blocks a penalty jump. The analysis in § 3 shows that the increase persists on held-out tokens not used to detect the jump and is concentrated in the first block after it. Across penalty jumps, the block-averaged penalty increases while teacher confidence decreases, as reflected in rising entropy and falling top-1 probability. We call this observed pattern penalty–confidence tension. Control boundaries entering blocks with high penalties do not exhibit comparable changes in teacher confidence. This tension motivates examining whether supervision in the block after a jump should be reduced.

Motivated by penalty–confidence tension, we propose Adaptive Pullback OPD (AP-OPD), illustrated in Figure 1. The method uses the preceding block as a local reference and, at each detected jump, uniformly scales down all signed supervision coefficients in the following block so that its weighted block-averaged penalty matches the preceding block’s original mean. This rule uses an adjacent segment of the same response to set the reference. A positive scaling factor preserves the signs and relative proportions of the token coefficients. The reference determines the scaling factor without an additional strength constant. AP-OPD detects jumps using log-probabilities already computed by OPD and requires no additional forward passes.

Training experiments support the value of this intervention. AP-OPD improves performance, whereas a magnitude-cap rule that selects blocks by their penalty values and removes a similar amount of penalty brings no gain over vanilla OPD. The removal amount alone therefore does not explain the improvement. We also compare AP-OPD with FiRe-OPD, which weights supervision using teacher and student confidence (Li et al., 2026b), and IW-OPD, which uses accumulated prefix discrepancy (Xie et al., 2026b). On mathematical reasoning, AP-OPD consistently improves macro-averaged Avg@k over vanilla OPD under both jump detectors and obtains a higher macro average than the compared reweighting methods. On code generation, with a different teacher and student, AP-OPD also achieves the highest macro average among the evaluated methods. Our contributions are: (1) We identify and diagnose penalty–confidence tension at penalty jumps in OPD, showing that local penalty changes provide information about changes in teacher predictions beyond the current penalty value. (2) We propose Adaptive Pullback OPD, which uniformly scales supervision in the block after a jump using the preceding block as a local reference. (3) We demonstrate gains on mathematical reasoning and code generation, and show that removing a comparable amount of penalty with a magnitude-cap rule does not yield the same benefit.

Preliminaries

Token-Level OPD

Let $x$ be a prompt, $\pi_{\theta}$ the student, and $\pi_{T}$ a frozen teacher. The student samples $\hat{y}_{1:T} \sim\pi_{\theta}(\cdot\mid x)$, and $c_t = (x,\hat{y}_{<t})$ denotes the context at position $t$. With $\ell_t^S = \log\pi_{\theta}(\hat{y}_t \mid c_t)$ and $\ell_t^T = \log\pi_T(\hat{y}_t \mid c_t)$, the sampled log-ratio is

The sequence-level reverse KL is $\mathbb{E}[\sum_t r_t]$ (Gu et al., [5]); its policy gradient uses suffix returns (Williams, 1992; Sutton et al., 1999). Token-level OPD uses only the local term, with stop-gradient advantage $A_t=-\operatorname{sg}(r_t)$ (Fu et al., [4]). Averaging over the $N_{\mathrm{tok}}$ valid tokens in a mini-batch (Yu et al., 2025) gives

Vanilla OPD sets $w_{n,t}=1$. AP-OPD and the compared baselines all change the stop-gradient weights $w_{n,t}$.

Penalizing supervision. A single realization of $r_t$ can take either sign. When $r_t>0$, the student assigns the sampled token a higher probability than the teacher, so $A_t<0$ and the update pushes that token down. This case dominates in practice: in the vanilla OPD rollouts analyzed below, tokens with $r_t>0$ carry 83.5% of $\sum_t|A_t|$. We define the token penalty as

Block Profile and Jumps

Single log-ratios fluctuate sharply between neighboring tokens, so we summarize a response at the level of contiguous blocks. We partition its valid tokens into blocks $\mathcal{B}_1,\ldots,\mathcal{B}_M$ of $B$ tokens; only the last block may be shorter, and $n_i=|\mathcal{B}_i|$. The block-averaged penalty $D_i$ is the mean positive log-ratio within a block, including zeros for nonpositive values; $s_i^2$ is the within-block sample variance of $d_t$ for $n_i\ge2$:

At boundary $i$, let $\Delta_i=D_{i+1}-D_i$. A jump is a positive change judged sufficiently large by a chosen detector. Let $\mathcal{P}$ denote the response’s block penalty profile and required statistics, and $\operatorname{Det}(i;\mathcal{P})\in\{0,1\}$ the detector’s decision. Over boundaries $i=1,\ldots,M-1$, the jump set is

The default $z$-gate accepts $i$ when $n_i,n_{i+1}\ge2$ and $\Delta_i>z\hat{\sigma}_i$, where

We fix $z=2$; because adjacent tokens are correlated, this is a standardized threshold, not a calibrated significance level. For $i\in\mathcal{E}$, $\mathcal{B}_i$ and $\mathcal{B}_{i+1}$ are the pre- and post-jump blocks, and $\Delta_i$ is the rise. Blocks only localize where to intervene; the loss in Eq. (2) remains token-level.

Diagnostic Analysis

We examine whether local changes in block-averaged penalty reveal changes in teacher predictions beyond the current penalty value. We analyze 512 rollouts saved during vanilla OPD training at steps 70, 80, 90, and 100, one per distinct prompt, with the mathematical-reasoning student, teacher, and data of §5.1. Jumps are detected with the default $z$-gate at $B=64$ and $z=2$, about twice per response. We compare each jump with control boundaries from the same response at a similar relative position: the high-magnitude boundary, which enters the block with the highest block-averaged penalty among the candidates; a random positive-change boundary with $\Delta_i>0$; and a random boundary. The random boundary measures background variation. The random positive-change boundary asks whether selecting a rise alone explains the effect. The high-magnitude boundary asks whether entering a block with a high penalty is sufficient to account for the observations. Appendix § 10 gives the full protocol.

The Penalty Jumps

Jumps are selected for a large $\Delta_i$, so measuring the rise on the same tokens would also count the noise that made the boundary pass the gate. We therefore use a held-out measurement: jumps are detected on the odd-position tokens of each block and the profile is measured on the even-position tokens, and vice versa. The held-out profiles in Figure 2 show an abrupt penalty increase at jumps, larger than at any of the three controls. The rise is concentrated in the first block after a jump. By the second block, the held-out block-averaged penalty is back at the level of a position-matched random boundary. The analysis in § 5.4 shows that the rise persists after token-level clipping. Appendix § 10 reports the same pattern in two further training runs.

The post-jump block carries about twice its token share of the penalty: 6.5% of tokens hold 13.5% of the penalty mass. The blocks selected by magnitude-cap are even more concentrated, so concentration alone does not single out jumps.

Penalty–Confidence Tension

We measure teacher prediction changes using entropy and top-1 probability separately. For the teacher distribution $p_t(v) = \pi_T(v \mid c_t)$ over vocabulary $\mathcal{V}$, these are $H_t = -\sum_{v \in\mathcal{V}} p_t(v) \log p_t(v)$ and $P_{\max,t} = \max_{v \in\mathcal{V}} p_t(v)$. We run full-vocabulary teacher forward passes over the analysis rollouts and report their block means as $H$ and $P_{\max}$. Panels a and b of Figure 3 show that across the detected jumps, $H$ rises and $P_{\max}$ falls, indicating a decrease in teacher confidence. Together with the increase in block-averaged penalty, this observed pattern constitutes penalty–confidence tension. Compared with high-magnitude boundaries, jumps show an additional entropy increase of 0.261 nats and an additional top-1 probability decrease of 0.083. These paired contrasts compare the post-minus-pre changes within matched groups. They quantify how teacher predictions change across the boundary, rather than how confident the teacher is in the following block alone. Local penalty changes therefore reveal differences in teacher predictions beyond entering a block with a high penalty.

Figure 3c shows that, on the held-out tokens, the log-probabilities that both the student and the teacher assign to the sampled tokens fall across a jump. For both models, the drop is larger than across either control. This pattern cannot be explained by an increase in the student’s sampled-token probability alone.

Implications for the method. Penalty–confidence tension motivates testing whether supervision in the post-jump block should be reduced. We use the preceding block in the same response as a local reference, without assuming that its penalty is correct or optimal. The rule in § 4 detects penalty jumps rather than teacher uncertainty and does not use teacher confidence as an input. Training experiments in § 5.3 evaluate the intervention: AP-OPD improves performance, whereas magnitude-cap removes a similar amount of penalty without a gain. This comparison does not establish a causal role for the confidence decrease.

Methodology

Adaptive Pullback OPD keeps the vanilla advantage $A_t$ and the objective in Eq. (2), and changes only the token weights $w_t$. For each response, it computes the block profile in Eq. (4) and detects the jumps $\mathcal{E}$ using Eq. (5). It then uniformly scales supervision in each post-jump block using the preceding block as a local reference. Algorithm 1 gives the complete procedure.

Pullback Rule

At a jump $i \in\mathcal{E}$, AP-OPD uses the preceding block’s original block-averaged penalty $D_i$ as a local reference. It uniformly scales all signed supervision coefficients in the post-jump block by one factor:

The weighted block-averaged penalty therefore matches the reference:

A positive weight preserves the signs and relative proportions of the coefficients; a zero weight sets them all to zero. The scaling also reduces rewarding coefficients when $r_t < 0$. In contrast, setting every $r_t$ to $D_i$ erases token differences and turns negative coefficients into penalties when $D_i > 0$; subtracting $\Delta_i$ turns tokens with $0 < r_t < \Delta_i$ from penalized to rewarded.

The rule triggers on detected local increases in block-averaged penalty. Detection and weights both use the observed, unweighted profile. A block that stays high after a jump therefore keeps weight one. If the next boundary is also a jump, the post-jump block serves as its pre-jump block, again with its original, unweighted block-averaged penalty. Each block is weighted at most once.

Properties

Bounded weights. Since $0 \le D_i < D_{i+1}$ for every jump, post-jump weights lie in $[0,1)$ and all other weights are one: AP-OPD never upweights a token or flips the sign of a coefficient.

Strength set by the data. The penalty mass removed from a post-jump block is $\sum_{t\in\mathcal{B}_{i+1}}(1-w_t)d_t=n_{i+1}\Delta_i$. Given the detector and its parameters, the local reference sets the scaling factor without an additional strength constant. For the default detector, $B=64$ and $z=2$ are fixed before training. The weights have no additional normalization, clipping, or floor. Removed penalty mass is a coefficient statistic, not a gradient or parameter-update magnitude.

Selective intervention. As reported in § 5.4, about a quarter of training responses contain no jump and are trained exactly as in vanilla OPD. As quantified in Appendix § 10, applying the same ratio at every boundary with $\Delta_i > 0$ would remove 36.1% of the penalty mass instead of 11.6% under the default detector on the analysis rollouts.

Ranking detector. As an alternative to the $z$-gate, the top-$\alpha$ detector ranks the boundaries of each response by $\Delta_i$. It selects the

largest positive changes, where $C$ is the number of block boundaries and $\alpha=5\%$, and applies the same pullback in Eq. (7).

Training Objective and Complexity

Substituting Eq. (7) into Eq. (2) gives the AP-OPD objective. The weights are stop-gradient quantities computed in one $O(T)$ pass over log-probabilities that OPD already produces; they need no additional rollout, model forward pass, or full-vocabulary statistic, and inference is unchanged. The weights are not renormalized.

Experiments

Experimental Setup

Training. For mathematical reasoning, the student is Qwen3-1.7B-Base ((Yang et al., 2025)). The frozen teacher is Qwen3-8B-RL-Math, which is Qwen3-8B-Base trained on mathematics with GSPO ((Zheng et al., 2025)). Training prompts come from DAPO-Math-17K ((Yu et al., 2025)), and each method trains for 200 steps. For code generation,

Evaluation. For mathematical reasoning, we report Avg@$k$, the accuracy averaged over $k$ sampled responses per problem, on MATH500 ([7] (Lightman et al. 2024)), AMC23, Minerva ((Lewkowycz et al., 2022)), OlympiadBench ([6]), AIME24, and AIME25. Answers are graded with the rule-based math checker of verl ((Sheng et al., 2025)). For code generation, we report Avg@4 on HumanEval+, MBPP+ ((Liu et al., 2023)), and LiveCodeBench v6 ([10]), running each program against the benchmark’s tests. All models share one decoding setup. Mathematical results average two training runs that differ in data order. Appendix § 9 gives the full protocol, and Appendix § 11 reports each run.

Qwen3-4B-Instruct-2507 is distilled into Qwen3-1.7B on the code subset of Eurus-2-RL-Data ([3]), and each method trains for 100 steps. Within each domain, every method starts from the same student checkpoint. In both domains, our runs use 128 prompts per step, one rollout per prompt of at most 8,192 tokens, and AdamW with a constant learning rate of $10^{-6}$. AP-OPD uses $B=64$ and $z=2$ in both domains, fixed before training. Details are shown in Appendix § 9.

Baselines. We compare with vanilla token-level OPD ([4]) and two supervision reweighting methods. IW-OPD uses accumulated absolute log-ratios, normalized within each response, to assign decreasing prefix weights ((Xie et al., 2026b)). FiRe-OPD filters trajectories and reweights tokens by teacher and student confidence ((Li et al., 2026b)). Both use their authors’ code and default optimization settings, and share the student, teacher, training prompts, number of steps, and evaluation with our runs. On code generation, we compare with vanilla OPD and IW-OPD.

Main Results

Mathematical reasoning. Table 1 reports AP-OPD with each of its two jump detectors, the $z$-gate of § 2.2 and the top-$\alpha$ ranking of Eq. (9). Both detectors improve the macro average over vanilla OPD, by 0.80 points for top-$\alpha$ and 0.57 for the $z$-gate. The improvement holds in each training run (Appendix § 11). In the two-run mean, top-$\alpha$ improves on every benchmark. Both detectors also obtain higher macro averages than IW-OPD and FiRe-OPD in this mean. FiRe-OPD, the stronger baseline, improves on vanilla OPD by less than either detector. Computing its token weights requires the full-vocabulary entropies of the teacher and the student. AP-OPD uses only the sampled-token log-ratios that OPD already computes.

Method	MATH500	AMC23	Minerva	Olympiad	AIME24	AIME25	Macro
Teacher (Qwen3-8B-RL-Math)	86.80	74.22	35.75	50.19	32.08	21.67	50.12
Student (Qwen3-1.7B-Base)	42.15	24.06	11.67	16.63	3.54	1.67	16.62
Vanilla OPD	72.90	43.75	20.96	33.63	12.40	7.29	31.82
IW-OPD	71.60	43.28	22.15	32.31	11.35	7.19	31.31
FiRe-OPD	72.80	45.39	21.74	32.83	11.88	8.44	32.18
AP-OPD (top-$\alpha$)	72.97	44.77	<u>22.06</u>	34.26	13.02	8.65	32.62
AP-OPD ($z$-gate)	73.08	46.72	21.42	32.94	12.19	8.02	<u>32.39</u>

Choice of detector. The selected boundary sets of the two detectors have a Jaccard overlap of about one third on the analysis rollouts. Since both improve over vanilla OPD, the gain is not tied to one detector. We keep the $z$-gate as the default as it needs no selection ratio and leaves responses without a clear rise untouched.

caption: Code generation uses Avg@4 (%). Qwen3-4B-Instruct-2507 (Teacher) and undistilled Qwen3-1.7B (Student) are unranked references. HE+ is HumanEval+; LCB is LiveCodeBench v6. Macro is the unweighted mean; best trained scores are bold.

:::

Code generation. Code generation tests AP-OPD with a different student, teacher, and training data. The student is already post-trained. The teacher is a general instruction model. With the default $z$-gate, AP-OPD obtains the highest macro average, 2.05 points above vanilla OPD and 0.96 above IW-OPD (Table 2). It improves over vanilla OPD on all three benchmarks. It leads on HumanEval+ and on LiveCodeBench, the hardest benchmark, and is close to IW-OPD on MBPP+. The gain over vanilla OPD is larger than on mathematical reasoning. Across both domains, down-weighting supervision after detected jumps improves over vanilla OPD. Section 5.3 compares AP-OPD with magnitude-cap to test whether removing a similar amount of penalty is sufficient to obtain the observed gains.

Ablation Studies

Comparison with magnitude-cap. Magnitude-cap selects the $\lceil\alpha C \rceil$ post-boundary blocks with the largest $D_{i+1}$ and assigns weight $\min(1, \bar{D}/D_{i+1})$, where $\bar{D}$ is the response’s mean penalty. We measure the removed fraction as $q = \sum_t(1-w_t)d_t/\sum_t d_t$, averaged equally over responses with nonzero penalty mass. Magnitude-cap removes 13.75%, between the two AP-OPD detectors, but brings no gain and trails both by about one point (Table 3a; paired intervals in Appendix C). An earlier experiment also gains from a fixed post-jump weight of 0.3, whereas pre-jump upweighting brings no gain (Appendix D). The tested gains thus occur with two detectors and two post-jump weights; a similar removal amount alone is insufficient. These rule comparisons do not isolate selection, reference, and scaling depth.

Variant	Cov.	Unsel.	$q$	Macro	$\Delta$
(a) Comparison with magnitude-cap
Vanilla OPD	–	–	–	31.55	–
Magnitude-cap	7.17	0.00	13.75	31.40	−0.15
AP-OPD (top-$\alpha$)	7.19	0.00	14.63	32.50	+0.95
AP-OPD ($z$-gate)	6.49	28.71	11.60	32.50	+0.95
(b) Block size, $z$-gate
$B = 32$	4.23	24.22	9.51	32.06	+0.50
$B = 64$	6.49	28.71	11.60	32.50	+0.95
$B = 128$	8.60	36.72	11.78	31.96	+0.40

Block size. The block-size ablation in Table 3b shows improvements over vanilla OPD and magnitude-cap for all three block sizes. Larger

blocks select fewer jumps but weight longer spans, so the removed mass $q$ stays within 9.5–11.8%. The default $B = 64$ gives the largest gain. Shorter blocks may make block means noisier, while longer blocks may average over local increases.

Analysis

Relation to token-level clipping. The rise is not carried by a few extreme tokens, so clipping each token’s log-ratio, as REPOPLD does ([16]), leaves it largely in place. At a clipping bound that removes as much penalty mass as AP-OPD on the analysis set, jumps are still detected and about 90% of the held-out rise remains; RE-OPOLD’s default bound removes more than twice as much and still keeps about 70%. The two rules also remove different supervision. Appendix E reports that only 14.5% of AP-OPD’s removal lies above the bound, close to the 12.5% expected if its removal were unrelated to token size.

Training statistics. During training, AP-OPD marks 6.8% of boundaries as jumps and leaves 24.2% of responses unweighted. Gradient clipping fires less often than in vanilla OPD. The training statistics in Appendix C show that response length and student entropy stay close to vanilla OPD. The mean token weight is 0.945. Magnitude-cap has a similar mean weight, 0.961, but brings no gain. The removal amount alone therefore does not explain the performance difference.

Related Work

On-policy distillation. Unlike distillation from fixed or teacher-generated data [8, 15, 9] (Wang et al. 2021; Xie et al. 2026a), OPD scores the student’s own rollouts with the teacher [1]. GKD supports on-policy learning with alternative loss functions [1]. MiniLLM minimizes reverse KL with policy gradients to prevent overestimating low-probability regions of the teacher distribution [5]. DistiLLM uses an adaptive off-policy approach to reuse student-generated outputs (Ko et al., 2024). Recent work extends OPD to reasoning post-training [16] (Zhu et al. 2026; Zhang et al. 2026a; Li et al. 2026a) and studies its supervision and optimization [11, 14] (Song and Zheng 2026; Yang et al. 2026a; Zhang et al. 2026b; Wang et al. 2026). RevisitingOPD documents failure modes of sampled-token comparison, including unreliable guidance on student-drifted prefixes [4].

Reweighting, pruning, and down-weighting in OPD. TIP selects tokens by student uncertainty and teacher–student divergence (Xu et al., 2026). FiRe-OPD filters trajectories and softly reweights tokens by teacher and student confidence (Li et al., 2026b). IW-OPD uses accumulated absolute log-ratios, normalized within each response, to assign decreasing prefix weights (Xie et al., 2026b). Prune-OPD decays the supervision of a suffix once teacher–student support mismatch accumulates (Yang et al., 2026b). REOPOLD clips token rewards at a fixed bound [16]. Rock Tokens studies tokens whose distillation loss persists over training [13], Blockwise Gating weights 64-token blocks by student policy drift (Zheng and Jiang, 2026), and entropy-aware OPD adds a forward-KL term where teacher entropy is high [14]. TrOPD gates the penalty by the teacher–student probability ratio (Xing et al., 2026). AP-OPD instead uses changes between consecutive blocks in the same response and scales post-jump supervision using the preceding block as a local reference. Unlike confidence-based weighting, its rule uses penalty changes without measuring teacher confidence.

Temporal information. Return-to-go OPD propagates later log-ratios to earlier positions, at the cost of higher gradient variance [5, 4].

TOPD and AC-OPD obtain near-future evidence by generating continuations from selected prefixes [12] (Zhou et al. 2026). LGR evaluates the student’s top-$K$ candidate tokens by the one-step-ahead teacher confidence they induce and adds a group-normalized reward (Liu et al., 2026). AP-OPD uses only changes realized along the completed rollout and needs no additional generation or forward pass.

Choice of baselines. We compare with IW-OPD and FiRe-OPD as supervision reweighting baselines on student-generated rollouts; FiRe-OPD also filters trajectories. Other methods introduce changes to objectives, rollout generation, or token selection. TrOPD and entropy-aware OPD add a forward-KL term, and TrOPD also uses teacher-generated prefixes. LGR, TOPD, and AC-OPD add teacher passes or continuations. Prune-OPD truncates rollouts, and TIP keeps a subset of tokens to reduce training memory. Rock Tokens selects token types by their loss over training. Blockwise Gating gates on the drift between the rollout and the current student, which is zero when each batch is used for one update, as in our setting. REOPOLD also adds stage-dependent token masks, so we compare only its token-level clipping, directly on our rollouts (§5.4). Magnitude-cap selects blocks by their block-averaged penalties and removes a similar amount of penalty mass, trained with the same configuration as AP-OPD (§5.3).

Conclusion

We studied local penalty changes in on-policy distillation, where the current penalty value does not distinguish sustained high penalties from abrupt increases along a response. Comparing consecutive token blocks reveals penalty jumps that persist on held-out tokens. Across these jumps, penalties rise while teacher confidence decreases, whereas boundaries into high-penalty blocks do not exhibit comparable confidence changes. This penalty–confidence tension shows that local changes provide information beyond the current penalty value. Motivated by this observation, we proposed AP-OPD, which uniformly scales supervision coefficients in each post-jump block using the preceding block as a local reference. The weighted average penalty matches the preceding block’s original mean, setting the scaling factor without an additional strength constant. AP-OPD improves performance on mathematical reasoning and code generation.

Limitations

We evaluate AP-OPD on mathematical reasoning and code generation, and have not yet applied it to other settings such as agentic or multi-turn tasks. Our diagnostic evidence for penalty–confidence tension is observational and does not establish that decreased teacher confidence causes the gains from pullback. AP-OPD also leaves room for improvement in how it forms blocks. It uses a fixed block length, and adapting the block size to each response or aligning blocks with reasoning steps is a natural extension. Finally, AP-OPD responds only to abrupt rises, so it does not address reward-hacking-like behavior in OPD such as repetitive looping, where the penalty fades rather than jumps.

Experimental Details

Student Training

Table 4 lists the mathematical-reasoning configuration shared by vanilla OPD, both AP-OPD detectors, magnitude-cap, and the block-size runs; the code experiment is described in Appendix § 9.§ 4.

Setting	Value
Student initialization	Qwen3-1.7B-Base
Frozen teacher	Qwen3-8B-RL-Math; see Table 5
Training prompts	DAPO-Math-17K; 17,398 prompts
Prompt / response limit	2,048 / 8,192 tokens
Steps	200, from the base checkpoint
Batch	128 prompts, one rollout each; one update per step
Rollout sampling	temperature 1.0, top-$p$ 1.0, no top-$k$; thinking disabled
Optimizer	AdamW, $\beta= (0.9, 0.999)$, weight decay 0.01
Learning rate	$10^{-6}$, constant, no warmup
Gradient clipping	global norm 1.0
Loss	token mean; no task reward, KL loss, or entropy bonus
Data order	not shuffled; run 1: data seed 42; run 2: data seed 43
AP-OPD	$B = 64$, $z = 2$; top-$\alpha$ uses $\alpha= 5\%$

Teacher

The mathematical-reasoning teacher, Qwen3-8B-RL-Math, is Qwen3-8B-Base trained with GSPO (Zheng et al., 2025) on DAPO-Math-17K. Table 5 lists its configuration. The teacher is frozen during distillation.

Setting	Value
Initialization	Qwen3-8B-Base
Training prompts	DAPO-Math-17K
Policy optimization	GSPO
Steps	200
Prompt / response limit	2,048 / 4,096 tokens
Batch	512 prompts, 16 responses each; mini-batch of 32 prompts
Clipping range	$3 \times10^{-4}$ lower, $4 \times10^{-4}$ upper
Learning rate	$10^{-6}$, 10 warmup steps
Weight decay	0.1
Reward	+1 for a correct answer, −1 otherwise; no overlong shaping

Evaluation

All models use the evaluation configuration in Table 6, with 7,388 responses per model. Prompts are capped at 2,048 tokens and responses at 16,384; sampling uses temperature 0.7, top-$p$ 0.8, top-$k$ 20, and seed 42, with thinking mode disabled. For a benchmark with $N$ problems and correctness $c_{q,j} \in\{0,1\}$ of sample $j$ on problem $q$,

Benchmark	Problems	$k$	Metric
MATH500	500	4	Avg@4
AMC23	40	16	Avg@16
Minerva	272	4	Avg@4
OlympiadBench	675	4	Avg@4
AIME24	30	16	Avg@16
AIME25	30	16	Avg@16

Answers are graded with the rule-based math checker of verl.

Paired bootstrap. For a contrast between two models, we resample the problems of each benchmark with replacement 10,000 times using seed 42, keep the existing $k$ scores of each resampled problem for both models, and recompute the per-benchmark and macro differences. We report percentile intervals. The intervals use the run-1 checkpoints and cover evaluation randomness, not training variance.

Code Generation

The code experiment distills Qwen3-4B-Instruct-2507 into Qwen3-1.7B, with the configuration in Table 7. Training prompts are drawn from the 25,276 training problems in the code subset of Eurus-2-RL-Data (Cui et al., [3]), competitive-programming problems with test cases. All methods train for 100 steps from the same student checkpoint, and we evaluate the step-100 checkpoints. We evaluate on 164 HumanEval+ problems, 378 MBPP+ problems, and 175 LiveCodeBench v6 problems, with four samples per problem, responses capped at 16,384 tokens, temperature 0.7, top-$p$ 0.8, top-$k$ 20, and thinking disabled. Each program is executed in a sandbox against the benchmark’s tests, and Avg@4 is the share of passing samples.

caption: Training configuration of our code-generation runs.

:::

Diagnostic Analysis Details

Rollouts

The analysis set contains 512 rollouts, one per distinct prompt, saved by the vanilla OPD run at steps 70, 80, 90, and 100, with 128 rollouts per step. For replication we use two further training runs with rollouts saved every 10 steps over 300 steps, with 3,840 rollouts per run: a vanilla OPD run and a run trained with the paired rule of Appendix § 12. All statistics are computed offline and do not affect training.

Detector Statistics

Table 8 compares jump detectors and other selection rules on the analysis set. The two detectors select overlapping but different boundaries, with a Jaccard index of 0.33.

Rule	Selected/resp.	Unselected (%)	Token cov. (%)	$q$ (%)
Top-$\alpha$ (5%)	1.85	0.00	7.19	14.63
$z = 1.5$	3.93	8.98	13.37	20.04
$z = 2$ (AP-OPD)	1.91	28.71	6.49	11.60
$z = 3$	0.36	75.78	1.10	2.56
All $\Delta_i > 0$	—	—	—	36.11
Magnitude-cap	1.85	0.00	7.17	13.75
Detect	− High-mag. bound.	− Rand. pos.	− Rand. bound.
odd	0.019 [0.006, 0.031]	0.041 [0.025, 0.058]	0.135 [0.105, 0.168]
even	0.036 [0.016, 0.057]	0.050 [0.034, 0.066]	0.145 [0.116, 0.177]

Split-Half Protocol

Profiles use the window $[-2B, 2B)$ around each boundary, with offset 0 at the first token of the following block, and never cross a mask break or the response boundary. Controls come from the same response and the same decile of relative position, defined as the boundary offset divided by valid length. Within each control type, boundaries are drawn without replacement; the high-magnitude boundary is the candidate with the highest $D_{i+1}$.

For split-half measurement, detection uses only the odd-indexed tokens of each 64-token block, including the token counts in the gate, and controls are also selected on odd tokens; all quantities are then measured on even tokens. The reverse direction swaps the roles, and the two directions are averaged with shared bootstrap weights. Estimates are averaged within each response, then over prompts; intervals come from 10,000 prompt-clustered re-samples using seed 42. The held-out rise $U_{a:b}$ is the measured mean of $d_t$ over offsets $[a,b)$ minus the measured mean of the block preceding the selected boundary. For a jump, this reference is the pre-jump block; each control uses its own preceding block. On the analysis set, 910 of the 978 $z = 2$ jumps have a complete window, covering 346 responses; with odd and even detection there are 478 and 453 complete jumps from 247 and 230 responses, and 324 prompts in the union. Each detection half of a block has only 32 tokens, and adjacent tokens remain correlated, so split-half measurement reduces but does not remove selection effects.

Teacher Confidence

Of the 512 analysis prompts, 362 contain an eligible $z = 2$ jump, and 353 of these yield a complete group with a position-matched high-magnitude boundary and a random boundary. 153 groups

Rollouts	$U_{0:64}$	$U_{64:128}$	Ratio
Analysis set	0.190 [0.172, 0.209]	0.096 [0.081, 0.111]	50.3%
Vanilla, 300 steps	0.188 [0.183, 0.194]	0.101 [0.096, 0.106]	53.6%
Paired rule, 300 steps	0.190 [0.184, 0.196]	0.103 [0.098, 0.108]	54.5%

share no prompt or control prefix with one another within the subset; the full set comprises 80 connected components. The teacher is run causally over the original student history, without re-encoding prefixes or truncating early context, and $H$ and $P_{\max}$ are averaged over the 64 tokens of each block. Intervals resample the 80 components for all groups and resample groups within training steps for the independent subset, using 10,000 re-samples with seed 42. Table 11 reports both sets. These distributional changes support the observed penalty–confidence tension for the default $z = 2$ jump set; they are not held-out measurements or a combined confidence score.

Metric	Control	All 353 groups	Independent 153
$H$ (nats)	random	0.300 [0.275, 0.342]	0.311 [0.263, 0.365]
$H$ (nats)	high-magnitude	0.172 [0.122, 0.265]	0.261 [0.212, 0.317]
$P_{\max}$	random	−0.099 [−0.112, −0.091]	−0.102 [−0.116, −0.088]
$P_{\max}$	high-magnitude	−0.061 [−0.085, −0.050]	−0.083 [−0.098, −0.069]

Log-probability decomposition. Writing $\Delta S$ and $\Delta T$ for the post-minus-pre changes in the block means of $\ell^{S}$ and $\ell^{T}$, the change in the mean signed log-ratio is $\Delta S-\Delta T$. This identity does not directly give the change in block-averaged penalty, which averages $[r_t]_+$. In the held-out setting of Figure 3c, the teacher’s drop exceeds that at high-magnitude boundaries by 0.052 [0.028, 0.077] with odd detection and 0.077 [0.043, 0.112] with even detection; the student’s drop also exceeds that of both controls.

Supplementary Results

Table 12 lists the per-benchmark scores of all trained models, and Table 13 gives per-benchmark paired intervals for the contrasts among our own runs. The five methods in the main comparison are reported for each run and for their mean. Magnitude-cap and the additional block sizes were trained in run 1 only. The paired intervals and training dynamics also refer to run 1.

Alternative Reweighting Rules

AP-OPD uniformly scales supervision in the post-jump block using the preceding block’s original block-averaged penalty as a local reference. Two alternatives are natural: compensating by upweighting the pre-jump block and down-weighting the post-jump block by a fixed constant. We evaluate three such rules at step 100 with 8,192-token evaluation responses, so their numbers are not comparable with Table 1. All three select boundaries with the top-$\alpha$ ranking of Eq. (9) with $\alpha= 5\%$. The paired rule raises each selected pre-jump block and lowers its post-jump block by the same amount, with the weights normalized over the response. The two one-sided rules multiply every selected pre-jump block by 4 or every selected post-jump block by 0.3. Table 15 shows that upweighting the pre-jump block brings no gain. The pre-jump $\times4$ rule triggers gradient clipping in 88 of 100 steps. The paired rule has a macro average close to vanilla OPD. Down-weighting the post-jump block by a constant 0.3, fixed before training, improves over vanilla OPD and has the best macro average. AP-OPD uses the preceding block’s original mean to determine the post-jump weight, $D_i/D_{i+1}$, without an additional scaling constant. The fixed-weight result shows that a gain can also occur under another post-jump weighting rule; it does not establish that the ratio is superior, since the protocols differ.

Rule	MATH500	AMC23	Minerva	Olympiad	AIME24	AIME25	Macro	Clipped	Mean $w$
Vanilla OPD	69.90	43.12	20.31	30.67	11.46	6.25	30.29	–	1
Paired (+pre, −post)	70.00	41.88	20.77	30.33	11.67	7.71	30.39	32 / 100	1.006
Pre-jump ×4	69.15	41.88	20.50	31.26	12.29	6.25	30.22	88 / 100	1.200
Post-jump ×0.3	70.25	44.38	21.23	31.11	12.29	6.88	31.02	30 / 100	0.955

Model	MATH500	AMC23	Minerva	Olympiad	AIME24	AIME25	Macro
Mean of the two runs
Vanilla OPD	72.90	43.75	20.96	33.63	12.40	7.29	31.82
IW-OPD	71.60	43.28	22.15	32.31	11.35	7.19	31.31
FiRe-OPD	72.80	45.39	21.74	32.83	11.88	8.44	32.18
AP-OPD (top-$\alpha$)	72.97	44.77	22.06	34.26	13.02	8.65	32.62
AP-OPD ($z$-gate, $B = 64$)	73.08	46.72	21.42	32.94	12.19	8.02	32.39
Run 1
Vanilla OPD	73.50	43.75	20.22	33.30	12.08	6.46	31.55
IW-OPD	73.05	43.91	21.60	31.78	11.46	8.75	31.76
FiRe-OPD	73.00	44.22	21.42	33.00	11.46	8.12	31.87
AP-OPD (top-$\alpha$)	73.05	45.94	21.42	34.41	13.12	7.08	32.50
AP-OPD ($z$-gate, $B = 64$)	72.75	46.56	21.32	33.33	12.71	8.33	32.50
Run 2
Vanilla OPD	72.30	43.75	21.69	33.96	12.71	8.12	32.09
IW-OPD	70.15	42.66	22.70	32.85	11.25	5.62	30.87
FiRe-OPD	72.60	46.56	22.06	32.67	12.29	8.75	32.49
AP-OPD (top-$\alpha$)	72.90	43.59	22.70	34.11	12.92	10.21	32.74
AP-OPD ($z$-gate, $B = 64$)	73.40	46.88	21.51	32.56	11.67	7.71	32.29
Additional ablations
Magnitude-cap	72.55	44.53	21.88	32.56	9.58	7.29	31.40
AP-OPD ($z$-gate, $B = 32$)	71.30	45.16	22.61	32.85	12.29	8.13	32.06
AP-OPD ($z$-gate, $B = 128$)	73.15	42.81	22.79	33.81	11.25	7.92	31.96

Contrast	MATH500	AMC23	Minerva	Olympiad	AIME24	AIME25	Macro
$z$-gate − Vanilla	−0.75	+2.81	+1.10	+0.04	+0.63	+1.88	+0.95
	[−2.40, 0.80]	[−0.47, 6.25]	[−1.19, 3.31]	[−1.44, 1.48]	[−1.04, 2.29]	[−0.42, 5.00]	[0.02, 1.90]
Top-$\alpha$ − Vanilla	−0.45	+2.19	+1.19	+1.11	+1.04	+0.63	+0.95
	[−2.05, 1.10]	[−0.94, 5.47]	[−0.92, 3.31]	[−0.33, 2.52]	[−1.67, 4.17]	[−1.25, 2.71]	[0.01, 1.90]
Magnitude-cap − Vanilla	−0.95	+0.78	+1.65	−0.74	−2.50	+0.83	−0.15
	[−2.60, 0.65]	[−1.56, 3.28]	[−0.46, 3.86]	[−2.22, 0.70]	[−5.00, −0.21]	[−1.46, 3.33]	[−0.99, 0.70]
$z$-gate − Magnitude-cap	+0.20	+2.03	−0.55	+0.78	+3.13	+1.04	+1.10
	[−1.55, 1.95]	[−0.31, 4.69]	[−2.57, 1.38]	[−0.81, 2.41]	[0.42, 6.46]	[−1.04, 4.17]	[0.20, 2.07]
Top-$\alpha$ − Magnitude-cap	+0.50	+1.41	−0.46	+1.85	+3.54	−0.21	+1.11
	[−1.15, 2.20]	[−1.72, 4.22]	[−2.67, 1.75]	[0.30, 3.44]	[0.21, 7.50]	[−1.67, 1.04]	[0.13, 2.12]
$z$-gate − Top-$\alpha$	−0.30	+0.63	−0.09	−1.07	−0.42	+1.25	−0.00
	[−2.00, 1.35]	[−2.19, 3.59]	[−2.30, 2.02]	[−2.56, 0.41]	[−3.75, 2.50]	[−1.04, 4.38]	[−1.01, 1.01]

Block-Level versus Token-Level Down-Weighting

This appendix gives the protocol and additional numbers for § 5.4.

Setup. REPOPLD bounds each token’s penalty from above [16]; in our notation it replaces $r_t$ by $\min(r_t,\kappa)$ with

We use two bounds. In the removal-matched setting, $\lambda= 0.051$ gives $\kappa= 3.131$, which removes the same 11.6% of penalty mass as AP-OPD on

caption: tbl:14: Run-1 training dynamics over 200 steps, reported as means of logged values. Clipped steps have pre-clipping gradient norm above 1.

:::

analysis set. Under REPOOLD’s default, $\lambda= 0.3$ gives $\kappa= 1.720$, which removes 27.0%. On the two 300-step runs of Appendix § 10 we keep both bounds fixed rather than recalibrating them, so the matched bound removes somewhat more than AP-OPD there: 13.2% and 13.4%, compared with 10.7% and 10.6% for AP-OPD, respectively. After clipping, we recompute the block profile, rerun the $z = 2$ detector, and repeat the split-half measurement of Appendix § 10.§ 3.

Rise after clipping. Table 16 shows that at both bounds and on all three sets of rollouts, nearly all jumps are detected again and most of the held-out rise remains. At the matched bound on the analysis set, the held-out rise still exceeds each of the three matched controls in both detection directions under the paired bootstrap of Appendix § 10.§ 3, as it does at the default bound. Clipping also changes block variances and creates new jumps. The detector finds 1,124 jumps at the matched bound and 1,296 at the default bound, compared with 978 without clipping. The retained rise therefore compares two detected sets rather than following each original jump.

Rollouts	Bound	Removed	Jumps kept	Rise kept	Overlap	Expected
Analysis set	matched	11.6	99.0	89.8 [86.1, 93.8]	14.5 [13.1, 15.9]	12.5
Analysis set	default	27.0	96.3	71.1 [67.0, 75.8]	32.3 [30.9, 33.6]	28.6
Vanilla, 300 steps	matched	13.2	98.9	86.1 [84.8, 87.4]	13.3 [13.1, 13.6]	13.1
Vanilla, 300 steps	default	30.1	95.9	68.0 [66.7, 69.4]	31.9 [31.6, 32.2]	30.2
Paired rule, 300 steps	matched	13.4	98.7	84.2 [82.9, 85.5]	13.7 [13.5, 14.0]	13.1
Paired rule, 300 steps	default	30.3	95.5	66.4 [65.0, 67.8]	32.5 [32.1, 32.8]	30.3

Overlap of removed mass. With AP-OPD’s selections and weights fixed on the unclipped signal, we split each removed amount $(1 - w_t)d_t$ into the part above $\kappa$, which clipping would also remove, and the part below it. If AP-OPD’s removal were unrelated to token size, the part above $\kappa$ would equal the share of all penalty mass that lies above $\kappa$. Table 16 shows that the overlap stays close to this reference at both bounds and on all three sets of rollouts. On the two 300-step runs at the matched bound, it nearly coincides with the reference. The two rules therefore remove largely different supervision.