Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets

Introduction

Post-training adapts general-purpose language models to reason, follow instructions, and use tools. The dominant approaches are reinforcement learning from verifiable rewards (RLVR; Shao et al., 2024; Schulman et al., 2017), distillation from a stronger teacher (Agarwal et al., [1]), and supervised fine-tuning (Wei et al., 2022). Across these approaches, the training signal is most effective when it remains on-policy, that is, close to the model’s own output distribution. Updates near this distribution can strengthen the target behavior while preserving capabilities acquired during pretraining (Agarwal et al., [1]; Yang et al., 2024). Post-training is particularly important for models with only a few billion parameters: although they are inexpensive to run and easy to deploy, they often

Accepted at two NeurIPS 2026 workshops: Towards Test-Time Continual Learning Agents (TTCL) and Foundations of LLM Post-Training in Changing Environments (FLLMPT).

Preprint. lack the proficiency required for specialized tasks. We study how to teach such models to solve hard problems that they currently answer only rarely, if at all, without degrading their existing capabilities.

A fundamental challenge of on-policy RL is that on very hard problems the on-policy signal disappears. RLVR requires the model to succeed occasionally, so when every sampled attempt fails the reward is zero everywhere and nothing is reinforced. On-policy distillation (OPD; Agarwal et al., [1]) keeps the update on the model’s own rollouts and corrects them token by token to be closer to a response of a stronger teacher. However, the student rollout of a difficult task contains many mistakes: feedback on the later tokens is computed on top of an already-wrong prefix, so it largely patches earlier errors instead of teaching the step at hand (Zhang et al., 2026). Additionally, the later-token feedback in OPD is inefficient: as the student learns, the prefix on which later-token feedback was given and training budget was spent becomes less probable.

Alternatively, the training signal can come from a complete correct answer supplied from outside, either ground-truth data or a stronger teacher’s solution. Unlike OPD, which corrects the student’s own rollout token by token, this target replaces the rollout entirely: the whole training budget goes on producing a correct answer rather than on patching the one the student wrote. Such a target is drawn from a distribution other than the student’s own, so it is off-policy. The further it lies from what the student would produce, the more the training signal specializes on the task while eroding its abilities elsewhere. This suggests keeping the target a complete correct answer but choosing the one closest to what the student already writes, so the supervision stays near-on-policy.

Several recent works pursue similar ideas. SPOT (Lin and Han, [21]), SODA (Chen et al., [5]), and SuperCorrect (Yang et al., 2025) fix the student’s mistakes with a strong black-box teacher and learn from the corrections. What they leave unexamined is the on-policy-ness of the correction itself, how it changes with the size of the edit, and whether refreshing the corrections online against the evolving student matters. We focus on these aspects. We call the recipe of minimal, online corrections projected distillation (Figure 1) and study it along five axes: how well it learns from close, correct solutions relative to other post-training approaches; why the edit must be minimal to keep the correction near-on-policy; why the corrections must be refreshed against the still-training student; which objective learns best from a correction; and how the projector itself can be improved. A regret analysis in the imitation-learning setting shows the importance of the edit size.

Our contributions

1. We introduce projected distillation, a post-training recipe that learns from minimal, online corrections of the student’s mistakes. It needs only the correction text, so any capable black-box model can serve as the projector. Across two students and five benchmarks it attains the best specialization–retention trade-off, and retains prior abilities where standard post-training forgets on continual-learning metrics (§ 4.1).

2. Holding correctness fixed, we show that the correction must be both minimal and online. Pushing it off-policy monotonically worsens the trade-off (Sec. 4.2), and freezing the pool instead of refreshing it makes it collapse as the student drifts (Sec. 4.3). We further give an efficient training objective, a diff-localized contrast with self-training on correct rollouts (Sec. 4.4), and a

refinement procedure that improves the projections from a mid-capacity (32B) teacher without updating its parameters (Sec. 4.5).

3. We theoretically analyze projected distillation through imitation learning. The edit size interpolates between DAgger’s linear-in-horizon regret and behavior cloning’s quadratic regret (§ 5).

Related Work

Methods for improving a model on hard problems. Approaches differ in where the correct target comes from and how far that target lies from the model’s own distribution; together these largely determine the specialization–retention trade-off. Reinforcement learning from verifiable rewards (RLVR; GRPO (Shao et al., 2024), PPO (Schulman et al., 2017)) draws the target from the model’s own rollouts, but on the pass@$k=0$ set every rollout is wrong and no signal remains. On-policy distillation (GKD Agarwal et al., [1]) and self-distillation (Yang et al., 2024) keep the update on those rollouts while matching a teacher’s token distribution, which requires teacher logits. Supervised fine-tuning on an external gold or teacher solution supplies correctness directly but is maximally off-policy, so fitting it forgets. Between these, there is scope to manufacture a correct target based on the student’s attempt. This opens up the possibility of treating the correct solutions along a continuum from on-policy to off-policy with respect to the model’s current generation style. Our work studies how far the edits may go, whether they need to be recomputed as the model changes, and what objective to use for effective learning.

Closest work. The literature presents approaches similar to ours: LeMa An et al., [2] mines corrections once, from an assortment of models rather than the student, with free-form rewrites and an objective that uses response likelihood maximization. SPOT Lin and Han, [21] gathers mistakes of the base model, rectifies the mistakes using minimal edits from an oracle, creates a preference dataset and learns with a reward-based binary cross-entropy objective. SODA Chen et al., [5] creates mistake–correction pairs without emphasizing minimal edits, performs an SFT warm-up, and then, with the SFT’ed policy fixed, learns from static preference pairs. SuperCorrect (Yang et al., 2025) distills thought templates from a teacher and applies collaborative DPO over the teacher’s correction traces. We also work in the hard-task regime where the model fails. However, our paper focuses mainly on the regime of online learning from close, correct targets to teach the student how to effectively solve hard tasks. We perform a controlled study of the edit sizes used to provide corrections (Sec. 4.2), static vs. online-refreshed pools of corrections (Sec. 4.3) and effective learning objectives for black-box learning (Sec. 4.4). In addition, we theoretically analyze how the edit-size influences the learning regret, which distinguishes our work from the previous ones. Concurrent work Karan et al., [15] explores self-projection with an MCMC sampler, in which the student itself rewrites its response with the reference solution in view; Appendix § 15 compares it with black-box projection on its math benchmark.

Projected Distillation

We study post-training on problems the model rarely solves, where its own samples earn no reward and correctness has to come from outside.

Setup. Given a problem $x$, a solution is a token sequence $y \in\mathcal{V}^{\le H}$ drawn from a vocabulary $\mathcal{V}$ with a generation horizon $H$. The student produces $y$ autoregressively via $\pi_{\theta}(y \mid x) = \prod_t \pi_{\theta}(y_t \mid x, y_{<t})$, where $y_t$ denotes the $t$-th token, $y_{<t}$ its prefix, and $\pi_{\mathrm{ref}}$ the frozen initialization of the student. Correctness is judged by a verifier $R(x,y) \in\{0,1\}$ (e.g., exact-match checking or unit tests), inducing the set of accepted solutions $\mathcal{C}(x) = \{y : R(x,y) = 1\}$. Our focus is on problems that lie beyond the student’s current ability, i.e., pass@$k=0$ set, for which all $k$ sampled attempts fail. Further, in § 5, we formalize the generation process using a dynamic-token sequence MDP to provide theoretical insights.

Projection. The accepted set $\mathcal{C}(x)$ typically contains many solutions, but these solutions can differ substantially in likelihood under the student. Training toward a solution the student already considers likely requires less movement of its output distribution and therefore better preserves unrelated capabilities Kirkpatrick et al., [17]; Kumar et al. [18]. We therefore choose, among the verifier-accepted solutions in $\mathcal{C}(x)$, the one with highest likelihood under the student.

Practically, searching the true projection within $C(x)$ is intractable. We propose to find a close approximation with the help of the student’s attempt itself. The intuition behind this is that a wrong attempt is a sample from the student and is therefore already likely under it; a correction that preserves most of its tokens preserves most of that likelihood, while the verifier places the result in $C(x)$. Minimal editing is thus a tractable surrogate for Definition 1, with the verifier supplying correctness and the retained tokens supplying proximity. We test the surrogate directly in Sec. 4.2, varying the edit size and recording the likelihood shift $\Lambda= \log\pi_{\theta_{S}}(y^{+} \mid x) - \log\pi_{\theta_{S}}(y^{-} \mid x)$ that each size demands. A projector receives the problem and the attempt in context and is prompted to make the smallest change that corrects the answer (examples in Appendix § 8; prompts in Appendix § 9.§ 3).

Learning from projection. In each round we prompt the student on the hard problems and collect a batch of responses. Most of them are wrong. We write $y^{-}$ for a wrong response and $y^{+}$ for its projection, and $y^{c}$ for the responses the student gets correct on its own. We have access only to the text the projector returns, not to its logits, so we cannot match its token distribution. We learn from the pair $(y^{-}, y^{+})$ by preference instead. The projection edits only a small part of the attempt, so $y^{-}$ and $y^{+}$ agree almost everywhere, and a contrast over their full length would spend most of its gradient on text that was already correct. We therefore restrict the contrast to the tokens the edit changed. A token-diff of $y^{-}$ against $y^{+}$ marks a mistake span $E^{-}$ and a fix span $E^{+}$; for instance, for $y^{-} = \mathrm{ABCD}$ and $y^{+} = \mathrm{AXCZ}$, $E^{-} = \{\mathrm{B}, \mathrm{D}\}$ and $E^{+} = \{\mathrm{X}, \mathrm{Z}\}$. We apply a DPO contrast [26] to those spans alone, using the frozen initialization $\pi_{\mathrm{ref}}$ as reference and $\beta$ for its strength:

Here $\sigma$ is the logistic function. The contrast constrains only how likely $y^{+}$ is relative to $y^{-}$, so we add a term that supervises the student’s own correct responses directly:

Therefore, the wrong responses supply the contrast and the correct ones the supervised term. The next round re-prompts the student on the same problems and projects whatever it gets wrong then, so the targets follow the current policy. Figure 2 summarizes one round of our approach. Sec. 4.3 compares rebuilding the targets each round against reusing a static correction pool, and Sec. 4.4 compares our loss (2) against five alternatives on the same data.

Experiments

First, we compare projected distillation with standard post-training approaches in the hard-task regime, and ask whether what a student learns on one benchmark’s hard set carries over to the hard problems of other benchmarks (§ 4.1). The remaining sections examine the recipe itself, each component of which could have been set differently: the correction may change more or fewer of the attempt’s tokens (Sec. 4.2), it may be computed once or recomputed as the student learns (Sec. 4.3), and several objectives may be applied to it (Sec. 4.4). We vary each in turn while holding the remainder fixed, then ask whether the projections can be improved further by test-time scaling (Sec. 4.5), and finally whether the same model can serve as its own projector when given privileged information (Sec. 4.6).

Experimental Setup: We study two instruction-tuned students from different families, Qwen3-8B (Qwen Team, [25]) and OLMo-2-7B (OLMo Team, [24]), on five benchmarks spanning math, code, tools and science: GSM8K (Cobbe et al., [6]), MATH (Hendrycks et al., [12]), LiveCodeBench v6 (Jain et al., [14]), SciKnowEval (Feng et al., [7]), and ToolAlpaca (Tang et al., 2023). For each (student, benchmark) pair we train only on its hard set: the problems on which the base student fails all $k=16$ sampled attempts at temperature 0.8. The hard set is mined from the benchmark’s training split (Appendix § 9.§ 2), and retention is measured in-domain on a disjoint held-out slice of the same benchmark. All methods use matched budgets for optimization updates and per-step generation. The remaining configuration is in Appendix § 9.§ 1, with prompt templates in Appendix § 9.§ 3 and further training details in Appendix § 9.§ 4. Unless noted, results are averaged over 3 training seeds, and Appendix § 9.§ 5 separates training-seed from evaluation-seed spread. Appendix § 10.§ 3 additionally evaluates the trained checkpoints on suites they never saw, and Appendix § 10.§ 5 on the hard problems of benchmarks they never saw.

Baselines. We compare against four baselines, each probing a different axis of the post-training design space. GRPO (Shao et al., 2024) performs on-policy RL with verifiable rewards, where positive rewards are scarce. OPD (Agarwal et al., [1]) performs white-box distillation from the token-level logits of a strong teacher conditioned on the student’s own prefixes, using a same-family 32B teacher (Qwen3-32B or OLMo-2-32B) for each student; our method uses these same 32B teachers as its projection models, and both methods are given the gold answer as privileged information. Surgical Post-Training (SPOT; Lin and Han, [21]) minimally corrects the student’s initial attempts and trains on the resulting frozen preference data, while Semi-On-policy Distillation (SODA; Chen et al., [5]) fine-tunes the student on teacher corrections and then contrasts a teacher response against a frozen copy of that fine-tuned student. Against SPOT and SODA every method draws its corrections from one shared black-box model, which isolates the learning algorithm from the projector.

Metrics. We report specialization, pass@1 on the hard set, and retention, pass@1 on the held-out slice. A method can raise either at the other’s expense, so neither effectively ranks methods on its own. Our headline metric is the area under the specialization–retention Pareto frontier (AUPF), the retention a method sustains as it specializes further, anchored at a shared untrained base. AUPF is low both for a method that never specializes and for one that specializes by giving up retention. We report it alongside three continual-learning summaries of the retention trajectory, backward transfer (Lopez-Paz and Ranzato, [23]), forgetting (Chaudhry et al., [3]) and training-averaged accuracy (Kemker et al., [16]), and retention-safe specialization, the highest specialization reached while retention stays within $\varepsilon$ of base. Full definitions are in Appendix § 10.

RQ1: How does projected distillation compare with baselines?

We compare against the four baselines, drawing our corrections first from a same-family 32B teacher and then from a shared black-box projector, and report AUPF in Table 1, the continual-learning metrics in Table 3, and the pooled frontiers in Figure 3.

	RLVR same-family 32B teacher			black-box projector
	GRPO	OPD	Ours	SPOT	SODA	Ours
Qwen3-8B
GSM8K	0.061±.011	0.145±.006	0.205±.012	0.209±.030	0.231±.016	0.262±.020
MATH	0.000±.000	0.003±.001	0.008±.002	0.145±.010	0.159±.009	0.035±.010
LCB	0.000±.000	0.068±.018	0.138±.003	0.095±.005	0.091±.001	0.123±.006
SciKnow	0.045±.008	0.027±.002	0.179±.005	0.124±.008	0.155±.013	0.124±.005
Tool	0.248±.152	0.128±.004	0.676±.013	0.594±.010	0.635±.014	0.667±.015
OLMo-2-7B
GSM8K	0.195±.084	0.417±.011	0.217±.014	0.186±.007	0.210±.004	0.336±.067
MATH	0.009±.003	0.014±.002	0.017±.002	0.013±.001	0.015±.002	0.016±.001
SciKnow	0.198±.022	0.060±.013	0.236±.010	0.116±.011	0.101±.001	0.234±.015
Tool	0.171±.111	0.319±.038	0.626±.018	0.222±.031	0.488±.029	0.682±.038

Table 3: RQ1: continual-learning metrics, pooled over students, benchmarks and seeds (mean $\pm$ s.e.), with arrows marking the better direction. $\bar{x}$ and $\bar{y}_{\mathrm{norm}}$ are specialization and retention averaged over training, BWT is backward transfer against the base, $F$ the drop from a run’s retention peak, and SafeSpec$_{\epsilon}$ the most specialization reached at any checkpoint whose retention is within $\epsilon$ of base (Appendix § 10). Bold = best, _underline_ = second best per column. Our variants specialize most while holding retention at base, where the baselines either barely specialize or forget.

Method	$\bar{x}\mathord{\uparrow}$	$\bar{y}_{\mathrm{norm}}\mathord{\uparrow}$	BWT$\mathord{\uparrow}$	$F\mathord{\downarrow}$	SafeSpec$_{2\%}\mathord{\uparrow}$	SafeSpec$_{5\%}\mathord{\uparrow}$
GRPO	$0.114\pm0.034$	$0.826\pm0.048$	$-0.203\pm0.058$	$0.221\pm0.057$	$0.035\pm0.012$	$0.056\pm0.022$
OPD	$0.118\pm0.022$	$0.951\pm0.031$	$-0.062\pm0.043$	$0.136\pm0.041$	$0.101\pm0.019$	$0.135\pm0.026$
SPOT	$0.184\pm0.031$	$0.706\pm0.074$	$-0.380\pm0.082$	$0.509\pm0.064$	$0.147\pm0.020$	$0.149\pm0.020$
SODA	$0.224\pm0.031$	$0.776\pm0.053$	$-0.266\pm0.059$	$0.355\pm0.044$	$0.140\pm0.021$	$0.190\pm0.022$
Ours (32B)	$0.264\pm0.042$	$1.000$\pm0.020$ | $0.009$$\pm0.028$ | $0.084$$\pm0.015$ | $0.202$$\pm0.040$ | $0.295$$\pm0.048$
Ours (bb)	$0.276$$\pm0.042$	_0.995_ $\pm0.024$	$-0.004\pm0.034$	_0.107_ $\pm0.027$	_0.160_ $\pm0.037$	_0.235_ $\pm0.040$

Projected distillation attains the highest AUPF on eight of nine cells against GRPO and OPD and on seven of nine against SPOT and SODA (Table 1), under both projector settings. The margins are largest on SciKnowEval and ToolAlpaca, reaching 0.179 against OPD’s 0.027 on Qwen SciKnowEval and 0.676 against 0.128 on Qwen ToolAlpaca. The baselines are not short of specialization; SPOT and SODA both reach real hard-set accuracy (Table 3). However, they retain poorly. In Figure 3 our frontier runs flat along the untrained base across the whole specialization range while every baseline’s falls away from it. Pooled across projectors, our mean AUPF is 0.256 with the mid-capacity same-family 32B teacher and 0.275 with the black-box projector, so the mid-capacity projector performs slightly worse. In Sec. 4.5, we investigate how to improve learning with such a teacher.

The two baseline families fall short for different reasons. GRPO has little to learn from, since the hard set is mined at pass@k=0; continued sampling still recovers occasional correct rollouts, enough for the highest specialization of any method on OLMo SciKnowEval, but its retention is the lowest there as well (Table 8), because reinforcing a rare correct rollout sharpens the student onto that solution at the expense of what it produces elsewhere (Yue et al., 2025). SPOT and SODA hold correct targets throughout, so they are not limited by correctness. They are rather limited by how far the correction moves the attempt and how often it is rebuilt, as shown further in sections 4.2 and 4.3. Three AUPF result cells are not in our favor. On Qwen MATH, SPOT and SODA fit the hard-set answers more closely, SODA reaching 0.136 specialization against our 0.031, and give up some retention for it (0.664 against our 0.760; Table 8). On Qwen SciKnowEval SODA leads on both axes, in the one cell whose retention carries a standard error of 0.131 across seeds, several times that of any other method there. On OLMo GSM8K, OPD reaches 0.496 specialization against our 0.236 at comparable retention, so token-level feedback apparently keeps its effectiveness there.

Table 8: RQ1 per-cell final accuracies at the 250-step checkpoint, mean ± s.e. across seeds. Hard = hard-set pass@1 (specialization); Held = held-out pass@1 (in-domain retention), measured on a disjoint slice of the same benchmark’s training split, with sizes in Tabs. 6 and 7. These slices are not the public test suites used in Tab. 11, so the two are not comparable cell by cell. Higher is better on both, and a method is preferable where it raises Hard without lowering Held. Bold = best on each axis within a (student, benchmark) block. Our two variants take the best Hard in 6 of 9 blocks and the best Held in 5 of 9, so the AUPF ordering in Tab. 1 rests on the two axes together rather than on either alone. † This is the one cell reported from a single training seed: OLMo-2-7B/MATH GRPO seeds 0 and 1 stopped with CUDA out-of-memory before the 250-step checkpoint, so only seed 2 is reported and the ± there is the standard error of one value, not of three. Every other cell is three seeds.

Student	Benchmark	Method	Hard ↑	Held ↑
Qwen3-8B	GSM8K	GRPO	0.044 ± 0.012	0.950 ± 0.006
Qwen3-8B	GSM8K	OPD	0.123 ± 0.013	0.953 ± 0.007
Qwen3-8B	GSM8K	Ours (32B)	0.197 ± 0.019	0.943 ± 0.019
Qwen3-8B	GSM8K	SPOT	0.090 ± 0.005	0.216 ± 0.039
Qwen3-8B	GSM8K	SODA	0.246 ± 0.021	0.604 ± 0.140
Qwen3-8B	GSM8K	Ours (bb)	0.273 ± 0.022	0.943 ± 0.009
Qwen3-8B	MATH	GRPO	0.000 ± 0.000	0.627 ± 0.003
Qwen3-8B	MATH	OPD	0.002 ± 0.002	0.633 ± 0.017
Qwen3-8B	MATH	Ours (32B)	0.007 ± 0.004	0.700 ± 0.015
Qwen3-8B	MATH	SPOT	0.087 ± 0.008	0.258 ± 0.028
Qwen3-8B	MATH	SODA	0.136 ± 0.023	0.664 ± 0.002
Qwen3-8B	MATH	Ours (bb)	0.031 ± 0.004	0.760 ± 0.006
Qwen3-8B	LiveCodeBench	GRPO	0.000 ± 0.000	0.371 ± 0.000
Qwen3-8B	LiveCodeBench	OPD	0.093 ± 0.032	0.367 ± 0.018
Qwen3-8B	LiveCodeBench	Ours (32B)	0.325 ± 0.014	0.387 ± 0.013
Qwen3-8B	LiveCodeBench	SPOT	0.217 ± 0.003	0.413 ± 0.027
Qwen3-8B	LiveCodeBench	SODA	0.237 ± 0.009	0.313 ± 0.041
Qwen3-8B	LiveCodeBench	Ours (bb)	0.308 ± 0.008	0.353 ± 0.007
Qwen3-8B	SciKnowEval	GRPO	0.067 ± 0.024	0.560 ± 0.000
Qwen3-8B	SciKnowEval	OPD	0.027 ± 0.004	0.567 ± 0.026
Qwen3-8B	SciKnowEval	Ours (32B)	0.293 ± 0.015	0.607 ± 0.022
Qwen3-8B	SciKnowEval	SPOT	0.164 ± 0.026	0.596 ± 0.004
Qwen3-8B	SciKnowEval	SODA	0.227 ± 0.033	0.573 ± 0.037
Qwen3-8B	SciKnowEval	Ours (bb)	0.204 ± 0.019	0.497 ± 0.131
Qwen3-8B	ToolAlpaca	GRPO	0.249 ± 0.154	0.937 ± 0.009
Qwen3-8B	ToolAlpaca	OPD	0.078 ± 0.039	0.463 ± 0.274
Qwen3-8B	ToolAlpaca	Ours (32B)	0.720 ± 0.010	0.897 ± 0.015
Qwen3-8B	ToolAlpaca	SPOT	0.691 ± 0.006	0.678 ± 0.073
Qwen3-8B	ToolAlpaca	SODA	0.667 ± 0.023	0.691 ± 0.067
Qwen3-8B	ToolAlpaca	Ours (bb)	0.716 ± 0.008	0.847 ± 0.023
OLMo-2-7B	GSM8K	GRPO	0.220 ± 0.070	0.637 ± 0.048
OLMo-2-7B	GSM8K	OPD	0.496 ± 0.016	0.803 ± 0.015
OLMo-2-7B	GSM8K	Ours (32B)	0.236 ± 0.008	0.830 ± 0.006
OLMo-2-7B	GSM8K	SPOT	0.115 ± 0.045	0.313 ± 0.153
OLMo-2-7B	GSM8K	SODA	0.153 ± 0.007	0.527 ± 0.020
OLMo-2-7B	GSM8K	Ours (bb)	0.378 ± 0.091	0.774 ± 0.018
OLMo-2-7B	MATH	GRPO†	0.020 ± 0.000	0.330 ± 0.000
OLMo-2-7B	MATH	OPD	0.013 ± 0.004	0.370 ± 0.025
OLMo-2-7B	MATH	Ours (32B)	0.036 ± 0.008	0.307 ± 0.009
OLMo-2-7B	MATH	SPOT	0.009 ± 0.006	0.133 ± 0.020
OLMo-2-7B	MATH	SODA	0.011 ± 0.002	0.151 ± 0.008
OLMo-2-7B	MATH	Ours (bb)	0.016 ± 0.006	0.380 ± 0.010
OLMo-2-7B	SciKnowEval	GRPO	0.658 ± 0.037	0.217 ± 0.047
OLMo-2-7B	SciKnowEval	OPD	0.113 ± 0.024	0.353 ± 0.033
OLMo-2-7B	SciKnowEval	Ours (32B)	0.573 ± 0.023	0.447 ± 0.003
OLMo-2-7B	SciKnowEval	SPOT	0.224 ± 0.008	0.431 ± 0.019
OLMo-2-7B	SciKnowEval	SODA	0.089 ± 0.016	0.371 ± 0.002
OLMo-2-7B	SciKnowEval	Ours (bb)	0.564 ± 0.035	0.424 ± 0.024
OLMo-2-7B	ToolAlpaca	GRPO	0.171 ± 0.171	0.140 ± 0.140
OLMo-2-7B	ToolAlpaca	OPD	0.273 ± 0.015	0.840 ± 0.015
OLMo-2-7B	ToolAlpaca	Ours (32B)	0.798 ± 0.006	0.707 ± 0.047
OLMo-2-7B	ToolAlpaca	SPOT	0.193 ± 0.112	0.069 ± 0.049
OLMo-2-7B	ToolAlpaca	SODA	0.220 ± 0.070	0.189 ± 0.059
OLMo-2-7B	ToolAlpaca	Ours (bb)	0.856 ± 0.026	0.749 ± 0.036

Training on a hard set risks catastrophic forgetting due to limited model capacity, so Table 3 reports four continual-learning measures, each summarizing a whole run. Training-averaged specialization and retention give the mean of each axis over the run: ours is highest on specialization ($\bar{x} = 0.264$ and 0.276 against SODA’s 0.224) and holds retention at base ($\bar{y}_{\mathrm{norm}} = 1.000$ and 0.995 against 0.776), so what it gains is not paid for out of retention during training. Backward transfer is the net change in retention from the untrained base to the final checkpoint, and ours are the only values that are not significantly negative (0.009 and −0.004). Forgetting is the drop from a run’s highest retention to its final retention, so it registers ability lost during training even when the endpoint looks acceptable: ours are 0.084 and 0.107 against 0.509 for SPOT and 0.355 for SODA, which specialize early and lose retention through the rest of training (Figure 10). Retention-safe specialization, the most a run reaches at any checkpoint whose retention is within $\varepsilon$ of base, is highest for both our variants at either tolerance. Retention here is measured on a held-out slice of the trained benchmark; Appendix § 10.§ 3 evaluates the math-trained checkpoints on suites they never saw.

Transfer to other benchmarks’ hard problems. We test whether performance improvement on one benchmark’s hard problems extends to hard problems from other benchmarks. For this experiment, we train Qwen3-8B with each black-box method and with GRPO on 501 MATH problems that it fails on all 16 attempts. We evaluate it on 127 held-out hard MATH problems not seen during training, and on 2,885 hard problems from OlympiadBench (He et al., [10]), Omni-MATH (Gao et al., [8]), AIME 1983–2023 and DAPO-Math (Yu et al., 2025). More details of the setup are provided in Appendix § 10.§ 5.

We report the results in Table 2. Projected distillation shows stronger performance than the baselines in hard-to-hard generalization. On the held-out MATH problems it raises pass@1 from 0.013 to 0.056 and pass@16 from 0.150 to 0.354, against 0.031 and 0.291 for the best baseline. On the four other benchmarks it raises pass@1 from 0.011 to 0.020 (+0.009 $\pm$ 0.001) and pass@16 from 0.114 to 0.154, where the best baseline gains 0.005 in pass@1 and SPOT and GRPO gain nothing.

Method	MATH held-out, hard (127)		Other benchmarks, hard (2,885)
	pass@1	pass@16	pass@1	pass@16
Base	0.013	0.150	0.011	0.114
GRPO	0.020	0.205	0.011	0.108
SPOT	0.020	0.197	0.012	0.114
SODA	0.031	0.291	0.016	0.127
Ours	0.056	0.354	0.020	0.154

What the student learns on one benchmark’s hard problems therefore carries to hard problems it never trained on. In Appendix § 10.§ 5, we provide a breakdown of the gain across individual benchmarks and an analysis of a projected distillation model trained on Omni-MATH.

RQ2: How does edit size influence learning in projected distillation?

A correction may be correct and still lie far from what the student would have produced, and this question concerns the cost of that distance. We construct four arms that hold correctness fixed and vary only how much the correction rewrites the attempt: a minimal edit, two progressively larger rewrites obtained by re-prompting the projector, and gold, the dataset’s GSM8K reference solution, which discards the attempt altogether. Each arm trains a separate Qwen3-8B student from the same base under identical hyperparameters, so only the target $y^{+}$ varies; the rejected side remains the student’s own wrong rollout $\bar{y}^{-}$. The projector rather than the prompt settles how much text is replaced, so we identify the arms by the edit size they realize and measure the distance a correction imposes by the likelihood shift it demands, $\Lambda= \log\pi_{\theta}(y^{+} \mid x) - \log\pi_{\theta}(y^{-} \mid x)$, reported as $\lvert\Lambda\rvert$.

Figure 4 shows that among the three arms that differ only in edit size, relative specialization falls to 0.54 and then 0.45 of the minimal-edit arm and retention from 0.916 to 0.807, while $\Lambda$ falls from $-86$ to $-433$. The arms are equally correct, so what the larger rewrites cost is distance from the student. Gold sits outside that ordering: its edit is 926 tokens, between the two rewrites, yet it specializes better than either (0.87 of the minimal edit) and returns the lowest retention of any arm (0.744). The token count understates its distance, since gold keeps none of the attempt and counts small only because a reference solution is short; its likelihood shift does not say the same: at $\Lambda= +3969$ it is the one arm the student finds more likely than its own attempt, the sign opposite to every rewrite, and that value is dominated by the length difference rather than lying on the rewrites’ scale. We therefore read edit size as the quantity governing the trade-off among matched rewrites, with the minimal edit best balanced, and report _gold_ on the behavioral axes only; Appendix § 11 gives the per-arm values and the mechanism measurements.

RQ3: How much does refreshing corrections online help?

Projected distillation recomputes its corrections at every round, so that each target is an edit of an attempt the current student has just produced. Naturally, we ask what role this mechanism plays in projected distillation learning. For comparison, we gather a _static pool_: minimal-edit corrections of the initial checkpoint’s responses, mined once before training and reused for the remainder of the run. This is the same regime occupied by methods like SPOT [21] and LeMa [2]. The projector for both static and online-refreshed arms is the same black-box projector. We report results pooled over all model$\times$benchmark cells and three seeds in Figure 5.

Refreshing the corrections is essential. Online refresh improves specialization steadily throughout training, reaching 0.37 pass@1, whereas the static pool rises over the first few updates and then flattens near 0.20 (Figure 5, left). Retention separates more sharply: online refresh holds it at base for the whole run, while the static pool climbs slightly above base over the first 25 steps, falls to 0.76 by step 50, and declines from there to 0.60 (Figure 5, right). The fall sets in once the pool has been reused for several rounds. A static pool cannot follow the student to the failures it develops afterwards, so repeated passes fit it to a fixed set of targets, while its generalization degrades.

RQ4: How best to learn from projections of wrong responses?

In a typical round of projected distillation, we have access to wrong attempts $y^{-}$, their corrections $y^{+}$, and correct student responses $y^{c}$. These three quantities admit several learning objectives, and in this section we study an effective objective for learning. The simplest is supervised fine-tuning on the projection, optionally combined with a term on the student’s own correct responses. Keeping only that second term is rejection fine-tuning (RFT; Yuan et al., 2023), which learns from the student’s successes alone and ignores the projection. Further, because the wrong attempt is also retained, a contrastive objective is available: DPO (Rafailov et al., [26]) over the full response, again optionally combined with the self-SFT term. Since a minimal edit leaves most of the response unchanged, the contrast may instead be localized to the edited spans ((1)). We compare the six resulting recipes on the GSM8K hard set and report the outcome in Figure 6.

The projection is necessary: training on the student’s own correct responses alone recovers 0.16 of the adopted recipe, since the hard set is mined at pass@$k$=0 and the student’s rollouts stay correct on about 6% of attempts through the run, so RFT has almost nothing to train on. Among objectives that do use the projection, the contrast is the more reliable choice: SFT on the projection reaches 0.81 on average but carries a standard error of 0.078 across seeds, three to sixteen times that of any contrastive arm. The self-SFT term then settles the ordering among those arms. A contrast constrains the likelihood of $y^{+}$ only relative to $y^{-}$, and a localized one does so over a handful of tokens, so both gain from a term that raises the likelihood of correct text outright, and the localized one gains more: without it the diff-localized contrast underperforms the full-sequence one (0.73 against 0.78), and with it the ordering reverses (1.00 against 0.92). We adopt the diff-localized contrast with self-SFT as the default recipe (§ 3); per-arm objectives and absolute values are in Appendix § 12.

RQ5: Can we improve the quality of projections for better learning?

In this section, we explore ways to make projections from a mid-capacity projector more effective. The natural approach would be to fine-tune the projector; we instead take a computationally efficient test-time route and ask a frozen Qwen3-32B to refine its own correction over several rounds. At each round it is re-prompted with its previous correction and the spans it edited, and across rounds we retain the valid correction most likely under the base student, measured by log-likelihood over the edited tokens. We sweep the refinement budget over $K \in\{0,1,2\}$ on a Qwen3-8B student and the GSM8K hard set with the default recipe.

Refinement improves the corrections in two ways (Figure 7). More of the student’s failures become correctable, the correction rate rising from 9.2% at $K=0$ to 15.4% at $K=1$ and 16.7% at $K=2$ (Figure 7, left), so a larger part of the hard set contributes a training pair at all. At the same time the edits become smaller, from 256 to 130 tokens (Figure 7, right), which by the account of Sec. 4.2 should make each target easier to learn from. Downstream, hard-set pass@1 rises from 0.37 to 0.44 while retention stays near 0.91–0.93, with most of the gain arriving in the first round. Test-time refinement can therefore improve the projections, leading to effective performance. Additional details are in Appendix § 13.

RQ6: Can we use the same model for projection given privileged information?

Use of the same model to provide the learning signal is a popular choice in post-training. One well-known example is on-policy self-distillation (OPSD) (Zhao et al., 2026). Similarly, in this section, we test how much we can gain by replacing external projections with the model’s own self-projections given the ground truth answer. We experiment with Qwen3-8B on the GSM8K dataset. The full setup is provided in Appendix § 14.

Table 4 reports the result. Self-projection does produce usable corrections: the student lifts its hard set from 0.000 to 0.219 while retention stays at 0.924, so a model that is told the answer can repair enough of its own reasoning to learn from it. It does not, however, substitute for a stronger projector. The black-box projector reaches 0.560 on the same problems, and even the student-agnostic gold solution reaches 0.489. Concurrent work [15] proposes a self-projection scheme of its own, in which an MCMC sampler has the student rewrite its response with a reference solution in view, and Appendix § 15 compares it against black-box projection on its own math benchmark.

Projector	Hard set	Held-out
none (untrained student)	0.000	$\approx$0.955
the student itself	$0.219_{\pm0.012}$	$0.924_{\pm0.021}$
gold solution	0.489	0.744
black-box projector	0.560	0.916

A Patch-MDP Analysis of Projected Distillation

Setup. We model generation as a deterministic patch MDP with a token budget $H$ (Definition 2). A state is a prompt together with the tokens generated so far. An action is a patch, a span of tokens appended to that state, and every token carries a reward in $[0,1]$. A patch may span several tokens, but the reward and every bound below decompose token by token (Lemma 1), so decisions are indexed by token position throughout and $H$ counts them. The student is a policy $\pi$ in a class $\Pi$, with expected return $J(\pi)$ and optimum $J^\star$. Choosing action $w$ at state $s$ forfeits some return against the best action available in $s$. We denote this optimality gap with $\Delta(s,w)$, which is upper-bounded by a constant $\Delta_{\max}$ (Definition 3). Owing to telescoping (Lemma 1), a full rollout’s shortfall can be accounted for one token at a time. The regret $J^\star-J(\pi)$ then equals $H$ times the average per-token optimality gap, taken over $\pi$’s state occupancy (Lemma 2). We write $m$ for the edit size (Definition 4): the tokens the correction changes relative to the attempt, counted over the $H$ decisions Definition 2 gives both and so including the padding that equalizes their lengths. It relates to the token diff $|E^+|+|E^-|$ of § 3, the quantity Sec. 4.2 reports, by $m=|E^+|+|E^-|+\lvert|E^+|-|E^-|\rvert=2\max(|E^+|,|E^-|)$, the extra term being the change in answer length. Appendices J.1 and J.2 give the full treatment.

Learning protocol. Training runs for $M$ rounds. Each round is one batch of rollouts, the corrections made to them, and the gradient update that follows. $\pi_i$ is the policy entering round $i$. Ideally we would query the projector at every student mistake as the student generates a response. That would let us minimize the gap under the student’s own state occupancy. This is exactly the DAgger (Ross et al., [27]) approach, where an expert labels states under the learner’s own occupancy, leading to a learning regret of $O(H)$, linear in the horizon. Per-mistake labeling as an LLM student responds is infeasible, since it demands a projector call and a forward pass at every mistake. In projected distillation, we instead correct the rollout once and perform a single forward pass over the response. This minimizes the optimality gap under a state occupancy drifted away from the student’s own.

Fitting error. The projector edits the student’s wrong response such that for each state in the rewritten response, the next token has zero optimality gap. We define two further quantities: (i) the policy fit error $\tilde{\varepsilon}$, the irreducible error in fitting the best policy in the class to the expert projections; and (ii) the optimization term $\tilde{\gamma}_{M}$, the student’s excess disagreement over that best policy, which vanishes for a no-regret learner (Definition 5). Since a single disagreement costs at most $\Delta_{\max}$, the gap loss at these prefixes is bounded by $\Delta_{\max}$ times the disagreement rate (Lemma 3).

We assume that the drift caused by earlier token edits at a given decision point has a smooth, bounded effect on the optimality gap. We denote the maximum per-token effect on the optimality gap by $\eta$.

The first term collects the fitting and optimization costs. The second is drift, which does not decrease with $M$; a smaller edit is the only lever on it.

Horizon regimes. For a minimal edit the edit size is $m=O(1)$, making the drift $O(\eta H)$ and hence the regret $O(H)$, linear in the horizon. A complete rewrite of the response leads to $m=\Theta(H)$, making the drift $O(\eta H^2)$ and therefore the regret $O(H^2)$, quadratic in the horizon (Corollary 1). In a controlled experiment in Sec. 4.2 we vary $m$ (Definition 4) and observe that the student’s performance suffers as the edit size increases. The two endpoints recover the rates of DAgger and of behavior cloning, though not those algorithms themselves.

Conclusion

We introduced projected distillation, a post-training recipe to enable lower-capacity LLMs to solve hard problems beyond their current reach. We demonstrated that standard on-policy post-training methods such as RLVR and OPD struggle to specialize on the hard task, while training on gold solutions leads to catastrophic forgetting. Our proposed approach avoids these issues by maintaining rich learning targets close to the student’s current policy. This is done by iteratively transforming the student’s failed attempts into correct solutions through minimal edits. Our extensive empirical analyses showed that our approach achieved a better specialization–retention Pareto frontier than previous approaches. We showed that it is important to learn from minimal edits rather than from larger edits or full rewrites, and from the mistakes of the latest checkpoint rather than only those of the base model. Theoretical analysis with a smoothness assumption on the effects of token edits described a spectrum where minimal edits lead to DAgger-like linear-in-horizon regret, while full rewrites lead to behavior-cloning-like quadratic regret. Thus, our work provides empirical and theoretical evidence on the utility of the underlying continuum of on-policyness for continual learning on hard tasks.

Limitations and future work. We instantiate the projection as a minimal token edit; exploring likelihood-based alternatives is also an important future direction. The uniform per-token sensitivity assumed in Assumption 1 may be strong and needs further investigation. Our recipe assumes a cheap correctness check, leaving open the domains without a readily available verifier. We also study single-turn tasks, where the student produces one response and the projector edits that response. Multi-turn tasks, in which a failure can originate several turns before it becomes visible in the outcome, would require deciding which turn to project, and we leave this to future work.

Notation

Table 5 consolidates the notation used throughout the paper, with each symbol’s meaning and where it is defined or used.

Problem, student, verifier (§ 3)
$x$	problem / prompt (drawn from a problem distribution)
$y \in\mathcal{V}^{\le H}$	answer: a token sequence of length $\le H$
$\pi_{\theta_S}$	student policy (autoregressive, parameters $\theta_S$)
$\pi_{\mathrm{ref}} = \pi_{\theta_{S,0}}$	frozen initial student, used as the DPO reference
$R(x,y) \in\{0,1\}$	verifier (marks an answer correct or not)
$C(x)$	set of correct answers $\{y : R(x,y) = 1\}$
$\operatorname{pass}@k=0$	problems on which all $k$ sampled attempts are wrong
Projection and objective (Definition 1, Equation (1))
$\Lambda$	likelihood shift the correction demands of the student
$\operatorname{proj}(x)$	projection: the most likely correct answer (Definition 1)
$y^{-}=y$	student’s wrong attempt (rejected)
$y^{+}$	the projector’s minimal-edit correction (chosen)
$y^{c}$	student’s own correct rollout (self-SFT target)
$E^{-}, E^{+}$	mistake span (unlearn) and fix span (learn) from the token-diff
$\beta$	DPO contrast strength (temperature)
$\mathcal{L}_{\mathrm{Diff\text{-}Local\text{-}DPO}},\mathcal{L}_{\mathrm{SFT}},\mathcal{L}$	diff-local contrast; self-SFT; full objective
Patch MDP (Definition 2, Appendix § 16.§ 1)
$\mathcal{V}, H$	vocabulary; horizon, the token budget of a rollout
$s_t=(x,y_{1:t})$	state before decision $t$: the prompt and the first $t$ generated tokens
$w$	action: a variable-length patch (a continuation that fits the budget)
$A(s)$	admissible actions $\{w \in\mathcal{V}^{h}: 1 \le h \le H-|s|\}$
$p:s^{\prime}=s\oplus w$	deterministic transition (append the patch)
$r_{\mathrm{tok}} \in[0,1]$	per-token reward; $r(s,w)$ sums it over the patch
$\rho$	initial state distribution: the distribution over problems
$\gamma=1$	no discounting
$\pi,\pi_{\theta}$	policy; underlying token model
$V^{\star},Q^{\star}$	optimal value and action-value
$J(\pi),J^{\star}$	expected return of $\pi$; its optimum, (6)
$d^{\pi},\widetilde{d}_i$	occupancy induced by $\pi$; the corrected prefixes of round $i$
$\Delta(s,w)$	optimality gap $V^{\star}(s)-Q^{\star}(s,w) \in[0,\Delta_{\max}]$
The regret analysis (§ 5, Theorem 1)
$M,\pi_i$	rounds of training; the learner entering round $i$
$\Pi,\Theta$	policy class and its parameter set (Appendix § 16.§ 1)
$w^{\star}(s)$	the projector’s token at $s$, assumed optimal there
$m$	edit size of a correction (Definition 4); in Theorem 1, its mean over rounds
$e(s,s^{\prime})$	edit count between two histories (Definition 4)
$e_t,\bar{e}_i$	edit count at decision $t$; its mean over round $i$ (Lemma 4)
$\eta$	per-edited-token drift (Assumption 1)
$\Delta_{\max}$	largest return one decision can forfeit (Definition 3)
$L(d,\pi)$	mean gap of $\pi$ under a state distribution $d$, (7)
$\widetilde{\varepsilon}_i(\pi)$	rate at which $\pi$ disagrees with $w^{\star}$ at those prefixes
$\widetilde{\varepsilon}$	approximation error: the disagreement of the best policy in the class
$\widetilde{\gamma}_M$	optimization error: the learner’s excess disagreement over it
$D,G$	parameter-set diameter; gradient bound (Proposition 1)

Projection examples

This appendix shows complete wrong$\to$projected pairs, so the minimal-edit property can be read at the level of full solutions. In each pair the projector reuses the student’s derivation verbatim and changes only what is needed to make the answer correct; the changed spans are the token-diff $E^{-}$ (mistake) and $E^{+}$ (fix) that the diff-local DPO loss (1) contrasts. All student answers are from Qwen3-8B (base); all projections are from the black-box projector. Appendix § 8.§ 1 works through the single-edit example of Figure 8 in full, and Appendix § 8.§ 2 a multi-edit one.

A single edit: Cindy’s wishing pond

Problem. Cindy tosses 5 dimes into the wishing pond. Eric flips 3 quarters into the pond. Garrick throws in 8 nickels. Ivy then drops 60 pennies in. If Eric dips his hands into the water and pulls out a quarter, how much money, in cents, did they put into the pond? (Gold: 200.)

$y^{-}$ (rejected, wrong — #### 225)

Dimes: 5 × 10 = 50. Quarters: 3 × 25 = 75. Nickels: 8 × 5 = 40. Pennies: 60 × 1 = 60. Total = 50 + 75 + 40 + 60 = 225. #### 225

$y^{+}$ (chosen, minimal edit — #### 200)

Dimes: 5 × 10 = 50. Quarters: 3 × 25 = 75. Nickels: 8 × 5 = 40. Pennies: 60 × 1 = 60. Total = 50 + 75 + 40 + 60 = 225. Eric removes a quarter: 225 − 25 = 200. #### 200

The entire count is reused verbatim; the projector inserts the single step the student omitted (subtracting the withdrawn quarter). The token-diff is a one-span edit: E^{-} = \{\text{#### 225}\}, E^{+} = \{\text{“Eric removes. . . = 200”, #### 200}\}.

Several edits: John’s computer

Problem. John’s new computer cost $1500. He needs a monitor and other peripherals that cost 1/5 that much. He also wants to upgrade the $300 video card in the base computer with one that costs twice as much. How much did he spend? (Gold: 2100.)

$y^{-}$ (rejected, wrong — #### 2400)

Computer: $1500. Peripherals: $\frac{1}{5} \times1500 = 300$. Upgraded video card: $2 \times300 = 600$. Total: $1500 + 300 + 600 = 2400$. #### 2400

$y^{+}$ (chosen, minimal edit — #### 2100)

Computer: $1500. Peripherals: $\frac{1}{5} \times1500 = 300$. Upgraded video card: $2 \times300 = 600$. Since the base computer already includes the $300 card, he pays only the extra: $600 - 300 = 300$. Total: $1500 + 300 + 300 = 2100$. #### 2100

Here two spans change—the video-card line and the total—while the computer and peripheral steps are reused unchanged. This is a multi-edit projection: the correction threads the fix (charging only the upgrade difference) through to the final sum.

Experimental setup

Reproducibility Details

Model families and projectors. The two students are open-weight instruction-tuned models from different families, Qwen3-8B ([25]) and OLMo-2-7B ([24]), chosen so the effect is tested as a property of the method rather than of one pretraining recipe. The same-family projector shares the student’s tokenizer, as OPD requires, and unless noted it is Qwen3-32B for the Qwen student and OLMo-2-32B for the OLMo student. The black-box path (§ 3) needs only the correction text, so it is tokenizer-free and draws its corrections from a single black-box model shared across students. The exact checkpoints are: students Qwen/Qwen3-8B and allenai/OLMo-2-1124-7B-Instruct; same-family 32B teachers Qwen/Qwen3-32B and allenai/OLMo-2-0325-32B-Instruct.

Benchmarks. Five benchmarks spanning math, code, tools, and science, each contributing a mined hard set for training and a held-out slice for evaluation:

• GSM8K ([6]): grade-school math, numeric answers graded by exact match. Mine from the 7,473-example train split; evaluate on a 200-problem held-out slice of the test split.

• MATH ([12]): competition math, $\boxed{}$ answers graded by symbolic equivalence. Mine from the 12k train split; evaluate on the MATH-500 slice ([20]).

• ToolAlpaca ((Tang et al., 2023)): tool use and API calling. Mine from the 468-API, $\sim$4,255-instance release. The model must emit the first tool call for a request, and a call counts as correct when the action is a valid function of that API, its arguments parse as JSON with all required parameters present, and the action matches the reference call.

• LiveCodeBench v6 ([14]): contamination-resistant competitive-programming problems, graded by unit-test execution (pass@1). Mine hard problems from the v6 release; evaluate on a held-out slice.

• SciKnowEval ([7]): multi-level scientific knowledge. We use the L3 four-choice subset (4,135 items) of the 28,392-item release, graded by a letter match against the reference option.

Evaluation protocol. Specialization and retention are evaluated identically at step 0 and every 25 steps thereafter, so each student and benchmark serves as its own reference; the continual-learning summaries built from these two series are defined in Appendix § 10. Every benchmark is scored by the grader listed above, always as sampled pass@1 on its held-out slice, a 200-problem slice for GSM8K. The full-suite evaluations of Appendix § 10.§ 3 instead score the final checkpoints on complete test splits. Answer extraction is per benchmark: the ####-delimited final number for GSM8K, with a last-number fallback for free-form generations; the $\boxed{}$ expression for MATH, compared by symbolic equivalence; the final Answer: letter for SciKnowEval; the first Action and Action Input pair for ToolAlpaca; and the extracted program for LiveCodeBench, executed against the unit tests.

Hardware and software. The main experiments ran on a single pod of 8$\times$ NVIDIA A100-SXM4 (40 GB) GPUs, which was the compute available to us. The stack is PyTorch 2.8.0 (CUDA 12.8), TRL 1.6.0 ((von Werra et al., 2020)), Transformers 5.12.1, and Accelerate 1.14.0. Multi-GPU training uses fully-sharded data parallelism (FSDP2, full-shard) with gradient checkpointing. Final checkpoints are gathered and consolidated from the sharded (DTensor) state.

Training configuration. Unless noted, all methods train for 250 update steps at a constant learning rate with the Adam optimizer in bf16, computing loss on assistant response tokens only. GRPO and OPD use $5 \times10^{-6}$, and SPOT, SODA, and projected distillation use $1 \times10^{-5}$. The per-step generation budget is matched at 8 rollouts: the distillation methods (OPD, SPOT, SODA, and ours) draw 8 problems per step, while GRPO draws a group of 8 samples on a single problem. The DPO-based objectives use $\beta=0.1$; SODA runs a brief SFT warmup on $y^{+}$ (first 25 steps) before its preference phase. GRPO keeps TRL’s default KL coefficient of zero toward the reference policy.

Hard-Set Construction and Statistics

Mining rule. Training targets problems the student cannot already solve. For each candidate problem we draw $k=16$ samples from the untrained student at temperature 0.8 (seed 0) and keep the problem only if all 16 are graded wrong by that benchmark’s grader, i.e. pass@$16 = 0$. The retained problems form the frozen hard set; the 16 failed samples are stored with each record, which is what allows an offline arm to be built without re-generating.

Because the criterion is defined against a specific student, hard sets are per student: the same benchmark yields a different hard set for Qwen3-8B than for OLMo-2-7B, and a checkpoint is only ever scored against the hard set mined for its own base model.

Table 6 reports the mining statistics for Qwen3-8B and Table 7 the split sizes for OLMo-2-7B. Hard-set size is a property of the student, not of the benchmark: GSM8K is hard for only 1.63% of Qwen3-8B’s training pool, whereas MATH yields roughly twice as many hard problems for OLMo-2-7B (4,046) as for Qwen3-8B (1,991).

Benchmark	Screened	Hard set	Hard %	Base sample acc.	Eval slice
GSM8K	7,473	122	1.63	0.949	200
MATH	12,000	1,991	16.59	0.703	500
SciKnowEval	3,721	1,003	26.96	0.323	400
ToolAlpaca	3,584	467	13.03	0.845	300
LiveCodeBench	—	100	—	—	50

Benchmark	Hard set	Eval slice
GSM8K	230	200
MATH	4,046	200
SciKnowEval	493	200
ToolAlpaca	471	200
LiveCodeBench	810	131

Train/evaluation disjointness. We verify rather than assume separation. Comparing question text lower-cased with $\mathrm{\LaTeX}$ commands stripped and non-alphanumeric characters removed — so that a problem reformatted between two source releases is still caught, which a byte-level comparison would miss — gives zero overlap between hard set and evaluation slice for both math benchmarks: 1,991 versus 500 for MATH and 122 versus 200 for GSM8K. AMC23, MMLU (Hendrycks et al., [11]) and HumanEval are evaluation-only and were never trained on by any method. Four problems appear twice inside the MATH hard set (1,991 records, 1,987 unique) and therefore carry double weight during training.

Prompt Templates

All prompts are single user turns passed through the model’s own chat template. For Qwen3 the thinking mode is disabled in mining, training and evaluation alike, so the same decoding regime holds throughout. The instruction appended to a question is selected by the benchmark’s answer type, which fixes the output format the grader expects; the grader always follows the evaluation benchmark, not the benchmark a checkpoint was trained on.

Student prompts

The student prompt is the question followed by one of the four instructions below.

Numeric (GSM8K):

Solve the problem by showing your full step-by-step reasoning. After the reasoning, write the final answer on a new line by itself in the exact format "#### N", where N is just the final number (no units, words, or commas). Always include this #### line as the last line.

Boxed (MATH, MATH-500, AMC23):

Solve the problem by showing your full step-by-step reasoning. Put your final answer inside \boxed{...} at the end. The content of the box must be exactly the final answer (a number or a simplified expression), with no surrounding words.

Multiple choice (SciKnowEval, MMLU):

Answer the multiple-choice question. Think step by step, then write the final answer on a new line by itself as "Answer: X" where X is A, B, C, or D.

Code (LiveCodeBench, HumanEval):

Complete the Python function. Return the full function definition inside a ```python code block. Do not include tests or explanation.

Projection prompt

The projector is instructed to edit the student’s wrong attempt rather than to solve the problem afresh; this is what keeps the target near the student’s own distribution and makes the token-level diff small. The role is set per domain (“expert math tutor” for numeric and boxed answers, “expert science tutor” for multiple choice, “expert assistant for tool/function calling” for tool use).

System message:

You are an {role}. A student answered a problem incorrectly. PROJECT their attempt onto a correct solution: keep the student’s overall approach, wording, and structure as much as possible, changing ONLY what is needed to make it correct. Do NOT rewrite from scratch in your own style -- stay as close to how the student writes as possible. {end-format requirement}

User message:

Problem: {problem} Student’s incorrect attempt: {wrong attempt} Minimally edit the student’s attempt into a correct solution, staying as close to their style as possible. {end-format requirement}

The end-format requirement names the graded target exactly: End with the correct final answer, {gold}. for math, the correct option for multiple choice, and the required Action / Action Input pair for tool calling. A projection that does not end in the graded format is rejected by the verifier, so a correction cannot be accepted on the strength of its prose alone; every accepted projection has been re-graded by the same grader used for evaluation.

Code is different in kind. For programming problems the correctness signal given to the projector is execution feedback on the student’s own attempt — which tests failed, with input, expected and actual values — and not the hidden test suite:

You are an expert competitive programmer helping a student. The student wrote a solution that FAILS some tests. PROJECT their attempt onto a correct one: keep the student’s overall approach, structure, and variable names as much as possible, changing ONLY what is needed to make it pass. Do NOT rewrite from scratch in your own style -- stay as close to how the student writes as possible. Return the corrected solution as a single code block.

Training Configuration: Additional Details

Appendix § 9.§ 1 gives the models, learning rates, optimizer, per-step generation budget and hardware. This section records the details that a re-implementation additionally needs.

Generation budget per domain, and why it varies. The rollout budget must be at least the budget the hard set was mined with. A problem is in the hard set precisely because the student failed it under the mining budget, so giving the student a smaller budget during training truncates its derivation mid-proof and the problem cannot be solved even in principle. Competition MATH attempts run a median of roughly 805 tokens, with 61% needing at least 690, so a 700-token budget both pinned hard-set accuracy near zero and trained the model on truncated derivations, collapsing held-out accuracy from 0.39 to 0.16. The experiments of § 4 therefore use a uniform 2,048-token rollout budget, across both students and all five benchmarks, which is above the mining budget everywhere. Rollouts are sampled at $T=0.8$. This is also why the mean deleted span reported for the gold arm in Table 15 exceeds 700 tokens. The transfer runs of Appendix § 10.§ 5 and the replication of Appendix § 15 use their own budgets, stated in those sections.

Arm	$|E^{+}|+|E^{-}|$	$|E^{+}|/|E^{-}|$	hard	held	$\Lambda$
minimal (ours)	276	160 / 116	0.560±0.015	0.916±0.016	−86±4
3.2×	879	308 / 572	0.301±0.065	__0.847__±0.067	−372±67
3.9×	1061	359 / 702	0.254±0.021	0.807±0.069	−433±35
gold (anchor)	926	85 / 841	__0.489__±0.090	0.744±0.016	+3969±247$^{\dagger}$

Projector configuration. The black-box projector is Claude Opus 4.8, accessed through a hosted API and run off-GPU, which is what allows the whole pod to be spent on the student. Each wrong sample gets at most two projection attempts and the first verified-correct projection is kept; if neither verifies, the sample contributes no preference pair for that step. The same-family projector variant is served locally.

GPU allocation. Of the eight GPUs in the pod, seven hold FSDP2 shards of the student and one is reserved for the in-loop vLLM (Kwon et al., [19]) generator that produces on-policy rollouts and runs the periodic evaluations. Sharing the generator GPU with a training rank starves the generator and stalls the step, so the split is fixed rather than dynamic.

Loss variants. The arms and their objectives are given in Appendix § 12. Two further switches are fixed throughout: the adopted default uses unnormalized rather than token-mean-normalized DPO log-probabilities, and the auxiliary SFT term acts on the student’s own correct rollouts $y^c$, as in (2), not on the projection $y^+$.

Evaluation: Statistical Resolution and Seed Variance

Two kinds of seed. Two sources of variation are easy to conflate and differ by roughly a factor of fifty in magnitude.

• A training seed indexes an independently trained checkpoint. Different weights. The spread is large: the three math-trained projector-distillation seeds score 0.749, 0.311 and 0.688 on MATH-500.

• An evaluation seed indexes which generations were sampled from one fixed checkpoint. The model is identical across replicates. The spread is small, typically $\pm0.006$ on the large suites.

The two are reported in different places. The $\pm$ in Table 11 and Table 12 is the standard error over evaluation seeds of a single training-seed-0 checkpoint: it certifies that a measurement would reproduce under re-sampling, and says nothing about whether a different training run would land near it. Everywhere else, including Table 8 to Table 10, the $\pm$ is the standard error over the three training seeds. Claims about a method rather than about a particular checkpoint have to be read against training-seed spread, which is the dominant term.

Student	Benchmark	RLVR GRPO	same-family 32B teacher OPD	same-family 32B teacher Ours	black-box projector SPOT	black-box projector SODA	black-box projector Ours
Qwen3-8B	GSM8K	0.020±0.006	0.016±0.007	0.030±0.015	0.751±0.039	0.362±0.140	0.024±0.009
Qwen3-8B	MATH	0.001±0.001	0.030±0.012	0.027±0.015	0.533±0.030	0.133±0.004	0.010±0.006
Qwen3-8B	LiveCodeBench	0.000±0.000	0.073±0.029	0.047±0.018	0.040±0.020	0.070±0.038	0.053±0.007
Qwen3-8B	SciKnowEval	0.040±0.010	0.047±0.026	0.023±0.012	0.084±0.009	0.109±0.046	0.153±0.129
Qwen3-8B	ToolAlpaca	0.013±0.009	0.490±0.272	0.053±0.015	0.272±0.073	0.259±0.067	0.103±0.023
OLMo-2-7B	GSM8K	0.233±0.048	0.070±0.016	0.043±0.007	0.556±0.153	0.342±0.020	0.095±0.018
OLMo-2-7B	MATH	0.147±0.038	0.023±0.015	0.077±0.009	0.240±0.022	0.219±0.008	0.027±0.015
OLMo-2-7B	SciKnowEval	0.132±0.047	0.063±0.032	0.013±0.009	0.062±0.006	0.060±0.014	0.033±0.018
OLMo-2-7B	ToolAlpaca	0.704±0.140	0.030±0.006	0.139±0.048	0.789±0.048	0.669±0.058	0.098±0.037

Method	GSM8K	MATH-500	AMC23	MMLU	HumanEval
Base (Qwen3-8B)	$0.934 \pm0.003$	$0.709 \pm0.005$	$0.433 \pm0.030$	$0.702 \pm0.002$	$0.821 \pm0.002$
OURS	$0.915 \pm0.001$	$0.749 \pm0.003$	$0.492 \pm0.036$	$0.765 \pm0.001$	$0.787 \pm0.000$
OURS-32B	$0.917 \pm0.001$	$0.615 \pm0.005$	$0.400 \pm0.050$	$0.724 \pm0.001$	$0.827 \pm0.005$
SODA	$0.882 \pm0.004$	$0.590 \pm0.019$	$0.342 \pm0.022$	$0.766 \pm0.002$	$0.797 \pm0.005$
SPOT	$0.916 \pm0.002$	$0.513 \pm0.011$	$0.325 \pm0.029$	$0.750 \pm0.002$	$0.813 \pm0.014$
GRPO	$0.932 \pm0.004$	$0.700 \pm0.008$	$0.425 \pm0.029$	$0.702 \pm0.002$	$0.821 \pm0.002$
OPD	$0.927 \pm0.002$	$0.692 \pm0.005$	$0.408 \pm0.008$	$0.677 \pm0.002$	$0.815 \pm0.005$

Method	GSM8K	MATH-500	AMC23	MMLU	HumanEval
OURS	−0.019	+0.041	+0.058	<u>+0.063</u>	−0.035
OURS-32B	−0.017	−0.093	−0.033	+0.022	+0.006
SODA	−0.052	−0.119	−0.092	+0.064	−0.024
SPOT	−0.018	−0.195	−0.108	+0.047	−0.008
GRPO	−0.002	<u>−0.009</u>	−0.008	−0.001	<u>+0.000</u>
OPD	<u>−0.006</u>	−0.017	−0.025	−0.025	−0.006

Greedy decoding cannot be averaged this way. It is deterministic, so replicates return the same number and a standard deviation of exactly zero, which reports nothing about how well the mean is pinned down. Sampling one generation per problem and repeating over independent seeds gives an error bar that means something. For pass@$k$ estimated from $n > k$ samples we use the unbiased estimator of Chen et al. [4], $\operatorname{pass}@k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$. The $\operatorname{pass}@16$ figures in this paper are not estimated that way: they are computed from $n=16$ draws as the share of problems solved at least once. That indicator is unbiased for $\operatorname{pass}@16$ when $n=k$, but it carries more variance than an estimate from more draws would.

Resolution limits of the small suites. Suite size sets the granularity of any difference that can be claimed. AMC23 has only 40 problems, so one problem is 0.025 and the base standard error alone is $\pm0.030$; differences below roughly 0.06 there should not be argued. GSM8K (1,319) and MMLU (14,042) are the suites where small differences resolve.

A note on greedy reproducibility. Repeating a greedy evaluation with identical prompts and identical suite size did not return bit-identical scores. The cause is continuous-batching non-determinism in the inference server, which changes numerics enough to flip borderline argmax tokens. Greedy numbers therefore carry a small amount of unreported jitter of their own, which is a further reason to report a sampled mean with an explicit error bar.

RQ1: Comparison with baselines

Metrics

Area under the Pareto frontier (AUPF). For a single run, evaluation at checkpoints $i=1,\ldots,n$ yields points $(x_i,y_i)\in[0,1]^2$, where $x_i$ is specialization (hard-set $\operatorname{pass}@1$) and $y_i$ is retention (held-out $\operatorname{pass}@1$). A point is dominated if another point is at least as good on both axes, and the Pareto frontier is the non-dominated set

We anchor the frontier at a shared untrained base $B=(0,H_0)$: specialization 0 and retention $H_0$, the untrained held-out accuracy, both estimated as the mean step-0 value over the methods that start untrained. Specialization at the anchor is measured at step 0 rather than inferred from the mining criterion; it averages 0.012 across those methods, which we round to zero. $B$ is a forced left anchor (it is never subject to domination); using a shared base rather than each run’s own first checkpoint prevents methods that begin with an SFT warmup (e.g. SODA) from being scored against an already-elevated starting specialization. Indexing $\{B\}\cup F$ in order of increasing specialization, $0=\tilde{x}_0<\tilde{x}_1<\cdots<\tilde{x}_K$, AUPF is the area under the piecewise-linear frontier $y_F(\cdot)$ by the trapezoidal rule:

We compute AUPF per seed and report the mean $\pm$ s.e. across seeds. Because retention is integrated over specialization from the base at $x=0$, a method that never specializes ($\tilde{x}_K\approx0$) has AUPF $\approx0$; a higher AUPF means pushing specialization further while holding retention.

Continual-learning metrics (Table 3). For a run with per-step specialization $x_t$ and retention $y_t$ ($t=1,\ldots,T$) and the shared untrained base $(0,H_0)$, define

$\bar{x},\bar{y}$ are training-averaged accuracies in the spirit of the AUC metrics of Kemker et al. [16]; BWT is the backward transfer of Lopez-Paz and Ranzato [23] and $F$ the forgetting measure of Chaudhry et al. [3]. $\operatorname{SafeSpec}_{\varepsilon}$ is the highest specialization reached at any checkpoint whose retention is within $\varepsilon$ of the untrained base. For the aggregate row in Table 3, retention is expressed as a fraction of base so panels with different retention scales are commensurate: $\bar{y}_{\mathrm{norm}}=\bar{y}/H_0$, $F_{\mathrm{norm}}=F/H_0$, $\operatorname{BWT}_{\mathrm{norm}}=(y_T-H_0)/H_0$, and pooled $\operatorname{SafeSpec}$ uses $y_t/H_0\ge1-\varepsilon_{\mathrm{rel}}$ with $\varepsilon_{\mathrm{rel}}\in\{0.02,0.05\}$. Each metric is computed per seed and averaged over 3 seeds; the aggregate pools further over (family, benchmark), with GRPO on Qwen-LiveCodeBench entered as no learning.

Per-cell results

Tab. 8 gives the exact final-checkpoint accuracies behind Figure 3: hard-set pass@1 (specialization) and held-out pass@1 (retention), as mean $\pm$ standard error across seeds at 250 steps. Figure 9 breaks that figure down per benchmark, Figure 10 gives the pooled trajectories over training, and Tabs. 9 and 10 break the two continual-learning summaries of Table 3 down per cell.

Student	Benchmark	RLVR GRPO	same-family 32B teacher OPD	same-family 32B teacher Ours	black-box projector SPOT	black-box projector SODA	black-box projector Ours
Qwen3-8B	GSM8K	−0.017±0.006	−0.013±0.007	−0.023±0.019	−0.751±0.039	−0.362±0.140	−0.023±0.009
Qwen3-8B	MATH	+0.003±0.003	−0.010±0.017	+0.077±0.015	−0.366±0.028	+0.041±0.002	+0.137±0.006
Qwen3-8B	LiveCodeBench	+0.000±0.000	−0.004±0.018	+0.016±0.013	+0.042±0.027	−0.058±0.041	−0.018±0.007
Qwen3-8B	SciKnowEval	+0.007±0.000	+0.013±0.026	+0.053±0.022	+0.042±0.004	+0.020±0.037	−0.057±0.131
Qwen3-8B	ToolAlpaca	−0.013±0.009	−0.487±0.274	−0.053±0.015	−0.272±0.073	−0.259±0.067	−0.103±0.023
OLMo-2-7B	GSM8K	−0.232±0.048	−0.066±0.015	−0.039±0.006	−0.556±0.153	−0.342±0.020	−0.094±0.018
OLMo-2-7B	MATH	−0.123±0.049	−0.000±0.025	−0.063±0.009	−0.237±0.020	−0.219±0.008	+0.010±0.010
OLMo-2-7B	SciKnowEval	−0.132±0.047	+0.005±0.033	+0.098±0.003	+0.083±0.019	+0.023±0.002	+0.076±0.024
OLMo-2-7B	ToolAlpaca	−0.704±0.140	−0.004±0.015	−0.137±0.047	−0.775±0.049	−0.655±0.059	−0.095±0.036

Math-domain evaluation: in-domain and out-of-domain behavior

Sec. 4.1 measures retention on a held-out slice of the benchmark a method was trained on. Here we ask what training on one domain does to abilities elsewhere. For each math-trained Qwen3-8B method we evaluate its final checkpoint on five full test suites: GSM8K (1,319 problems) and MATH-500 (500) in the trained domain, and AMC23 (40), MMLU (14,042) and HumanEval (164), none of which any method was trained on. The metric is completion pass@1, one generation per problem at $T=0.8$ and top-$p$ 0.95. Each evaluation is repeated with three evaluation seeds and averaged, with the checkpoint held fixed: an evaluation seed changes only which generations are sampled from that one model. The reported ± is the standard error of that mean across the three replicates. Training and evaluation data are disjoint, with zero overlap between the 1,991-problem math hard set and the 500-problem evaluation slice on normalized, LaTeX-stripped text. Table 11 reports the absolute scores and Table 12 the change relative to the untrained student.

Discussion

§ 4.1 names three cells that go to a baseline; the full rows are in Table 8. On Qwen MATH, SODA reaches 0.136 specialization at 0.664 retention and SPOT 0.087 at 0.258, against our 0.031 at 0.760: both fit the hard set further than we do, and SPOT gives up most of its held-out accuracy doing so. On Qwen SciKnowEval, SODA reaches 0.227 at 0.573 against our 0.204 at 0.497, leading on both axes; our retention there carries a standard error of 0.131 across seeds, against 0.037 for SODA and 0.004 for SPOT, so that cell is the least stable in the panel. On OLMo GSM8K, OPD reaches 0.496 at 0.803 against our 0.236 at 0.830.

The OLMo-2-7B LiveCodeBench cell is omitted from every RQ1 table and figure. No method moves specialization appreciably above zero there, so a frontier anchored at the base is degenerate and the cell contributes an area near zero for every method, our own included.

Transfer to other benchmarks’ hard problems

This subsection gives the setup and the full results of the transfer experiment summarized in § 4.1 and Table 2.

Hard sets. For every problem of every pool we draw 16 samples from the untrained Qwen3-8B at $T=0.8$ (seed 0, thinking mode disabled, 3,072 new tokens) and grade them with the MATH grader. A problem is hard when all 16 are wrong, as in Appendix § 9.§ 2. The budget is set so that a hard problem is one the student cannot solve rather than one it cannot finish (Appendix § 9.§ 4). This leaves 628 of the 11,998 MATH training problems hard. We split the 628 into 501 training and 127 held-out problems, stratified by subject and level. Every other pool is used only for testing. A test problem is removed when a MATH training problem covers at least 60% of its 13-word phrases with the same answer, which removes 410 AIME and 474 DAPO-Math problems.

Training and evaluation. Each method trains Qwen3-8B on the 501 problems for 250 updates (about four passes) with the configuration of Appendix § 9.§ 1 and a 3,072-token rollout budget, saving a checkpoint every 50 updates, under one training seed. Ours, SPOT and SODA draw their corrections from the black-box projector, which corrects 1,798 of our 1,844 wrong attempts (97.5%). The base model and every checkpoint draw 16 samples per problem at $T=0.8$ and top-$p$ 0.95 (seed 1, so the base scores slightly above zero on its own hard set); pass@1 is the mean over the samples and pass@16 the share of problems solved at least once. A change against the base is marked when it is at least twice its paired standard error. The four other benchmarks contribute 2,885 hard problems in all, by benchmark in Table 13, out of 6,874 that survive filtering and decontamination.

Benchmark	Hard	Base → Ours	Change	pass@16	change
OlympiadBench	150	0.011 → 0.017	+0.006±0.004	0.133 → 0.147	+0.000±0.013
Omni-MATH	2,178	0.010 → 0.019	+0.009±0.001	0.096 → 0.132	+0.015±0.003
AIME 1983–2023	199	0.016 → 0.013	−0.003±0.003	0.146 → 0.126	−0.010±0.010
DAPO-Math	358	0.018 → 0.033	+0.016±0.004	0.201 → 0.307	+0.034±0.010
All four	2,885	0.011 → 0.020	+0.009±0.001	0.114 → 0.154	+0.015±0.003

Which benchmarks transfer. Table 13 splits the cross-benchmark gain of our best checkpoint by benchmark. It concentrates in Omni-MATH and DAPO-Math, which together hold 2,536 of the 2,885 hard problems, and it holds under sampling and under greedy decoding alike. OlympiadBench moves within its standard error and the AIME problems do not move.

A second training benchmark. To check that the transfer is not specific to MATH, we repeat the experiment for our method with Omni-MATH as the training benchmark: 501 of its 2,178 hard problems, drawn in proportion to its difficulty levels and topics, and otherwise the same recipe and protocol. The projector corrects 86 to 96% of the wrong attempts. The two runs share most test problems, so Table 14 compares them on identical sets: the 127 held-out MATH hard problems, the 1,677 Omni-MATH hard problems that neither run trained on, and the three benchmarks that neither run trained on (OlympiadBench, AIME 1983–2023 and DAPO-Math). Transfer runs in both directions and to a similar degree. Training on Omni-MATH lifts the held-out MATH hard problems about as much as training on MATH does (pass@16 0.362 against 0.354), and training on MATH lifts the Omni-MATH hard problems nearly as much as training on Omni-MATH does (pass@1 0.018 against 0.022). The Omni-MATH run transfers more to the three olympiad-style benchmarks (0.030 against 0.024). The direction of training therefore does not decide where the gain appears.

Trained on	MATH held-out, hard pass@1	MATH held-out, hard pass@16	Omni-MATH held-out, hard pass@1	Omni-MATH held-out, hard pass@16	3 others hard
Base	0.013	0.150	0.009	0.092	0.016
MATH	0.056$^{*}$	0.354	0.018$^{*}$	0.128	0.024$^{*}$
Omni-MATH	0.045$^{*}$	0.362	0.022$^{*}$	0.146	0.030$^{*}$

RQ2: Edit size

Setup and measurement

Qwen3-8B student, GSM8K hard set (mined pass@k=0, 122 problems), black-box projector, one objective shared by every arm, lr $1\mathrm{e}{-5}$, $\beta=0.1$, 250 steps, 3 seeds. The axis is realized by edit magnitude: the minimal edit, then re-prompting the projector for progressively larger rewrites of the same attempt, plus the student-agnostic gold anchor. Because the projector rather than the prompt determines how much text it actually replaces, arms are identified by their measured edit size (Table 15); the two larger arms realize $3.2\times$ and $3.9\times$ the minimal edit. Off-policyness is the log-likelihood difference between the correction and the attempt under the current student $\pi_\theta$,

and a larger $|\Lambda|$ means the correction sits farther from what the student would itself produce. We summarize $\Lambda$ by its median over training steps rather than its mean: the per-step quantity is heavy-tailed, since a rollout is scored against a policy that has advanced by up to five optimizer steps since it was sampled, and occasional large excursions dominate an average. The median is stable across seeds (Table 15) whereas the mean is not. For gold, which replaces a long attempt with a short reference solution, $\Lambda$ moves chiefly through length; Table 15 reports its value with that caveat, and it is omitted from the $|\Lambda|$ panel of Figure 4.

Ablation arms

Table 15 lists the per-arm endpoint accuracies with the off-policyness scalar $|\Lambda|$ and the mean edit-span sizes $|E^{+}|$, $|E^{-}|$ that define the arms. Both spans grow together across the edit arms, so the two are not separable here. gold is the exception in kind rather than in degree, inserting the fewest tokens of any arm while deleting the entire attempt.

Mechanism

Full-utterance and diff-localized shift. Figure 11 compares $\Lambda$ measured over the whole utterance against the same quantity restricted to the changed spans $E^{-}/E^{+}$. The two agree at $r = 0.9999$ across all 2,249 logged steps of the edit arms, with the localized quantity retaining 94 to 95% of the magnitude, because the tokens before the first edit are both identical and identically conditioned, so their contributions cancel exactly. Shared tokens after an edit do not: they are scored on prefixes that have already diverged, and they are what the remaining 5 to 6% measures. Localizing the measurement, as the objective of (1) does, therefore keeps almost all of the signal.

Edit size and the likelihood shift, step by step. Figure 12 plots every logged step of the edit arms, with the edit size realized at that step against the likelihood shift it demands. The relation is strongly monotone but not linear. Pooling the arms, the median $|\Lambda|$ rises across every edit-size decile, from 82 in the smallest to 570 in the largest (Spearman $\rho= 0.83$), while the Pearson correlation is only 0.28. Both concavity and a heavy tail depress it, and the tail dominates: dropping the largest 2% of $|\Lambda|$ raises Pearson to 0.71, whereas linearizing the concavity alone, by correlating against log $|\Lambda|$, reaches 0.50. The effect also diminishes with the size of the edit: the per-step Theil–Sen slope is 0.72 [0.63, 0.82] at the minimal edit against 0.17 [0.16, 0.18] and 0.14 [0.13, 0.15] at the two larger rewrites, and the relation holds within each arm on its own ($\rho= 0.52, 0.82, 0.78$). We therefore read edit size as a monotone indicator of the likelihood shift rather than a proportional one.

Insertion and deletion across the arms. Figure 13 reports the per-step ratio $|E^{+}|/|E^{-}|$. It falls monotonically along the axis, from 2.46 at the minimal edit through 1.33 and 0.99 at the larger rewrites to 0.13 for gold, so what changes across the arms is not only the amount rewritten but whether the correction predominantly adds to the attempt or replaces it.

The two edit-size measures. That asymmetry is also what separates the quantity Sec. 4.2 reports from the one Theorem 1 uses. Table 16 gives both for every arm.

Arm	$|E^{+}|$	$|E^{-}|$	token diff	length change	$m$
minimal (ours)	160	116	276	44	319
3.2×	308	572	879	264	1144
3.9×	359	702	1061	343	1405
gold	85	841	926	755	1681

Additional results

Specialization–retention plane. Figure 14 plots the two axes against each other. The minimal edit occupies the upper-right region, dominating both larger rewrites on both axes, while gold attains intermediate specialization at the lowest retention of any arm.

Held-out trajectory. Figure 15 tracks held-out accuracy over training. The edit arms separate in edit-size order and decline smoothly, the minimal edit holding near its base value throughout. gold instead destabilizes early, falling to 0.54 by step 75 and remaining volatile for the rest of the run before settling at 0.744.

RQ4: Loss-recipe details

Objectives

Rollout terminology. Each training step, the student rolls out attempts on the hard problems and the verifier splits them: a correct rollout $y^{c}$ is one it solves on its own ($R(x,y^{c}) = 1$); a wrong rollout $y^{-}$ is one it fails ($R = 0$). For each wrong rollout the frozen projector produces the minimal-edit projection $y^{+}$ (verified-correct, § 3), and a token-diff aligns $(y^{-}, y^{+})$ into the mistake span $E^{-}$ and fix span $E^{+}$ used by (1). The full-sequence DPO variant replaces $E^{+}, E^{-}$ in (1) with the full sequences $y^{+}, y^{-}$.

Per-arm objectives. The four contrastive arms optimize $\mathcal{L}_{\mathrm{DPO}}^{E}$ (diff-local, on the changed spans $E^{\pm}$) or $\mathcal{L}_{\mathrm{DPO}}^{\mathrm{seq}}$ (full-sequence, on $y^{\pm}$), each with or without an added self-SFT term $\mathcal{L}_{\mathrm{SFT}}(y^{c})$ on the student’s own correct rollouts. RFT optimizes $\mathcal{L}_{\mathrm{SFT}}(y^{c})$ alone (no projected target, no contrast); SFT on projection + self-SFT optimizes $\mathcal{L}_{\mathrm{SFT}}(y^{+}) + \mathcal{L}_{\mathrm{SFT}}(y^{c})$ (SFT on the projected correction $y^{+}$ plus self-SFT on the student’s own correct rollouts, no contrast).

Results

Compute (batch size), number of epochs and learning rate are matched across arms. Table 17 gives the absolute numbers behind Figure 6.

Arm	hard $\uparrow$	rel. $\uparrow$	held $\uparrow$
Diff-local DPO + self-SFT	0.601$\pm$0.005	1.000	0.938$\pm$0.002
Full-sequence DPO + self-SFT	0.552$\pm$0.022	0.918	0.911$\pm$0.014
SFT on projection + self-SFT	0.484$\pm$0.078	0.805	0.877$\pm$0.054
Full-sequence DPO	0.470$\pm$0.026	0.782	0.902$\pm$0.002
Diff-local DPO	0.440$\pm$0.015	0.732	0.920$\pm$0.014
RFT	0.096$\pm$0.011	0.159	0.944$\pm$0.002

Three things the table does not show. RFT is limited by how rarely the student succeeds unaided: the hard set is mined at pass@$k=0$, and its rollouts stay correct on about 6% of attempts through the run, so it trains on very little. The instability of SFT on the projection runs across both axes, its standard error of 0.078 on specialization and 0.054 on retention being three to sixteen and four to twenty-seven times those of any other arm. And the self-SFT term is worth +0.082 to the full-sequence contrast and +0.161 to the diff-local one in absolute pass@1: because the projection edits few tokens, a full-sequence contrast is spread over spans that are already correct while a diff-local one acts only where the edit fell, and that concentration leaves it with less absolute signal of its own, so it gains more from a term that raises the likelihood of correct text outright.

RQ5: Refinement details

Refinement protocol

A correction is valid only if a reasoning check confirms that its derivation process actually yields the answer, so a projection that reaches the right answer by an unsound route is discarded. Round 0 is a single correction pass. In each subsequent round the frozen projector is re-prompted with its own previous correction together with the spans it edited in that correction, and returns a refined correction. Across rounds we keep the valid correction that the student itself is most likely to produce, scored by the base student’s log-likelihood on the edited tokens, which selects for corrections the student can absorb rather than for the ones the projector prefers. The projector is never updated. All arms train the Qwen3-8B student for 250 steps with the default recipe of § 3 over 3 seeds.

Results

Projection quality. Refinement improves the corrections on three counts at once, and the three are linked. Re-prompting the projector with the spans it has already edited pushes it to consolidate rather than rewrite, so the correction it returns reaches the right answer more often and departs from the student’s own attempt less: the median $|\Lambda|$ falls from 38.1 to 26.2 over the two rounds, placing the refined targets nearer the student in the sense Sec. 4.2 measures, and the edit size falls with it. The $K=0$ edit size carries a standard error of 70 tokens across seeds, so the level of that row is the least settled number in the table; the ordering across budgets is what the comparison rests on.

Downstream trade-off. Figure 16 traces the attainable frontier over training for each budget. Refinement extends it: the furthest specialization reached on the frontier rises from 0.393 at $K=0$ to 0.459 at $K=1$ and 0.475 at $K=2$. The $K=2$ frontier also lies above the other two over the whole sweep, and is the only one that rises above the shared base in retention, reaching 0.960 against the base’s 0.955, while $K=0$ and $K=1$ only decline from it; the extension in specialization is therefore not paid for out of retention. At the final checkpoint (Table 18) the two budgets that refine are level on specialization and $K=2$ holds retention slightly higher; against standard errors of 0.010–0.021 neither difference is resolved, and what separates the three budgets is the first round.

Table 18: RQ5 per-budget results on the GSM8K hard set with Qwen3-8B, mean ± s.e. across 3 seeds. The first three columns describe the corrections the frozen projector supplies, averaged over training; the last two the student that learns from them, at the final checkpoint. $corr.$ is the running correction rate $\sum_t n_t^{\mathrm{corr}}/\sum_t n_t^{\mathrm{wrong}}$ plotted in Figure 7 (left), and edit and median $|\Lambda|$ are the two bars in Figure 7 (right). Bold = best per column. Every column describing the corrections improves monotonically with the budget; downstream, the specialization gain arrives in the first round.

	Corrections			Student
$K$	$corr.\ (\%) \uparrow$	$edit\ |E^{+}|+|E^{-}| \downarrow$	$median\ |\Lambda| \downarrow$	$hard \uparrow$	$held \uparrow$
0	$9.2 \pm0.2$	$256 \pm70$	$38.1 \pm1.8$	$0.366 \pm0.023$	$0.911 \pm0.014$
1	$15.4 \pm0.7$	$182 \pm16$	$34.2 \pm3.0$	$0.440 \pm0.010$	$0.909 \pm0.010$
2	$16.7 \pm1.1$	$130 \pm24$	$26.2 \pm4.8$	$0.440 \pm0.021$	$0.929 \pm0.006$

Correction rate over training. Figure 17 shows the per-step correction rate for $K \in\{0,1,2\}$. For every $K$ the rate rises early and then declines as training proceeds, because the student learns the easily-correctable problems and the wrong attempts that remain in later training are the hardest ones, which even the refined projector rarely corrects. The main-text Figure 7 reports the running (cumulative) version of this quantity.

RQ6: Self-projection details

Everywhere else in this paper the projector is a separate and stronger model. Here we make the same model write the correction. We give the student its own wrong attempt together with privileged information, the verified final answer, and ask it for a correct solution that stays close to what it wrote. That output is then used exactly as a projector’s correction would be, with the same objective and the same training loop. Nothing else changes, so the run measures how much of the method depends on the projector being stronger than the student.

The student is Qwen3-8B and the hard set is the 122 GSM8K problems it fails on all 16 attempts. The teacher is a frozen copy of the student’s own base weights, so it does not improve as the student does. We train for 250 updates over 3 seeds, with the same data, budget and evaluation as the edit-size runs of Sec. 4.2, from which the reference rows are taken. The answer is supplied with an instruction to derive it rather than assert it; without that, the model appends the answer line to unchanged wrong reasoning and the grader accepts it. We also run the same idea at the logit level, where the frozen base model scores the student’s own generation under a prompt carrying the answer and the student trains on that token distribution instead of on edited text.

Teacher	Correction enters as	Hard set	Held-out
none (base student)	–	0.000	≈0.955
the student itself	edited text	$0.219 \pm0.012$	$0.924 \pm0.021$
the student itself	token distribution	$0.046 \pm0.010$	$0.937 \pm0.003$
gold solution	full replacement	0.489	0.744
black-box projector	minimal edit	0.560	0.916

Self-projection works, but it does not match a stronger projector. It lifts the hard set from 0.000 to 0.219 while retention stays at 0.924, so an 8B model that is told the answer can repair its own reasoning often enough to learn from it. A black-box projector reaches 0.560 on the same problems, and even the gold solution reaches 0.489. Self-projection recovers about 40% of the specialization, which says that the projector’s strength, and not the privileged answer by itself, is what the method mostly relies on.

The logit version gives almost nothing, 0.046 after 250 updates. The answer does reach the teacher, and the student never sees it, so the two token distributions differ from the first step; what is missing is not information but capability. The teacher is the student’s own base weights, and being told which answer is correct does not let those weights produce a correct derivation, so the distribution it supplies over the student’s own tokens carries little the student can use. The measured loss stays between 0.003 and 0.006 throughout, and making the target denser, a pure forward KL against the teacher’s top-20 tokens rather than its top-1, roughly doubled the result from a very small base.

Two points help in reading the table. The text-level and logit-level arms inherit different learning rates from their respective families in the main experiments ($10^{-5}$ and $5 \times10^{-6}$), so each is comparable to its own column rather than to the other; the comparison against the black-box projector is at a matched $10^{-5}$. The 8B also does not edit minimally. It appends a fresh derivation rather than repairing the attempt, which is part of why it falls short: producing a small, targeted edit is precisely what the weaker projector cannot do.

Comparison with concurrent self-projection

Concurrent work ([15]) also moves the supervision toward the student, but does so by self-projection: an MCMC sampler in which the student itself, shown the problem, a reference solution and a partial response, regenerates the response block by block (32 blocks of 58 tokens, 10 proposals per block), accepting a proposal when it raises the student’s mean log-likelihood. The student is then fine-tuned on the resulting traces (Sampling SFT), optionally followed by GRPO (Sampling SFT + RL). We reproduce the math setting of that work’s Table 1 with its released code, data, training recipes and evaluation scripts, and change only where the correction comes from: the black-box projector minimally edits each wrong attempt given the reference answer (Appendix § 9.§ 3), and that work’s grader must accept the edit.

Setup. The student is Qwen2.5-3B, trained on that work’s 7,962 MATH training problems. It answers each once with that work’s prompt ($T=0.6$, 1,856 tokens) and solves 3,675; the projector corrects 4,265 of the other 4,287 attempts within two tries (4,317 calls, 11.2M tokens). We train four arms from these data, one training seed each:

• Black-box projection + SFT: that work’s SFT recipe (learning rate $5 \times10^{-5}$, 2 epochs, batch 16, cosine schedule with 10 warm-up steps, sequence ups to 2,048 tokens) on the 3,675 correct answers and the 4,265 projections, in place of its MCMC traces.

• Black-box projection + SFT + RL: that work’s RL stage on the previous arm (GRPO for 500 updates of 256 problems $\times$ 4 rollouts, learning rate $10^{-6}$, KL coefficient 0.001).

• Ours, static pool: the objective of (2) on the same data, the 4,265 pairs ($y^{-}, y^{+}$) supplying the contrast and the 3,675 correct answers the self-SFT term, for 1,985 updates of 8 records (2 passes,

as for SFT) at learning rate $10^{-5}$. As in Sec. 4.3, the corrections are mined once against the untrained student.

• Ours, online refresh: the full recipe of § 3, in which every update samples the current student on 8 problems and projects its wrong answers, for 250 updates.

Evaluation follows that work’s protocol: one sample per problem at $T=0.6$ with up to 1,856 tokens, on MATH levels 3–5 (1,024 problems), AMC (83), MATH-500 (500) and GSM8K (1,320) for the trained skill, and on SciKnowEval chemistry (600) and MMLU (lm-evaluation-harness Gao et al., [9], 0-shot) for prior capabilities. Our SFT ran in TRL rather than verl (Sheng et al., 2024), with the same settings.

Findings. SFT on black-box projections reaches a math average of 0.545 against 0.534 for Sampling SFT, and 0.579 against 0.567 once the same RL stage is added (Table 20). Within our own runs, the objective of (2) on the static pool beats SFT on the same data (0.557 against 0.545; +0.039 ± 0.014 on MATH(3,4,5)) and matches SFT + RL on MATH(3,4,5) to within noise, at 1/24 of its compute (Table 21). Prior capabilities hold for every arm: MMLU stays within 0.006 of base, and no change in Chemistry is significant.

	MATH(3,4,5)	AMC	MATH-500	GSM8K	Avg.	Chemistry	MMLU
Base (reported)	0.315	0.133	0.245	0.579	0.318	0.283	0.651
SFT	0.243	0.100	0.168	0.455	0.241	0.222	0.648
OPSD (Zhao et al., 2026)	0.267	0.084	0.332	0.524	0.302	0.242	0.652
GRPO	0.457	0.249	0.313	0.808	0.457	0.278	0.652
UFT (Liu et al., [22])	0.470	0.293	0.297	0.746	0.452	0.283	0.653
Sampling SFT	0.495	0.277	0.582	0.782	0.534	0.266	0.651
Sampling SFT + RL	0.545	0.241	0.652	0.830	0.567	0.285	0.652
Black-box projection + SFT	0.514*	0.277	0.578	0.812*	0.545	0.287	0.649
Black-box projection + SFT + RL	0.561*	0.301	0.622*	0.832*	0.579	0.268	0.645
Ours, static pool	0.553*	0.277	0.586	0.812*	0.557	0.262	0.651
Ours, online refresh	0.488*	0.277	0.536	0.805*	0.527	0.250	0.654

Arm	Student FLOPs	Against Sampling SFT
Sampling SFT	$8.4 \times10^{18}$	1×
Sampling SFT + RL	$2.1 \times10^{19}$	–
Black-box projection + SFT	$2.8 \times10^{17}$	30× less
Black-box projection + SFT + RL	$1.3 \times10^{19}$	–
Ours, static pool	$5.3 \times10^{17}$	16× less
Ours, online refresh	$6.9 \times10^{16}$	121× less

Two features of this setting bear on the online arm. The static pool covers 7,962 problems and is passed over only twice, so it never reaches the regime in which a pool has been reused for several rounds, which is where the static arm of Sec. 4.3 falls; and the online run trains on an eighth as many records. Under these conditions the static pool scores higher (0.557 against 0.527).

Compute. The MCMC sampler accounts for 98% of Sampling SFT’s compute: each proposal re-reads a 521-token prompt, regenerates part of the response and is re-scored, about $2.5 \times10^{5}$ token passes per training problem. Black-box projection uses at least 30× less compute on the student’s hardware. The RL stage costs about as much as the whole MCMC step, so it dominates both arms that include it.

Theoretical analysis

The Patch MDP: Model and Assumption

This appendix gives the model and the assumption behind § 5; Appendix § 16.§ 2 gives the proofs.

Each decision emits one token, so a rollout takes exactly $H$ of them, indexed $t = 0, \ldots, H - 1$ with $s_H$ absorbing. Then $d^{\pi}(s) = \frac{1}{H}\sum_{t=0}^{H-1} \Pr_{\pi}[s_t = s]$ is a probability distribution over states, the normalized occupancy of $\pi$.

Write $V^{\star}$ and $Q^{\star}$ for the optimal value and action-value of $\mathcal{M}$. The return of a policy and its optimum are

the second equality because an optimal policy is optimal from every state.

Definition 3 (Optimality gap). $\Delta(s, w) := V^{\star}(s) - Q^{\star}(s, w)$ is the return an action forfeits against the best one available at $s$, and $\Delta_{\max} := \sup_{s,w} \Delta(s, w)$, finite because returns lie in $[0, H]$. It is a constant of $\mathcal{M}$, and plays the role of the cost-to-go constant $u$ of Ross et al. [27].

Definition 4 (Edit count and edit size). For histories $s, s^{\prime}$,

where $\operatorname{LCS}(s, s^{\prime})$ is a longest common subsequence [13] of their generated tokens. This counts the insertions and deletions carrying one into the other, the string-to-string correction distance of Wagner and Fischer (1974) restricted to those two operations. The edit size of a correction is $m := e(y^{-}, y^{+})$, the two rollouts compared at the $H$ decisions Definition 2 gives them, absorbing padding included. It exceeds the token-diff count $|E^{-}| + |E^{+}|$ of § 3, which Sec. 4.2 measures, by exactly the change in answer length: $m = |E^{-}| + |E^{+}| + \lvert|y^{-}| - |y^{+}| \rvert$.

The learning problem. Training runs for $M$ rounds. In round $i$ the learner $\pi_i$ samples rollouts, a frozen projector rewrites the wrong ones, and the learner is scored on those rewrites and updated once. The learner ranges over a policy class $\Pi= \{\pi_{\theta} : \theta\in\Theta\}$ with $\Theta\subseteq\mathbb{R}^{d}$ convex of Euclidean diameter $D$, so minimizing over $\pi\in\Pi$ and over $\theta\in\Theta$ are the same operation; only Proposition 1 uses this.

Where the loss is read. The learner corrects a whole rollout once and scores the correction in a single forward pass, so the loss is evaluated at corrected prefixes, not at the states the learner visits. Write $\tilde d_i(\tilde s) = \frac{1}{H}\sum_{t=0}^{H-1} \Pr[\tilde s_t = \tilde s]$ for the occupancy of those corrected prefixes in round $i$, the state-visitation distribution the loss is read under and the counterpart of $d^{\pi_i}$. For any distribution $d$ over states, let

be the mean gap of $\pi$ under $d$. Performance is controlled by $L(d^{\pi_i},\pi_i)$, but the algorithm can only reduce $L(\tilde d_i,\pi)$.

The projector. The projector edits a wrong attempt into a correct one, and $w^\star(s)$ denotes its token at $s$. We idealize it as an expert: its token is optimal wherever we read it, so $\Delta(s,w^\star(s))=0$. Corollary 2 gives the bound for a projector that is suboptimal.

Definition 5 (Surrogate, realizability, average regret). The surrogate objective $\tilde\varepsilon_i(\pi)$ for policy $\pi$ is the 0–1 disagreement with the projector, measured under $\tilde d_i$:

The approximation and optimization terms are

so that $\frac{1}{M}\sum_i\tilde\varepsilon_i(\pi_i)=\tilde\varepsilon+\tilde\gamma_M$ and the learner is no-regret when $\tilde\gamma_M\to0$. Here $\tilde\varepsilon$ is a minimum over the policy class, so it measures how well the class can reproduce the projector’s tokens. It is unaffected by how much return those tokens forfeit; that enters separately, as $\bar\delta$ in Corollary 2.

The assumption

The analysis rests on Assumption 1, stated in § 5: the map $s\mapsto\mathbb{E}_{w\sim\pi(\cdot\mid s)}[\Delta(s,w)]$ is $\eta$-Lipschitz in the edit count $e(\cdot,\cdot)$, uniformly over $\pi\in\Pi$. It is the only assumption the analysis makes about the MDP. Three points about it follow.

The Lipschitz constant is uniform over $\Pi$ and over pairs of histories, but the proof invokes it only at the pairs $(s_t,\tilde s_t)$ that Lemma 4 compares, so $\eta$ may be taken as the largest shift over those pairs.

The condition is placed on the averaged gap rather than on $\Delta(s,w)$ at a fixed $w$. The proof compares $\mathbb{E}_{w\sim\pi}[\Delta]$ at two different prefixes, and moving from one to the other shifts the action distribution as well as the gap, so both must be controlled. Two pointwise conditions suffice. Suppose the gap is itself Lipschitz, $|\Delta(s',w)-\Delta(s,w)|\le\eta_\Delta e(s,s')$, and the policy is smooth in the same count, $\operatorname{TV}(\pi(\cdot\mid s),\pi(\cdot\mid s'))\le\kappa e(s,s')$. Adding and subtracting $\mathbb{E}_{w\sim\pi(\cdot\mid s')}[\Delta(s,w)]$ splits the difference in two, bounded by $\eta_\Delta e$ and, since $\Delta$ takes values in $[0,\Delta_{\max}]$ and its oscillation is therefore at most $\Delta_{\max}$, by $\Delta_{\max}\kappa e$. Together they give Assumption 1 with $\eta\le\eta_\Delta+\Delta_{\max}\kappa$.

Assumption 1 is also the only reason any result below depends on the size of an edit. An argument that knows only which decisions sit downstream of an edit pays $\Delta_{\max}$ at each of them, which is identical for a one-token edit and for a hundred-token edit at the same position: “the prefix differs” is a binary predicate and it saturates. Assumption 1 instead charges in proportion to $e(\cdot,\cdot)$, which every edit-size claim below uses.

Proofs

The gap decomposes across a patch

Lemma 1 (Patch additivity). For patch MDP, $\Delta(s,w_{1:h})=\sum_{j=1}^{h}\Delta(s\oplus w_{<j},w_j)$.

Proof. Since the transitions of Patch MDP are deterministic (c.f. Definition 2), $Q^\star(s,w_{1:h})=r(s,w_{1:h})+V^\star(s\oplus w_{1:h})$, and by Definition 2 the reward splits as $r(s,w_{1:h})=\sum_{j=1}^{h}r_{\mathrm{tok}}(s\oplus w_{<j},w_j)$ $w_{<j}, w_j$). Hence

where the last step uses $Q^\star(s\oplus w_{<j},w_j)=r_{\mathrm{tok}}(s\oplus w_{<j},w_j)+V^\star(s\oplus w_{<j+1})$.

The total gap along a rollout is therefore the same however the rollout is cut into patches; only the per-step decomposition changes.

Regret is the accumulated gap, under the learner’s own occupancy

Both expectations in $L(d^{\pi_i},\pi_i)$ are taken under $\pi_i$’s own occupancy. No change of measure is performed, so no concentrability coefficient appears.

Disagreement with the projector dominates the gap

Proof. Fix a corrected state $\tilde{s}$ and split the expectation over the projector’s token and the rest:

Taking $\mathbb{E}_{\tilde{s}\sim\tilde{d}_i}$ of both sides and applying Definition 5 gives the claim. □

This is the reduction of Ross et al. [27], applied at the corrected prefixes rather than at the learner’s own states.

Transfer from the corrected prefixes to the learner’s own

Proof. Pair the two distributions decision by decision: the $t$-th term pairs the learner’s own prefix $s_t$ with the corrected prefix $\tilde{s}_t$, which differ by $e_t$ tokens (Definition 4). By Definition 2 both sides of (7) are averages of exactly $H$ such terms, so

□

How much edited text a rollout carries

Proof. Let $u$ index the tokens of $y$ and $v$ those of $\tilde{y}$. A monotone matching between them is a set $\Phi$ of pairs $(u,v)$ with $y_u=\tilde{y}_v$, whose order agrees on both sides: $u<u' \Longleftrightarrow v<v'$ for distinct $(u,v),(u',v')\in\Phi$. The matched tokens read in order form a common subsequence of $y$ and $\tilde{y}$, so a largest matching has $|\Phi|=|\operatorname{LCS}(y,\tilde{y})|$. Take such a $\Phi$. Since $y$ and $\tilde{y}$ each have $H$ tokens, $m=2H-2|\Phi|$ by Definition 4. For a decision $t$ put $\Phi_t=\{(u,v)\in\Phi:u\leq t,\ v\leq t\}$, the pairs falling inside both prefixes, whose matched tokens form a common subsequence of $y_{1:t}$ and $\tilde{y}_{1:t}$. Both coordinates increase along $\Phi$, so $\max(u,v)$ does too, strictly; every pair outside $\Phi_t$ has $\max(u,v)\geq t+1$, and distinct pairs take distinct values there, so

Therefore

Summing over the $H$ decisions gives the claim. □

An insertion early in the answer shifts every later prefix, and a count of mismatched positions would read that shift as a rewrite of the whole suffix. Definition 4 charges it at the size of the insertion instead, which is why the drift of Theorem 1 scales with $m$ and not with $H$.

The bound

Theorem 1 of § 5 states that under Definitions 2 and 3, Assumption 1, an idealized projector and $m$ the mean edit size over the rounds,

Proof of Theorem 1.

where $\bar{e}=\frac{1}{M}\sum_i\bar{e}_i$. Lemma 5 bounds round $i$ by its own edit size, $H\bar{e}_i=E_i\left[\sum_{t=0}^{H-1}e_t\right]\leq m_iH$, so $\bar{e}\leq\frac{1}{M}\sum_i m_i=m$; no bound on the largest round is needed. □

Rate in the number of rounds

Proposition 1 (Convex reference rate). Let $\Pi$ and $\Theta$ be as in Appendix § 16.§ 1, and suppose in addition that $\theta\mapsto\tilde{\varepsilon}_i(\pi_\theta)$ is convex on $\Theta$ for every $i$, with gradient bound $G=\max_i\sup_{\theta\in\Theta}\|\nabla_\theta\tilde{\varepsilon}_i(\pi_\theta)\|$. Then projected online gradient descent on $\Theta$ with step size $\alpha=D/(G\sqrt{M})$ attains $\tilde{\gamma}_M\leq DG/\sqrt{M}$. Proof. Since $\Pi$ is parameterized by $\Theta$, the minimum over $\pi\in\Pi$ in Definition 5 is a minimum over $\theta\in\Theta$. Write $\theta^\star\in\arg\min_{\theta\in\Theta} \sum_i \widetilde{\varepsilon}_i(\pi_\theta)$ for a minimiser, so that the comparator in the definition and the comparator in the analysis are the same point. Then, with $g_i = \nabla_\theta\widetilde{\varepsilon}_i(\pi_{\theta_i})$,

(by Definition 5)

(convexity)

(potential $\frac{1}{2\alpha}\lVert\theta_i - \theta^\star\rVert^2$ telescopes)

(at $\alpha= D/(G\sqrt{M})$)

where $D$ is the Euclidean diameter of $\Theta$ and $G$ the gradient bound. Dividing by $M$ gives $\widetilde{\gamma}_M \le DG/\sqrt{M}$.

Remark 1 (Scope of the rate). Proposition 1 is the standard online-convex-optimization guarantee, stated to show how regret decays in the number of rounds. Its convexity hypothesis does not hold for a neural policy. Theorem 1 uses only $\widetilde{\gamma}_M$ as defined, so nothing below depends on Proposition 1; the $O(1/\sqrt{M})$ form should be read as a convex reference point.

Reading the bound

Scope of the analysis

Remark 2 (The analysis assumes a dense per-token reward). Definition 2 makes the reward token-additive with $r_{\mathrm{tok}} \in[0,1]$, so a rollout’s return is at most $H$ and the trivial bound on $J^\star- J(\pi)$ is $H$. The experiments verify only the final answer, a single 0–1 outcome, under which the trivial bound is 1. The drift term $\eta mH$ is therefore informative only where $\eta m$ is small: the guarantee is a statement about the dense-reward model, and the per-token credit it assumes is not what the verifier supplies.

Remark 3 (A statement about horizon dependence, not about algorithm identity). Setting $e_t \equiv0$, which corresponds to a projector queried at every mistake on the learner’s own prefix so that no edited token enters a conditioning prefix, removes the drift and leaves $H\Delta_{\max}(\widetilde{\varepsilon} + \widetilde{\gamma}_M)$, the shape of the DAgger guarantee (Ross et al. [27]). The linear–quadratic dichotomy of Corollary 1 is the same dichotomy that separates DAgger from behavior cloning. It does not follow that projected distillation is either algorithm: minimal-edit projection does not query a projector at every visited state and carries a drift term DAgger does not have, and a full rewrite is not behavior cloning for the reason in Remark 4. The correspondence is between rates.

Remark 4 (The drift does not vanish under a perfect fit). In Ross et al. [27] the quadratic term for behavior cloning scales with the fit error, so a learner that imitates perfectly suffers no compounding. The drift $\eta mH$ of Theorem 1 carries no factor of $\widetilde{\varepsilon}$. A learner that matches the projector exactly at every corrected prefix still pays it, because the mismatch between $\widetilde{d}_i$ and $d^\pi$ is exogenous, fixed by the edit that was chosen, rather than a consequence of imperfect fitting.

Remark 5 (The analyzed loss is not the deployed loss). Theorem 1 is proved for the 0–1 disagreement of Definition 5. Three consequences.

First, no gradient method minimizes a 0–1 loss. Following Ross et al. [27] we state the guarantee for the 0–1 loss and note that any convex upper bound may be substituted, with $\widetilde{\varepsilon}$ and $\widetilde{\gamma}_M$ read as the floor and regret of that surrogate; the bound holds verbatim with a larger $\widetilde{\varepsilon}$. Second, cross-entropy on the correction is such a bound. Since $\log p \le p - 1$ for $p \in(0,1]$, we have $1 - \pi(w \mid\tilde{s}) \le-\log\pi(w \mid\tilde{s})$, so Theorem 1 covers maximum-likelihood fine-tuning on the projection.

Third, the deployed objective is not covered. The recipe minimizes a diff-localized contrast on the changed spans together with a self-SFT term on the learner’s own correct rollouts. A preference contrast is not an upper bound on the disagreement with the projector: it can be reduced by lowering the likelihood of $y^{-}$ without raising that of $y^{+}$, so a small contrast loss does not imply a small $\tilde{\varepsilon}$. The analysis characterizes the target: how far the correction lies from the attempt, the quantity the edit-size experiment varies and the one both objectives share. It does not characterize the contrast.

Proof. The term $\pi(w^{\star} \mid\tilde{s})\Delta(\tilde{s}, w^{\star})$ dropped in Lemma 3 no longer vanishes, so that lemma reads $L(\tilde{d}_i,\pi) \le\bar{\delta} + \Delta_{\max}\tilde{\varepsilon}_i(\pi)$. Substituting it at the Lemma 3 step of Theorem 1 adds $H\bar{\delta}$, every term there already carrying the factor $H$ from Lemma 2, and leaves the rest of the chain unchanged.

The two error sources stay separate: $\bar{\delta}$ measures the projector and shrinks only with a better one, while $\tilde{\varepsilon}$ measures the policy class and shrinks with a larger one.