WorksComputer Science
GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL
Abstract
Training large language models for complex reasoning tasks relies on two complementary signals: sparse, outcome-grounded reinforcement learning (RL) for correctness, and dense, token-level distillation for guidance. However, complementarity in supervision does not guarantee compatibility in optimization; imitation-focused distillation can conflict with the outcome-driven RL objective, and existing hybrids lack a principled resolution. We propose GIFT (Gradient-Informed FilTering for Distillation-Augmented Reasoning RL), a novel method that directly addresses the allocation problem of which teacher updates to apply. GIFT performs a token-level geometric analysis, decomposing the distillation gradient into a component parallel to the RL gradient, which measures first-order conflict and provides a safety filter, and a component orthogonal to it, which quantifies novel directional guidance and provides a measure of value. This decomposition yields a principled scoring rule to rank token-level guidance. We combine this with a progressive coverage annealing schedule that shrinks the budget of tokens receiving distillation, ensuring that the most conflicting updates are discarded first while the most complementary ones are retained at full strength. This facilitates a smooth transition from dense, geometrically-filtered guidance to a pure, outcome-grounded objective at convergence. Extensive experiments on mathematical and logical reasoning benchmarks show that GIFT consistently outperforms strong baselines across multiple model scales, achieving relative gains of up to 9.5% on difficult mathematical reasoning tasks. Code: https://github.com/lilifree/GIFT
Introduction
“Thoughts without content are empty, intuitions without concepts are blind. Only through their union can knowledge arise.”
— Immanuel Kant, Critique of Pure Reason
Training large language models for complex reasoning relies on two complementary forms of supervision: outcome-grounded reinforcement learning and dense, token-level distillation. Reinforcement learning with verifiable rewards (RLVR), such as GRPO [20], anchors learning to task success, but offers little guidance about the intermediate decisions that produce an outcome. A single reward supervises an entire response, and GRPO’s group-relative signal vanishes when all responses receive the same reward. On-policy self-distillation (OPSD; [34]) provides more direct guidance: the same model, conditioned on privileged information such as a reference solution, supplies teacher distributions along the student’s own rollout. Yet this dense supervision targets imitation rather than correctness; following it can introduce information leakage and late-stage instability ([28, 21]). We seek the best of both worlds: dense teacher guidance to accelerate learning, with verifiable outcomes determining what the model ultimately optimizes.
Realizing this promise requires more than combining the two signals: complementarity in supervision does not guarantee compatibility in optimization. RLSD and RL CSD use teacher-derived signals to modulate verifier-based advantages ([28, 18]), while OPDVR gates sampled-token distillation rewards by trajectory correctness [12]. For a nonzero RL gradient at a fixed token, such scalar changes cannot introduce a direction outside its logit-space span. An auxiliary distributional loss, as in SDPG [14], can supply such directions, yet admitting a response or token does not establish how its teacher gradient interacts with the current RL update. Teacher withdrawal addresses persistence: RLSD and SDPG anneal teacher influence, and Sequential Beats Joint separates distillation and RLVR into successive stages [11]. But withdrawal alone leaves an allocation problem. Uniformly weakening or abruptly removing guidance can discard complementary updates together with redundant or conflicting ones. Selecting surviving tokens by disagreement alone can retain a strong teacher signal precisely where it obstructs the outcome update. Which token-level teacher contributions should survive as the available supervision budget contracts?
We address this allocation problem with our method, GIFT (Gradient Informed FilTering for Distillation Augmented Reasoning RL), which uses token-level gradient geometry to determine where auxiliary distillation participates. Instead of relying on heuristics like disagreement or uncertainty, GIFT directly analyzes the interaction between the teacher’s distillation gradient and the student’s RL gradient. At each token, we decompose the teacher’s update into two key components: (1) a component parallel to the RL gradient, which measures first-order conflict or assistance, and (2) a component orthogonal to the RL gradient, which represents novel directional guidance that cannot be produced by simply rescaling the RL update. Our central hypothesis is that the most effective guidance is both non-conflicting and contains significant novel direction. However, since privileged teacher guidance (as in OPSD) targets imitation rather than guaranteed correctness, its influence must be carefully managed to avoid biasing the final policy. GIFT addresses this by implementing its geometric selection under a progressively shrinking token budget. This annealing process ensures that as training matures, the model transitions from dense but potentially flawed guidance towards the verifiable outcome signal. Only the most geometrically complementary teacher contributions survive before the model converges to the pure, unprivileged RL objective. This approach leads to a method that is both principled and effective. Our main contributions are:
We propose GIFT, a method that uses the geometric relationship between teacher and student gradients to selectively apply distillation. It is the first method to explicitly distinguish between conflicting, redundant, and complementary token-level guidance to solve the allocation problem.
We derive a closed-form, token-level scoring rule based on this geometry, allowing for efficient selection without needing to compute auxiliary backward passes. We combine this with a progressive coverage annealing schedule that smoothly transitions from dense guidance to pure reinforcement learning, ensuring stability and preserving the primacy of the outcome-based objective.
GIFT achieves the highest average scores on challenging mathematical reasoning benchmarks across four model scales, outperforming eight strong baselines. Ablation studies confirm that its combination of geometric selection and coverage annealing is critical to its success, yielding significant gains over simpler selection criteria or withdrawal strategies.
Related Work
Dense teacher guidance can supplement outcome supervision, but its usefulness depends on how it is incorporated, not simply on how much is provided. We situate GIFT along four related design choices: how teacher information enters the update, which tokens receive it, how its interaction with RL is assessed, and how its influence is withdrawn.
Outcome Supervision and Self-Distillation. RL from verifiable outcomes, or RLVR, forms the basis of our primary objective. GRPO [20] derives group-relative advantages from verified outcomes; subsequent works like Dr. GRPO [15] and DAPO [30] refine its normalization and loss aggregation to improve stability and reduce bias. While these methods provide an unbiased, outcome-grounded learning signal, their supervision is sparse and vanishes for rollout groups where all responses are correct or incorrect. On-policy self-distillation (OPSD; [34]) offers a dense, token-level alternative, supplying teacher feedback along the student’s trajectory. Hybrid methods, such as RLSD [28], RLCSD [18], and AMR-SD [21], attempt to combine these two signals by using teacher-derived signals to perform scalar modulation of the RL advantages. However, they are fundamentally limited in expressiveness: re-weighting the RL gradient cannot introduce an update direction outside its original logit-space span. GIFT uses OPSD-RLVR geometry to prioritize complementary guidance under exact quotas, without excluding all conflicts.
Token-Selective Teacher Guidance. Given a dense teacher signal, a critical question is which tokens should receive it. Many methods rely on statistical heuristics. TIP Xu et al., [24] ranks tokens by student entropy and teacher-student divergence; EOPD Jin et al., [9] gates supervision by teacher entropy; RSTG Han et al., [5] combines these heuristics for failed rollouts; and CRPO Wu et al., [22] uses predictive entropy to distinguish exploration from exploitation bias. While these statistics identify uncertainty or disagreement, they do not directly establish compatibility with the current outcome-driven update. Other works highlight that simple agreement is not the goal. RLRT Kim et al., [10] reinforces successful student departures from the teacher, demonstrating teacher disagreement can be valuable. SKALD Jiang et al., [8] focuses on zero-variance groups where the RL signal is absent, providing distillation as a fallback. This aligns with GIFT’s treatment of such cases (Corollary 1), but GIFT’s geometric criterion is more general. Instead of relying on heuristics, GIFT directly assesses the first-order interaction between the teacher’s distillation gradient and student’s RL gradient at each token, providing a principled selection basis.
Gradient Geometry for Auxiliary Learning. The geometric relationship between gradients from multiple objectives is a well-established concept in multi-task learning. Methods like PCGrad Yu et al., [31] project away conflicting gradient components, while ATTITTUD Dery et al., [2] decomposes an auxiliary gradient into components agreeing, conflicting, or orthogonal to the primary task’s gradient subspace, and re-weights them. ATTITTUD is the direct intellectual predecessor to our geometric analysis. However, GIFT makes three critical and novel adaptations for the RLVR context: (1) Usage: We use geometry for discrete selection of tokens, not continuous modification of the auxiliary gradient, which avoids distorting the teacher signal. (2) Granularity and Efficiency: Our analysis is performed at the token level in logit space, where we derive a closed-form, solution (Proposition 1) for the geometric components, avoiding expensive, parameter-space SVD approximations required by ATTITTUD. (3) Value of Orthogonality: We identify the orthogonal component () as the most valuable form of complementary guidance—representing novel directions unreachable by the RL signal (Theorem 2)—whereas ATTITTUD treats it as neutral.
Demonstration-Guided RL and Teacher Withdrawal. Privileged teacher guidance must be carefully managed to avoid biasing the final policy towards imitation. This withdrawal can be implemented in several ways. Methods like LUFFY Yan et al., [25] and UFT Liu et al., [13] introduce teacher guidance through trajectory composition or annotated prefixes, and anneal the length or proportion of this guidance. Other methods anneal the auxiliary loss coefficient () to zero, as seen in Kickstarting Schmitt et al., [19], RLSD, and SDPG. However, such coefficient annealing dilutes the signal for all tokens simultaneously, discarding complementary updates along with conflicting ones. Sequential Beats Joint Li et al., [11] avoids this by separating distillation and RLVR into successive stages, but forgoes the benefits of joint training. GIFT addresses this optimization tension through coverage annealing, shrinking the selected-token quota while fixing the loss weight. At each ranking, positive residual scores precede zero scores and conflicts; within conflicts, smaller normalized opposition receives higher priority. This controls teacher participation while preserving retained-token loss weights, before the auxiliary loss vanishes at teacher exit.
Method
In the introduction, we established the central challenge: to effectively merge the dense, token-level guidance from self-distillation Liu et al., [14]; Agarwal et al. [1]; Gu et al. [3] with the sparse, outcome-grounded supervision from reinforcement learning Shao et al., [20]; Guo et al. [4] (RL). Simply combining them is insufficient; we need a principled mechanism to decide when and where to apply the teacher’s guidance to ensure it complements, rather than corrupts, the primary RL objective. This section introduces our method, GIFT, which resolves this ”allocation problem” through a geometric analysis of the underlying gradients, as shown in Figure 1.
Figure 1. GIFT decomposes the distillation gradient into conflict () and complement () components relative to the RL gradient, ranks tokens, and anneals the selection budget from full to zero.
A Geometric Approach to Guidance
Our methodology is built on a direct analysis of the geometric interaction between the teacher and student gradients Dery et al., [2]; Yu et al. [31] at each token. Let be the student’s logits at position , with probability distribution . At this position, we consider two distinct logit-space gradients. The first is the outcome-driven gradient from the clipped GRPO loss [20], . This gradient, denoted , represents the direction of steepest ascent for the RL objective, steering the model toward task success determined by the verifier. The second is the imitation-driven gradient from the distillation loss, [1, 3, 34, 14]. This gradient, denoted , points in the direction that best aligns the student’s distribution with the privileged teacher’s distribution . These gradients are:
where is the one-hot vector for the sampled token, is the advantage, indicates an active clipped branch, and is the teacher distribution. The gradient serves as our reference direction. On the set of tokens where a non-zero RL gradient exists, we decompose the teacher gradient with respect to the normalized RL gradient :
Here, the scalar is the signed magnitude of the parallel component, measuring first-order assistance or conflict. The scalar is the norm of the orthogonal component, quantifying novel directional guidance. Together, they satisfy . These two scalars form the core of our selection mechanism, as explained next.
Safety: The Role of the Parallel Component ()
Our first priority is to ensure that auxiliary guidance does not undermine the primary, outcome-driven objective. The parallel component provides a direct, first-order measure of this safety. Theorem 1 formalizes this intuition.
Theorem 1 (Local first-order non-interference). For a given token, the change in the primary GRPO loss from a combined update [2, 31] is given by
The implications are direct:
Assistance (): The teacher’s guidance is aligned with the RL gradient and accelerates the decrease of the primary loss. This guidance is beneficial.
Neutrality (): The teacher’s guidance is orthogonal to the RL gradient and has no first-order effect on the primary loss. It is safe, regardless of its magnitude .
Conflict (): The teacher’s guidance opposes the RL gradient, slowing or even reversing progress on the primary objective. This guidance is harmful.
This geometric criterion provides a more precise filter than heuristics like student-teacher divergence (e.g., TIP [24]) or disagreement. While a large divergence might indicate valuable guidance, it does not distinguish between productive, orthogonal disagreement () and destructive, head-on conflict (). Our use of makes this crucial distinction. Therefore, the sign of serves as a fundamental safety gate. We define any token with as conflicting and give it lowest priority. This raises the next question: among the ”safe” tokens (where ), which are most valuable to select? This is decided by the orthogonal component, .
Value: Why the Orthogonal Component () Matters
Having established safety with the parallel component , we now examine value. Among ”safe” tokens (), the most useful guidance lies in the orthogonal component, quantified by . Crucially, ’s value does not stem from a direct, first-order contribution to the primary RL objective—as established in Theorem 1, its contribution to the primary loss descent is zero. Its profound importance relates instead to the guidance’s novelty and expressiveness. This value is best understood through its counterpart. A purely parallel teacher gradient () is fundamentally redundant. The resulting update, , simplifies to merely rescaling the original RL gradient . This critical limitation offers no new directional information unattainable by simply adjusting the learning rate or advantage scaling for that token. An entirely collinear teacher tells the optimizer nothing it could not learn from the RL signal alone.
All genuinely new geometric information provided by the teacher resides in the orthogonal component . Therefore, selecting for a large is equivalent to prioritizing the most valuable form of teacher guidance—the part that is structurally inaccessible to simpler advantage-modulation schemes. The following theorem formalizes this concept of complementarity in three distinct ways.
Theorem 2 (Complementarity of the orthogonal residual). Under the assumptions of Theorem 1:
(Maximal locally GRPO-neutral distillation decrease.) Among unit-norm logit perturbations that are locally neutral to the token-level primary surrogate,
If , the maximum is attained by . If , every feasible attains the value zero. Thus, is the maximal first-order distillation decrease among unit-norm logit perturbations that are locally neutral to the token-level primary surrogate; no finite-step or shared-parameter conclusion follows.
(Irreducibility to token-local GRPO rescaling.) Fix the sampled tokens, model state, and reference gradients, and equip with the Euclidean product norm . Let be the space of unrestricted signed, token-local rescalings of the fixed GRPO reference field. For any fixed binary mask , write . Then
In particular, for and , the residuals defined in Equation (2) satisfy
Thus, selecting any with certifies that the selected teacher field cannot be represented by token-local scalar rescalings of the fixed GRPO field. This is a sufficient, not a necessary, condition. A selected zero-reference block with also certifies irreducibility, as can a noncollinear small nonzero-reference block.
(Gradient diversity.) For a collection of gradients summed into one update, define
(Definition 1; when the are identical and when they are mutually orthogonal, so larger means less redundancy among the summands). Applying this to the two gradients combined at token , i.e. : at fixed , and , is strictly decreasing in , hence strictly increasing in on the safe branch , where it rises from at to at at . Unconstrained over all , however, is maximized as and diverges when .
This theorem justifies prioritizing the orthogonal component for three reasons. First, measures maximal progress on the imitation objective that is “free” to the outcome-driven task, having no first-order cost to the primary RL objective. Second, it represents guidance advantage-modulation methods like RLSD [28] and OPDVR [12] cannot express, since they only rescale the RL gradient without introducing novel directions. Third, in sparse-reward RLVR [20, 4], a large indicates high gradient diversity (), injecting new dimensions to prevent stagnation. However, the most diverse signal directly opposes the primary gradient (), which would be catastrophic. Our method resolves this by assigning conflicting signals negative scores, ensuring their early discard as the token budget anneals. This provides early dense guidance while progressively enforcing safety.
GIFT resolves this tension by framing token selection as an exact-quota lexicographic program: at each step, a quota of eligible tokens must be selected, and GIFT chooses the size- subset of the eligible set that lexicographically minimizes
where collects the conflicting eligible tokens, the remaining eligible ones, their number, their normalized opposition, and the injected orthogonal guidance.
Scoring and Progressive Selection
The lexicographic exact-quota program in (7) admits an exact, closed-form solution in the form of a piecewise scoring rule, providing a principled ranking for all eligible tokens. Each token receives a score defined as:
This structure imposes a strict hierarchy that directly reflects our optimization criteria. For tokens with conflicting guidance (), the score is negative, ensuring they are ranked lowest. For all safe tokens (), the score is simply the orthogonal magnitude , prioritizing novel, complementary information. Tokens with no usable RL gradient () receive a neutral score of zero. The optimality of this ranking is formally guaranteed by the following theorem.
Theorem 3 (Exact-quota lexicographic optimality). With the notation of (7), for any integer quota , the set of highest scores lexicographically minimizes over size- subsets, and
If the selection is conflict-free and maximizes over size- subsets of ; otherwise it takes plus the conflicting tokens of smallest normalized opposition . Both hold under arbitrary tie-breaking.
Theorem 3 provides the formal guarantee that this scoring rule is the exact solution to the stated lexicographic exact-quota problem: it first minimizes the number of conflicting tokens admitted, then their normalized opposition, and only then maximizes the injected orthogonal guidance .
This static ranking is embedded within a dynamic withdrawal strategy, implemented by progressively shrinking the token budget over the course of training via an annealing schedule.
where is the retention ratio, is an initial full-coverage phase, and is the step at which all teacher guidance is withdrawn. This annealing of the coverage, rather than a loss coefficient, makes the trade-off between exploration and exploitation time-varying. Early in training, when the budget is large and the RL signal is sparse, the model benefits from broad, diverse guidance. As training matures and the budget contracts, the lowest-scoring tokens—the most conflicting ones—are the first to be excluded. The most complementary tokens (high , ) are retained the longest.
This provides a principled, state-aware withdrawal order. Unlike uniform coefficient annealing ([14, 28]) which dilutes all guidance simultaneously, or staged training ([11]) which forgoes joint optimization, GIFT’s geometric ranking and coverage annealing ensures that as the supervision budget contracts, it is precisely the most harmful guidance (lowest-scoring tokens) that is shed first, while the most complementary guidance is retained at full strength.
The GIFT Objective
The principles of geometric selection and progressive withdrawal are implemented in the final GIFT objective function:
Here, selects the top- eligible tokens per sub-batch; contains all valid positions and . Several critical design choices apply. First, fixing ensures selected tokens receive full-strength, unprojected distillation loss. Control operates via selection (who receives guidance), not dilution (guidance strength). The teacher loses reach, not voice. Second, normalizing over all valid tokens () rather than just selected ones ensures stability, keeping each retained token’s contribution independent of budget size. Otherwise, a shrinking budget would artificially inflate surviving tokens, undermining smooth withdrawal. Finally, as the budget anneals to zero (), the mask becomes universally zero, guaranteeing a clean transition. The auxiliary term vanishes, reducing identically to the native GRPO loss. This grounds final behavior purely in the outcome-driven objective. The teacher acts as a temporary, geometrically-filtered scaffold, harnessing dense guidance before complete withdrawal without corrupting the final policy.
Experiments
We conduct experiments to validate GIFT’s core principles. Our evaluation addresses three central questions: (1) Effectiveness: Does GIFT’s approach of gradient-informed filtering and progressive withdrawal achieve state-of-the-art performance on challenging reasoning tasks compared to existing methods? (2) Generalization: Do GIFT’s benefits extend across different architectures, scales, and reasoning domains, including out-of-distribution scenarios? (3) Component Necessity: Are GIFT’s key components—specifically the geometric scoring rule based on conflict () and novelty (), and the coverage annealing schedule—critical to its success?
Experimental Setup
Models and tasks. We evaluate on two families of models: three scales of Qwen3 (1.7B, 4B, 8B) [27] and OLMo-3-7B-Think [17]. For mathematical reasoning, we train on the DeepMath-103K dataset [6], using progressively harder difficulty splits for larger models, and evaluate on the AMC23 (math-ai, 2023), AIME24 [32], and AIME25 [33] benchmarks. For logical reasoning, we train on Knights-and-Knaves puzzles [23] (4-8 roles) and evaluate on both in-distribution (ID) and out-of-distribution (OOD, 9-11 roles) splits to test generalization.
Baselines. We compare GIFT with the initial model (BASE) and eight training baselines, grouped by supervision source and how it enters optimization (Appendix G): GRPO, using verifier-derived advantages without teacher feedback [20]; distributional self-distillation, matching next-token distributions of a conditioned self-teacher (OPSD [34], SDPO (Hübõtler et al., 2026)); verifier-guided token-level credit assignment, modulating advantages or gating rewards with teacher-derived scalars (RLSD [28], RLCSD [18], OPDVR [12]); and demonstration-guided methods (LUFFY [25], UFT [13]).
Evaluation. For mathematics, we report mean@12, the average correctness over 12 sampled responses per problem, averaged across problems, using rule-based answer matching. For logic, we report pass@1 on 500 ID problems and 100 problems for each of the three OOD role counts (9, 10, and 11). The reported Avg. equally weights the three mathematical benchmarks of the four logical-reasoning splits. All scores are percentages, and gains are relative percentage-point differences.
Main Results
Our main results demonstrate that GIFT’s principled filtering mechanism consistently outperforms existing methods across multiple tasks and model scales, providing a superior solution to the core tension between dense guidance and outcome-grounded learning.
Mathematical Reasoning. As shown in Table 1, GIFT achieves the highest average score on mathematical reasoning across all three Qwen3 models. Compared to the base model, it achieves average relative gains of 5.03%, 3.39%, and 2.99% for the 1.7B, 4B, and 8B models, respectively. The inferiority of the baselines stems from their distinct, yet suboptimal, strategies for combining supervision.
| Size | Bench. | BASE | GRPO | OPSD | SDPO | RLSD | RLCSD | OPDVR | LUFFY | UFT | GIFT |
| 1.7B | AMC23 | 75.50 | 75.70 | 75.60 | 75.30 | 75.30 | 75.80 | 76.51 | 75.40 | 74.60 | 75.80 (+0.40%) |
| 1.7B | AIME24 | 44.44 | 45.28 | 45.56 | 44.72 | 44.72 | 46.94 | 46.67 | 48.33 | 48.33 | 48.61 (+9.38%) |
| 1.7B | AIME25 | 35.00 | 35.56 | 33.33 | 36.67 | 34.72 | 34.72 | 35.83 | 34.17 | 36.67 | 38.33 (+9.51%) |
| 1.7B | Avg. | 51.65 | 52.18 | 51.50 | 52.23 | 51.58 | 52.49 | 53.00 | 52.63 | 53.20 | 54.25 (+5.03%) |
| 4B | AMC23 | 88.45 | 89.16 | 88.25 | 88.96 | 88.86 | 90.26 | 88.45 | 89.16 | 89.66 | 88.55 (+0.11%) |
| 4B | AIME24 | 71.67 | 74.17 | 73.06 | 70.83 | 73.61 | 72.22 | 73.61 | 76.11 | 70.56 | 73.33 (+2.32%) |
| 4B | AIME25 | 63.89 | 63.61 | 67.22 | 67.22 | 63.61 | 64.17 | 64.72 | 65.28 | 64.72 | 69.72 (+9.13%) |
| 4B | Avg. | 74.67 | 75.65 | 76.18 | 75.67 | 75.36 | 75.55 | 75.59 | 76.85 | 74.98 | 77.20 (+3.39%) |
| 8B | AMC23 | 89.16 | 89.36 | 88.76 | 89.56 | 88.76 | 89.26 | 89.96 | 89.96 | 88.86 | 89.66 (+0.56%) |
| 8B | AIME24 | 75.00 | 74.72 | 77.22 | 75.83 | 75.28 | 76.11 | 74.44 | 73.33 | 76.11 | 76.11 (+1.48%) |
| 8B | AIME25 | 65.83 | 65.83 | 66.39 | 66.39 | 69.72 | 68.06 | 69.72 | 67.50 | 66.39 | 71.11 (+8.02%) |
| 8B | Avg. | 76.67 | 76.64 | 77.46 | 77.26 | 77.92 | 77.81 | 78.04 | 76.93 | 77.12 | 78.96 (+2.99%) |
Table 1. Mathematical reasoning across Qwen3 scales (mean@12, %). Avg. is the mean over AMC23, AIME24, and AIME25. Green denotes relative gains over BASE; bold and underlined values indicate the best and second-best distinct scores. Bench. denotes benchmark.
The two extremes of the supervision spectrum, pure RL and pure distillation, highlight the core challenge. GRPO provides an unbiased, outcome-grounded learning signal, which is why we retain it as our primary objective. However, its supervision is sparse—a single reward for an entire trajectory—and its group-relative signal vanishes in uniformly correct or incorrect groups, leading to sample inefficiency and learning stagnation. Conversely, on-policy self-distillation offers dense, token-level guidance but optimizes for imitation of a privileged teacher rather than verifiable correctness: OPSD conditions its self-teacher on a reference solution, while SDPO conditions it on environment feedback or verified successful rollouts. In both, the auxiliary loss is applied unconditionally along the entire trajectory—with no check for compatibility with the outcome-driven update and no mechanism for withdrawing teacher influence.
Hybrid methods attempt to bridge this gap, but their approaches lead to distinct compromises. One family, including RLSD, RLCSD, and OPDVR, ultimately converts the teacher signal into a scalar that modulates the RL update: RLSD and RLCSD use teacher-derived signals to modulate verifier-based advantages, while OPDVR reformulates the implicit reward of sampled-token distillation based on trajectory correctness and applies ReLU gating, so that the teacher enters the update only through a per-token scalar reward. By collapsing the teacher’s rich distributional information into a scalar reward or weight, these methods are structurally incapable of introducing novel update directions, as established in our theoretical analysis (Theorem 2(ii)); they can only re-weight the token-level components of the existing RL update, whereas GIFT’s auxiliary loss can provide valuable orthogonal guidance (). Other methods like LUFFY and UFT introduce teacher guidance through trajectory composition or annotated prefixes, which likewise lack the fine-grained, geometry-aware conflict checking that GIFT performs at every token.
In contrast, GIFT synthesizes the strengths of these approaches: it complements GRPO’s reliable but sparse signal with dense, expressive guidance from an auxiliary loss, but subjects that guidance to a rigorous geometric filter. This unique combination explains its consistent, high-level performance, particularly on the challenging AIME25 benchmark where it establishes a new state-of-the-art.
Logical Reasoning and Generalization. On the Knights-and-Knaves task (Table 2), GIFT again achieves the highest average score, improving over the base model with a relative gain of 3.71%. Crucially, this improvement is not confined to the in-distribution test set; GIFT shows a substantial average relative gain of 4.70% on the OOD splits. This indicates that the reasoning capabilities learned via gradient-informed guidance generalize to more complex problems not seen during training. The consistent failure of baselines to match this performance further underscores our analysis: methods that are either structurally limited (scalar modulation), ungrounded (pure imitation), or lack a robust filtering mechanism (unfiltered or coarse-grained guidance) cannot as effectively harness teacher guidance to build generalizable reasoning skills. This result, combined with strong performance on the OLMo-3 model family (Appendix § 14, Figure 2), confirms that GIFT’s mechanism is not specific to a single model architecture or reasoning domain but offers a generally applicable framework for improving complex reasoning.
| Benchmark | BASE | GRPO | OPSD | SDPO | RLSD | RLCSD | OPDVR | LUFFY | UFT | GIFT |
| In-distribution (ID) | ||||||||||
| 4–8 roles | 72.40 | 73.40 | 72.80 | 75.20 | 73.60 | 74.00 | 73.00 | 74.60 | 74.20 | 73.40 (+1.38%) |
| Out-of-distribution (OOD) | ||||||||||
| 9 roles | 69.00 | 72.00 | 75.00 | 73.00 | 71.00 | 69.00 | 71.00 | 67.00 | 71.00 | 73.00 (+5.80%) |
| 10 roles | 59.00 | 56.00 | 59.00 | 56.00 | 55.00 | 59.00 | 57.00 | 59.00 | 56.00 | 61.00 (+3.39%) |
| 11 roles | 42.00 | 46.00 | 41.00 | 40.00 | 41.00 | 44.00 | 44.00 | 43.00 | 40.00 | 44.00 (+4.76%) |
| Avg. | 60.60 | 61.85 | 61.95 | 61.05 | 60.15 | 61.50 | 61.25 | 60.90 | 60.30 | 62.85 (+3.71%) |
Table 2. Logical reasoning on Knights-and-Knaves (pass@1, %). The ID split contains 4–8 roles, while the OOD splits contain 9, 10, and 11 roles. Green denotes relative gains over BASE.

Figure 2. Results on OLMo3-7B, reported as mean@12. Avg. is the equally weighted average over the two benchmarks. Green numbers indicate absolute gains over Base.
Ablation Study
Here We present key findings from the ablations detailed in Appendix § 13.§ 4.
The Geometric Scoring Rule is Essential. Table 3 shows heuristic criteria (e.g., entropy-KL divergence, raw gradient cosine similarity) fail to match GIFT, confirming statistical proxies for disagreement are less effective than our direct geometric analysis. Table 4 shows its constituent parts are necessary, though their importance varies with task difficulty. On simpler AMC23 and AIME24 benchmarks, “Conflict-Free” and “Conflict-only” ablations perform comparably or better than GIFT. When the primary RL gradient is reliable, simpler strategies prioritizing safety by strictly rejecting conflicts (“Conflict-Free”) or accelerating agreement (“Conflict-only”) are highly effective. However, these strategies falter on the challenging AIME25 benchmark, where GIFT excels. The “Residual-only” ablation—prioritizing novel guidance () while ignoring conflict signs—underperforms across all settings, demonstrating pursuing novelty without safety checks is detrimental. Conversely, “Conflict-Free” proves too restrictive on AIME25, likely discarding valuable teacher guidance needed to escape local optima simply because it appears conflicting. GIFT’s superiority on AIME25 stems from its synthesis: using the conflict signal () for safety while leveraging its exact-quota system to judiciously admit the least-opposing conflicting tokens when necessary, providing a vital mechanism for exploration. These results confirm GIFT’s complete geometric scoring rule provides a more robust approach than isolated heuristics, validating a design that places conflicting tokens below zero- and positive-score candidates.
| Method | Mathematical Reasoning (mean@12) | Logical Reasoning (pass@1) | |||||||
| AMC23 | AIME24 | AIME25 | Avg. | ID (4–8) | OOD-9 | OOD-10 | OOD-11 | Avg. | |
| All Tokens | 89.96 | 76.11 | 66.39 | 77.49 | 73.80 | 69.00 | 59.00 | 45.00 | 61.70 |
| Random Tokens | 89.66 | 75.28 | 66.67 | 77.20 | 73.60 | 71.00 | 58.00 | 46.00 | 62.15 |
| Entropy-KL Selection | 89.36 | 75.28 | 66.11 | 76.92 | 71.80 | 63.00 | 54.00 | 52.00 | 60.20 |
| Gradient-cosine Selection | 89.56 | 77.50 | 65.83 | 77.63 | 72.80 | 68.00 | 61.00 | 39.00 | 60.20 |
| Full Model | 89.66 | 76.11 | 71.11 | 78.96 | 73.40 | 73.00 | 61.00 | 44.00 | 62.85 |
Table 3. Ablation of token-selection strategies. Mathematical reasoning uses Qwen3-8B and is evaluated by mean@12. Logical reasoning uses Qwen3-4B and is evaluated by pass@1.
| Benchmark | Conflict-only | Residual-only | Teacher-norm | Conflict-Free | GIFT |
| AMC23 | 89.86 | 89.26 | 88.76 | 89.96 | 89.66 |
| AIME24 | 75.56 | 75.00 | 73.89 | 76.11 | 76.11 |
| AIME25 | 65.83 | 66.67 | 68.06 | 67.11 | 71.11 |
| Avg. | 77.08 | 76.97 | 76.90 | 77.73 | 78.96 |
Table 4. Ablation of selection strategies on Qwen3-8B. Mean@12 scores on AMC23, AIME24, and AIME25.
Coverage Annealing is Superior to Alternative Annealing Strategies. Withdrawing teacher influence is as critical as applying it. We thus compare GIFT’s coverage annealing with two alternatives. First, sequential training ([11]) trains with OPSD distillation before switching to pure GRPO. Second, weight annealing combines objectives but uniformly applies distillation, gradually decaying the loss coefficient to zero. As shown in Table 7 (Appendix § 14), GIFT significantly outperforms both strategies, averaging 1.59 and 2.87 percentage point gains, respectively. Unlike sequential training, GIFT benefits from joint optimization synergy. Furthermore, unlike indiscriminate weight annealing, GIFT’s coverage annealing preserves the most valuable teacher signals at full strength while selectively pruning unhelpful ones. This confirms that selective, state-aware withdrawal is superior to both staged training and uniform decay.
| Training schedule | AMC23 | AIME24 | AIME25 | Avg. |
| OPSD GRPO | 89.06 | 75.28 | 67.78 | 77.37 |
| Weight annealing | 89.66 | 72.22 | 66.39 | 76.09 |
| GIFT | 89.66 | 76.11 | 71.11 | 78.96 |
Table 7. Teacher-withdrawal schedules on Qwen3-8B (mean@12, %). Avg. equally weights the three benchmarks. Bold marks the best score in each column, including ties.
A Broader Allocation Scope is More Effective. We find that allocating the token budget across the entire training sub-batch outperforms narrower scopes like per-rollout or per-prompt (Table 5). This allows for a more efficient global allocation of supervision, enabling tokens to compete for guidance across different prompts and responses. The model can thus focus the limited supervision budget on the most complementary updates available in the entire sub-batch, rather than being constrained by a locally optimal but globally suboptimal choice.
| Benchmark | Per-rollout | Per-prompt | GIFT |
| AMC23 | 89.09 | 89.56 | 89.66 |
| AIME24 | 75.28 | 76.39 | 76.11 |
| AIME25 | 67.78 | 66.39 | 71.11 |
| Avg. | 77.37 | 77.45 | 78.96 |
Table 5. Ablation of token selection scopes on Qwen3-8B.
Conclusion
We introduced GIFT, a method that resolves sparse RL and dense distillation conflicts by filtering teacher guidance via token-level gradient geometry, decomposing distillation gradients into components parallel (conflict) and orthogonal (novelty) to the RL gradient for principled scoring. Paired with coverage annealing, this selectively withdraws conflicting updates, smoothly transitioning to a pure RL objective. On complex reasoning benchmarks, GIFT consistently outperforms strong baselines, validating this integration of dense supervision.
AI Use Statement
During the preparation of this manuscript, large language models (LLMs) were utilized solely to refine grammar, readability, and formatting. The core scientific contributions—encompassing the research design, methodology, data analysis, and conclusions—were entirely conceptualized and executed by the human authors. We have carefully verified all AI-assisted language refinements and assume full responsibility for the manuscript’s final content and integrity.
Reproducibility Statement
To ensure our findings are fully reproducible, all source code and trained model weights will be made publicly available upon the acceptance of this manuscript. Furthermore, we will provide comprehensive documentation and evaluation protocols to assist the research community in replicating our experiments.
Main Notation
For clarity, Table 6 summarizes the main notations used in GIFT.
| Notation | Type | Meaning |
| index / set | Token position and the set of all valid token positions. | |
| vectors | Student logits and predictive distribution, where . | |
| distribution | Privileged teacher distribution at position . | |
| token / vector | Sampled token and its corresponding one-hot vector. | |
| scalars | Token weight, advantage, importance ratio, and active-branch indicator in the clipped GRPO objective. | |
| loss | Token-level clipped GRPO loss providing outcome-grounded supervision. | |
| loss | Token-level self-distillation loss aligning with . | |
| vector | Outcome-driven logit gradient, . | |
| vector | Imitation-driven logit gradient, . | |
| scalar | Signed parallel component, , measuring assistance or conflict with the RL gradient. | |
| scalar | Orthogonal magnitude, , measuring complementary teacher guidance. | |
| set | Token positions with a usable non-zero RL reference gradient. | |
| set | Distillation-eligible token positions in training sub-batch . | |
| sets | Conflicting and non-conflicting eligible tokens, respectively. | |
| scalar | GIFT score: for conflicting tokens, for non-conflicting tokens, and zero when no usable RL reference exists. | |
| integer | Number of tokens retained in training sub-batch at step . | |
| indicator | Binary mask indicating whether token is selected at step . | |
| scalar | Retention ratio controlling the progressively shrinking token budget. | |
| integers | End of the full-coverage phase and step of complete teacher withdrawal. | |
| scalar | Distillation coefficient, fixed at . | |
| integer | Total number of valid token positions used for loss normalization. |
Table 6. Summary of the main notations used in GIFT.
Supporting Lemmas
Throughout we fix a token position and drop the index where unambiguous. Let , let be the teacher support with , and set
so the objective is finite and differentiable. We use .
Lemma 1. (Sparse forward-KL gradient.) The loss satisfies , where the tail-completed teacher
is a probability distribution.
Proof. Only and depend on , as and are detached. For the first,
with . For the second, since ,
so . Adding the two and using gives . Finally and .
Proposition 1 (Closed-form scoring). With and ,
and . Consequently and are computable from , , and sums over , with and . No -dimensional gradient is materialized and no auxiliary backward pass is needed.
Proof. Each identity splits the sum over and its complement and substitutes (13): e.g. and ; identically for with . For the last, expand . The only vocabulary-wide quantity is , a scalar reduction of probabilities already formed in the forward pass. The expression for follows from (orthogonality).
Implementation note. if , else ; is clipped at zero before the square root to absorb cancellation when .
Proposition 2 (Invariance of the score). For ,
so and are invariant to and to positive rescaling of , provided the active-branch and norm-threshold decisions are unchanged; is in addition independent of .
Proof. On the active branch with and . Since we have and
so the magnitude cancels. With from Lemma 1, gives the display. involves neither nor , and depends on only through , which is insensitive to .
Gradient Diversity
Theorem 2(iii) uses the following quantity, introduced by Yin et al. [29] to characterize how far mini-batch SGD can be scaled before departing from the behaviour of its serial counterpart. We state it and the two facts we rely on, so that the argument is self-contained.
Definition 1 (Gradient diversity). For gradients summed into a single update with ,
Lemma 2 (Endpoints of ). If then . If the are nonzero and mutually orthogonal then , whatever their norms. In general if and only if .
Proof. For identical gradients the numerator is and the denominator , giving . For mutually orthogonal gradients every cross term in the second form of (19) vanishes, leaving . The final claim is immediate from that same form, since the numerator is positive.
So is a redundancy measure: it is smallest () when the summands duplicate one another, equals exactly when they are mutually non-interfering, and exceeds only when they partially cancel—the last case being why an unconstrained preference for large is unsafe. The regime relevant here is with and . We use only Definition 1 and Lemma 2; the convergence and stability bounds of Yin et al. [29] are cited as context for why low redundancy is desirable, but no statement in Theorem 2(iii) depends on them.
Proofs of the Main Results
Theorem 1 (non-interference)
Proof. Let . By the chain rule and ,
using and . This is . The GRPO-only step is , with value ; the teacher’s contribution is the residual , whose sign is opposite to that of since , .
For (1): if then , a strictly larger decrease; if then exactly, independently of , and in particular the orthogonal component has no first-order effect at any magnitude. For and any we get , so the primary descent cannot be cancelled.
For (2): if then , which exceeds by for : the decrease is necessarily smaller. It becomes nonnegative, i.e. the primary loss no longer decreases to first order, exactly when , i.e. ; strict reversal () holds when .
Theorem 2 (complementarity)
Part (i). Decompose . Feasibility is equivalent to , whence . By Cauchy–Schwarz this is at most . If , is feasible (, ) and attains . If then and every feasible gives . (By Theorem 1 with the perturbation , the constraint is precisely first-order neutrality of the primary loss.)
Part (ii). Equip with the Euclidean product norm . Since is finite, is a closed linear subspace of , where . Distinct token blocks are orthogonal, so the orthogonal projection of acts blockwise:
By the Pythagorean identity in each block and ,
For , where , the parenthesized term equals by the definition of the orthogonal residual. All remaining terms are nonnegative. Consequently,
If and for some , this lower bound is strictly positive, hence . Thus the selected teacher field cannot be represented by token-local scalar rescalings of the fixed GRPO reference field.
Scope of Part (ii). The comparison space permits arbitrary real-valued rescalings of the fixed token-level GRPO gradients. It therefore contains the fields produced by scalar-modulation rules whose per-token gradients remain multiples of the same , but need not equal the reachable set of a particular algorithm with nonnegative or bounded multipliers. No reference direction is truncated: a small but nonzero still spans a line, whereas contributes the zero subspace. The threshold enters only through the lower bound involving , not through the definition of .
The result concerns independent token blocks in logit space. Non-membership in does not imply non-membership in its image after mapping through the shared model Jacobians; distinct token-level fields can induce the same parameter gradient. Thus, the theorem establishes local directional irreducibility, not shared-parameter or function-space novelty, nor improved verifier performance. Collinear teacher guidance can still affect update magnitudes and optimization behavior.
Finally, the identity holds for any fixed binary mask and does not assert that GIFT globally maximizes the distance to . Its third lexicographic criterion maximizes only the selected residual energy on the defined-reference, non-conflicting branch, after the first two objective coordinates have been optimized. This restricted selection property is distinct from maximizing the total residual energy over all tokens.
Part (iii). Apply Definition 1 with , and , so . Write and , and expand the denominator using :
Hence has a numerator independent of and a denominator strictly increasing in (slope ), so is strictly decreasing in wherever the denominator is positive. On the safe branch we have , strictly decreasing in ; composing the two monotonicities, is strictly increasing in there. At (so , the two gradients collinear) the value is , which lies in for : the upper bound is strict by AM–GM, and the lower bound follows from with equality iff . At (orthogonal) the value is exactly 1, consistent with Lemma 2.
For the unconstrained statement, is strictly decreasing in over the whole range , hence maximized as ; there and the denominator equals , which vanishes when , i.e. exactly when , at which point diverges. By Lemma 2 this is the regime , in which the two gradients partially cancel.
Theorem 3 (selection)
Proof. Write , , and , with the sets and objectives defined before Equation equation 7. Feasibility means and , for an integer .
Separation. If , then . If , then ; if , then . Thus every candidate in precedes every candidate in .
First coordinate. Every size- subset satisfies
A top- set attains this bound: it contains when , and contains all of otherwise. This proves Equation equation 9.
Second coordinate. Let . If , then for every first-coordinate minimizer. If , every such minimizer contains all of and exactly elements of . Since on , top- selection chooses the smallest , minimizing .
Third coordinate. If , all first-coordinate minimizers contain , so is constant. If , define on and on . Then on , and for . Because squaring preserves order on , the largest scores maximize .
Equal scores give equal contributions to the relevant objective coordinate, including ties between and no-usable-reference positions. Hence arbitrary tie-breaking preserves all three objective values. The cases and are included, as is the empty pool with .
Remark (Normalized opposition is not primary-loss damage). By Theorem 1, the first-order change in the primary loss induced by an admitted conflicting token is , which scales with as well as . The coordinate is therefore a normalized opposition—first-order damage per unit of RL-gradient norm (with )—and serves only as a ranking key. It coincides with aggregate primary-loss damage only if all admitted conflicting tokens share the same , which is non-generic; we therefore avoid calling it harm or severity.
Remark (Exact quota vs. hard constraint). Equation 7 imposes rather than the hard-constrained program subject to for selected and , which admits no conflicting token and may select fewer than tokens. That hard-constrained rule is implemented as the Conflict-Free ablation (Appendix § 13.§ 4, Table 4). The two coincide whenever the quota does not exceed the number of non-conflicting eligible tokens (, the first case of Theorem 3); they differ only when the quota forces admission of conflicting tokens, in which case GIFT admits those with the smallest normalized opposition. The exact quota (Equation (29)) moreover keeps realized sub-batch coverage a deterministic function of the schedule and the eligible set, which the exposure accounting of Appendix § 9 relies on.
Behaviour at GRPO Stationarity
Corollary 1 (Local behavior when the primary token gradient vanishes). Suppose *and the token-level primary surrogate is differentiable. Then every local logit perturbation is first-order neutral to that token-level surrogate. Under the GIFT objective, the auxiliary logit-gradient block at token is
Hence token contributes a nonzero distillation update only if , , , and . Eligibility or a zero score alone does not guarantee selection.
Proof. Because , the directional derivative of along any local logit perturbation is $r_i^\mathsf{T}v = 0. The auxiliary term in the GIFT objective is multiplied by the detached mask and normalized by , which gives the stated block. When no usable reference direction exists, the implementation assigns token score zero; its update therefore depends on whether it is selected under the exact quota.
When the sampled GRPO gradient vanishes, GIFT assigns the token a zero score; whether it receives distillation still depends on the exact-quota selection mask ((Corollary 1)).
We record this as a structural observation rather than a claim about aggregate behaviour. Whether a prompt whose group is uniformly correct or wrong should receive teacher supervision is an empirical question—the supervision is first-order free with respect to GRPO, but freeness is not the same as usefulness, and the group-level reward carries no information about the teacher’s reliability there. The corollary establishes only that GIFT’s scoring is well defined in this regime and that it assigns such tokens neutral rather than negative priority.
Termination and Bounded Exposure
Corollary 2 (Termination and nominal exposure). For all masks vanish and equals the native GRPO objective identically, so teacher and scoring passes may be skipped and the terminal phase is unprivileged. The nominal teacher exposure over actor steps is
equal to for .
Proof. For , gives , so TopK returns and for all ; the auxiliary sum in Equation (11) vanishes identically and . Since the teacher and scoring computations enter the loss only through that sum, both may be skipped without changing the objective or its gradient. For the exposure, the first steps contribute , and substituting ,
while steps contribute zero. Dividing by gives Equation (26), with it is .
Remark. This is nominal exposure. Realized exposure can be strictly smaller because , so (26) is an upper bound on the fraction of token-steps receiving privileged supervision.
Coverage Annealing Does Not Renormalize Survivors
Proposition 3. Holding the rollout, model parameters, teacher targets, detached selection mask, and token weights under (11), the auxiliary gradient contributed by a retained token is , independent of the budget and of which other tokens are retained. Under coefficient annealing at full coverage the contribution is , decaying at every position simultaneously. Under selected-set normalization with , it is , which grows as the budget shrinks.
Proof. Differentiating (11) with , and teacher targets detached, the auxiliary gradient is . The coefficient of a retained term is , and is fixed by the rollout, independent of , and the mask; is constant. Replacing by scales all terms by the common factor , so no position retains full strength once . Replacing by gives per-token weight ; since is nondecreasing in , is nondecreasing in the budget and the weight is nonincreasing in it, strictly whenever strictly decreases. □
Remark. This concerns per-token coefficients, not the aggregate auxiliary gradient norm, which need not decrease monotonically under coverage annealing because the retained set concentrates on the largest .
Baselines
We compare GIFT with the initial model (BASE) and eight training baselines, grouped by supervision source and how that supervision enters optimization.
Initial model and outcome-based optimization. These baselines establish performance before additional training and under verifier-based reinforcement learning without teacher feedback.
BASE: The initial model checkpoint without the additional task-specific training considered in our experiments.
GRPO [20]: Performs pure reinforcement learning using verifier-derived group-relative advantages, without teacher-derived token-level supervision.
Distributional self-distillation. These methods provide dense supervision by matching next-token distributions from a self-teacher conditioned on additional reference information or environment feedback.
OPSD [34]: Conditions a self-teacher on a correct reference answer or reasoning trace and matches its next-token distributions along student-generated trajectories. Its original formulation uses a forward-KL distillation objective without verifier-derived advantages.
SDPO [7]: Conditions a self-teacher on environment feedback or verified successful solutions and distills its next-token distributions along existing student trajectories.
Verifier-guided token-level credit assignment. These methods combine verifiable outcomes with scalar teacher-derived signals by modulating token-level advantages or gating token-level rewards.
RLSD [28]: Uses sampled-token probability ratios between a privileged self-teacher and the student to construct positive, clipped weights for verifier-derived advantages. This sign-preserving modulation redistributes credit across tokens and is annealed during training.
RLCSD [18]: Contrasts sampled-token likelihoods under correct- and incorrect-solution-conditioned teachers and applies bounded, threshold-gated modulation to verifier-derived advantages while preserving their signs.
OPDVR [12]: Gates sampled-token teacher–student log-ratio rewards according to trajectory correctness, retaining non-negative rewards for correct trajectories and non-positive rewards for incorrect trajectories.
Demonstration-guided methods. These methods incorporate reference reasoning trajectories into training samples or rollout contexts rather than relying only on teacher distributions along unassisted student rollouts.
LUFFY [26]: Combines off-policy teacher demonstrations with on-policy student rollouts and uses regularized importance sampling to learn from demonstrations while retaining student exploration.
UFT [13]: Initializes rollouts with prefixes of reference solutions, applies supervised learning to the provided prefixes, and performs reinforcement learning on the generated continuations. The expected prefix length is progressively reduced until training becomes hint-free.
These comparisons separate the source of teacher guidance from how it enters optimization. In contrast, GIFT uses gradient-informed token selection and coverage annealing to control the contribution of an auxiliary distributional loss while retaining the GRPO objective throughout training.
Detailed Experimental Setup
This appendix describes the training data, evaluation benchmarks, GIFT configuration, and evaluation protocol. The model-specific optimization and systems settings reported below refer to the OLMo-3-7B-Think configuration. They are listed separately from the settings shared by the experimental design.
Training Datasets
DeepMath-103K. DeepMath-103K [6] contains 103,022 English mathematical-reasoning examples covering six major topics. Each problem is paired with a final answer, a GPT-4o-estimated difficulty score, a topic label, and three DeepSeek-R1 reasoning traces. The released dataset undergoes semantic decontamination, question standardization, and answer-consistency filtering, supporting both outcome verification and reference-conditioned distillation.
Knights and Knaves. We adopt the synthetic Knights and Knaves (K&K) task from Logic-RL [23], where each character is either a truth-telling knight or a lying knave. Given their logically composed statements, the model must infer the unique role assignment, which can be verified using deterministic rules. We train on puzzles with 4–8 characters and evaluate role-count generalization on larger instances of the same task.
Evaluation Benchmarks
AMC 2023. AMC evaluates mathematical problem solving across algebra, geometry, number theory, counting, and probability (math-ai, 2023). Our evaluation set contains 83 problems, denoted as AMC23 in the result tables, and is scored by matching the generated final answer.
AIME24 and AIME25. AIME24 and AIME25 each contain 30 problems from the corresponding year’s AIME I and AIME II examinations [32, 33]. Each answer is an integer from 0 to 999 and is evaluated using final-answer matching. Results for the two years are reported separately.
In-distribution and out-of-distribution logic. For logical reasoning, we train on Knights-and-Knaves puzzles (K&K) [23] with 4–8 roles. We evaluate on 500 in-distribution problems and three out-of-distribution splits with 9, 10, and 11 roles, each containing 100 problems, to assess generalization beyond the training range.
GIFT Configuration
Privileged self-teacher. The student generates responses without access to the reference solution. A self-teacher then re-evaluates the same response prefixes with the reference solution and answer included in its context. The teacher shares the current actor parameters, with gradients stopped through the teacher branch, and is used only during training.
Sparse forward-KL distillation. Let and be the student and teacher distributions at position , and let contain the teacher’s top-100 tokens. We aggregate the remaining vocabulary into a single tail and compute
where and likewise for . Teacher probabilities are treated as fixed targets.
Token ranking and allocation. For eligible response tokens, we compute the gradient geometry statistics and defined in the main text. The ranking score is when and otherwise; tokens without a usable GRPO direction () are unclassified and receive score zero. Thus, conflicting tokens remain eligible but are withdrawn first as coverage decreases: when the quota exceeds the number of non-conflicting and unclassified eligible tokens, the least opposing conflicting tokens fill the remaining capacity, as guaranteed by Theorem 3. The quota in Equation 25 is exact— whenever —so realized coverage tracks deterministically at the sub-batch level. Selection is performed independently within each training sub-batch :
where and denote its valid and eligible response tokens, respectively. The highest-scoring eligible tokens are selected.
Coverage schedule and objective. Training spans 96 actor steps with
We fix the distillation coefficient to . Selected tokens receive the complete sparse forward-KL loss, normalized by the number of all valid response tokens, while GRPO remains active throughout training. From step 78, teacher evaluation is skipped and optimization reduces to GRPO alone.
Ablation Coverage and Qualitative Diagnostics
We examine the token-selection criterion, allocation scope, and teacher-withdrawal strategy. Let and denote the GRPO and OPSD loss gradients with respect to the student logits. The signed parallel component and orthogonal residual norm follow the definitions in the method section.
Token-selection strategies.
All Tokens: Adds the OPSD loss directly to the GRPO objective at all valid response positions, without token selection. This variant evaluates joint training with unfiltered distillation.
Random Tokens: Selects token positions at random under the distillation budget, without using uncertainty or gradient geometry. It tests whether selective coverage alone is sufficient.
Entropy–KL Selection: Following the Q3 criterion in TIP (Xu et al. [24]), this variant prioritizes positions where the student is confident but disagrees with the teacher. Let
denote the normalized student entropy. The selection score is
Tokens with larger scores have greater teacher–student divergence and lower student uncertainty. They are selected under the same coverage budget as GIFT.
Gradient-cone Selection: Uses the cosine similarity between the OPSD and GRPO logit gradients, , at positions with usable gradients. This criterion measures directional agreement without separately accounting for the magnitude of the orthogonal residual.
Conflict-only: Uses only the conflict information provided by the parallel component for token selection, omitting residual-based prioritization. It isolates the contribution of conflict assessment.
Residual-only: Selects tokens solely according to the orthogonal residual norm , without distinguishing conflicting from non-conflicting positions through the sign of .
Teacher-norm: Selects tokens according to , the magnitude of the teacher-induced distillation gradient with respect to the student logits. It does not separate parallel and orthogonal contributions.
Conflict-Free: Excludes every token with from distillation, regardless of the available budget. Unlike GIFT’s conflict-sensitive ordering, this variant never admits conflicting tokens to fill unused capacity.
Allocation scopes. These variants change the set of tokens that compete for the distillation budget.
Per-rollout: Ranks and selects tokens independently within each generated response. Tokens from different responses do not compete for a shared quota.
Per-prompt: Ranks tokens jointly across the group of responses generated for the same prompt. Tokens can compete across responses within the group, but not across different prompts.
Training sub-batch (GIFT): Ranks tokens jointly within each training sub-batch, allowing competition across the responses and prompts it contains. Selection budgets remain independent across training sub-batches.
Teacher-withdrawal strategies.
Sequential OPSDGRPO: Performs OPSD training first and then switches to GRPO, separating distillation and outcome-based optimization into successive stages.
Weight Annealing: Gradually reduces the distillation-loss coefficient, weakening teacher supervision through its weight rather than progressively removing token positions.
Coverage Annealing (GIFT): Progressively reduces the token-retention budget while keeping the distillation coefficient fixed at . Selected positions receive the full distillation loss, and training becomes pure GRPO when coverage reaches zero.
The preliminary schedule sweep evaluates linear decay with equal to , , , , and , and cosine decay with and . Here, denotes the initial full-coverage period and the teacher exit step. All-token and conflict-free variants also change coverage or eligibility; schedule comparisons do not imply matched cumulative teacher exposure or training cost.
Additional Experiments
We provide additional experiments on model-family generalization, teacher withdrawal, and schedule sensitivity. These analyses examine the central design of GIFT: dense teacher guidance is used early, progressively restricted to higher-priority tokens, and eventually removed to recover pure outcome-grounded optimization. We evaluate whether this strategy transfers beyond Qwen3, outperforms alternative withdrawal mechanisms, and remains effective under different annealing configurations.
Evaluation beyond the Qwen3 family. On OLMo-3-7B-Think, GIFT achieves the highest mean@12 on both AMC23 and AIME25 (Figure 2). Its two-benchmark average of 80.70% exceeds Base and the strongest baseline, RLCSD, by 2.34 and 1.12 percentage points, respectively. The gain over Base is larger on AIME25 than on AMC23 (3.50 versus 1.18 points), consistent with the trend observed across the Qwen3 models. These improvements are consistent with GIFT’s central motivation: rather than uniformly applying teacher supervision, GIFT prioritizes non-conflicting updates
Qualitative Analysis
that contain directional information beyond the local RL gradient, while progressively reducing their coverage. This allows dense guidance to complement the outcome-grounded objective during early training without remaining the final optimization target. The consistent gains on OLMo-3-7B-Think further suggest that this mechanism is not specific to the Qwen3 model family.
Coverage annealing versus alternative withdrawal strategies. Table 7 compares GIFT with two alternative teacher-withdrawal strategies. In sequential OPSDGRPO training, the model is first optimized using only the OPSD objective and then switched to pure GRPO, so dense teacher supervision and outcome-based optimization are applied in separate stages. Weight annealing instead combines OPSD with GRPO throughout the transition but applies distillation to all valid tokens and gradually decays its global loss coefficient, without annealing token coverage. GIFT outperforms these two alternatives by 1.59 and 2.87 percentage points on average, respectively. Unlike sequential training, GIFT keeps the outcome-grounded GRPO objective active while teacher guidance is available. Unlike weight annealing, it progressively removes lower-priority positions while preserving the full distillation strength at selected tokens. This design retains concentrated teacher guidance where it is more compatible with and complementary to the RL update, rather than uniformly weakening every teacher contribution. Once coverage reaches zero, the auxiliary loss vanishes and optimization continues with pure GRPO. The results support joint optimization with selective coverage annealing over either stage-wise training or uniform coefficient decay.
Sensitivity to warm-up and exit steps. Table 8 presents a preliminary sweep of the full-coverage warm-up and teacher exit step . With , removing warm-up or extending it to 20 steps reduces the average from 78.96% to 78.03% or 77.26%, respectively. With , moving teacher exit to step 50 or 90 gives 77.46% or 77.64%. The default configuration therefore performs best among the tested linear schedules. These results favor an initial full-coverage period followed by gradual withdrawal, while showing that either shortening or extending teacher guidance can reduce performance.
| Schedule | AMC23 | AIME24 | AIME25 | Avg. | ||
| Linear (GIFT) | 10 | 78 | 89.66 | 76.11 | 71.11 | 78.96 |
| Linear | 0 | 78 | 89.36 | 75.56 | 69.17 | 78.03 |
| Linear | 20 | 78 | 89.56 | 75.28 | 66.94 | 77.26 |
| Linear | 10 | 50 | 90.16 | 76.39 | 65.83 | 77.46 |
| Linear | 10 | 90 | 89.86 | 75.00 | 68.06 | 77.64 |
| Cosine | 10 | 78 | 90.46 | 76.11 | 67.22 | 77.93 |
| Cosine | 0 | 78 | 89.96 | 73.89 | 67.22 | 77.02 |
Table 8. Preliminary sensitivity to the full-coverage period , teacher exit step , and decay shape on Qwen3-8B (mean@12, %). Avg. equally weights the three benchmarks. Bold marks the best score in each evaluation column.
Together, these experiments support GIFT’s combination of conflict-aware token selection, shared training-sub-batch allocation, and progressive teacher withdrawal. The evidence concerns reasoning performance and observed selection behavior, rather than direct measurements of parameter-space gradient compatibility or optimization plasticity.
Qualitative Analysis
We provide three qualitative analyses to examine both the reasoning outcomes and token-selection behavior of GIFT. First, we compare representative solutions from GIFT, GRPO, and OPSD to illustrate how combining dense guidance with outcome-grounded optimization affects reasoning accuracy. Second, we visualize the token positions retained at different training steps, showing how supervision becomes progressively sparse within a response. Third, we analyze retention in the space of the signed parallel component and orthogonal residual norm , connecting the observed selection pattern to GIFT’s piecewise ranking rule. Together, these visualizations illustrate how GIFT transitions from broad teacher supervision to concentrated guidance on higher-priority positions as the coverage budget contracts.
Qualitative analysis of reasoning accuracy. Figure 3 compares solutions to a combinatorial counting problem. All three displayed solutions separate strings according to whether they contain a digit 4, but differ in how accurately they execute the subsequent counting. GRPO undercounts the no-4 case as 283 instead of 671, while OPSD miscalculates intersection terms in inclusion–exclusion and obtains an incorrect total of 1336. GIFT correctly handles both cases, yielding 671 + 625 = 1296. Across 12 sampled responses to this problem, GIFT produces 12 correct answers, compared with 7 for GRPO and 6 for OPSD. The contrast connects to our introductory motivation: a plausible reasoning structure does not ensure correct intermediate decisions, and neither outcome-based training nor self-distillation alone eliminates such errors in these examples. GIFT’s more consistent answers provide a qualitative illustration of the goal of combining dense guidance with outcome-grounded optimization, complementing the quantitative ablations of its selection and withdrawal mechanisms.

Figure 3. Example solutions to a combinatorial counting problem. The displayed GIFT solution is correct, while the GRPO and OPSD solutions contain counting errors. Correct-sample counts summarize 12 generations per method.
Qualitative analysis of token selection. Figure 4 illustrates how distillation coverage changes within a reasoning excerpt about convex functions. All 69 displayed tokens are selected at step 10, compared with 40 at step 40 and 17 at step 70. The remaining positions include words such as “convex,” “outside,” and “continuity,” while many surrounding words and mathematical symbols no longer receive distillation. Supervision therefore becomes sparse within sentences rather than being removed at the sentence level. This example illustrates the allocation problem raised in the introduction: reducing teacher influence involves deciding where guidance remains, not merely weakening every teacher contribution. GIFT makes this decision through gradient-based ranking within each training sub-batch. Selected positions retain the full distillation loss, while all valid tokens remain included in the GRPO objective. The visualization shows where supervision is applied, rather than establishing the semantic importance or correctness of individual tokens.

Figure 4. Token-level distillation selection in a reasoning excerpt. Red marks selected tokens and gray marks unselected tokens. The number of selected positions decreases from 69 to 40 to 17 across the displayed steps; counts refer only to the shown excerpt.
Token selection under budget annealing. Figure 5 shows how GIFT concentrates distillation as the token budget contracts. At step 10, all eligible tokens are retained, including conflicting and neutral positions. At step 40, conflicting-token retention is zero in the displayed statistics, while 97.4% of non-conflicting tokens and 60.9% of neutral tokens remain selected. By step 70, these rates decrease to 21.0% and 10.8%, respectively, with non-conflicting selection concentrated at larger orthogonal residuals. This pattern follows the piecewise ranking: conflicting tokens receive negative scores, non-conflicting tokens are prioritized by residual magnitude, and neutral tokens remain eligible with zero scores. Competition within each training sub-batch therefore produces a selective withdrawal of supervision rather than uniform attenuation. Retained tokens receive the full, unprojected distillation loss, while all valid response tokens remain included in the GRPO objective.

Figure 5. Token retention under coverage annealing. The horizontal and vertical axes show the signed parallel component and orthogonal residual norm of the distillation gradient relative to the GRPO direction. Colors indicate the fraction of eligible tokens retained within each bin. Annotations summarize retention for conflicting () and non-conflicting () tokens with usable GRPO reference directions. Bottom bars separately report retention for neutral tokens without such a direction.
References
- [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pp. 21246–21263, 2024.arxiv.org/abs/2306.13649
- [2]Lucio M. Dery, Yann Dauphin, and David Grangier. {AUXILIARY} {task} {update} {decomposition}: {THE} {good}, {the} {bad} {and} {the} {neutral}. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=1GTma8Hw1P.arxiv.org/abs/2108.11346
- [3]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minllm: On-policy distillation of large language models. arXiv preprint arXiv:2306.08543, 2023.arxiv.org/abs/2306.08543
- [4]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihuao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Wu Yu, Z. F. Wu, Zhibin Guo, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochaо Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecheng Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaoјun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzუ Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zhou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zhihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z.
- [5]Zhuowen Han, Jinwei Xiao, Zhengxi Lu, Renren Jin, Zhiyuan Yao, Yuxin Liu, Hongyan Hao, Yueqing Sun, Yu Yang, Qi GU, Xunliang Cai, and Deyi Xiong. Distill where you fail: Recovering learning signals of negative rl-groups from adaptive teacher guidance, 2026. URL https://arxiv.org/abs/2608.00782.
- [6]Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning, 2025. URL https://arxiv.org/abs/2504.11456.
- [7]Jonas Hübötter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation, 2026. URL https://arxiv.org/abs/2601.20802.
- [8]Yubo Jiang, Fengying Xie, Zhiguo Jiang, and Haopeng Zhang. Distill skills into weights, not prompts: Abstract skills as privileged signals for on-policy self-distillation, 2026. URL https://arxiv.org/abs/2608.09826.
- [9]Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026.arxiv.org/abs/2603.07079
- [10]Jeonghye Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Rebellious student: Reversing teacher signals for reasoning exploration with self-distilled rlvr. arXiv preprint arXiv:2605.10781, 2026.arxiv.org/abs/2605.10781
- [11]Boyan Li, Bingsen Chen, Chenghao Yang, Ping Nie, Chen Zhao, and Xi Ye. Sequential beats joint: On the interplay between on-policy distillation and rlvr, 2026. URL https://arxiv.org/abs/2609.04108.
- [12]Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, and Gao Huang. On-policy distillation with verifiable reward. arXiv preprint arXiv:2608.24696, 2026.arxiv.org/abs/2608.24696
- [13]Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and reinforcement fine-tuning. arXiv preprint arXiv:2505.16984, 2025a.arxiv.org/abs/2505.16984
- [14]Yifeng Liu, Shiyuan Zhang, Yifan Zhang, and Quanquan Gu. Self-distilled policy gradient. arXiv preprint arXiv:2606.04036, 2026.arxiv.org/abs/2606.04036
- [15]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
- [17]Team Olmo, ; Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Shashank Gupta, Taira Anderson, Teng Xiao, Tyler Murray, Tyler Romero, Victoria Graf, Akari Asai, Akshita Bhagia, Alexander Wettig, Alisa Liu, Aman Rangapur, Chloe Anastasiades, Costa Huang, Dustin Schwenk, Harsh Trivedi, Ian Magnusson, Jaron Lochner, Jiacheng Liu, Lester James V. Miranda, Maarten Sap, Malia Morgan, Michael Schmitz, Michal Guerguin, Michael Wilson, Regan Huff, Ronan Le Bras, Rui Xin, Rulin Shao, Sam Skjonsberg, Shannon Zejiang Shen, Shuyue Stella Li, Tucker Wilde, Valentina Pyatkin, Will Merrill, Yapei Chang, Yuling Gu, Zhiyuan Zeng, Ashish Sabharwal, Luke Zettlemoyer, Pang Wei Koh, Ali Farhadi, Noah A. Smith, and Hannahneh Hajishirzi. Olmo 3, 2026. URL https://arxiv.org/abs/2512.13961.
- [18]Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, and Lijie Wen. Rlc sd: Reinforcement learning with contrastive on-policy self-distillation. arXiv preprint arXiv:2606.11709, 2026.arxiv.org/abs/2606.11709
- [19]Simon Schmitt, Jonathan J. Hudson, Augustin Židek, Simon Osindero, Carl Doersch, Wojciech M. Czarnecki, Joel Z. Leibo, Heinrich Küttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting deep reinforcement learning. ArXiv, abs/1803.03835, 2018. URL https://api.semanticscholar.org/CorpusID:3883991.arxiv.org/abs/1803.03835
- [20]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
- [21]Zhenlin Wei, Pu Jian, Yingzhuo Deng, Xiaohan Wang, Jiajun Chai, Zhexin Hu, Wei Lin, Shanbin Zhang, and Guojun Yin. Amr-sd: Asymmetric meta-reflective self-distillation for token-level credit assignment, 2026. URL https://arxiv.org/abs/2605.18529.
- [22]Xingjian Wu, Junlin Liu, Xingchen Liu, Xuhang Zhu, Jianing Wang, Linsen Guo, Xiaoyu Li, Xuezhi Cao, and Xunliang Cai. Contrastive reinforced policy optimization via privileged self-distillation. 2026. URL https://arxiv.org/abs/2607.28026.
- [23]Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, 2025. URL https://arxiv.org/abs/2502.14768.
- [24]Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026.arxiv.org/abs/2604.14084
- [25]Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance, 2025. URL https://arxiv.org/abs/2504.14945, 2025a.
- [26]Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance, 2025b. URL https://arxiv.org/abs/2504.14945.
- [27]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengyuan Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
- [28]Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr. arXiv preprint arXiv:2604.03128, 2026.arxiv.org/abs/2604.03128
- [29]Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient diversity: a key ingredient for scalable distributed learning, 2018. URL https://arxiv.org/abs/1706.05699.
- [30]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xianpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale. 2025. URL https://arxiv.org/abs/2503.14476.
- [31]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782, 2020.arxiv.org/abs/2001.06782
- [32]Yifan Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2024, 2024. URL https://huggingface.co/datasets/math-ai/aime24.
- [33]Yifan Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2025, 2025. URL https://huggingface.co/datasets/math-ai/aime25.
- [34]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026.arxiv.org/abs/2601.18734
