betaFeedback

WorksComputer Science

GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL

Abstract

Training large language models for complex reasoning tasks relies on two complementary signals: sparse, outcome-grounded reinforcement learning (RL) for correctness, and dense, token-level distillation for guidance. However, complementarity in supervision does not guarantee compatibility in optimization; imitation-focused distillation can conflict with the outcome-driven RL objective, and existing hybrids lack a principled resolution. We propose GIFT (Gradient-Informed FilTering for Distillation-Augmented Reasoning RL), a novel method that directly addresses the allocation problem of which teacher updates to apply. GIFT performs a token-level geometric analysis, decomposing the distillation gradient into a component parallel to the RL gradient, which measures first-order conflict and provides a safety filter, and a component orthogonal to it, which quantifies novel directional guidance and provides a measure of value. This decomposition yields a principled scoring rule to rank token-level guidance. We combine this with a progressive coverage annealing schedule that shrinks the budget of tokens receiving distillation, ensuring that the most conflicting updates are discarded first while the most complementary ones are retained at full strength. This facilitates a smooth transition from dense, geometrically-filtered guidance to a pure, outcome-grounded objective at convergence. Extensive experiments on mathematical and logical reasoning benchmarks show that GIFT consistently outperforms strong baselines across multiple model scales, achieving relative gains of up to 9.5% on difficult mathematical reasoning tasks. Code: https://github.com/lilifree/GIFT

Introduction

“Thoughts without content are empty, intuitions without concepts are blind. Only through their union can knowledge arise.”

— Immanuel Kant, Critique of Pure Reason

Training large language models for complex reasoning relies on two complementary forms of supervision: outcome-grounded reinforcement learning and dense, token-level distillation. Reinforcement learning with verifiable rewards (RLVR), such as GRPO [20], anchors learning to task success, but offers little guidance about the intermediate decisions that produce an outcome. A single reward supervises an entire response, and GRPO’s group-relative signal vanishes when all responses receive the same reward. On-policy self-distillation (OPSD; [34]) provides more direct guidance: the same model, conditioned on privileged information such as a reference solution, supplies teacher distributions along the student’s own rollout. Yet this dense supervision targets imitation rather than correctness; following it can introduce information leakage and late-stage instability ([28, 21]). We seek the best of both worlds: dense teacher guidance to accelerate learning, with verifiable outcomes determining what the model ultimately optimizes.

Realizing this promise requires more than combining the two signals: complementarity in supervision does not guarantee compatibility in optimization. RLSD and RL CSD use teacher-derived signals to modulate verifier-based advantages ([28, 18]), while OPDVR gates sampled-token distillation rewards by trajectory correctness [12]. For a nonzero RL gradient at a fixed token, such scalar changes cannot introduce a direction outside its logit-space span. An auxiliary distributional loss, as in SDPG [14], can supply such directions, yet admitting a response or token does not establish how its teacher gradient interacts with the current RL update. Teacher withdrawal addresses persistence: RLSD and SDPG anneal teacher influence, and Sequential Beats Joint separates distillation and RLVR into successive stages [11]. But withdrawal alone leaves an allocation problem. Uniformly weakening or abruptly removing guidance can discard complementary updates together with redundant or conflicting ones. Selecting surviving tokens by disagreement alone can retain a strong teacher signal precisely where it obstructs the outcome update. Which token-level teacher contributions should survive as the available supervision budget contracts?

We address this allocation problem with our method, GIFT (Gradient Informed FilTering for Distillation Augmented Reasoning RL), which uses token-level gradient geometry to determine where auxiliary distillation participates. Instead of relying on heuristics like disagreement or uncertainty, GIFT directly analyzes the interaction between the teacher’s distillation gradient and the student’s RL gradient. At each token, we decompose the teacher’s update into two key components: (1) a component parallel to the RL gradient, which measures first-order conflict or assistance, and (2) a component orthogonal to the RL gradient, which represents novel directional guidance that cannot be produced by simply rescaling the RL update. Our central hypothesis is that the most effective guidance is both non-conflicting and contains significant novel direction. However, since privileged teacher guidance (as in OPSD) targets imitation rather than guaranteed correctness, its influence must be carefully managed to avoid biasing the final policy. GIFT addresses this by implementing its geometric selection under a progressively shrinking token budget. This annealing process ensures that as training matures, the model transitions from dense but potentially flawed guidance towards the verifiable outcome signal. Only the most geometrically complementary teacher contributions survive before the model converges to the pure, unprivileged RL objective. This approach leads to a method that is both principled and effective. Our main contributions are:

  • We propose GIFT, a method that uses the geometric relationship between teacher and student gradients to selectively apply distillation. It is the first method to explicitly distinguish between conflicting, redundant, and complementary token-level guidance to solve the allocation problem.

  • We derive a closed-form, token-level scoring rule based on this geometry, allowing for efficient selection without needing to compute auxiliary backward passes. We combine this with a progressive coverage annealing schedule that smoothly transitions from dense guidance to pure reinforcement learning, ensuring stability and preserving the primacy of the outcome-based objective.

  • GIFT achieves the highest average scores on challenging mathematical reasoning benchmarks across four model scales, outperforming eight strong baselines. Ablation studies confirm that its combination of geometric selection and coverage annealing is critical to its success, yielding significant gains over simpler selection criteria or withdrawal strategies.

Related Work

Dense teacher guidance can supplement outcome supervision, but its usefulness depends on how it is incorporated, not simply on how much is provided. We situate GIFT along four related design choices: how teacher information enters the update, which tokens receive it, how its interaction with RL is assessed, and how its influence is withdrawn.

Outcome Supervision and Self-Distillation. RL from verifiable outcomes, or RLVR, forms the basis of our primary objective. GRPO [20] derives group-relative advantages from verified outcomes; subsequent works like Dr. GRPO [15] and DAPO [30] refine its normalization and loss aggregation to improve stability and reduce bias. While these methods provide an unbiased, outcome-grounded learning signal, their supervision is sparse and vanishes for rollout groups where all responses are correct or incorrect. On-policy self-distillation (OPSD; [34]) offers a dense, token-level alternative, supplying teacher feedback along the student’s trajectory. Hybrid methods, such as RLSD [28], RLCSD [18], and AMR-SD [21], attempt to combine these two signals by using teacher-derived signals to perform scalar modulation of the RL advantages. However, they are fundamentally limited in expressiveness: re-weighting the RL gradient cannot introduce an update direction outside its original logit-space span. GIFT uses OPSD-RLVR geometry to prioritize complementary guidance under exact quotas, without excluding all conflicts.

Token-Selective Teacher Guidance. Given a dense teacher signal, a critical question is which tokens should receive it. Many methods rely on statistical heuristics. TIP Xu et al., [24] ranks tokens by student entropy and teacher-student divergence; EOPD Jin et al., [9] gates supervision by teacher entropy; RSTG Han et al., [5] combines these heuristics for failed rollouts; and CRPO Wu et al., [22] uses predictive entropy to distinguish exploration from exploitation bias. While these statistics identify uncertainty or disagreement, they do not directly establish compatibility with the current outcome-driven update. Other works highlight that simple agreement is not the goal. RLRT Kim et al., [10] reinforces successful student departures from the teacher, demonstrating teacher disagreement can be valuable. SKALD Jiang et al., [8] focuses on zero-variance groups where the RL signal is absent, providing distillation as a fallback. This aligns with GIFT’s treatment of such cases (Corollary 1), but GIFT’s geometric criterion is more general. Instead of relying on heuristics, GIFT directly assesses the first-order interaction between the teacher’s distillation gradient and student’s RL gradient at each token, providing a principled selection basis.

Gradient Geometry for Auxiliary Learning. The geometric relationship between gradients from multiple objectives is a well-established concept in multi-task learning. Methods like PCGrad Yu et al., [31] project away conflicting gradient components, while ATTITTUD Dery et al., [2] decomposes an auxiliary gradient into components agreeing, conflicting, or orthogonal to the primary task’s gradient subspace, and re-weights them. ATTITTUD is the direct intellectual predecessor to our geometric analysis. However, GIFT makes three critical and novel adaptations for the RLVR context: (1) Usage: We use geometry for discrete selection of tokens, not continuous modification of the auxiliary gradient, which avoids distorting the teacher signal. (2) Granularity and Efficiency: Our analysis is performed at the token level in logit space, where we derive a closed-form, O(M)O(M) solution (Proposition 1) for the geometric components, avoiding expensive, parameter-space SVD approximations required by ATTITTUD. (3) Value of Orthogonality: We identify the orthogonal component (bib_i) as the most valuable form of complementary guidance—representing novel directions unreachable by the RL signal (Theorem 2)—whereas ATTITTUD treats it as neutral.

Demonstration-Guided RL and Teacher Withdrawal. Privileged teacher guidance must be carefully managed to avoid biasing the final policy towards imitation. This withdrawal can be implemented in several ways. Methods like LUFFY Yan et al., [25] and UFT Liu et al., [13] introduce teacher guidance through trajectory composition or annotated prefixes, and anneal the length or proportion of this guidance. Other methods anneal the auxiliary loss coefficient (λ\lambda) to zero, as seen in Kickstarting Schmitt et al., [19], RLSD, and SDPG. However, such coefficient annealing dilutes the signal for all tokens simultaneously, discarding complementary updates along with conflicting ones. Sequential Beats Joint Li et al., [11] avoids this by separating distillation and RLVR into successive stages, but forgoes the benefits of joint training. GIFT addresses this optimization tension through coverage annealing, shrinking the selected-token quota while fixing the loss weight. At each ranking, positive residual scores precede zero scores and conflicts; within conflicts, smaller normalized opposition receives higher priority. This controls teacher participation while preserving retained-token loss weights, before the auxiliary loss vanishes at teacher exit.

Method

In the introduction, we established the central challenge: to effectively merge the dense, token-level guidance from self-distillation Liu et al., [14]; Agarwal et al. [1]; Gu et al. [3] with the sparse, outcome-grounded supervision from reinforcement learning Shao et al., [20]; Guo et al. [4] (RL). Simply combining them is insufficient; we need a principled mechanism to decide when and where to apply the teacher’s guidance to ensure it complements, rather than corrupts, the primary RL objective. This section introduces our method, GIFT, which resolves this ”allocation problem” through a geometric analysis of the underlying gradients, as shown in Figure 1.

Figure 1. GIFT decomposes the distillation gradient into conflict (aia_i) and complement (bib_i) components relative to the RL gradient, ranks tokens, and anneals the selection budget from full to zero.

A Geometric Approach to Guidance

Our methodology is built on a direct analysis of the geometric interaction between the teacher and student gradients Dery et al., [2]; Yu et al. [31] at each token. Let zi∈R∣V∣z_i \in\mathbb{R}^{|V|} be the student’s logits at position ii, with probability distribution pi=softmax⁡(zi)p_i = \operatorname{softmax}(z_i). At this position, we consider two distinct logit-space gradients. The first is the outcome-driven gradient from the clipped GRPO loss [20], ℓiGRPO\ell_i^{\mathrm{GRPO}}. This gradient, denoted rir_i, represents the direction of steepest ascent for the RL objective, steering the model toward task success determined by the verifier. The second is the imitation-driven gradient from the distillation loss, ℓiSKL\ell_i^{\mathrm{SKL}} [1, 3, 34, 14]. This gradient, denoted did_i, points in the direction that best aligns the student’s distribution pip_i with the privileged teacher’s distribution q~i\tilde{q}_i. These gradients are:

ri=∇ziℓiGRPO=wiAiρihi(pi−eui),di=∇zi(wiℓiSKL)=wi(pi−q~i),(1)r_i = \nabla_{z_i}\ell_i^{\mathrm{GRPO}} = w_i A_i \rho_i h_i (p_i-e_{u_i}), \qquad d_i = \nabla_{z_i}(w_i\ell_i^{\mathrm{SKL}}) = w_i(p_i-\tilde{q}_i), \tag*{(1)}

where euie_{u_i} is the one-hot vector for the sampled token, AiA_i is the advantage, hih_i indicates an active clipped branch, and q~i\tilde{q}_i is the teacher distribution. The gradient rir_i serves as our reference direction. On the set of tokens D\mathcal{D} where a non-zero RL gradient exists, we decompose the teacher gradient did_i with respect to the normalized RL gradient r^i=ri/∥ri∥2\hat{r}_i=r_i/\lVert r_i\rVert_2:

ai=⟨di,r^i⟩,di⊥=di−air^i,bi=∥di⊥∥2.(2)a_i = \langle d_i,\hat{r}_i\rangle,\qquad d_i^{\perp}=d_i-a_i\hat{r}_i,\qquad b_i=\lVert d_i^{\perp}\rVert_2. \tag*{(2)}

Here, the scalar aia_i is the signed magnitude of the parallel component, measuring first-order assistance or conflict. The scalar bib_i is the norm of the orthogonal component, quantifying novel directional guidance. Together, they satisfy ai2+bi2=∥di∥22a_i^2+b_i^2=\lVert d_i\rVert_2^2. These two scalars form the core of our selection mechanism, as explained next.

Safety: The Role of the Parallel Component (aia_i)

Our first priority is to ensure that auxiliary guidance does not undermine the primary, outcome-driven objective. The parallel component aia_i provides a direct, first-order measure of this safety. Theorem 1 formalizes this intuition.

Theorem 1 (Local first-order non-interference). For a given token, the change in the primary GRPO loss from a combined update [2, 31] is given by

ddηfi(zi(η))∣η=0=−∥ri∥22−λ∥ri∥2ai.\left.\frac{d}{d\eta}f_i(z_i(\eta))\right|_{\eta=0}=-\lVert r_i\rVert_2^2-\lambda\lVert r_i\rVert_2a_i.

The implications are direct:

  1. Assistance (ai>0a_i>0): The teacher’s guidance is aligned with the RL gradient and accelerates the decrease of the primary loss. This guidance is beneficial.

  1. Neutrality (ai=0a_i=0): The teacher’s guidance is orthogonal to the RL gradient and has no first-order effect on the primary loss. It is safe, regardless of its magnitude ∥di∥2\lVert d_i\rVert_2.

  1. Conflict (ai<0a_i<0): The teacher’s guidance opposes the RL gradient, slowing or even reversing progress on the primary objective. This guidance is harmful.

This geometric criterion provides a more precise filter than heuristics like student-teacher divergence (e.g., TIP [24]) or disagreement. While a large divergence might indicate valuable guidance, it does not distinguish between productive, orthogonal disagreement (ai≥0,bi≫0a_i\geq0,b_i\gg0) and destructive, head-on conflict (ai<0a_i<0). Our use of aia_i makes this crucial distinction. Therefore, the sign of aia_i serves as a fundamental safety gate. We define any token with ai<0a_i<0 as conflicting and give it lowest priority. This raises the next question: among the ”safe” tokens (where ai≥0a_i\geq0), which are most valuable to select? This is decided by the orthogonal component, bib_i.

Value: Why the Orthogonal Component (bib_i) Matters

Having established safety with the parallel component aia_i, we now examine value. Among ”safe” tokens (ai≥0a_i\geq0), the most useful guidance lies in the orthogonal component, quantified by bib_i. Crucially, bib_i’s value does not stem from a direct, first-order contribution to the primary RL objective—as established in Theorem 1, its contribution to the primary loss descent is zero. Its profound importance relates instead to the guidance’s novelty and expressiveness. This value is best understood through its counterpart. A purely parallel teacher gradient (bi=0b_i = 0) is fundamentally redundant. The resulting update, ri+λdir_i + \lambda d_i, simplifies to merely rescaling the original RL gradient rir_i. This critical limitation offers no new directional information unattainable by simply adjusting the learning rate or advantage scaling for that token. An entirely collinear teacher tells the optimizer nothing it could not learn from the RL signal alone.

All genuinely new geometric information provided by the teacher resides in the orthogonal component di⊥d_i^\perp. Therefore, selecting for a large bib_i is equivalent to prioritizing the most valuable form of teacher guidance—the part that is structurally inaccessible to simpler advantage-modulation schemes. The following theorem formalizes this concept of complementarity in three distinct ways.

Theorem 2 (Complementarity of the orthogonal residual). Under the assumptions of Theorem 1:

  1. (Maximal locally GRPO-neutral distillation decrease.) Among unit-norm logit perturbations that are locally neutral to the token-level primary surrogate,

max⁡∥v∥2≤1, ri⊤v=0(−di⊤v)=bi.(3)\max_{\lVert v\rVert_2\leq1,\ r_i^\top v=0}(-d_i^\top v)=b_i. \tag*{(3)}

If bi>0b_i > 0, the maximum is attained by v∗=−di⊥/biv^*=-d_i^\perp/b_i. If bi=0b_i = 0, every feasible vv attains the value zero. Thus, bib_i is the maximal first-order distillation decrease among unit-norm logit perturbations that are locally neutral to the token-level primary surrogate; no finite-step or shared-parameter conclusion follows.

  1. (Irreducibility to token-local GRPO rescaling.) Fix the sampled tokens, model state, and reference gradients, and equip H=⨁i∈IR∣V∣\mathcal{H}=\bigoplus_{i\in I}\mathbb{R}^{|V|} with the Euclidean product norm ∥(xi)i∥H2=∑i∥xi∥22\lVert(x_i)_i\rVert_{\mathcal{H}}^2=\sum_i\lVert x_i\rVert_2^2. Let M0={(ciri)i∈I:ci∈R}\mathcal{M}_0=\{(c_i r_i)_{i\in I}:c_i\in\mathbb{R}\} be the space of unrestricted signed, token-local rescalings of the fixed GRPO reference field. For any fixed binary mask m∈{0,1}Im\in\{0,1\}^{I}, write Dm=(midi)i∈ID_m=(m_i d_i)_{i\in I}. Then

dist⁡2(Dm,M0)=∑i:ri≠0mi(∥di∥22−⟨di,ri⟩2∥ri∥22)+∑i:ri=0mi∥di∥22.(4)\operatorname{dist}^2(D_m,\mathcal{M}_0)=\sum_{i:r_i\ne0}m_i\left(\lVert d_i\rVert_2^2-\frac{\langle d_i,r_i\rangle^2}{\lVert r_i\rVert_2^2}\right)+\sum_{i:r_i=0}m_i\lVert d_i\rVert_2^2. \tag*{(4)}

In particular, for ϵg≥0\epsilon_g\geq0 and Dϵ={i:∥ri∥2>ϵg}D_{\epsilon}=\{i:\lVert r_i\rVert_2>\epsilon_g\}, the residuals defined in Equation (2) satisfy

dist⁡2(Dm,M0)≥∑i∈Dϵmibi2.(5)\operatorname{dist}^2(D_m,\mathcal{M}_0)\geq\sum_{i\in D_{\epsilon}}m_i b_i^2. \tag*{(5)}

Thus, selecting any i∈Dϵi\in D_{\epsilon} with bi>0b_i>0 certifies that the selected teacher field cannot be represented by token-local scalar rescalings of the fixed GRPO field. This is a sufficient, not a necessary, condition. A selected zero-reference block with di≠0d_i\ne0 also certifies irreducibility, as can a noncollinear small nonzero-reference block.

  1. (Gradient diversity.) For a collection of gradients {gj}j=1n\{g_j\}_{j=1}^{n} summed into one update, define

Δ({gj})=∑j∥gj∥22∥∑jgj∥22(6)\Delta(\{g_j\})=\frac{\sum_j\lVert g_j\rVert_2^2}{\left\lVert\sum_j g_j\right\rVert_2^2} \tag*{(6)}

(Definition 1; Δ=1/n\Delta=1/n when the gjg_j are identical and Δ=1\Delta=1 when they are mutually orthogonal, so larger Δ\Delta means less redundancy among the summands). Applying this to the two gradients combined at token ii, i.e. Δi=Δ({ri,λdi})\Delta_i=\Delta(\{r_i,\lambda d_i\}): at fixed ∥ri∥2\lVert r_i\rVert_2, ∥di∥2\lVert d_i\rVert_2 and λ>0\lambda>0, Δi\Delta_i is strictly decreasing in aia_i, hence strictly increasing in bib_i on the safe branch ai∈[0,∥di∥2]a_i\in[0,\lVert d_i\rVert_2], where it rises from (∥ri∥22+∥di∥22)/(∥ri∥2+λ∥di∥2)2(\lVert r_i\rVert_2^2+\lVert d_i\rVert_2^2)/(\lVert r_i\rVert_2+\lambda\lVert d_i\rVert_2)^2 at bi=0b_i=0 to 11 at bi=0→1b_i=0\to1 at bi=∥di∥2b_i=\lVert d_i\rVert_2. Unconstrained over all aia_i, however, Δi\Delta_i is maximized as ai→−∥di∥2a_i\to-\lVert d_i\rVert_2 and diverges when ri+λdi=0r_i+\lambda d_i=0.

This theorem justifies prioritizing the orthogonal component bib_i for three reasons. First, bib_i measures maximal progress on the imitation objective that is “free” to the outcome-driven task, having no first-order cost to the primary RL objective. Second, it represents guidance advantage-modulation methods like RLSD [28] and OPDVR [12] cannot express, since they only rescale the RL gradient without introducing novel directions. Third, in sparse-reward RLVR [20, 4], a large bib_i indicates high gradient diversity (Δi\Delta_i), injecting new dimensions to prevent stagnation. However, the most diverse signal directly opposes the primary gradient (ai<0a_i<0), which would be catastrophic. Our method resolves this by assigning conflicting signals negative scores, ensuring their early discard as the token budget anneals. This provides early dense guidance while progressively enforcing safety.

GIFT resolves this tension by framing token selection as an exact-quota lexicographic program: at each step, a quota of kk eligible tokens must be selected, and GIFT chooses the size-kk subset of the eligible set EmE_m that lexicographically minimizes

Am⋆∈arg⁡min⁡lexA⊆Em,∣A∣=k(N−(A),O−(A),−E+(A)),(7)A_m^\star\in\mathop{\arg\min\nolimits^{\mathrm{lex}}}_{A\subseteq E_m,\lvert A\rvert=k}\left(N^-(A),O^-(A),-E^+(A)\right), \tag*{(7)}

where Cm={i∈Em∩D:ai<0}C_m=\{i\in E_m\cap\mathcal{D}:a_i<0\} collects the conflicting eligible tokens, Um=Em∖CmU_m=E_m\setminus C_m the remaining eligible ones, N−(A)=∣A∩Cm∣N^-(A)=\lvert A\cap C_m\rvert their number, O−(A)=∑i∈A∩Cm∣ai∣O^-(A)=\sum_{i\in A\cap C_m}\lvert a_i\rvert their normalized opposition, and E+(A)=∑i∈A∩Um∩Dbi2E^+(A)=\sum_{i\in A\cap U_m\cap\mathcal{D}}b_i^2 the injected orthogonal guidance.

Scoring and Progressive Selection

The lexicographic exact-quota program in (7) admits an exact, closed-form solution in the form of a piecewise scoring rule, providing a principled ranking for all eligible tokens. Each token receives a score sis_i defined as:

si={ai,i∈D, ai<0,bi,i∈D, ai≥0,0,i∈E∖D,(8)s_i= \begin{cases} a_i, & i\in\mathcal{D},\ a_i<0,\\ b_i, & i\in\mathcal{D},\ a_i\ge0,\\ 0, & i\in\mathcal{E}\setminus\mathcal{D}, \end{cases} \tag*{(8)}

This structure imposes a strict hierarchy that directly reflects our optimization criteria. For tokens with conflicting guidance (ai<0a_i<0), the score is negative, ensuring they are ranked lowest. For all safe tokens (ai≥0a_i\ge0), the score is simply the orthogonal magnitude bib_i, prioritizing novel, complementary information. Tokens with no usable RL gradient (i∉Di\notin\mathcal{D}) receive a neutral score of zero. The optimality of this ranking is formally guaranteed by the following theorem.

Theorem 3 (Exact-quota lexicographic optimality). With the notation of (7), for any integer quota k∈{0,…,∣Em∣}k\in\{0,\ldots,\lvert E_m\rvert\}, the set Am⋆A_m^\star of kk highest scores lexicographically minimizes (N−,O−,−E+)(N^-,O^-,-E^+) over size-kk subsets, and

∣Am⋆∩Cm∣=max⁡{0,k−∣Um∣}.(9)\lvert A_m^\star\cap C_m\rvert=\max\{0,k-\lvert U_m\rvert\}. \tag*{(9)}

If k≤∣Um∣k\le\lvert U_m\rvert the selection is conflict-free and maximizes E+E^+ over size-kk subsets of UmU_m; otherwise it takes UmU_m plus the k−∣Um∣k-\lvert U_m\rvert conflicting tokens of smallest normalized opposition ∣ai∣\lvert a_i\rvert. Both hold under arbitrary tie-breaking.

Theorem 3 provides the formal guarantee that this scoring rule is the exact solution to the stated lexicographic exact-quota problem: it first minimizes the number of conflicting tokens admitted, then their normalized opposition, and only then maximizes the injected orthogonal guidance (∑ibi2)(\sum_i b_i^2).

This static ranking is embedded within a dynamic withdrawal strategy, implemented by progressively shrinking the token budget ktk_t over the course of training via an annealing schedule.

κt=Texit−tTexit−Hfor H<t<Texit,(10)\kappa_t=\frac{T_{\mathrm{exit}}-t}{T_{\mathrm{exit}}-H}\quad\text{for }H<t<T_{\mathrm{exit}}, \tag*{(10)}

where κt\kappa_t is the retention ratio, HH is an initial full-coverage phase, and TexitT_{\mathrm{exit}} is the step at which all teacher guidance is withdrawn. This annealing of the coverage, rather than a loss coefficient, makes the trade-off between exploration and exploitation time-varying. Early in training, when the budget is large and the RL signal is sparse, the model benefits from broad, diverse guidance. As training matures and the budget contracts, the lowest-scoring tokens—the most conflicting ones—are the first to be excluded. The most complementary tokens (high bib_i, ai≥0a_i\ge0) are retained the longest.

This provides a principled, state-aware withdrawal order. Unlike uniform coefficient annealing ([14, 28]) which dilutes all guidance simultaneously, or staged training ([11]) which forgoes joint optimization, GIFT’s geometric ranking and coverage annealing ensures that as the supervision budget contracts, it is precisely the most harmful guidance (lowest-scoring tokens) that is shed first, while the most complementary guidance is retained at full strength.

The GIFT Objective

The principles of geometric selection and progressive withdrawal are implemented in the final GIFT objective function:

LtGIFT=LtGRPO+λNvalid∑i∈Imi(t)wiℓiSKL,λ=1.(11)\mathcal{L}^{\mathrm{GIFT}}_t=\mathcal{L}^{\mathrm{GRPO}}_t+\frac{\lambda}{N_{\mathrm{valid}}}\sum_{i\in\mathcal{I}}m_i(t)w_i\ell_i^{\mathrm{SKL}},\qquad\lambda=1. \tag*{(11)}

Here, mi(t)m_i(t) selects the top-kt,mk_{t,m} eligible tokens per sub-batch; I\mathcal{I} contains all valid positions and Nvalid=∣I∣N_{\mathrm{valid}}=\lvert\mathcal{I}\rvert. Several critical design choices apply. First, fixing λ=1\lambda=1 ensures selected tokens receive full-strength, unprojected distillation loss. Control operates via selection (who receives guidance), not dilution (guidance strength). The teacher loses reach, not voice. Second, normalizing over all valid tokens (NvalidN_{\mathrm{valid}}) rather than just selected ones ensures stability, keeping each retained token’s contribution independent of budget size. Otherwise, a shrinking budget would artificially inflate surviving tokens, undermining smooth withdrawal. Finally, as the budget anneals to zero (t≥Texitt \ge T_{\mathrm{exit}}), the mask mi(t)m_i(t) becomes universally zero, guaranteeing a clean transition. The auxiliary term vanishes, reducing LtGIFTL_t^{\mathrm{GIFT}} identically to the native GRPO loss. This grounds final behavior purely in the outcome-driven objective. The teacher acts as a temporary, geometrically-filtered scaffold, harnessing dense guidance before complete withdrawal without corrupting the final policy.

Experiments

We conduct experiments to validate GIFT’s core principles. Our evaluation addresses three central questions: (1) Effectiveness: Does GIFT’s approach of gradient-informed filtering and progressive withdrawal achieve state-of-the-art performance on challenging reasoning tasks compared to existing methods? (2) Generalization: Do GIFT’s benefits extend across different architectures, scales, and reasoning domains, including out-of-distribution scenarios? (3) Component Necessity: Are GIFT’s key components—specifically the geometric scoring rule based on conflict (aia_i) and novelty (bib_i), and the coverage annealing schedule—critical to its success?

Experimental Setup

Models and tasks. We evaluate on two families of models: three scales of Qwen3 (1.7B, 4B, 8B) [27] and OLMo-3-7B-Think [17]. For mathematical reasoning, we train on the DeepMath-103K dataset [6], using progressively harder difficulty splits for larger models, and evaluate on the AMC23 (math-ai, 2023), AIME24 [32], and AIME25 [33] benchmarks. For logical reasoning, we train on Knights-and-Knaves puzzles [23] (4-8 roles) and evaluate on both in-distribution (ID) and out-of-distribution (OOD, 9-11 roles) splits to test generalization.

Baselines. We compare GIFT with the initial model (BASE) and eight training baselines, grouped by supervision source and how it enters optimization (Appendix G): GRPO, using verifier-derived advantages without teacher feedback [20]; distributional self-distillation, matching next-token distributions of a conditioned self-teacher (OPSD [34], SDPO (Hübõtler et al., 2026)); verifier-guided token-level credit assignment, modulating advantages or gating rewards with teacher-derived scalars (RLSD [28], RLCSD [18], OPDVR [12]); and demonstration-guided methods (LUFFY [25], UFT [13]).

Evaluation. For mathematics, we report mean@12, the average correctness over 12 sampled responses per problem, averaged across problems, using rule-based answer matching. For logic, we report pass@1 on 500 ID problems and 100 problems for each of the three OOD role counts (9, 10, and 11). The reported Avg. equally weights the three mathematical benchmarks of the four logical-reasoning splits. All scores are percentages, and gains are relative percentage-point differences.

Main Results

Our main results demonstrate that GIFT’s principled filtering mechanism consistently outperforms existing methods across multiple tasks and model scales, providing a superior solution to the core tension between dense guidance and outcome-grounded learning.

Mathematical Reasoning. As shown in Table 1, GIFT achieves the highest average score on mathematical reasoning across all three Qwen3 models. Compared to the base model, it achieves average relative gains of 5.03%, 3.39%, and 2.99% for the 1.7B, 4B, and 8B models, respectively. The inferiority of the baselines stems from their distinct, yet suboptimal, strategies for combining supervision.

SizeBench.BASEGRPOOPSDSDPORLSDRLCSDOPDVRLUFFYUFTGIFT
1.7BAMC2375.5075.7075.6075.3075.3075.8076.5175.4074.6075.80 (+0.40%)
1.7BAIME2444.4445.2845.5644.7244.7246.9446.6748.3348.3348.61 (+9.38%)
1.7BAIME2535.0035.5633.3336.6734.7234.7235.8334.1736.6738.33 (+9.51%)
1.7BAvg.51.6552.1851.5052.2351.5852.4953.0052.6353.2054.25 (+5.03%)
4BAMC2388.4589.1688.2588.9688.8690.2688.4589.1689.6688.55 (+0.11%)
4BAIME2471.6774.1773.0670.8373.6172.2273.6176.1170.5673.33 (+2.32%)
4BAIME2563.8963.6167.2267.2263.6164.1764.7265.2864.7269.72 (+9.13%)
4BAvg.74.6775.6576.1875.6775.3675.5575.5976.8574.9877.20 (+3.39%)
8BAMC2389.1689.3688.7689.5688.7689.2689.9689.9688.8689.66 (+0.56%)
8BAIME2475.0074.7277.2275.8375.2876.1174.4473.3376.1176.11 (+1.48%)
8BAIME2565.8365.8366.3966.3969.7268.0669.7267.5066.3971.11 (+8.02%)
8BAvg.76.6776.6477.4677.2677.9277.8178.0476.9377.1278.96 (+2.99%)

Table 1. Mathematical reasoning across Qwen3 scales (mean@12, %). Avg. is the mean over AMC23, AIME24, and AIME25. Green denotes relative gains over BASE; bold and underlined values indicate the best and second-best distinct scores. Bench. denotes benchmark.

The two extremes of the supervision spectrum, pure RL and pure distillation, highlight the core challenge. GRPO provides an unbiased, outcome-grounded learning signal, which is why we retain it as our primary objective. However, its supervision is sparse—a single reward for an entire trajectory—and its group-relative signal vanishes in uniformly correct or incorrect groups, leading to sample inefficiency and learning stagnation. Conversely, on-policy self-distillation offers dense, token-level guidance but optimizes for imitation of a privileged teacher rather than verifiable correctness: OPSD conditions its self-teacher on a reference solution, while SDPO conditions it on environment feedback or verified successful rollouts. In both, the auxiliary loss is applied unconditionally along the entire trajectory—with no check for compatibility with the outcome-driven update and no mechanism for withdrawing teacher influence.

Hybrid methods attempt to bridge this gap, but their approaches lead to distinct compromises. One family, including RLSD, RLCSD, and OPDVR, ultimately converts the teacher signal into a scalar that modulates the RL update: RLSD and RLCSD use teacher-derived signals to modulate verifier-based advantages, while OPDVR reformulates the implicit reward of sampled-token distillation based on trajectory correctness and applies ReLU gating, so that the teacher enters the update only through a per-token scalar reward. By collapsing the teacher’s rich distributional information into a scalar reward or weight, these methods are structurally incapable of introducing novel update directions, as established in our theoretical analysis (Theorem 2(ii)); they can only re-weight the token-level components of the existing RL update, whereas GIFT’s auxiliary loss can provide valuable orthogonal guidance (bib_i). Other methods like LUFFY and UFT introduce teacher guidance through trajectory composition or annotated prefixes, which likewise lack the fine-grained, geometry-aware conflict checking that GIFT performs at every token.

In contrast, GIFT synthesizes the strengths of these approaches: it complements GRPO’s reliable but sparse signal with dense, expressive guidance from an auxiliary loss, but subjects that guidance to a rigorous geometric filter. This unique combination explains its consistent, high-level performance, particularly on the challenging AIME25 benchmark where it establishes a new state-of-the-art.

Logical Reasoning and Generalization. On the Knights-and-Knaves task (Table 2), GIFT again achieves the highest average score, improving over the base model with a relative gain of 3.71%. Crucially, this improvement is not confined to the in-distribution test set; GIFT shows a substantial average relative gain of 4.70% on the OOD splits. This indicates that the reasoning capabilities learned via gradient-informed guidance generalize to more complex problems not seen during training. The consistent failure of baselines to match this performance further underscores our analysis: methods that are either structurally limited (scalar modulation), ungrounded (pure imitation), or lack a robust filtering mechanism (unfiltered or coarse-grained guidance) cannot as effectively harness teacher guidance to build generalizable reasoning skills. This result, combined with strong performance on the OLMo-3 model family (Appendix § 14, Figure 2), confirms that GIFT’s mechanism is not specific to a single model architecture or reasoning domain but offers a generally applicable framework for improving complex reasoning.

BenchmarkBASEGRPOOPSDSDPORLSDRLCSDOPDVRLUFFYUFTGIFT
In-distribution (ID)
4–8 roles72.4073.4072.8075.2073.6074.0073.0074.6074.2073.40 (+1.38%)
Out-of-distribution (OOD)
9 roles69.0072.0075.0073.0071.0069.0071.0067.0071.0073.00 (+5.80%)
10 roles59.0056.0059.0056.0055.0059.0057.0059.0056.0061.00 (+3.39%)
11 roles42.0046.0041.0040.0041.0044.0044.0043.0040.0044.00 (+4.76%)
Avg.60.6061.8561.9561.0560.1561.5061.2560.9060.3062.85 (+3.71%)

Table 2. Logical reasoning on Knights-and-Knaves (pass@1, %). The ID split contains 4–8 roles, while the OOD splits contain 9, 10, and 11 roles. Green denotes relative gains over BASE.

Results on OLMo3-7B, reported as mean@12

Figure 2. Results on OLMo3-7B, reported as mean@12. Avg. is the equally weighted average over the two benchmarks. Green numbers indicate absolute gains over Base.

Ablation Study

Here We present key findings from the ablations detailed in Appendix § 13.§ 4.

The Geometric Scoring Rule is Essential. Table 3 shows heuristic criteria (e.g., entropy-KL divergence, raw gradient cosine similarity) fail to match GIFT, confirming statistical proxies for disagreement are less effective than our direct geometric analysis. Table 4 shows its constituent parts are necessary, though their importance varies with task difficulty. On simpler AMC23 and AIME24 benchmarks, “Conflict-Free” and “Conflict-only” ablations perform comparably or better than GIFT. When the primary RL gradient is reliable, simpler strategies prioritizing safety by strictly rejecting conflicts (“Conflict-Free”) or accelerating agreement (“Conflict-only”) are highly effective. However, these strategies falter on the challenging AIME25 benchmark, where GIFT excels. The “Residual-only” ablation—prioritizing novel guidance (bib_i) while ignoring conflict signs—underperforms across all settings, demonstrating pursuing novelty without safety checks is detrimental. Conversely, “Conflict-Free” proves too restrictive on AIME25, likely discarding valuable teacher guidance needed to escape local optima simply because it appears conflicting. GIFT’s superiority on AIME25 stems from its synthesis: using the conflict signal (aia_i) for safety while leveraging its exact-quota system to judiciously admit the least-opposing conflicting tokens when necessary, providing a vital mechanism for exploration. These results confirm GIFT’s complete geometric scoring rule provides a more robust approach than isolated heuristics, validating a design that places conflicting tokens below zero- and positive-score candidates.

MethodMathematical Reasoning (mean@12)Logical Reasoning (pass@1)
AMC23AIME24AIME25Avg.ID (4–8)OOD-9OOD-10OOD-11Avg.
All Tokens89.9676.1166.3977.4973.8069.0059.0045.0061.70
Random Tokens89.6675.2866.6777.2073.6071.0058.0046.0062.15
Entropy-KL Selection89.3675.2866.1176.9271.8063.0054.0052.0060.20
Gradient-cosine Selection89.5677.5065.8377.6372.8068.0061.0039.0060.20
Full Model89.6676.1171.1178.9673.4073.0061.0044.0062.85

Table 3. Ablation of token-selection strategies. Mathematical reasoning uses Qwen3-8B and is evaluated by mean@12. Logical reasoning uses Qwen3-4B and is evaluated by pass@1.

BenchmarkConflict-onlyResidual-onlyTeacher-normConflict-FreeGIFT
AMC2389.8689.2688.7689.9689.66
AIME2475.5675.0073.8976.1176.11
AIME2565.8366.6768.0667.1171.11
Avg.77.0876.9776.9077.7378.96

Table 4. Ablation of selection strategies on Qwen3-8B. Mean@12 scores on AMC23, AIME24, and AIME25.

Coverage Annealing is Superior to Alternative Annealing Strategies. Withdrawing teacher influence is as critical as applying it. We thus compare GIFT’s coverage annealing with two alternatives. First, sequential training ([11]) trains with OPSD distillation before switching to pure GRPO. Second, weight annealing combines objectives but uniformly applies distillation, gradually decaying the loss coefficient λ\lambda to zero. As shown in Table 7 (Appendix § 14), GIFT significantly outperforms both strategies, averaging 1.59 and 2.87 percentage point gains, respectively. Unlike sequential training, GIFT benefits from joint optimization synergy. Furthermore, unlike indiscriminate weight annealing, GIFT’s coverage annealing preserves the most valuable teacher signals at full strength while selectively pruning unhelpful ones. This confirms that selective, state-aware withdrawal is superior to both staged training and uniform decay.

Training scheduleAMC23AIME24AIME25Avg.
OPSD →\to GRPO89.0675.2867.7877.37
Weight annealing89.6672.2266.3976.09
GIFT89.6676.1171.1178.96

Table 7. Teacher-withdrawal schedules on Qwen3-8B (mean@12, %). Avg. equally weights the three benchmarks. Bold marks the best score in each column, including ties.

A Broader Allocation Scope is More Effective. We find that allocating the token budget across the entire training sub-batch outperforms narrower scopes like per-rollout or per-prompt (Table 5). This allows for a more efficient global allocation of supervision, enabling tokens to compete for guidance across different prompts and responses. The model can thus focus the limited supervision budget on the most complementary updates available in the entire sub-batch, rather than being constrained by a locally optimal but globally suboptimal choice.

BenchmarkPer-rolloutPer-promptGIFT
AMC2389.0989.5689.66
AIME2475.2876.3976.11
AIME2567.7866.3971.11
Avg.77.3777.4578.96

Table 5. Ablation of token selection scopes on Qwen3-8B.

Conclusion

We introduced GIFT, a method that resolves sparse RL and dense distillation conflicts by filtering teacher guidance via token-level gradient geometry, decomposing distillation gradients into components parallel (conflict) and orthogonal (novelty) to the RL gradient for principled scoring. Paired with coverage annealing, this selectively withdraws conflicting updates, smoothly transitioning to a pure RL objective. On complex reasoning benchmarks, GIFT consistently outperforms strong baselines, validating this integration of dense supervision.

AI Use Statement

During the preparation of this manuscript, large language models (LLMs) were utilized solely to refine grammar, readability, and formatting. The core scientific contributions—encompassing the research design, methodology, data analysis, and conclusions—were entirely conceptualized and executed by the human authors. We have carefully verified all AI-assisted language refinements and assume full responsibility for the manuscript’s final content and integrity.

Reproducibility Statement

To ensure our findings are fully reproducible, all source code and trained model weights will be made publicly available upon the acceptance of this manuscript. Furthermore, we will provide comprehensive documentation and evaluation protocols to assist the research community in replicating our experiments.

Main Notation

For clarity, Table 6 summarizes the main notations used in GIFT.

NotationTypeMeaning
i,Ii, \mathcal{I}index / setToken position ii and the set I\mathcal{I} of all valid token positions.
zi,piz_i, p_ivectorsStudent logits and predictive distribution, where pi=softmax⁡(zi)p_i = \operatorname{softmax}(z_i).
q~i\tilde{q}_idistributionPrivileged teacher distribution at position ii.
ui,euiu_i, e_{u_i}token / vectorSampled token and its corresponding one-hot vector.
wi,Ai,ρi,hiw_i, A_i, \rho_i, h_iscalarsToken weight, advantage, importance ratio, and active-branch indicator in the clipped GRPO objective.
ℓiGRPO\ell_i^{\mathrm{GRPO}}lossToken-level clipped GRPO loss providing outcome-grounded supervision.
ℓisKL\ell_i^{\mathrm{sKL}}lossToken-level self-distillation loss aligning pip_i with q~i\tilde{q}_i.
rir_ivectorOutcome-driven logit gradient, ri=∇ziℓiGRPOr_i = \nabla_{z_i}\ell_i^{\mathrm{GRPO}}.
did_ivectorImitation-driven logit gradient, di=∇zi(wiℓisKL)d_i = \nabla_{z_i}(w_i\ell_i^{\mathrm{sKL}}).
aia_iscalarSigned parallel component, ai=⟨di,r^i⟩a_i = \langle d_i, \hat{r}_i\rangle, measuring assistance or conflict with the RL gradient.
bib_iscalarOrthogonal magnitude, bi=∥di−air^i∥2b_i = \lVert d_i - a_i\hat{r}_i\rVert_2, measuring complementary teacher guidance.
D\mathcal{D}setToken positions with a usable non-zero RL reference gradient.
Em\mathcal{E}_msetDistillation-eligible token positions in training sub-batch mm.
Cm,Um\mathcal{C}_m,\mathcal{U}_msetsConflicting and non-conflicting eligible tokens, respectively.
sis_iscalarGIFT score: aia_i for conflicting tokens, bib_i for non-conflicting tokens, and zero when no usable RL reference exists.
kt,mk_{t,m}integerNumber of tokens retained in training sub-batch mm at step tt.
mi(t)m_i(t)indicatorBinary mask indicating whether token ii is selected at step tt.
κt\kappa_tscalarRetention ratio controlling the progressively shrinking token budget.
H,TexitH, T_{\mathrm{exit}}integersEnd of the full-coverage phase and step of complete teacher withdrawal.
λ\lambdascalarDistillation coefficient, fixed at λ=1\lambda= 1.
NvalidN_{\mathrm{valid}}integerTotal number of valid token positions used for loss normalization.

Table 6. Summary of the main notations used in GIFT.

Supporting Lemmas

Throughout we fix a token position and drop the index where unambiguous. Let p=softmax⁡(z)p = \operatorname{softmax}(z), let SS be the teacher support with ∣S∣=M|S| = M, and set

Ptail=1−∑v∈Sp(v)>0,Qtail=1−∑v∈Sq(v)>0,σ=QtailPtail,(12)P_{\mathrm{tail}} = 1 - \sum_{v \in S} p(v) > 0,\qquad Q_{\mathrm{tail}} = 1 - \sum_{v \in S} q(v) > 0,\qquad\sigma= \frac{Q_{\mathrm{tail}}}{P_{\mathrm{tail}},} \tag*{(12)}

so the objective is finite and differentiable. We use ∂p(v)/∂z(k)=p(v)(δvk−p(k))\partial p(v)/\partial z(k) = p(v)(\delta_{vk} - p(k)).

Lemma 1. (Sparse forward-KL gradient.) The loss ℓsKL=∑v∈Sq(v)log⁡q(v)p(v)+Qtaillog⁡QtailPtail\ell^{\mathrm{sKL}} = \sum_{v \in S} q(v)\log\frac{q(v)}{p(v)} + Q_{\mathrm{tail}}\log\frac{Q_{\mathrm{tail}}}{P_{\mathrm{tail}}} satisfies ∇zℓsKL=p−q~\nabla_z\ell^{\mathrm{sKL}} = p - \tilde{q}, where the tail-completed teacher

q~(v)=q(v)(v∈S),q~(v)=σp(v)(v∉S)(13)\tilde{q}(v) = q(v)\quad(v \in S),\qquad\tilde{q}(v) = \sigma p(v)\quad(v \notin S) \tag*{(13)}

is a probability distribution.

Proof. Only −∑v∈Sq(v)log⁡p(v)-\sum_{v \in S}q(v)\log p(v) and −Qtaillog⁡Ptail-Q_{\mathrm{tail}}\log P_{\mathrm{tail}} depend on zz, as qq and QtailQ_{\mathrm{tail}} are detached. For the first,

∂∂z(k)(−∑v∈Sq(v)log⁡p(v))=−∑v∈Sq(v)(δvk−p(k))=−qS(k)+p(k)∑v∈Sq(v),(14)\frac{\partial}{\partial z(k)}\left(-\sum_{v \in S}q(v)\log p(v)\right) = -\sum_{v \in S}q(v)(\delta_{vk} - p(k)) = -q_S(k) + p(k)\sum_{v \in S}q(v), \tag*{(14)}

with qS(k)=q(k)1{k∈S}q_S(k) = q(k)\mathbf{1}\{k \in S\}. For the second, since Ptail=∑v∉Sp(v)P_{\mathrm{tail}} = \sum_{v \notin S}p(v),

∂Ptail∂z(k)=1{k∉S}p(k)−p(k)Ptail,(15)\frac{\partial P_{\mathrm{tail}}}{\partial z(k)} = \mathbf{1}\{k \notin S\}p(k) - p(k)P_{\mathrm{tail}}, \tag*{(15)}

so ∂∂z(k)(−Qtaillog⁡Ptail)=−σ1{k∉S}p(k)+p(k)Qtail\frac{\partial}{\partial z^{(k)}}(-Q_{\mathrm{tail}}\log P_{\mathrm{tail}})=-\sigma\mathbf{1}\{k\notin S\}p(k)+p(k)Q_{\mathrm{tail}}. Adding the two and using ∑v∈Sq(v)+Qtail=1\sum_{v\in S}q(v)+Q_{\mathrm{tail}}=1 gives q~(k)=p(k)−q~(k)\tilde{q}(k)=p(k)-\tilde{q}(k). Finally q~≥0\tilde{q}\ge0 and ∑vq~(v)=∑v∈Sq(v)+σPtail=∑v∈Sq(v)+Qtail=1\sum_v\tilde{q}(v)=\sum_{v\in S}q(v)+\sigma P_{\mathrm{tail}}=\sum_{v\in S}q(v)+Q_{\mathrm{tail}}=1.

Proposition 1 (Closed-form scoring). With Π=∑v∈Vp(v)2\Pi=\sum_{v\in V}p(v)^2 and πS=∑v∈Sp(v)2\pi_S=\sum_{v\in S}p(v)^2,

∥p−eu∥22=Π−2p(u)+1,⟨p,q~⟩=∑v∈Sp(v)q(v)+σ(Π−πS),∥q~∥22=∑v∈Sq(v)2+σ2(Π−πS),(16)\lVert p-e_u\rVert_2^2=\Pi-2p(u)+1,\qquad\langle p,\tilde{q}\rangle=\sum_{v\in S}p(v)q(v)+\sigma(\Pi-\pi_S),\qquad\lVert\tilde{q}\rVert_2^2=\sum_{v\in S}q(v)^2+\sigma^2(\Pi-\pi_S), \tag*{(16)}

and ⟨p−q~,p−eu⟩=Π−p(u)−⟨p,q~⟩+q~(u)\langle p-\tilde{q},p-e_u\rangle=\Pi-p(u)-\langle p,\tilde{q}\rangle+\tilde{q}(u). Consequently aa and bb are computable from Π\Pi, p(u)p(u), q~(u)\tilde{q}(u) and O(M)O(M) sums over SS, with ∥d∥22=w2(Π−2⟨p,q~⟩+∥q~∥22)\lVert d\rVert_2^2=w^2(\Pi-2\langle p,\tilde{q}\rangle+\lVert\tilde{q}\rVert_2^2) and b2=∥d∥22−a2b^2=\lVert d\rVert_2^2-a^2. No ∣V∣|V|-dimensional gradient is materialized and no auxiliary backward pass is needed.

Proof. Each identity splits the sum over SS and its complement and substitutes (13): e.g. ⟨p,q~⟩=∑v∈Sp(v)q(v)+σ∑v∉Sp(v)2\langle p,\tilde{q}\rangle=\sum_{v\in S}p(v)q(v)+\sigma\sum_{v\notin S}p(v)^2 and ∑v∉Sp(v)2=Π−πS\sum_{v\notin S}p(v)^2=\Pi-\pi_S; identically for ∥q~∥22\lVert\tilde{q}\rVert_2^2 with σ2\sigma^2. For the last, expand ⟨p−q~,p−eu⟩=∥p∥22−⟨p,eu⟩−⟨q~,p⟩+⟨q~,eu⟩\langle p-\tilde{q},p-e_u\rangle=\lVert p\rVert_2^2-\langle p,e_u\rangle-\langle\tilde{q},p\rangle+\langle\tilde{q},e_u\rangle. The only vocabulary-wide quantity is Π\Pi, a scalar reduction of probabilities already formed in the forward pass. The expression for bb follows from a2+b2=∥d∥22a^2+b^2=\lVert d\rVert_2^2 (orthogonality).

Implementation note. q~(u)=q(u)\tilde{q}(u)=q(u) if u∈Su\in S, else σp(u)\sigma p(u); b2b^2 is clipped at zero before the square root to absorb cancellation when a2≈∥d∥22a^2\approx\lVert d\rVert_2^2.

Proposition 2 (Invariance of the score). For i∈Di\in D,

ai=wisign⁡(Ai)⟨pi−q~i,pi−eui⟩∥pi−eui∥22,bi2=∥di∥22−ai2,(17)a_i=w_i\operatorname{sign}(A_i)\frac{\langle p_i-\tilde{q}_i,p_i-e_{u_i}\rangle}{\lVert p_i-e_{u_i}\rVert_2^2},\qquad b_i^2=\lVert d_i\rVert_2^2-a_i^2, \tag*{(17)}

so aia_i and bib_i are invariant to ∣Ai∣|A_i| and to positive rescaling of ρi\rho_i, provided the active-branch and norm-threshold decisions are unchanged; bib_i is in addition independent of sign⁡(Ai)\operatorname{sign}(A_i).

Proof. On the active branch ri=cψr_i=c\psi with c=wiAiρi≠0c=w_iA_i\rho_i\ne0 and ψ=pi−eui≠0\psi=p_i-e_{u_i}\ne0. Since wi,ρi>0w_i,\rho_i>0 we have ∣c∣=wi∣Ai∣ρi|c|=w_i|A_i|\rho_i and

r^i=cψ∣c∣∥ψ∥2=sign⁡(Ai)ψ∥ψ∥2,(18)\hat{r}_i=\frac{c\psi}{|c|\lVert\psi\rVert_2}=\operatorname{sign}(A_i)\frac{\psi}{\lVert\psi\rVert_2}, \tag*{(18)}

so the magnitude cancels. With di=wi(pi−q~i)d_i=w_i(p_i-\tilde{q}_i) from Lemma 1, ai=⟨di,r^i⟩a_i=\langle d_i,\hat{r}_i\rangle gives the display. ∥di∥2\lVert d_i\rVert_2 involves neither AiA_i nor ρi\rho_i, and bi2=∥di∥22−ai2b_i^2=\lVert d_i\rVert_2^2-a_i^2 depends on aia_i only through ai2a_i^2, which is insensitive to sign⁡(Ai)\operatorname{sign}(A_i).

Gradient Diversity

Theorem 2(iii) uses the following quantity, introduced by Yin et al. [29] to characterize how far mini-batch SGD can be scaled before departing from the behaviour of its serial counterpart. We state it and the two facts we rely on, so that the argument is self-contained.

Definition 1 (Gradient diversity). For gradients g1,…,gng_1,\ldots,g_n summed into a single update with ∑jgj≠0\sum_j g_j\ne0,

Δ({gj}):=∑j=1n∥gj∥22∥∑j=1ngj∥22=∑j∥gj∥22∑j∥gj∥22+∑j≠l⟨gj,gl⟩.(19)\Delta(\{g_j\}):=\frac{\sum_{j=1}^{n}\lVert g_j\rVert_2^2}{\left\lVert\sum_{j=1}^{n}g_j\right\rVert_2^2} =\frac{\sum_j\lVert g_j\rVert_2^2}{\sum_j\lVert g_j\rVert_2^2+\sum_{j\ne l}\langle g_j,g_l\rangle}. \tag*{(19)}

Lemma 2 (Endpoints of Δ\Delta). If g1=⋯=gn=g≠0g_1=\cdots=g_n=g\ne0 then Δ=1/n\Delta=1/n. If the gjg_j are nonzero and mutually orthogonal then Δ=1\Delta=1, whatever their norms. In general Δ>1\Delta>1 if and only if ∑j≠l⟨gj,gl⟩<0\sum_{j\ne l}\langle g_j,g_l\rangle<0.

Proof. For identical gradients the numerator is n∥g∥22n\lVert g\rVert_2^2 and the denominator ∥ng∥22=n2∥g∥22\lVert ng\rVert_2^2=n^2\lVert g\rVert_2^2, giving 1/n1/n. For mutually orthogonal gradients every cross term in the second form of (19) vanishes, leaving Δ=1\Delta=1. The final claim is immediate from that same form, since the numerator is positive.

So Δ\Delta is a redundancy measure: it is smallest (1/n1/n) when the summands duplicate one another, equals exactly 11 when they are mutually non-interfering, and exceeds 11 only when they partially cancel—the last case being why an unconstrained preference for large Δ\Delta is unsafe. The regime relevant here is n=2n = 2 with g1=rig_1 = r_i and g2=λdig_2 = \lambda d_i. We use only Definition 1 and Lemma 2; the convergence and stability bounds of Yin et al. [29] are cited as context for why low redundancy is desirable, but no statement in Theorem 2(iii) depends on them.

Proofs of the Main Results

Theorem 1 (non-interference)

Proof. Let ϕ(η)=fi(zi−η(ri+λdi))\phi(\eta) = f_i(z_i - \eta(r_i + \lambda d_i)). By the chain rule and ∇fi(zi)=ri\nabla f_i(z_i) = r_i,

ϕ′(0)=⟨ri,−(ri+λdi)⟩=−∥ri∥22−λ⟨ri,di⟩=−∥ri∥22−λ∥ri∥22ai,(20)\phi'(0) = \langle r_i, -(r_i + \lambda d_i)\rangle= -\lVert r_i\rVert_2^2 - \lambda\langle r_i, d_i\rangle= -\lVert r_i\rVert_2^2 - \lambda\lVert r_i\rVert_2^2 a_i, \tag*{(20)}

using ri=∥ri∥2r^ir_i = \lVert r_i\rVert_2\hat r_i and ai=⟨di,r^i⟩a_i = \langle d_i, \hat r_i\rangle. This is ddηfi(zi(η))∣η=0=−∥ri∥22−λ∥ri∥22ai\left.\frac{\mathrm{d}}{\mathrm{d}\eta}f_i(z_i(\eta))\right|_{\eta=0} = -\lVert r_i\rVert_2^2 - \lambda\lVert r_i\rVert_2^2a_i. The GRPO-only step is λ=0\lambda= 0, with value −∥ri∥22<0-\lVert r_i\rVert_2^2 < 0; the teacher’s contribution is the residual −λ∥ri∥22ai-\lambda\lVert r_i\rVert_2^2a_i, whose sign is opposite to that of aia_i since λ≥0\lambda\ge0, ∥ri∥2>0\lVert r_i\rVert_2 > 0.

For (1): if ai>0a_i > 0 then ϕ′(0)<−∥ri∥22\phi'(0) < -\lVert r_i\rVert_2^2, a strictly larger decrease; if ai=0a_i = 0 then ϕ′(0)=−∥ri∥22\phi'(0) = -\lVert r_i\rVert_2^2 exactly, independently of ∥di∥2\lVert d_i\rVert_2, and in particular the orthogonal component has no first-order effect at any magnitude. For ai≥0a_i \ge0 and any λ≥0\lambda\ge0 we get ϕ′(0)≤−∥ri∥22<0\phi'(0) \le-\lVert r_i\rVert_2^2 < 0, so the primary descent cannot be cancelled.

For (2): if ai<0a_i < 0 then ϕ′(0)=−∥ri∥22+λ∥ri∥22∣ai∣\phi'(0) = -\lVert r_i\rVert_2^2 + \lambda\lVert r_i\rVert_2^2\lvert a_i\rvert, which exceeds −∥ri∥22-\lVert r_i\rVert_2^2 by λ∥ri∥22∣ai∣>0\lambda\lVert r_i\rVert_2^2\lvert a_i\rvert> 0 for λ>0\lambda> 0: the decrease is necessarily smaller. It becomes nonnegative, i.e. the primary loss no longer decreases to first order, exactly when λ∥ri∥22∣ai∣≥∥ri∥22\lambda\lVert r_i\rVert_2^2\lvert a_i\rvert\ge\lVert r_i\rVert_2^2, i.e. λ∣ai∣≥∥ri∥22\lambda\lvert a_i\rvert\ge\lVert r_i\rVert_2^2; strict reversal (ϕ′(0)>0\phi'(0) > 0) holds when λ∣ai∣>∥ri∥22\lambda\lvert a_i\rvert> \lVert r_i\rVert_2^2.

Theorem 2 (complementarity)

Part (i). Decompose di=air^i+di⊥d_i = a_i\hat r_i + d_i^\perp. Feasibility ri⊤v=0r_i^\top v = 0 is equivalent to r^i⊤v=0\hat r_i^\top v = 0, whence −di⊤v=−(di⊥)⊤v-d_i^\top v = -(d_i^\perp)^\top v. By Cauchy–Schwarz this is at most ∥di⊥∥2∥v∥2≤bi\lVert d_i^\perp\rVert_2\lVert v\rVert_2 \le b_i. If bi>0b_i > 0, v⋆=−di⊥/biv^\star= -d_i^\perp/b_i is feasible (∥v⋆∥2=1\lVert v^\star\rVert_2 = 1, ri⊤v⋆=0r_i^\top v^\star= 0) and attains bib_i. If bi=0b_i = 0 then di⊥=0d_i^\perp= 0 and every feasible vv gives 0=bi0 = b_i. (By Theorem 1 with the perturbation zi+vz_i + v, the constraint ri⊤v=0r_i^\top v = 0 is precisely first-order neutrality of the primary loss.)

Part (ii). Equip H=⨁i∈IR∣V∣\mathcal{H} = \bigoplus_{i\in\mathcal{I}} \mathbb{R}^{|V|} with the Euclidean product norm ∥(xi)i∥H2=∑i∈I∥xi∥22\lVert(x_i)_i\rVert_{\mathcal{H}}^2 = \sum_{i\in\mathcal{I}}\lVert x_i\rVert_2^2. Since I\mathcal{I} is finite, M0=⨁i∈Ispan⁡(ri)\mathcal{M}_0 = \bigoplus_{i\in\mathcal{I}}\operatorname{span}(r_i) is a closed linear subspace of H\mathcal{H}, where span⁡(0)={0}\operatorname{span}(0) = \{0\}. Distinct token blocks are orthogonal, so the orthogonal projection of Dm=(midi)iD_m = (m_i d_i)_i acts blockwise:

(ΠM0Dm)i={mi⟨di,ri⟩∥ri∥22ri,ri≠0,0,ri=0.(21)(\Pi_{\mathcal{M}_0}D_m)_i = \begin{cases} m_i\frac{\langle d_i,r_i\rangle}{\lVert r_i\rVert_2^2}r_i, & r_i \ne0,\\ 0, & r_i = 0. \end{cases} \tag*{(21)}

By the Pythagorean identity in each block and mi2=mim_i^2 = m_i,

dist⁡2(Dm,M0)=∥Dm−ΠM0Dm∥H2=∑i:ri≠0mi(∥di∥22−⟨di,ri⟩2∥ri∥22)+∑i:ri=0mi∥di∥22.(22)\begin{aligned} \operatorname{dist}^2(D_m,\mathcal{M}_0) = \lVert D_m-\Pi_{\mathcal{M}_0}D_m\rVert_{\mathcal{H}}^2 \\ &= \sum_{i:r_i\ne0}m_i\left(\lVert d_i\rVert_2^2-\frac{\langle d_i,r_i\rangle^2}{\lVert r_i\rVert_2^2}\right)+\sum_{i:r_i=0}m_i\lVert d_i\rVert_2^2. \tag*{(22)} \end{aligned}

For i∈Dϵ={i:∥ri∥2>ϵg}i\in\mathcal{D}_{\epsilon}=\{i:\lVert r_i\rVert_2>\epsilon_g\}, where ϵg≥0\epsilon_g\ge0, the parenthesized term equals bi2b_i^2 by the definition of the orthogonal residual. All remaining terms are nonnegative. Consequently,

dist⁡2(Dm,M0)≥∑i∈Dϵmibi2.(23)\operatorname{dist}^2(D_m,\mathcal{M}_0)\ge\sum_{i\in\mathcal{D}_{\epsilon}}m_i b_i^2. \tag*{(23)}

If mi=1m_i=1 and bi>0b_i>0 for some i∈Dϵi\in\mathcal{D}_{\epsilon}, this lower bound is strictly positive, hence Dm∉M0D_m\notin\mathcal{M}_0. Thus the selected teacher field cannot be represented by token-local scalar rescalings of the fixed GRPO reference field.

Scope of Part (ii). The comparison space M0\mathcal{M}_0 permits arbitrary real-valued rescalings of the fixed token-level GRPO gradients. It therefore contains the fields produced by scalar-modulation rules whose per-token gradients remain multiples of the same rir_i, but need not equal the reachable set of a particular algorithm with nonnegative or bounded multipliers. No reference direction is truncated: a small but nonzero rir_i still spans a line, whereas ri=0r_i=0 contributes the zero subspace. The threshold ϵg\epsilon_g enters only through the lower bound involving Dϵ\mathcal{D}_{\epsilon}, not through the definition of M0\mathcal{M}_0.

The result concerns independent token blocks in logit space. Non-membership in M0\mathcal{M}_0 does not imply non-membership in its image after mapping through the shared model Jacobians; distinct token-level fields can induce the same parameter gradient. Thus, the theorem establishes local directional irreducibility, not shared-parameter or function-space novelty, nor improved verifier performance. Collinear teacher guidance can still affect update magnitudes and optimization behavior.

Finally, the identity holds for any fixed binary mask and does not assert that GIFT globally maximizes the distance to M0\mathcal{M}_0. Its third lexicographic criterion maximizes only the selected residual energy on the defined-reference, non-conflicting branch, after the first two objective coordinates have been optimized. This restricted selection property is distinct from maximizing the total residual energy over all tokens.

Part (iii). Apply Definition 1 with n=2n=2, g1=rig_1=r_i and g2=λdig_2=\lambda d_i, so Δi=(∥ri∥22+λ2∥di∥22)/∥ri+λdi∥22\Delta_i=(\lVert r_i\rVert_2^2+\lambda^2\lVert d_i\rVert_2^2)/\lVert r_i+\lambda d_i\rVert_2^2. Write ν=∥ri∥2>0\nu=\lVert r_i\rVert_2>0 and μ=λ∥di∥2\mu=\lambda\lVert d_i\rVert_2, and expand the denominator using ⟨ri,di⟩=νai\langle r_i,d_i\rangle=\nu a_i:

∥ri+λdi∥22=ν2+2λ⟨ri,di⟩+λ2∥di∥22=ν2+2λνai+μ2(24)\lVert r_i+\lambda d_i\rVert_2^2=\nu^2+2\lambda\langle r_i,d_i\rangle+\lambda^2\lVert d_i\rVert_2^2=\nu^2+2\lambda\nu a_i+\mu^2 \tag*{(24)}

Hence Δi=(ν2+μ2)/(ν2+μ2+2λνai)\Delta_i=(\nu^2+\mu^2)/(\nu^2+\mu^2+2\lambda\nu a_i) has a numerator independent of aia_i and a denominator strictly increasing in aia_i (slope 2λν>02\lambda\nu>0), so Δi\Delta_i is strictly decreasing in aia_i wherever the denominator is positive. On the safe branch ai∈[0,∥di∥2]a_i\in[0,\lVert d_i\rVert_2] we have ai=∥di∥22−bi2a_i=\sqrt{\lVert d_i\rVert_2^2-b_i^2}, strictly decreasing in bi∈[0,∥di∥2]b_i\in[0,\lVert d_i\rVert_2]; composing the two monotonicities, Δi\Delta_i is strictly increasing in bib_i there. At bi=0b_i=0 (so ai=∥di∥2a_i=\lVert d_i\rVert_2, the two gradients collinear) the value is (ν2+μ2)/(ν+μ)2(\nu^2+\mu^2)/(\nu+\mu)^2, which lies in [12,1)[\frac{1}{2},1) for ν,μ>0\nu,\mu>0: the upper bound is strict by AM–GM, and the lower bound follows from (ν+μ)2≤2(ν2+μ2)(\nu+\mu)^2\le2(\nu^2+\mu^2) with equality iff ν=μ\nu=\mu. At ai=0a_i=0 (orthogonal) the value is exactly 1, consistent with Lemma 2.

For the unconstrained statement, Δi\Delta_i is strictly decreasing in aia_i over the whole range ai∈[−∥di∥2,∥di∥2]a_i\in[-\lVert d_i\rVert_2,\lVert d_i\rVert_2], hence maximized as ai→−∥di∥2a_i\to-\lVert d_i\rVert_2; there di=−∥di∥2r^id_i=-\lVert d_i\rVert_2\hat r_i and the denominator equals (ν−μ)2(\nu-\mu)^2, which vanishes when μ=ν\mu=\nu, i.e. exactly when ri+λdi=0r_i+\lambda d_i=0, at which point Δi\Delta_i diverges. By Lemma 2 this is the regime Δi>1\Delta_i>1, in which the two gradients partially cancel.

Theorem 3 (selection)

Proof. Write C=Cm\mathcal{C}=\mathcal{C}_m, U=Um\mathcal{U}=\mathcal{U}_m, and E=Em\mathcal{E}=\mathcal{E}_m, with the sets and objectives defined before Equation equation 7. Feasibility means A⊆EA\subseteq\mathcal{E} and ∣A∣=k|A|=k, for an integer 0≤k≤∣E∣0\le k\le|\mathcal{E}|.

Separation. If i∈Ci\in\mathcal{C}, then si=ai<0s_i=a_i<0. If j∈U∩Dj\in\mathcal{U}\cap\mathcal{D}, then sj=bj≥0s_j=b_j\ge0; if j∈U∖Dj\in\mathcal{U}\setminus\mathcal{D}, then sj=0s_j=0. Thus every candidate in U\mathcal{U} precedes every candidate in C\mathcal{C}.

First coordinate. Every size-kk subset satisfies

N−(A)=k−∣A∩U∣≥max⁡{0,k−∣U∣}.N^-(A)=k-|A\cap\mathcal{U}|\ge\max\{0,k-|\mathcal{U}|\}.

A top-kk set attains this bound: it contains U\mathcal{U} when k≤∣U∣k\le|\mathcal{U}|, and contains all of U\mathcal{U} otherwise. This proves Equation equation 9.

Second coordinate. Let n=max⁡{0,k−∣U∣}n=\max\{0,k-|\mathcal{U}|\}. If n=0n=0, then C−=0C^-=0 for every first-coordinate minimizer. If n>0n>0, every such minimizer contains all of U\mathcal{U} and exactly nn elements of C\mathcal{C}. Since si=−∣ai∣s_i=-|a_i| on C\mathcal{C}, top-kk selection chooses the nn smallest ∣ai∣|a_i|, minimizing C−C^-.

Third coordinate. If k>∣U∣k>|\mathcal{U}|, all first-coordinate minimizers contain U\mathcal{U}, so E+E^+ is constant. If k≤∣U∣k\le|\mathcal{U}|, define βi=bi\beta_i=b_i on U∩D\mathcal{U}\cap\mathcal{D} and βi=0\beta_i=0 on U∖D\mathcal{U}\setminus\mathcal{D}. Then si=βi≥0s_i=\beta_i\ge0 on U\mathcal{U}, and E+(A)=∑i∈Aβi2E^+(A)=\sum_{i\in A}\beta_i^2 for A⊆UA\subseteq\mathcal{U}. Because squaring preserves order on [0,∞)[0,\infty), the kk largest scores maximize E+E^+.

Equal scores give equal contributions to the relevant objective coordinate, including ties between bi=0b_i=0 and no-usable-reference positions. Hence arbitrary tie-breaking preserves all three objective values. The cases k=0k=0 and k=∣E∣k=|\mathcal{E}| are included, as is the empty pool with k=0k=0.

Remark (Normalized opposition is not primary-loss damage). By Theorem 1, the first-order change in the primary loss induced by an admitted conflicting token is −λ∥ri∥22ai>0-\lambda\lVert r_i\rVert_2^2a_i>0, which scales with ∥ri∥22\lVert r_i\rVert_2^2 as well as aia_i. The coordinate O−(A)=∑i∈A∩Cm∣ai∣O^-(A)=\sum_{i\in A\cap\mathcal{C}_m}|a_i| is therefore a normalized opposition—first-order damage per unit of RL-gradient norm (with λ=1\lambda=1)—and serves only as a ranking key. It coincides with aggregate primary-loss damage only if all admitted conflicting tokens share the same ∥ri∥2\lVert r_i\rVert_2, which is non-generic; we therefore avoid calling it harm or severity.

Remark (Exact quota vs. hard constraint). Equation 7 imposes ∣A∣=k|A| = k rather than the hard-constrained program max⁡m∑imibi2\max_m \sum_i m_i b_i^2 subject to ai≥0a_i \ge0 for selected ii and ∑imi≤k\sum_i m_i \le k, which admits no conflicting token and may select fewer than kk tokens. That hard-constrained rule is implemented as the Conflict-Free ablation (Appendix § 13.§ 4, Table 4). The two coincide whenever the quota does not exceed the number of non-conflicting eligible tokens (k≤∣Um∣k \le|U_m|, the first case of Theorem 3); they differ only when the quota forces admission of conflicting tokens, in which case GIFT admits those with the smallest normalized opposition. The exact quota ∣At,m∣=min⁡{Kt,m,∣Em∣}|A_{t,m}| = \min\{K_{t,m}, |E_m|\} (Equation (29)) moreover keeps realized sub-batch coverage a deterministic function of the schedule κt\kappa_t and the eligible set, which the exposure accounting of Appendix § 9 relies on.

Behaviour at GRPO Stationarity

Corollary 1 (Local behavior when the primary token gradient vanishes). Suppose ri=0r_i = 0 *and the token-level primary surrogate is differentiable. Then every local logit perturbation is first-order neutral to that token-level surrogate. Under the GIFT objective, the auxiliary logit-gradient block at token ii is

mi(t)λwiNvaliddi.(25)m_i(t)\frac{\lambda w_i}{N_{\mathrm{valid}}}d_i. \tag*{(25)}

Hence token ii contributes a nonzero distillation update only if mi(t)=1m_i(t) = 1, wi≠0w_i \ne0, di≠0d_i \ne0, and t<Texitt < T_{\mathrm{exit}}. Eligibility or a zero score alone does not guarantee selection.

Proof. Because ri=∇zifi=0r_i = \nabla_{z_i} f_i = 0, the directional derivative of fif_i along any local logit perturbation vv is $r_i^\mathsf{T}v = 0. The auxiliary term in the GIFT objective is multiplied by the detached mask mi(t)m_i(t) and normalized by NvalidN_{\mathrm{valid}}, which gives the stated block. When no usable reference direction exists, the implementation assigns token ii score zero; its update therefore depends on whether it is selected under the exact quota. □\square

When the sampled GRPO gradient vanishes, GIFT assigns the token a zero score; whether it receives distillation still depends on the exact-quota selection mask ((Corollary 1)).

We record this as a structural observation rather than a claim about aggregate behaviour. Whether a prompt whose group is uniformly correct or wrong should receive teacher supervision is an empirical question—the supervision is first-order free with respect to GRPO, but freeness is not the same as usefulness, and the group-level reward carries no information about the teacher’s reliability there. The corollary establishes only that GIFT’s scoring is well defined in this regime and that it assigns such tokens neutral rather than negative priority.

Termination and Bounded Exposure

Corollary 2 (Termination and nominal exposure). For t≥Texitt \ge T_{\mathrm{exit}} all masks vanish and LtGIFTL_t^{\mathrm{GIFT}} equals the native GRPO objective identically, so teacher and scoring passes may be skipped and the terminal phase is unprivileged. The nominal teacher exposure over T≥TexitT \ge T_{\mathrm{exit}} actor steps is

1T∑t=1Tκt=1T(H+12(Texit−H−1)),(26)\frac{1}{T}\sum_{t=1}^{T}\kappa_t = \frac{1}{T}\left(H+\frac{1}{2}(T_{\mathrm{exit}}-H-1)\right), \tag*{(26)}

equal to 43.5/96≈45.3%43.5/96 \approx45.3\% for (H,Texit,T)=(10,78,96)(H,T_{\mathrm{exit}},T) = (10,78,96).

Proof. For t≥Texitt \ge T_{\mathrm{exit}}, κt=0\kappa_t = 0 gives km(t)=min⁡{0,∣Em∣}=0k_m(t) = \min\{0, |E_m|\} = 0, so TopK returns ∅\varnothing and mi(t)=0m_i(t) = 0 for all ii; the auxiliary sum in Equation (11) vanishes identically and LtGIFT=LtGRPOL_t^{\mathrm{GIFT}} = L_t^{\mathrm{GRPO}}. Since the teacher and scoring computations enter the loss only through that sum, both may be skipped without changing the objective or its gradient. For the exposure, the first HH steps contribute HH, and substituting j=Texit−tj = T_{\mathrm{exit}} - t,

∑t=H+1Texit−1Texit−tTexit−H=1Texit−H∑j=1Texit−H−1j=Texit−H−12,(27)\sum_{t=H+1}^{T_{\mathrm{exit}}-1}\frac{T_{\mathrm{exit}}-t}{T_{\mathrm{exit}}-H} = \frac{1}{T_{\mathrm{exit}}-H}\sum_{j=1}^{T_{\mathrm{exit}}-H-1}j = \frac{T_{\mathrm{exit}}-H-1}{2}, \tag*{(27)}

while steps t≥Texitt \ge T_{\mathrm{exit}} contribute zero. Dividing by TT gives Equation (26), with (H,Texit,T)=(10,78,96)(H,T_{\mathrm{exit}},T) = (10,78,96) it is (10+33.5)/96=43.5/96≈0.4531(10 + 33.5)/96 = 43.5/96 \approx0.4531. □\square

Remark. This is nominal exposure. Realized exposure 1T∑t∑imi(t)/∣Im∣\frac{1}{T}\sum_t\sum_i m_i(t)/|I_m| can be strictly smaller because km(t)=min⁡{⌊κt∣Im∣⌋,∣Em∣}k_m(t)=\min\{\lfloor\kappa_t|I_m|\rfloor,|\mathcal{E}_m|\}, so (26) is an upper bound on the fraction of token-steps receiving privileged supervision.

Coverage Annealing Does Not Renormalize Survivors

Proposition 3. Holding the rollout, model parameters, teacher targets, detached selection mask, and token weights under (11), the auxiliary gradient contributed by a retained token ii is (λ/Nvalid)wi∇θℓiSKL(\lambda/N_{\mathrm{valid}})w_i\nabla_\theta\ell_i^{\mathrm{SKL}}, independent of the budget and of which other tokens are retained. Under coefficient annealing λ→λt\lambda\to\lambda_t at full coverage the contribution is (λt/Nvalid)wi∇θℓiSKL(\lambda_t/N_{\mathrm{valid}})w_i\nabla_\theta\ell_i^{\mathrm{SKL}}, decaying at every position simultaneously. Under selected-set normalization 1/Nsel(t)1/N_{\mathrm{sel}}(t) with Nsel(t)=∑mkm(t)≥1N_{\mathrm{sel}}(t)=\sum_m k_m(t)\ge1, it is (λ/Nsel(t))wi∇θℓiSKL(\lambda/N_{\mathrm{sel}}(t))w_i\nabla_\theta\ell_i^{\mathrm{SKL}}, which grows as the budget shrinks.

Proof. Differentiating (11) with mi(t)m_i(t), wiw_i and teacher targets detached, the auxiliary gradient is (λ/Nvalid)∑i∈Imi(t)wi∇θℓiSKL(\lambda/N_{\mathrm{valid}})\sum_{i\in I}m_i(t)w_i\nabla_\theta\ell_i^{\mathrm{SKL}}. The coefficient of a retained term is λ/Nvalid\lambda/N_{\mathrm{valid}}, and Nvalid=∣I∣N_{\mathrm{valid}}=|I| is fixed by the rollout, independent of κt\kappa_t, km(t)k_m(t) and the mask; λ\lambda is constant. Replacing λ\lambda by λt\lambda_t scales all terms by the common factor λt/λ\lambda_t/\lambda, so no position retains full strength once λt<λ\lambda_t<\lambda. Replacing NvalidN_{\mathrm{valid}} by Nsel(t)N_{\mathrm{sel}}(t) gives per-token weight λ/Nsel(t)\lambda/N_{\mathrm{sel}}(t); since km(t)≤⌊κt∣Im∣⌋k_m(t)\le\lfloor\kappa_t|I_m|\rfloor is nondecreasing in κt\kappa_t, Nsel(t)N_{\mathrm{sel}}(t) is nondecreasing in the budget and the weight is nonincreasing in it, strictly whenever NselN_{\mathrm{sel}} strictly decreases. □

Remark. This concerns per-token coefficients, not the aggregate auxiliary gradient norm, which need not decrease monotonically under coverage annealing because the retained set concentrates on the largest bib_i.

Baselines

We compare GIFT with the initial model (BASE) and eight training baselines, grouped by supervision source and how that supervision enters optimization.

Initial model and outcome-based optimization. These baselines establish performance before additional training and under verifier-based reinforcement learning without teacher feedback.

  • BASE: The initial model checkpoint without the additional task-specific training considered in our experiments.

  • GRPO [20]: Performs pure reinforcement learning using verifier-derived group-relative advantages, without teacher-derived token-level supervision.

Distributional self-distillation. These methods provide dense supervision by matching next-token distributions from a self-teacher conditioned on additional reference information or environment feedback.

  • OPSD [34]: Conditions a self-teacher on a correct reference answer or reasoning trace and matches its next-token distributions along student-generated trajectories. Its original formulation uses a forward-KL distillation objective without verifier-derived advantages.

  • SDPO [7]: Conditions a self-teacher on environment feedback or verified successful solutions and distills its next-token distributions along existing student trajectories.

Verifier-guided token-level credit assignment. These methods combine verifiable outcomes with scalar teacher-derived signals by modulating token-level advantages or gating token-level rewards.

  • RLSD [28]: Uses sampled-token probability ratios between a privileged self-teacher and the student to construct positive, clipped weights for verifier-derived advantages. This sign-preserving modulation redistributes credit across tokens and is annealed during training.

  • RLCSD [18]: Contrasts sampled-token likelihoods under correct- and incorrect-solution-conditioned teachers and applies bounded, threshold-gated modulation to verifier-derived advantages while preserving their signs.

  • OPDVR [12]: Gates sampled-token teacher–student log-ratio rewards according to trajectory correctness, retaining non-negative rewards for correct trajectories and non-positive rewards for incorrect trajectories.

Demonstration-guided methods. These methods incorporate reference reasoning trajectories into training samples or rollout contexts rather than relying only on teacher distributions along unassisted student rollouts.

  • LUFFY [26]: Combines off-policy teacher demonstrations with on-policy student rollouts and uses regularized importance sampling to learn from demonstrations while retaining student exploration.

  • UFT [13]: Initializes rollouts with prefixes of reference solutions, applies supervised learning to the provided prefixes, and performs reinforcement learning on the generated continuations. The expected prefix length is progressively reduced until training becomes hint-free.

These comparisons separate the source of teacher guidance from how it enters optimization. In contrast, GIFT uses gradient-informed token selection and coverage annealing to control the contribution of an auxiliary distributional loss while retaining the GRPO objective throughout training.

Detailed Experimental Setup

This appendix describes the training data, evaluation benchmarks, GIFT configuration, and evaluation protocol. The model-specific optimization and systems settings reported below refer to the OLMo-3-7B-Think configuration. They are listed separately from the settings shared by the experimental design.

Training Datasets

DeepMath-103K. DeepMath-103K [6] contains 103,022 English mathematical-reasoning examples covering six major topics. Each problem is paired with a final answer, a GPT-4o-estimated difficulty score, a topic label, and three DeepSeek-R1 reasoning traces. The released dataset undergoes semantic decontamination, question standardization, and answer-consistency filtering, supporting both outcome verification and reference-conditioned distillation.

Knights and Knaves. We adopt the synthetic Knights and Knaves (K&K) task from Logic-RL [23], where each character is either a truth-telling knight or a lying knave. Given their logically composed statements, the model must infer the unique role assignment, which can be verified using deterministic rules. We train on puzzles with 4–8 characters and evaluate role-count generalization on larger instances of the same task.

Evaluation Benchmarks

AMC 2023. AMC evaluates mathematical problem solving across algebra, geometry, number theory, counting, and probability (math-ai, 2023). Our evaluation set contains 83 problems, denoted as AMC23 in the result tables, and is scored by matching the generated final answer.

AIME24 and AIME25. AIME24 and AIME25 each contain 30 problems from the corresponding year’s AIME I and AIME II examinations [32, 33]. Each answer is an integer from 0 to 999 and is evaluated using final-answer matching. Results for the two years are reported separately.

In-distribution and out-of-distribution logic. For logical reasoning, we train on Knights-and-Knaves puzzles (K&K) [23] with 4–8 roles. We evaluate on 500 in-distribution problems and three out-of-distribution splits with 9, 10, and 11 roles, each containing 100 problems, to assess generalization beyond the training range.

GIFT Configuration

Privileged self-teacher. The student generates responses without access to the reference solution. A self-teacher then re-evaluates the same response prefixes with the reference solution and answer included in its context. The teacher shares the current actor parameters, with gradients stopped through the teacher branch, and is used only during training.

Sparse forward-KL distillation. Let pip_i and qiq_i be the student and teacher distributions at position ii, and let SiS_i contain the teacher’s top-100 tokens. We aggregate the remaining vocabulary into a single tail and compute

ℓisKL=∑v∈Siqi(v)log⁡qi(v)pi(v)+qi,taillog⁡qi,tailpi,tail,(28)\ell_i^{\mathrm{sKL}} = \sum_{v \in S_i} q_i(v) \log\frac{q_i(v)}{p_i(v)} + q_{i,\mathrm{tail}} \log\frac{q_{i,\mathrm{tail}}}{p_{i,\mathrm{tail}}}, \tag*{(28)}

where pi,tail=1−∑v∈Sipi(v)p_{i,\mathrm{tail}} = 1 - \sum_{v \in S_i} p_i(v) and likewise for qi,tailq_{i,\mathrm{tail}}. Teacher probabilities are treated as fixed targets.

Token ranking and allocation. For eligible response tokens, we compute the gradient geometry statistics aia_i and bib_i defined in the main text. The ranking score is aia_i when ai<0a_i < 0 and bib_i otherwise; tokens without a usable GRPO direction (i∉Di \notin D) are unclassified and receive score zero. Thus, conflicting tokens remain eligible but are withdrawn first as coverage decreases: when the quota exceeds the number of non-conflicting and unclassified eligible tokens, the least opposing conflicting tokens fill the remaining capacity, as guaranteed by Theorem 3. The quota in Equation 25 is exact—∣At,m∣=Kt,m|A_{t,m}| = K_{t,m} whenever Kt,m≤∣Em∣K_{t,m} \le|E_m|—so realized coverage tracks κt\kappa_t deterministically at the sub-batch level. Selection is performed independently within each training sub-batch mm:

Kt,m=⌊κt∣Im∣⌋,∣At,m∣=min⁡{Kt,m,∣Em∣},(29)K_{t,m} = \left\lfloor\kappa_t |I_m| \right\rfloor,\qquad|A_{t,m}| = \min\{K_{t,m}, |E_m|\}, \tag*{(29)}

where ImI_m and EmE_m denote its valid and eligible response tokens, respectively. The highest-scoring eligible tokens are selected.

Coverage schedule and objective. Training spans 96 actor steps with

κt={1,1≤t≤10,78−t68,11≤t≤77,0,78≤t≤96.(30)\kappa_t = \begin{cases} 1, & 1 \le t \le10,\\ \frac{78-t}{68}, & 11 \le t \le77,\\ 0, & 78 \le t \le96. \end{cases} \tag*{(30)}

We fix the distillation coefficient to λ=1\lambda= 1. Selected tokens receive the complete sparse forward-KL loss, normalized by the number of all valid response tokens, while GRPO remains active throughout training. From step 78, teacher evaluation is skipped and optimization reduces to GRPO alone.

Ablation Coverage and Qualitative Diagnostics

We examine the token-selection criterion, allocation scope, and teacher-withdrawal strategy. Let rir_i and did_i denote the GRPO and OPSD loss gradients with respect to the student logits. The signed parallel component aia_i and orthogonal residual norm bib_i follow the definitions in the method section.

Token-selection strategies.

  • All Tokens: Adds the OPSD loss directly to the GRPO objective at all valid response positions, without token selection. This variant evaluates joint training with unfiltered distillation.

  • Random Tokens: Selects token positions at random under the distillation budget, without using uncertainty or gradient geometry. It tests whether selective coverage alone is sufficient.

  • Entropy–KL Selection: Following the Q3 criterion in TIP (Xu et al. [24]), this variant prioritizes positions where the student is confident but disagrees with the teacher. Let

hi=−1log⁡∣V∣∑v∈Vpi(v)log⁡pi(v)(31)h_i = -\frac{1}{\log|V|} \sum_{v \in V} p_i(v) \log p_i(v) \tag*{(31)}

denote the normalized student entropy. The selection score is

siQ3=DKL(qi∥pi)(1−hi).(32)s_i^{\mathrm{Q3}} = D_{\mathrm{KL}}(q_i \mathbin\Vert p_i)(1-h_i). \tag*{(32)}

Tokens with larger scores have greater teacher–student divergence and lower student uncertainty. They are selected under the same coverage budget as GIFT.

  • Gradient-cone Selection: Uses the cosine similarity between the OPSD and GRPO logit gradients, ci=⟨di,ri⟩/(∥di∥2∥ri∥2)c_i = \langle d_i, r_i \rangle/ (\lVert d_i \rVert_2 \lVert r_i \rVert_2), at positions with usable gradients. This criterion measures directional agreement without separately accounting for the magnitude of the orthogonal residual.

  • Conflict-only: Uses only the conflict information provided by the parallel component aia_i for token selection, omitting residual-based prioritization. It isolates the contribution of conflict assessment.

  • Residual-only: Selects tokens solely according to the orthogonal residual norm bib_i, without distinguishing conflicting from non-conflicting positions through the sign of aia_i.

  • Teacher-norm: Selects tokens according to ∥di∥2\lVert d_i \rVert_2, the magnitude of the teacher-induced distillation gradient with respect to the student logits. It does not separate parallel and orthogonal contributions.

  • Conflict-Free: Excludes every token with ai<0a_i < 0 from distillation, regardless of the available budget. Unlike GIFT’s conflict-sensitive ordering, this variant never admits conflicting tokens to fill unused capacity.

Allocation scopes. These variants change the set of tokens that compete for the distillation budget.

  • Per-rollout: Ranks and selects tokens independently within each generated response. Tokens from different responses do not compete for a shared quota.

  • Per-prompt: Ranks tokens jointly across the group of responses generated for the same prompt. Tokens can compete across responses within the group, but not across different prompts.

  • Training sub-batch (GIFT): Ranks tokens jointly within each training sub-batch, allowing competition across the responses and prompts it contains. Selection budgets remain independent across training sub-batches.

Teacher-withdrawal strategies.

  • Sequential OPSD→\toGRPO: Performs OPSD training first and then switches to GRPO, separating distillation and outcome-based optimization into successive stages.

  • Weight Annealing: Gradually reduces the distillation-loss coefficient, weakening teacher supervision through its weight rather than progressively removing token positions.

  • Coverage Annealing (GIFT): Progressively reduces the token-retention budget while keeping the distillation coefficient fixed at λ=1\lambda= 1. Selected positions receive the full distillation loss, and training becomes pure GRPO when coverage reaches zero.

The preliminary schedule sweep evaluates linear decay with (H,Texit)(H, T_{\mathrm{exit}}) equal to (10,78)(10, 78), (0,78)(0, 78), (20,78)(20, 78), (10,50)(10, 50), and (10,90)(10, 90), and cosine decay with (10,78)(10, 78) and (0,78)(0, 78). Here, HH denotes the initial full-coverage period and TexitT_{\mathrm{exit}} the teacher exit step. All-token and conflict-free variants also change coverage or eligibility; schedule comparisons do not imply matched cumulative teacher exposure or training cost.

Additional Experiments

We provide additional experiments on model-family generalization, teacher withdrawal, and schedule sensitivity. These analyses examine the central design of GIFT: dense teacher guidance is used early, progressively restricted to higher-priority tokens, and eventually removed to recover pure outcome-grounded optimization. We evaluate whether this strategy transfers beyond Qwen3, outperforms alternative withdrawal mechanisms, and remains effective under different annealing configurations.

Evaluation beyond the Qwen3 family. On OLMo-3-7B-Think, GIFT achieves the highest mean@12 on both AMC23 and AIME25 (Figure 2). Its two-benchmark average of 80.70% exceeds Base and the strongest baseline, RLCSD, by 2.34 and 1.12 percentage points, respectively. The gain over Base is larger on AIME25 than on AMC23 (3.50 versus 1.18 points), consistent with the trend observed across the Qwen3 models. These improvements are consistent with GIFT’s central motivation: rather than uniformly applying teacher supervision, GIFT prioritizes non-conflicting updates

Qualitative Analysis

that contain directional information beyond the local RL gradient, while progressively reducing their coverage. This allows dense guidance to complement the outcome-grounded objective during early training without remaining the final optimization target. The consistent gains on OLMo-3-7B-Think further suggest that this mechanism is not specific to the Qwen3 model family.

Coverage annealing versus alternative withdrawal strategies. Table 7 compares GIFT with two alternative teacher-withdrawal strategies. In sequential OPSD→\toGRPO training, the model is first optimized using only the OPSD objective and then switched to pure GRPO, so dense teacher supervision and outcome-based optimization are applied in separate stages. Weight annealing instead combines OPSD with GRPO throughout the transition but applies distillation to all valid tokens and gradually decays its global loss coefficient, without annealing token coverage. GIFT outperforms these two alternatives by 1.59 and 2.87 percentage points on average, respectively. Unlike sequential training, GIFT keeps the outcome-grounded GRPO objective active while teacher guidance is available. Unlike weight annealing, it progressively removes lower-priority positions while preserving the full distillation strength at selected tokens. This design retains concentrated teacher guidance where it is more compatible with and complementary to the RL update, rather than uniformly weakening every teacher contribution. Once coverage reaches zero, the auxiliary loss vanishes and optimization continues with pure GRPO. The results support joint optimization with selective coverage annealing over either stage-wise training or uniform coefficient decay.

Sensitivity to warm-up and exit steps. Table 8 presents a preliminary sweep of the full-coverage warm-up HH and teacher exit step TexitT_{\mathrm{exit}}. With Texit=78T_{\mathrm{exit}} = 78, removing warm-up or extending it to 20 steps reduces the average from 78.96% to 78.03% or 77.26%, respectively. With H=10H = 10, moving teacher exit to step 50 or 90 gives 77.46% or 77.64%. The default configuration therefore performs best among the tested linear schedules. These results favor an initial full-coverage period followed by gradual withdrawal, while showing that either shortening or extending teacher guidance can reduce performance.

ScheduleHHTexitT_{\mathrm{exit}}AMC23AIME24AIME25Avg.
Linear (GIFT)107889.6676.1171.1178.96
Linear07889.3675.5669.1778.03
Linear207889.5675.2866.9477.26
Linear105090.1676.3965.8377.46
Linear109089.8675.0068.0677.64
Cosine107890.4676.1167.2277.93
Cosine07889.9673.8967.2277.02

Table 8. Preliminary sensitivity to the full-coverage period HH, teacher exit step TexitT_{\mathrm{exit}}, and decay shape on Qwen3-8B (mean@12, %). Avg. equally weights the three benchmarks. Bold marks the best score in each evaluation column.

Together, these experiments support GIFT’s combination of conflict-aware token selection, shared training-sub-batch allocation, and progressive teacher withdrawal. The evidence concerns reasoning performance and observed selection behavior, rather than direct measurements of parameter-space gradient compatibility or optimization plasticity.

Qualitative Analysis

We provide three qualitative analyses to examine both the reasoning outcomes and token-selection behavior of GIFT. First, we compare representative solutions from GIFT, GRPO, and OPSD to illustrate how combining dense guidance with outcome-grounded optimization affects reasoning accuracy. Second, we visualize the token positions retained at different training steps, showing how supervision becomes progressively sparse within a response. Third, we analyze retention in the space of the signed parallel component aia_i and orthogonal residual norm bib_i, connecting the observed selection pattern to GIFT’s piecewise ranking rule. Together, these visualizations illustrate how GIFT transitions from broad teacher supervision to concentrated guidance on higher-priority positions as the coverage budget contracts.

Qualitative analysis of reasoning accuracy. Figure 3 compares solutions to a combinatorial counting problem. All three displayed solutions separate strings according to whether they contain a digit 4, but differ in how accurately they execute the subsequent counting. GRPO undercounts the no-4 case as 283 instead of 671, while OPSD miscalculates intersection terms in inclusion–exclusion and obtains an incorrect total of 1336. GIFT correctly handles both cases, yielding 671 + 625 = 1296. Across 12 sampled responses to this problem, GIFT produces 12 correct answers, compared with 7 for GRPO and 6 for OPSD. The contrast connects to our introductory motivation: a plausible reasoning structure does not ensure correct intermediate decisions, and neither outcome-based training nor self-distillation alone eliminates such errors in these examples. GIFT’s more consistent answers provide a qualitative illustration of the goal of combining dense guidance with outcome-grounded optimization, complementing the quantitative ablations of its selection and withdrawal mechanisms.

Example solutions to a combinatorial counting problem

Figure 3. Example solutions to a combinatorial counting problem. The displayed GIFT solution is correct, while the GRPO and OPSD solutions contain counting errors. Correct-sample counts summarize 12 generations per method.

Qualitative analysis of token selection. Figure 4 illustrates how distillation coverage changes within a reasoning excerpt about convex functions. All 69 displayed tokens are selected at step 10, compared with 40 at step 40 and 17 at step 70. The remaining positions include words such as “convex,” “outside,” and “continuity,” while many surrounding words and mathematical symbols no longer receive distillation. Supervision therefore becomes sparse within sentences rather than being removed at the sentence level. This example illustrates the allocation problem raised in the introduction: reducing teacher influence involves deciding where guidance remains, not merely weakening every teacher contribution. GIFT makes this decision through gradient-based ranking within each training sub-batch. Selected positions retain the full distillation loss, while all valid tokens remain included in the GRPO objective. The visualization shows where supervision is applied, rather than establishing the semantic importance or correctness of individual tokens.

Token-level distillation selection in a reasoning excerpt

Figure 4. Token-level distillation selection in a reasoning excerpt. Red marks selected tokens and gray marks unselected tokens. The number of selected positions decreases from 69 to 40 to 17 across the displayed steps; counts refer only to the shown excerpt.

Token selection under budget annealing. Figure 5 shows how GIFT concentrates distillation as the token budget contracts. At step 10, all eligible tokens are retained, including conflicting and neutral positions. At step 40, conflicting-token retention is zero in the displayed statistics, while 97.4% of non-conflicting tokens and 60.9% of neutral tokens remain selected. By step 70, these rates decrease to 21.0% and 10.8%, respectively, with non-conflicting selection concentrated at larger orthogonal residuals. This pattern follows the piecewise ranking: conflicting tokens receive negative scores, non-conflicting tokens are prioritized by residual magnitude, and neutral tokens remain eligible with zero scores. Competition within each training sub-batch therefore produces a selective withdrawal of supervision rather than uniform attenuation. Retained tokens receive the full, unprojected distillation loss, while all valid response tokens remain included in the GRPO objective.

Token retention under coverage annealing

Figure 5. Token retention under coverage annealing. The horizontal and vertical axes show the signed parallel component aia_i and orthogonal residual norm bib_i of the distillation gradient relative to the GRPO direction. Colors indicate the fraction of eligible tokens retained within each bin. Annotations summarize retention for conflicting (ai<0a_i < 0) and non-conflicting (ai≥0a_i \ge0) tokens with usable GRPO reference directions. Bottom bars separately report retention for neutral tokens without such a direction.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pp. 21246–21263, 2024.arxiv.org/abs/2306.13649
  2. [2]Lucio M. Dery, Yann Dauphin, and David Grangier. {AUXILIARY} {task} {update} {decomposition}: {THE} {good}, {the} {bad} {and} {the} {neutral}. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=1GTma8Hw1P.arxiv.org/abs/2108.11346
  3. [3]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minllm: On-policy distillation of large language models. arXiv preprint arXiv:2306.08543, 2023.arxiv.org/abs/2306.08543
  4. [4]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihuao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Wu Yu, Z. F. Wu, Zhibin Guo, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochaо Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecheng Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaoјun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzუ Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zhou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zhihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z.
  5. [5]Zhuowen Han, Jinwei Xiao, Zhengxi Lu, Renren Jin, Zhiyuan Yao, Yuxin Liu, Hongyan Hao, Yueqing Sun, Yu Yang, Qi GU, Xunliang Cai, and Deyi Xiong. Distill where you fail: Recovering learning signals of negative rl-groups from adaptive teacher guidance, 2026. URL https://arxiv.org/abs/2608.00782.
  6. [6]Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning, 2025. URL https://arxiv.org/abs/2504.11456.
  7. [7]Jonas Hübötter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation, 2026. URL https://arxiv.org/abs/2601.20802.
  8. [8]Yubo Jiang, Fengying Xie, Zhiguo Jiang, and Haopeng Zhang. Distill skills into weights, not prompts: Abstract skills as privileged signals for on-policy self-distillation, 2026. URL https://arxiv.org/abs/2608.09826.
  9. [9]Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026.arxiv.org/abs/2603.07079
  10. [10]Jeonghye Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Rebellious student: Reversing teacher signals for reasoning exploration with self-distilled rlvr. arXiv preprint arXiv:2605.10781, 2026.arxiv.org/abs/2605.10781
  11. [11]Boyan Li, Bingsen Chen, Chenghao Yang, Ping Nie, Chen Zhao, and Xi Ye. Sequential beats joint: On the interplay between on-policy distillation and rlvr, 2026. URL https://arxiv.org/abs/2609.04108.
  12. [12]Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, and Gao Huang. On-policy distillation with verifiable reward. arXiv preprint arXiv:2608.24696, 2026.arxiv.org/abs/2608.24696
  13. [13]Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and reinforcement fine-tuning. arXiv preprint arXiv:2505.16984, 2025a.arxiv.org/abs/2505.16984
  14. [14]Yifeng Liu, Shiyuan Zhang, Yifan Zhang, and Quanquan Gu. Self-distilled policy gradient. arXiv preprint arXiv:2606.04036, 2026.arxiv.org/abs/2606.04036
  15. [15]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
  16. [17]Team Olmo, ; Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Shashank Gupta, Taira Anderson, Teng Xiao, Tyler Murray, Tyler Romero, Victoria Graf, Akari Asai, Akshita Bhagia, Alexander Wettig, Alisa Liu, Aman Rangapur, Chloe Anastasiades, Costa Huang, Dustin Schwenk, Harsh Trivedi, Ian Magnusson, Jaron Lochner, Jiacheng Liu, Lester James V. Miranda, Maarten Sap, Malia Morgan, Michael Schmitz, Michal Guerguin, Michael Wilson, Regan Huff, Ronan Le Bras, Rui Xin, Rulin Shao, Sam Skjonsberg, Shannon Zejiang Shen, Shuyue Stella Li, Tucker Wilde, Valentina Pyatkin, Will Merrill, Yapei Chang, Yuling Gu, Zhiyuan Zeng, Ashish Sabharwal, Luke Zettlemoyer, Pang Wei Koh, Ali Farhadi, Noah A. Smith, and Hannahneh Hajishirzi. Olmo 3, 2026. URL https://arxiv.org/abs/2512.13961.
  17. [18]Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, and Lijie Wen. Rlc sd: Reinforcement learning with contrastive on-policy self-distillation. arXiv preprint arXiv:2606.11709, 2026.arxiv.org/abs/2606.11709
  18. [19]Simon Schmitt, Jonathan J. Hudson, Augustin Židek, Simon Osindero, Carl Doersch, Wojciech M. Czarnecki, Joel Z. Leibo, Heinrich Küttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting deep reinforcement learning. ArXiv, abs/1803.03835, 2018. URL https://api.semanticscholar.org/CorpusID:3883991.arxiv.org/abs/1803.03835
  19. [20]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.arxiv.org/abs/2402.03300
  20. [21]Zhenlin Wei, Pu Jian, Yingzhuo Deng, Xiaohan Wang, Jiajun Chai, Zhexin Hu, Wei Lin, Shanbin Zhang, and Guojun Yin. Amr-sd: Asymmetric meta-reflective self-distillation for token-level credit assignment, 2026. URL https://arxiv.org/abs/2605.18529.
  21. [22]Xingjian Wu, Junlin Liu, Xingchen Liu, Xuhang Zhu, Jianing Wang, Linsen Guo, Xiaoyu Li, Xuezhi Cao, and Xunliang Cai. Contrastive reinforced policy optimization via privileged self-distillation. 2026. URL https://arxiv.org/abs/2607.28026.
  22. [23]Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, 2025. URL https://arxiv.org/abs/2502.14768.
  23. [24]Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation. arXiv preprint arXiv:2604.14084, 2026.arxiv.org/abs/2604.14084
  24. [25]Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance, 2025. URL https://arxiv.org/abs/2504.14945, 2025a.
  25. [26]Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance, 2025b. URL https://arxiv.org/abs/2504.14945.
  26. [27]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengyuan Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  27. [28]Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr. arXiv preprint arXiv:2604.03128, 2026.arxiv.org/abs/2604.03128
  28. [29]Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient diversity: a key ingredient for scalable distributed learning, 2018. URL https://arxiv.org/abs/1706.05699.
  29. [30]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xianpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale. 2025. URL https://arxiv.org/abs/2503.14476.
  30. [31]Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782, 2020.arxiv.org/abs/2001.06782
  31. [32]Yifan Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2024, 2024. URL https://huggingface.co/datasets/math-ai/aime24.
  32. [33]Yifan Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2025, 2025. URL https://huggingface.co/datasets/math-ai/aime25.
  33. [34]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026.arxiv.org/abs/2601.18734

Paper details

vlnc:W_crsy2z97
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

large language models (LLMs)

View statement
Looking for related papers…
Contents