betaFeedback

WorksComputer Science

B-OPSD: On-Policy Self-Distillation with Self-Generated Problem Briefs

Abstract

On-Policy Self-Distillation (OPSD) has recently attracted interest because it provides dense teacher guidance without a separate, often larger teacher model. However, OPSD conditions its teacher on ground-truth solutions, which limits its use when such solutions are unavailable. We propose On-Policy Self-Distillation with Self-Generated Problem Briefs (B-OPSD). Using a prompt template, the initial model or a saved student checkpoint generates a problem brief from the question alone. The prompt asks for relevant principles, applicability conditions, and likely pitfalls, but not a final answer or a complete solution. A frozen teacher uses the brief as privileged context to guide the current student's on-policy rollouts through token-level distribution matching. B-OPSD uses no ground-truth solutions during training. Results on AIME 2024, AIME 2025, and HMMT 2025 show that both B-OPSD variants achieve higher average accuracy than OPSD at each of three Qwen3 model sizes. Code is available at https://github.com/ZevineXu/B-OPSD

Introduction

Large language models have shown strong performance in language understanding [2, 11, 30], mathematical reasoning [20, 12, 3], and code generation [14, 9, 4]. Their performance on specific tasks can be improved through supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy distillation (OPD) [15, 17, 1].

These methods provide different learning signals. SFT trains the model on reference responses supplied in the training data. In mathematical reasoning, RLVR can sample responses from the current model and score their final answers, but an outcome reward does not identify which reasoning steps caused an error. OPD also uses responses sampled from the student. A teacher then evaluates each student prefix and provides guidance through its next-token distribution [1, 13].

OPD commonly uses a separate teacher model to provide this guidance. On-Policy Self-Distillation (OPSD) instead gives the same base model two roles with different input contexts [31]. Its student sees the problem, while its teacher also sees a reference solution. Both roles evaluate the student’s trajectory, and the teacher’s distribution guides the student at each token. OPSD does not use a separate teacher model during distillation, but it still needs a reference solution for each training problem. Preparing reference solutions for a dataset of questions alone may require an external model and validation. This dependence raises a practical question: Can on-policy self-distillation work when no correct reference solution is available?

Preprint.

To answer this question, we propose On-Policy Self-Distillation with Self-Generated Problem Briefs (B-OPSD). The model generates a problem brief from the question alone. The brief is intended to connect relevant principles to the problem, state when they apply, and note likely pitfalls and checks. The prompt instructs the model not to provide a final answer or a complete solution. As Figure 1 shows, the student samples a solution trajectory from the question, while the teacher evaluates the same trajectory with the brief as additional context. Their token-level distribution difference updates only the student. We consider briefs generated once before training and briefs refreshed from earlier student checkpoints. Neither version uses a reference solution or an external model to prepare teacher context. On AIME 2024, AIME 2025, and HMMT 2025, both versions achieve higher average accuracy across the three benchmarks than OPSD at each of three Qwen3 model sizes in the reported runs. These results suggest that self-generated problem briefs can provide useful teacher context without a verified solution.

Overview of B-OPSD

Figure 1. Overview of B-OPSD. Student and teacher start from the same base model; the teacher stays frozen while the student is updated. The initial model generates static briefs, while a saved student checkpoint generates refreshed briefs. The current student generates its solution trajectory from the question alone. The frozen teacher receives the question and brief, evaluates the same student prefixes, and provides token-level feedback.

Related Work

Supervised fine-tuning (SFT). SFT trains on reference responses, predicting each token from the preceding reference tokens [15, 24, 5]. Sequence-level KD uses teacher responses as targets [10], while STaR and ReSTEM train on solutions selected with answer feedback [28, 18]. Training on reference prefixes but generating from the model’s own prefixes at inference creates exposure bias.

Reinforcement learning with verifiable rewards (RLVR). RLVR samples responses from the current policy and scores their final answers.[16, 22, 29] PPO uses clipped updates but typically trains a value model, which adds memory cost [16]. GRPO avoids this model by estimating advantages from groups of responses [17]. Building on GRPO, DAPO raises the upper clipping bound and filters groups with uniform rewards [26]. GSPO replaces GRPO’s token-level importance ratios and clipping with sequence-level versions [32]. Yet outcome rewards cannot locate a reasoning error or provide a teacher distribution at each token.

On-policy distillation (OPD). OPD addresses the limitation of RLVR by matching a teacher’s next-token distributions on trajectories generated by the current student [13, 27, 21]. GKD trains on student-generated sequences and allows different divergences to match the teacher and student distributions [1]. Entropy-Aware OPD adds forward KL to the reverse KL loss when the teacher distribution has high entropy [8]. Many existing OPD systems use a separate, stronger teacher model. The teacher is often larger than the student, which increases training cost.

On-policy self-distillation (OPSD). OPSD gives student and teacher roles to the same base model, with a reference answer or solution available only to the teacher [31]. Context distillation also uses extra information during training, and On-Policy Context Distillation conditions a teacher on historical solutions or optimized prompts [19, 25]. OPSD removes the separate teacher model but still needs a reference for each training problem. B-OPSD replaces that reference with a self-generated problem brief. The teacher uses the brief to guide student rollouts through token distributions; no reference answer is read during training. Table 1 compares the training setups.

MethodOn-policy dataDense learning signalNo separate teacher modelNo answer labels
SFT×✓✓×
GRPO✓×✓×
OPD✓✓×✓
OPSD✓✓✓×
B-OPSD✓✓✓✓

Table 1. B-OPSD combines on-policy student rollouts and token-level teacher feedback without a separate teacher model, reference solutions, or verified answers during training. The SFT and GRPO rows use the reference solutions and verified final answers, respectively, from the compared training setups. The OPD row represents a common setup with a separate teacher model and no answer labels during distillation.

Method

Generating Problem Briefs

We train B-OPSD on a set D={xi}i=1N\mathcal{D} = \{x_i\}_{i=1}^{N} of mathematical problems without reading reference solutions or using an outcome verifier. OPSD supplies its teacher with a reference solution [31]. B-OPSD instead prepares teacher context from the problem itself. Identifying relevant principles and checking their conditions may be possible even when producing a complete correct solution is difficult. We use a problem brief to make this information available to the teacher. Figure 2 shows the complete generation prompt PB\mathcal{P}_B.

Figure 2. Complete problem brief generation prompt PBP_B. The same prompt is used for static and refreshed briefs. The generator also receives the problem statement xx.

The prompt has four parts. First, it asks the generator to identify the target, givens, constraints, and main challenge. Second, it asks for relevant principles, including their formulas, meaning, applicability conditions, and connection to the problem. Third, it asks for a useful setup and main subgoals. Local setup equations are allowed, but the calculations and proof are left to the solver, and other valid approaches remain possible. Fourth, it asks for problem-specific pitfalls, conditions to verify, and ways to check the result. The prompt also asks the generator to avoid unrelated theory and invented assumptions, omit the final answer and complete solution, keep the brief within 700 tokens, and output only the brief.

Let θ0\theta_0 be the initial model parameters and θk\theta_k the student parameters after kk updates. The brief generator has parameters ϕk\phi_k from the initial model or a saved student checkpoint. Given a problem xx, it samples a brief as

b∼pϕk(⋅∣PB,x).(1)b \sim p_{\phi_k}(\cdot\mid\mathcal{P}_B, x). \tag*{(1)}

We use one brief for each problem when it is processed and pass the brief to the teacher as text. Static briefs are reused, while refreshed briefs are generated using the selected checkpoint. Gradients do not pass through brief generation. We do not verify or filter generated briefs. Appendix § 7 shows outputs from the generation records.

The teacher prompt presents the original problem followed by the brief and asks the model to keep each reasoning step valid. The student receives the problem without the brief. The brief is teacher context, not a target response for the student to reproduce. It is not checked against a reference solution and may contain mistakes. The method uses it to change the teacher’s conditioning context, not as a verified answer.

On-Policy Self-Distillation with Problem Briefs

We retain a frozen teacher with parameters θˉ=θ0\bar{\theta}=\theta_0 and update only the student. At training step kk, the student samples a solution trajectory from its current policy:

y^∼pθk(⋅∣x).(2)\hat{y} \sim p_{\theta_k}(\cdot\mid x). \tag*{(2)}

Let T=∣y^∣T = |\hat{y}| be its length and y^<t\hat{y}_{<t} its prefix at position tt. The student and teacher evaluate the same prefix, with the brief available only to the teacher:

pS,tθ=pθ(⋅∣x,y^<t),(3)p^{\theta}_{S,t} = p_{\theta}(\cdot\mid x,\hat{y}_{<t}), \tag*{(3)}
pT,t=pθˉ(⋅∣x,b,y^<t).(4)p_{T,t} = p_{\bar{\theta}}(\cdot\mid x,b,\hat{y}_{<t}). \tag*{(4)}

The role-specific prompts are omitted for clarity. To evaluate a trajectory, we append the same sampled prefix to each role’s prompt and compute next-token distributions. The teacher does not need to sample a separate solution. Its forward pass supplies a distribution at every position of the student’s trajectory. Thus, the loss can be computed for a trajectory without first judging its final answer. The teacher’s guidance still depends on the quality of the generated brief; it does not certify that a step or answer is correct.

Distribution matching. For a fixed brief and sampled trajectory, we average the divergence between the two next-token distributions over the positions in the trajectory:

ℓD(θ;x,b,y^)=1T∑t=1TD(pT,t∥pS,tθ).(5)\ell_D(\theta;x,b,\hat{y}) = \frac{1}{T}\sum_{t=1}^{T}D(p_{T,t}\Vert p^{\theta}_{S,t}). \tag*{(5)}

The loss compares full distributions rather than only the probabilities of the sampled tokens. We consider three choices for DD in Eq. (5). Forward KL weights the log ratio by the teacher distribution, while reverse KL weights it by the student distribution:

DFKL=DKL(pT,t∥pS,tθ),(6)D_{\mathrm{FKL}} = D_{\mathrm{KL}}(p_{T,t}\Vert p^{\theta}_{S,t}), \tag*{(6)}
DRKL=DKL(pS,tθ∥pT,t),(7)D_{\mathrm{RKL}} = D_{\mathrm{KL}}(p^{\theta}_{S,t}\Vert p_{T,t}), \tag*{(7)}

JSD compares each distribution with their mixture mt=βpT,t+(1−β)pS,tθm_t = \beta p_{T,t} + (1-\beta)p^{\theta}_{S,t}:

DJSDβ=βDKL(pT,t∥mt)+(1−β)DKL(pS,tθ∥mt).(8)D_{\mathrm{JSD}_{\beta}} = \beta D_{\mathrm{KL}}(p_{T,t}\Vert m_t) + (1-\beta)D_{\mathrm{KL}}(p^{\theta}_{S,t}\Vert m_t). \tag*{(8)}

Algorithm 1 B-OPSD training with static or refreshed problem briefs

Require: Problems D\mathcal{D}, initial parameters θ0\theta_0, brief prompt PBP_B, training steps KK Require: Brief mode, checkpoint schedule C\mathcal{C}, divergence D\mathcal{D}, threshold τ\tau

  1. Freeze teacher at θˉ←θ0\bar{\theta} \leftarrow\theta_0

  2. if brief mode is static then

  3. &nbsp;&nbsp;&nbsp;&nbsp;for all x∈Dx \in\mathcal{D} do

  4. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Generate and store b0(x)∼pθ0(⋅∣PB,x)b_0(x) \sim p_{\theta_0}(\cdot\mid P_B,x)

  5. &nbsp;&nbsp;&nbsp;&nbsp;end for

  6. end if

  7. for k=0,1,…,K−1k = 0,1,\ldots,K-1 do

  8. &nbsp;&nbsp;&nbsp;&nbsp;Sample a batch Bk\mathcal{B}_k from D\mathcal{D}

  9. &nbsp;&nbsp;&nbsp;&nbsp;if brief mode is refreshed then

  10. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Select generator ϕk←θs(k)\phi_k \leftarrow\theta_{s(k)} using Eq. (12)

  11. &nbsp;&nbsp;&nbsp;&nbsp;end if

  12. &nbsp;&nbsp;&nbsp;&nbsp;for all x∈Bkx \in\mathcal{B}_k do

  13. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if brief mode is static then

  14. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bk(x)←b0(x)b_k(x) \leftarrow b_0(x)

  15. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else

  16. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bk(x)∼pϕk(⋅∣PB,x)b_k(x) \sim p_{\phi_k}(\cdot\mid P_B,x)

  17. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;end if

  18. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Sample a student trajectory y^x∼pθk(⋅∣x)\hat{y}_x \sim p_{\theta_k}(\cdot\mid x)

  19. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Evaluate student and teacher distributions on y^x\hat{y}_x using Eqs. (3)–(4)

  20. &nbsp;&nbsp;&nbsp;&nbsp;end for

  21. &nbsp;&nbsp;&nbsp;&nbsp;Lk(θ)←1∣Bk∣∑x∈BkℓDT(θ;x,bk(x),y^x)\mathcal{L}_k(\theta) \leftarrow\frac{1}{|\mathcal{B}_k|}\sum_{x\in\mathcal{B}_k}\ell_D^T(\theta;x,b_k(x),\hat{y}_x)

  22. &nbsp;&nbsp;&nbsp;&nbsp;Update only the student to obtain θk+1\theta_{k+1} by minimizing Lk\mathcal{L}_k

  23. &nbsp;&nbsp;&nbsp;&nbsp;if k+1∈Ck+1 \in\mathcal{C} then

  24. &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Save student checkpoint θk+1\theta_{k+1}

  25. &nbsp;&nbsp;&nbsp;&nbsp;end if

  26. end for

  27. return pθKp_{\theta_K}

Here DKLD_{\mathrm{KL}} denotes KL divergence and β∈[0,1]\beta\in[0,1]. We use β=0.5\beta= 0.5 in the divergence comparison. The main experiments use forward KL. In the divergence comparison, we train a separate model with each loss under the same teacher conditioning.

Pointwise divergence clipping. We retain the pointwise clipping procedure of OPSD [31]. Let V\mathcal{V} be the model vocabulary and dt,vDd^D_{t,v} the contribution of entry vv to the selected divergence at position tt, so that

D(pT,t∥pθ,t)=∑v∈Vdt,vD.(9)D(p_{T,t} \Vert p_{\theta,t}) = \sum_{v\in\mathcal{V}} d^D_{t,v}. \tag*{(9)}

For a threshold τ\tau, the clipped loss is

ℓD(θ;x,b,y^)=1T∑t=1T∑v∈Vmin⁡(dt,vD,τ).(10)\ell_D(\theta;x,b,\hat{y}) = \frac{1}{T}\sum_{t=1}^{T}\sum_{v\in\mathcal{V}}\min(d^D_{t,v},\tau). \tag*{(10)}

For forward KL, dt,vDd^D_{t,v} is the teacher probability pT,t(v)p_{T,t}(v) times the log ratio of teacher to student probabilities at vv. Reverse KL exchanges the teacher and student probabilities. For JSD, dt,vDd^D_{t,v} is the sum of the two weighted KL contributions in Eq. (8), evaluated at vv. Clipping is applied to these vocabulary contributions before they are summed over entries and positions. It does not clip the completed sequence loss. The divergence choice and clipping threshold are reported in the experimental settings.

For a batch Bk\mathcal{B}_k, we average the clipped losses:

Lk(θ)=1∣Bk∣∑x∈BkℓDT(θ;x,bk(x),y^x),(11)\mathcal{L}_k(\theta) = \frac{1}{|\mathcal{B}_k|}\sum_{x\in\mathcal{B}_k}\ell_D^T(\theta;x,b_k(x),\hat{y}_x), \tag*{(11)}

where bk(x)b_k(x) is the brief assigned to xx at step kk and y^x\hat{y}_x is sampled according to Eq. (2). The optimizer updates θk\theta_k using this loss. Neither the teacher nor the brief generator receives gradients from the update. The sampled trajectory and teacher probabilities are held fixed during this update; gradients pass only through the student probabilities. A new trajectory is sampled from the updated student at later steps. Thus, the rollouts remain on-policy while the teacher stays at θ0\theta_0.

Static and Refreshed Briefs

We consider two ways to choose the brief generator. Both use the same generation prompt, frozen teacher, and distillation procedure. They differ in whether the generator remains at the initial model or follows saved student checkpoints.

B-OPSD with static briefs. Before training, the initial model generates one brief for every training problem. We retain each generated brief without format filtering and reuse it throughout training. Thus, ϕk=θ0\phi_k = \theta_0 for all kk, and bk(x)=b0(x)b_k(x) = b_0(x). This variant keeps the teacher context fixed for each problem. Its teacher distributions can still change across visits to that problem because the student may generate different trajectory prefixes.

B-OPSD with refreshed briefs. The generator uses the most recently saved student checkpoint. Let CC be the set of completed update counts at which checkpoints are saved, including 0. At step kk, we select

s(k)=max⁡{s∈C:s≤k},ϕk=θs(k).(12)s(k) = \max\{s \in C : s \leq k\}, \qquad\phi_k = \theta_{s(k)}. \tag*{(12)}

The initial model supplies briefs until a later checkpoint becomes available. Subsequent briefs use the selected checkpoint. This allows the brief content to reflect changes in the student during training without updating the teacher parameters. The checkpoint interval and generation settings are specified in the experiments.

Algorithm 1 summarizes the training procedure. In both variants, the current student generates the trajectories used for distillation. The source of the brief does not change this on-policy property. At inference, we use the trained student with the problem alone; brief generation and teacher evaluation are training operations.

Experiments

Experimental Setup

Models and data. Following OPSD [31], we use the instruct-tuned Qwen3-1.7B, Qwen3-4B, and Qwen3-8B models [23]. We sample up to 30,000 mathematical problems from the OpenThoughts reasoning subset [6]. B-OPSD reads only the problem statements during training. We evaluate on AIME 2024, AIME 2025, and HMMT 2025.

Baselines. We compare B-OPSD with the initial model, SFT, GRPO, and OPSD. We take the Base, SFT, and GRPO results from OPSD [31] and reproduce OPSD ourselves. In the original study, SFT trains on the reference solutions in OpenThoughts, while GRPO uses verified final answers to provide outcome rewards. Our OPSD reproduction uses the reference solutions as privileged teacher context.

Implementation details. Our OPSD reproduction and B-OPSD runs use LoRA on NVIDIA A800 GPUs [7]. Following OPSD, we use a learning rate of 5×10−65 \times10^{-6}, an effective batch size of 32, LoRA rank 64, and LoRA scaling factor 128. All three distillation runs train for 100 updates. We sample one student trajectory per problem at temperature 1.1, with a maximum length of 1,024 tokens. The teacher remains fixed at the initial model while the student is updated. The main B-OPSD runs use full-vocabulary forward KL with pointwise clipping at τ=0.05\tau= 0.05.

We generate problem briefs with temperature 0.0. B-OPSD with static briefs generates each brief before training. B-OPSD with refreshed briefs uses the most recently saved student checkpoint and saves a new checkpoint every 25 updates. Appendix § 6 gives further implementation details.

Evaluation. Following OPSD, our evaluations report Avg@12 accuracy from 12 samples per problem. We use thinking mode, sampling temperature 1.0, top-pp 0.95, and a maximum generation length of 38,912 tokens [31]. We evaluate our distillation checkpoints at steps 25, 50, 75, and 100 and report the highest score for each benchmark.

In the results, B-OPSD denotes the version with static briefs, and B-OPSD⋆ denotes the version with refreshed briefs.

Main Results

Table 2 shows that both B-OPSD variants have higher average accuracy than Base, SFT, GRPO, and OPSD at every model size tested. Compared with OPSD, B-OPSD gains 1.0, 0.6, and 0.6 percentage points at 1.7B, 4B, and 8B. B-OPSD⋆ gains 2.0, 1.3, and 1.3 points, respectively, and has the highest average at each size. These results show that problem briefs can provide useful teacher context without reference solutions during training in the reported settings.

Table 2: Avg@12 accuracy (%) across three benchmarks. B-OPSD uses static briefs; B-OPSD⋆ uses refreshed briefs. Average is computed from unrounded scores. Shaded rows show the B-OPSD variants, and bold marks the highest score for each model size and column.

MethodAIME 2024AIME 2025HMMT 2025Average
Qwen3-1.7B
Base51.536.723.137.1
SFT48.436.322.735.8
GRPO51.138.323.737.7
OPSD54.743.125.040.9
B-OPSD56.442.227.241.9
B-OPSD⋆57.542.528.642.9
Qwen3-4B
Base74.966.442.261.2
SFT70.262.343.458.6
GRPO75.668.144.462.7
OPSD75.667.545.863.0
B-OPSD76.468.346.163.6
B-OPSD⋆77.570.345.064.3
Qwen3-8B
Base75.865.643.961.8
SFT72.364.242.959.8
GRPO76.468.946.764.0
OPSD77.871.946.765.5
B-OPSD79.771.147.566.1
B-OPSD⋆78.371.950.066.8

Table 2.

Across the three benchmarks, B-OPSD⋆^\star achieves the highest average accuracy at every model size. On AIME 2024, both B-OPSD variants exceed OPSD at all three sizes. B-OPSD⋆^\star reaches 57.5 versus 54.7 for OPSD at 1.7B, and 77.5 versus 75.6 at 4B. On AIME 2025, B-OPSD⋆^\star reaches 70.3 versus 67.5 for OPSD at 4B. On HMMT 2025, B-OPSD exceeds OPSD at all three sizes, and B-OPSD⋆^\star reaches 50.0 versus 46.7 for OPSD at 8B.

Refreshing briefs raises the reported average over static briefs by 1.0, 0.7, and 0.7 points at 1.7B, 4B, and 8B. At 8B, refreshed briefs also achieve higher scores on AIME 2025 and HMMT 2025. Figure 3 shows the Qwen3-1.7B results across checkpoints. In its Average panel, B-OPSD⋆^\star exceeds both B-OPSD and OPSD at three of the four evaluated checkpoints. Briefs from later student checkpoints may provide teacher context that better reflects the student’s changing policy than briefs fixed before training. These results show that problem briefs can improve self-distillation without verified reference solutions in the reported settings.

Figure 3. Checkpoint Avg@12 accuracy (%) for Qwen3-1.7B. B-OPSD uses static briefs; B-OPSD* uses refreshed briefs. The Average panel combines scores from the same checkpoint; Table 2 averages each benchmark’s best score. The vertical axes start near the observed scores and differ across panels.

Problem Brief Generation Strategies

In this section, we examine whether problem briefs expected to be more accurate improve B-OPSD. A larger model may generate more accurate briefs, and access to a reference solution may also help the generator identify relevant information. We therefore compare two variants for distilling Qwen3-1.7B. In B-OPSD(8B), Qwen3-8B generates the briefs from the training problems. In B-OPSD(ref), the briefs are generated using the reference solutions supplied to OPSD. Both variants use the resulting briefs as privileged teacher context to train Qwen3-1.7B. B-OPSD(ref) uses reference solutions during brief preparation, while the main B-OPSD variants do not. All runs use the same training and evaluation settings for the 1.7B model. We report results on AIME 2024 and HMMT 2025 in Table 3.

Table 3: Avg@12 accuracy (%) for Qwen3-1.7B with different brief generation strategies. B-OPSD uses static briefs generated by the initial 1.7B model, and B-OPSD⋆^\star uses refreshed briefs. B-OPSD(8B) uses briefs generated by Qwen3-8B, while B-OPSD(ref) uses briefs generated using the reference solutions supplied to OPSD. Shaded rows show our main variants. Bold marks the best result in each column.

MethodAIME 2024HMMT 2025
OPSD54.725.0
B-OPSD(8B)56.427.8
B-OPSD(ref)56.427.5
B-OPSD56.427.2
B-OPSD⋆^\star57.528.6

Table 3.

Both alternative brief sources improve on OPSD. B-OPSD(8B) and B-OPSD(ref) each match B-OPSD at 56.4 on AIME 2024. On HMMT 2025, they exceed B-OPSD by 0.6 and 0.3 points, respectively. B-OPSD⋆^\star, which refreshes briefs using a recent student checkpoint, achieves the highest scores on both benchmarks. This pattern suggests that briefs adapted as the student learns may be more useful than briefs from a larger model or reference solutions.

Effect of Divergence Choice

Following OPSD [31], we compare forward KL DKL(pT∥pS)D_{\mathrm{KL}}(p_T \| p_S), reverse KL DKL(pS∥pT)D_{\mathrm{KL}}(p_S \| p_T), and Jensen–Shannon divergence (JSD) with β=0.5\beta= 0.5 for B-OPSD⋆^\star, B-OPSD, and OPSD on Qwen3-1.7B. We evaluate all three methods on AIME 2024 and HMMT 2025 and report Avg@12 accuracy, as in Table 2.

Forward KL achieves the highest score for both B-OPSD variants on AIME 2024 and HMMT 2025. For OPSD, forward KL scores highest on AIME 2024 and ties JSD on HMMT 2025. Under each of the three divergences, B-OPSD has the highest score among the three methods on both benchmarks and exceeds OPSD. These results support the use of forward KL in our main Qwen3-1.7B experiments. They also show that B-OPSD⋆^\star can outperform OPSD across the tested divergences without using reference solutions during training.

DivergenceOPSDB-OPSDB-OPSD⋆
AIME 2024
Forward KL54.756.457.5
Reverse KL51.152.754.2
JSD (β=0.5\beta= 0.5)50.050.052.9
HMMT 2025
Forward KL25.027.228.6
Reverse KL24.725.026.4
JSD (β=0.5\beta= 0.5)25.025.325.6

Table 4. Avg@12 accuracy (%) for Qwen3-1.7B with different divergences. B-OPSD uses static briefs; B-OPSD⋆ uses refreshed briefs. Bold marks the highest score in each row.

Problem Brief Component Ablation

We examine the four sections of the problem brief using Qwen3-1.7B. Each ablation omits one section: targets, relevant principles, suggested approach, or pitfalls and checks. We evaluate both B-OPSD with static briefs and B-OPSD with refreshed briefs on AIME 2024 and HMMT 2025. Table 5 compares each ablation with the corresponding full brief.

Omitted sectionB-OPSDB-OPSD⋆
AIME 2024HMMT 2025AIME 2024HMMT 2025
None (full brief)56.427.257.528.6
Targets57.228.656.127.8
Relevant principles55.826.756.726.7
Suggested approach57.528.357.026.9
Pitfalls and checks56.726.756.728.3

Table 5. Avg@12 accuracy (%) for Qwen3-1.7B when one problem brief section is omitted. B-OPSD uses static briefs, and B-OPSD⋆ uses refreshed briefs. The shaded row uses the full brief. Bold marks the highest score within each variant and benchmark.

Relevant principles are the most consistent contributor in this ablation. Removing them lowers accuracy on both benchmarks for both brief variants. With refreshed briefs, AIME 2024 falls from 57.5 to 56.7 and HMMT 2025 falls from 28.6 to 26.7. The complete refreshed brief also achieves the highest score on both benchmarks within its variant; removing any of its four sections lowers accuracy. These results identify relevant principles as useful teacher context and show the value of retaining the complete brief when it is refreshed from student checkpoints.

Conclusion

We introduced B-OPSD, which generates a problem brief from the question and gives it to a frozen teacher as additional context. The current student samples solution trajectories from the question alone. The teacher evaluates the same trajectories and provides token-level distributions that guide student updates. This process uses no reference solutions during training and requires no separate teacher model. Across three Qwen3 model sizes and three competition mathematics benchmarks, both B-OPSD with static briefs and B-OPSD with refreshed briefs achieved higher average accuracy than OPSD in the reported runs. Refreshed briefs achieved the highest average at every model size. These results show that problem briefs can match or exceed the performance of OPSD without verified reference solutions during training. We will further examine how problem briefs should be updated during training, including when to refresh them. We will also test B-OPSD on reasoning tasks beyond competition mathematics.

Additional Implementation Details

Our LoRA adapters cover the query, key, value, and output projections in attention and the gate, up, and down projections in the feed-forward layers. We use AdamW and bfloat16 precision. In the original OPSD study, SFT trains for 100 updates with a maximum sequence length of 16,000 tokens. GRPO trains for 500 updates and samples eight responses per problem at temperature 1.2, with a maximum length of 16,000 tokens [31].

The brief generation prompt asks for at most 700 tokens, but we set no separate hard length limit for brief decoding. We retain generated briefs without format filtering.

Generated Problem Brief Examples

We randomly sampled candidate problem briefs generated by Qwen3-1.7B and selected three for illustration. Their formatting is adapted for LaTeX.

Example 1

Problem. nn and cc are positive numbers such that increasing any positive number by n%n\% and then decreasing the result by c%c\% returns the original number. Prove that in this case

n2c2=100+n100−c.\frac{n^2}{c^2} = \frac{100+n}{100-c}.

Generated brief.

  1. Target and structure: The problem requires proving that n2c2=100+n100−c\frac{n^2}{c^2} = \frac{100+n}{100-c} given that applying n%n\% increase followed by c%c\% decrease returns the original number. The key is to derive a relationship between nn and cc from the condition that the operations are identity functions for all x>0x > 0.

  2. Relevant principles:

    • Algebraic manipulation: The condition that the operations return the original number leads to the equation (1+n100)(1−c100)=1\left(1+\frac{n}{100}\right)\left(1-\frac{c}{100}\right)=1.

    • Solving equations: Expanding and simplifying the equation to isolate nn in terms of cc (e.g., n=100c100−cn = \frac{100c}{100-c}) is critical.

    • Substitution: Substituting nn into the target expression to verify equality.

  3. Suggested approach:

    • Start with the identity condition: (1+n100)(1−c100)=1\left(1+\frac{n}{100}\right)\left(1-\frac{c}{100}\right)=1.

    • Expand and simplify to derive 100(n−c)=nc100(n-c)=nc.

    • Solve for nn in terms of cc: n=100c100−cn = \frac{100c}{100-c}

    • Substitute nn into n2c2\frac{n^2}{c^2} and 100+n100−c\frac{100+n}{100-c} to show they are equal.

  4. Pitfalls and checks:

    • Mistakes in algebra: Ensure correct expansion of (1+n100)(1−c100)\left(1+\frac{n}{100}\right)\left(1-\frac{c}{100}\right).

    • Domain constraints: Verify 100−c≠0100-c \ne0 to avoid division by zero.

    • Verification: Confirm that both sides of the equality simplify to the same expression after substitution.

The proof relies on algebraic manipulation and substitution, ensuring the derived relationship holds for all x>0x > 0.

Example 2

Problem. For what value of mm does the equation (x+4)(x+1)=m+2x(x + 4)(x + 1) = m + 2x have exactly one real solution? Express your answer as a common fraction.

Generated brief.

Problem Brief:

Target and structure: Determine the value of mm for which the equation (x+4)(x+1)=m+2x(x + 4)(x + 1) = m + 2x has exactly one real solution. The main challenge is to find the value of mm that makes the equation have a unique real root.

Relevant principles: The equation is a quadratic in xx, and the number of real solutions depends on the discriminant of the quadratic. The discriminant DD of a quadratic equation ax2+bx+c=0ax^2 + bx + c = 0 is given by D=b2−4acD = b^2 - 4ac. If D>0D > 0, there are two distinct real solutions; if D=0D = 0, there is exactly one real solution (a repeated root); and if D<0D < 0, there are no real solutions.

Suggested approach:

  • Expand and rearrange the equation to standard quadratic form.

  • Identify the coefficients aa, bb, and cc of the quadratic.

  • Compute the discriminant DD and set it equal to zero to find the value of mm that results in exactly one real solution.

  • Verify the result by substituting back into the original equation.

Pitfalls and checks:

  • Ensure the equation is correctly expanded and rearranged.

  • Confirm that the discriminant is correctly calculated.

  • Verify that the value of mm leads to a repeated root by substituting back into the original equation.

  • Check that the discriminant condition is applied correctly to the quadratic form.

Example 3

Problem. Let gg be a graph with nn vertices and mm edges that contains no 4-cycle. Show that:

m≤n4(1+4n−3).m \le\frac{n}{4}\left(1+\sqrt{4n-3}\right).

Generated brief.

1. Target and structure: The problem asks to prove that a 4-cycle-free graph with nn vertices and mm edges satisfies m≤n4(1+4n−3)m \le\frac{n}{4}\left(1+\sqrt{4n-3}\right). The key challenge is to derive this bound using extremal graph theory principles, leveraging inequalities and combinatorial counting.

2. Relevant principles:

  • Cauchy-Schwarz inequality: Relates the sum of squares of degrees to the total number of edges.

  • Counting paths of length 2: In a 4-cycle-free graph, the number of paths of length 2 between any two vertices is at most 1, leading to a bound on the sum of degrees squared.

  • Quadratic inequality derivation: Combines the above to derive the upper bound on mm.

3. Suggested approach:

  • Use the Cauchy-Schwarz inequality to relate ∑d(v)2\sum d(v)^2 to mm:

∑vd(v)2≥(2m)2n.\sum_v d(v)^2 \ge\frac{(2m)^2}{n}.
  • Count paths of length 2:

∑v(d(v)2)≤(n2).\sum_v \binom{d(v)}{2} \le\binom{n}{2}.
  • Combine these to derive the quadratic inequality:

4m2≤2mn+n2(n−1),4m^2 \le2mn+n^2(n-1),

leading to the bound m≤n4(1+4n−3)m \le\frac{n}{4}\left(1+\sqrt{4n-3}\right).

4. Pitfalls and checks:

  • Ensure the counting of paths of length 2 is valid for all 4-cycle-free graphs.

  • Verify that the derived quadratic inequality correctly bounds mm without overcounting.

  • Confirm the asymptotic behavior of the bound matches known extremal graph theory results.

This approach connects combinatorial counting with inequality techniques to derive the required bound.

References

  1. [1]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2306.13649
  2. [2]Lang Cao and Hanbing Liu. Tablemaster: A recipe to advance table understanding with language models. In International Conference on Learning Representations (ICLR). proceedings.iclr.cc, 2026.arxiv.org/abs/2501.19378
  3. [3]Jiangjie Chen, Wenxiang Chen, Jiacheng Du, et al. Seed-prover 1.5: Mastering undergraduate-level theorem proving via learning from experience. arXiv, abs/2512.17260, 2025.arxiv.org/abs/2512.17260
  4. [4]Bingquan Dai, Li Ray Luo, Qihong Tang, Jie Wang, Xinyu Lian, Hao Xu, Minghan Qin, Xudong Xu, Bo Dai, Haoqian Wang, Zhaoyang Lyu, and Jiangmiao Pang. Meshcoder: Llm-powered structured mesh code generation from point clouds. In Conference on Neural Information Processing Systems (NeurIPS), 2025.
  5. [5]Yuchen Fan, Yuzhong Hong, Qiushi Wang, Junwei Bao, Hongfei Jiang, and Yang Song. Preference-oriented supervised fine-tuning: Favoring target model over aligned large language models. In AAAI Conference on Artificial Intelligence (AAAI), pages 23859–23867. AAAI Press, 2025.arxiv.org/abs/2412.12865
  6. [6]Etash Kumar Guha et al. Openthoughts: Data recipes for reasoning models. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2506.04178
  7. [7]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2106.09685
  8. [8]Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv, abs/2603.07079, 2026.arxiv.org/abs/2603.07079
  9. [9]Minki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho, and Sung Ju Hwang. Distilling LLM agent into small models with retrieval and code tools. In Conference on Neural Information Processing Systems (NeurIPS), 2025.
  10. [10]Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Xavier Carreras, and Kevin Duh, editors, Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1317–1327. The Association for Computational Linguistics, 2016.arxiv.org/abs/1606.07947
  11. [11]Jewon Lee, Wooksu Shin, Seungmin Yang, Ki-Ung Song, Donguk Lim, Jaeyeon Kim, Tae-Ho Kim, and Bo-Kyeong Kim. ERGO: efficient high-resolution visual understanding for vision-language models. In International Conference on Learning Representations (ICLR). proceedings.iclr.cc, 2026.arxiv.org/abs/2509.21991
  12. [12]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Conference on Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2206.14858
  13. [13]Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connectionism, 2025.
  14. [14]OpenAI. GPT-4 technical report, 2023.arxiv.org/abs/2303.08774
  15. [15]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Conference on Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2203.02155
  16. [16]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, abs/1707.06347, 2017.arxiv.org/abs/1707.06347
  17. [17]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv, abs/2402.03300, 2024.arxiv.org/abs/2402.03300
  18. [18]Avi Singh et al. Beyond human data: Scaling self-training for problem-solving with language models. Trans. Mach. Learn. Res., 2024, 2024.arxiv.org/abs/2312.06585
  19. [19]Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv, abs/2209.15189, 2022.arxiv.org/abs/2209.15189
  20. [20]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Conference on Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2201.11903
  21. [21]Xueru Wen, Jie Lou, Xinyu Lu, Yuqiu Ji, Xinyan Guan, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, Debing Zhang, and Le Sun. On-policy self-alignment with fine-grained knowledge feedback for hallucination mitigation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Annual Meeting of the Association for Computational Linguistics (ACL), volume ACL 2025 of Findings of ACL, pages 5215–5231, 2025.
  22. [22]Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, JiaqI Liu, Jixuan Huang, Xun Deng, Zhihao Zhang, Honglin Guo, Zhikai Lei, Miao Zheng, Guoteng Wang, Peng Sun, Rui Zheng, Hang Yan, Yao Gui, Qi Zhang, and Xuanjing Huang. BAPO: stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping. In International Conference on Learning Representations (ICLR). proceedings.iclr.cc, 2026.arxiv.org/abs/2510.18927
  23. [23]An Yang et al. Qwen3 technical report, 2025.arxiv.org/abs/2505.09388
  24. [24]Yongqiang Yao, Jingru Tan, Kaihuan Liang, Feizhao Zhang, Jiahao Hu, Shuo Wu, Yazhe Niu, Ruihao Gong, Dahua Lin, and Ningyi Xu. Hierachical balance packing: Towards efficient supervised fine-tuning for long-context LLM. In Danielle Belgrave, Cheng Zhang, Laura N. Montoya, Hsuan-Tien Lin, Razvan Pascanu, Piotr Koniusz, Marzyeh Ghassemi, Nancy Chen, Ivan Vladimir Meza Ruiz, and Arturo Loaiza-Bonilla, editors, Conference on Neural Information Processing Systems (NeurIPS), 2025.DOI
  25. [25]Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv, abs/2602.12275, 2026.arxiv.org/abs/2602.12275
  26. [26]Qiying Yu et al. DAPO: an open-source LLM reinforcement learning system at scale. In Conference on Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2503.14476
  27. [27]Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, and Yaojie Lu. Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation. arXiv, abs/2605.18740, 2026.arxiv.org/abs/2605.18740
  28. [28]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.arxiv.org/abs/2203.14465
  29. [29]Jiazheng Zhang, Ziche Fu, Junrui Shen, Yunbin Zhao, Yunke Zhang, Zhiheng Xi, Long Ma, Chenxin An, Zhihao Zhang, Shichun Liu, Dingwei Zhu, Shihan Dou, Shaofan Liu, Han Li, Wiggin Zhou, Aiden Adams, Tao Gui, Fei Huang, Qi Zhang, and Xuanjing Huang. Entropy polarity in reinforcement fine-tuning: Direction, asymmetry, and control. arXiv, abs/2605.11775, 2026.arxiv.org/abs/2605.11775
  30. [30]Zizhuo Zhang, Qizhou Wang, Shanshan Ye, Jianing Zhu, Jiangchao Yao, Bo Han, and Masashi Sugiyama. Towards understanding valuable preference data for large language model alignment. In International Conference on Learning Representations (ICLR). proceedings.iclr.cc, 2026.arxiv.org/abs/2510.13212
  31. [31]Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models, 2026.arxiv.org/abs/2601.18734
  32. [32]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. arXiv, abs/2507.18071, 2025.arxiv.org/abs/2507.18071

Paper details

vlnc:W_pavrgkrh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents