betaFeedback

WorksComputer Science

On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

Abstract

On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.

Introduction

Post-training is a primary driver of capability gains in large language models [27], and its paradigms differ along two axes: whether training trajectories come from the model itself, and how dense the learning signal is. Supervised finetuning (SFT) imitates curated demonstrations, providing dense token-level targets on teacher-forced trajectories the model never visits at inference, which induces exposure bias [13, 5] and erodes existing capabilities [33]. Reinforcement learning with verifiable rewards (RLVR) trains on the model’s own rollouts [14, 31], avoiding this mismatch but compressing supervision into a single scalar per trajectory. On-policy distillation (OPD) combines the two: the student samples its own trajectories, and a teacher supplies dense next-token targets at the states it visits [5, 13]. On-policy self-distillation (OPSD) removes the need for a stronger teacher by constructing teacher and student from the same parameters under asymmetric conditioning [44, 32]. Across this progression, external supervision is reduced rather than removed: SFT requires reference solutions, RLVR gold answers, OPD a stronger teacher, and OPSD a privileged context, typically the ground-truth solution.

Curating this required supervision at scale is costly: process-level annotation of a single mathematics corpus required about 800,000 human step labels [25], and open instruction datasets rest on crowdsourcing efforts spanning over 13,000 volunteers [21]. This cost has motivated a shift toward synthetic supervision, in which the model generates its own instructions [36], rationales [40], or pseudo-solutions [24]. However, these methods still start from human-written problems, so human curation still decides which problems the model trains on and, in turn, what it can learn.

In language models, self-training is an established approach to self-improvement [40, 18, 34, 46]. The model solves problems, filters its solutions by agreement with a reference answer or its own majority vote, and is finetuned on those it retains. These methods, however, still train on a curated problem set, which must be collected anew for every task [34]. Yet a language model can pose problems, not only solve the ones it is given [43, 7]. Therefore, we investigate whether a language model can self-train without labels or a curated problem set, requiring only topic descriptors from which it generates its own training problems.

Image not extracted

Figure 1. On-Policy Self-Training loop. Seeded only by topic descriptors, the model generates and filters its own question-answer pairs, then trains on them via on-policy self-distillation from a privileged EMA teacher, iterating for data-free self-improvement.

In this work, we propose On-Policy Self-Training (OPST), an approach to self-train a language model with dense supervision derived from self-generated privileged information (Figure 1). We remove both ground-truth answers and the curated problem set: conditioned only on a coarse topic descriptor, the model synthesizes both a query and the privileged information (PI) used to train on it. Since the generated question-answer pairs are no longer human-curated, the quality of this data can be noisy. Hence, we filter the generated data along two axes: Pseudo-verification ensures that a problem is well-formed, requiring the model to reproduce its answer in a fresh context and under a paraphrase. Further, we apply a consensus gate to ensure that the problem is informative, discarding problems whose answer the student already consistently reproduces across on-policy rollouts. Because the gate is evaluated against the current student, the difficulty of the retained curriculum implicitly evolves as the student model improves. On the quality-filtered samples, we train the model with on-policy self-distillation (Figure 2), where the teacher provides high-quality token-level supervision to the student, limiting catastrophic forgetting of existing capabilities acquired from supervised data [33 a]. These effects come from only the teacher being conditioned on the privileged information, while the student receives the question alone.

This asymmetry is crucial in a purely synthetic setting. SFT on the generated answer, or RL rewarding agreement with it, treats the generated answer as the optimization target, which bounds the supervision by the quality of the generation. We show that providing the synthetic answer as context to the teacher instead enables it to produce a reasoning rollout from its own parametric knowledge that serves as an enriched learning signal to the student. The student’s supervision is therefore extended by what the teacher can infer from the privileged, generated context, exceeding what the generated answer alone provides. Iterating this loop improves the model over successive rounds, while an exponential moving average (EMA) teacher stabilizes against self-induced distribution drift.

In our experiments, we show that OPST improves language and multimodal reasoning without human-curated questions or answers, outperforming alternative training strategies. On Gemma 2 E2B [11], self-training increases MMLU [15] accuracy by +10.2 points, with consistent gains on GSM8K [9], GPQA [29], and HellaSwag [41]. We also find that self-training in-domain transfers to out-of-distribution domains, with performance either remaining stable or also improving. Applied to vision-language models with unlabeled images alone, Visual OPST improves VQAv2 accuracy by +6.9 points.

Related Work

On-policy self-distillation. OPSD [44] constructs both policies from a single model, conditioning the teacher on a verified reference solution. Since on-policy methods train on the model’s own samples, they forget less than SFT [33], and self-distillation has been shown to enable continual learning across multiple domains [32]. Subsequent work varies the privileged context between expert demonstrations [32], privileged observations [28], and environmental feedback [19]. Others address information leakage and instability [39] or extend the setting to agentic tasks [26]. Closest to our work, Li et al. [24] replace the gold solution with a majority-vote consensus over the model’s own rollouts, eliminating answer supervision. All of these, however, operate on a given distribution of problems. OPST removes that assumption: the model synthesizes its own queries from a coarse topic descriptor, so neither the problems nor their solutions are externally supplied.

Image not extracted

Figure 2. Model training. As in OPSD [44], the student sees only the question and matches the next-token distribution of a teacher that also sees privileged information. In OPST, this information is self-generated, and the teacher is an EMA of the student.

LLM Self-Training. Early self-training methods filter solutions by agreement with a reference answer [40, 34] or by self-consistency across samples [18], and label-free reinforcement learning has since replaced the reference answer with majority votes [46] or the model’s own confidence [45]. These methods remove answers but not problems: each still trains on a curated problem set, which must be collected anew for every task [34]. Self-play methods let a proposer generate tasks, seeded by as little as a topic description and checked by a code executor or by majority voting over the solver’s answers [43, 17, 7]. Self-generated problems, however, carry information that self-play reduces to a scalar reward: the answer and rationale the model produces while posing them. Used as privileged context for a self-distillation teacher, it instead yields dense, token-level supervision. On curated problems, Li et al. [24] show that self-generated answers are considerably more effective as teacher context than as a reward, and Zhang et al. [42] combine self-play with self-distillation for search agents. While their method requires no external training data, question construction is grounded in documents retrieved from an external corpus, whereas OPST generates problems from the model’s parametric knowledge alone. Whether a model can improve from problems it poses itself, with dense token-level supervision and without retrieval or other external grounding, has to our knowledge not yet been studied for general reasoning.

Method

In this section, we present On-Policy Self-Training. We first define the data-free post-training setting (§ 3.1) and review on-policy self-distillation (§ 3.2). Following, we describe how the model generates problems and privileged information from topic descriptors (§ 3.3), how it filters them intrinsically (§ 3.4), and how it trains on the result (§ 3.5). Repeating these steps yields multiple rounds of self-training (§ 3.6). Finally, we extend OPST to vision-language models (§ 3.7).

Problem Formulation: Data-Free Post-Training

Post-training typically relies on a curated dataset of human-written queries and ground-truth answers D={(xi,yi∗)}D = \{(x_i, y_i^*)\}, or at least on a curated prompt distribution Dprompts={xi}D_{\mathrm{prompts}} = \{x_i\}. In data-free post-training, neither is available. Training is seeded only by a set of topic descriptors T={τ1,…,τK}\mathcal{T} = \{\tau_1,\ldots,\tau_K\}, where each τk\tau_k names a broad subject area, such as “Abstract Algebra” or “Commonsense Causal Reasoning”. From T\mathcal{T}, the model generates a synthetic training set Dsyn={(xi,zi)}i=1ND_{\mathrm{syn}} = \{(x_i, z_i)\}_{i=1}^{N}, where xix_i is a generated query and ziz_i its privileged information, consisting of a rationale and a candidate answer (§ 3.3).

The setting has two objectives. First, the model must improve its reasoning in the seeded domains using only self-generated problems and privileged information. Second, because the training signal comes from its own generations without human oversight, the model must retain its general reasoning and previously acquired parametric knowledge on domains it is not self-trained on, a prerequisite for repeating the process over multiple rounds.

Preliminaries: On-Policy Self-Distillation

On-policy self-distillation (Zhao et al., [44]) derives the student πθ\pi_\theta and the teacher πϕ\pi_\phi from the same model and replaces the capability gap of standard distillation (Kim & Rush, [20]) with an information gap. Given a problem xx, the student samples a response y=(y1,…,yT)∼πθ(⋅∣x)y = (y_1, \ldots, y_T) \sim\pi_\theta(\cdot\mid x), with next-token distribution

pt=πθ(⋅∣x,y<t)(1)p_t = \pi_\theta(\cdot\mid x, y_{<t}) \tag*{(1)}

at step tt. The teacher evaluates the same prefix y<ty_{<t}, but is additionally conditioned on privileged information zz that the student does not see:

qt=πϕ(⋅∣x,z,y<t).(2)q_t = \pi_\phi(\cdot\mid x, z, y_{<t}). \tag*{(2)}

The student is trained to match the teacher along its own trajectory,

LOPSD(θ)=E(x,z)∼D, y∼πθ(⋅∣x)[1∑tMt∑t=1TMt D(pt∥qt)],(3)\mathcal{L}_{\mathrm{OPSD}}(\theta) = \mathbb{E}_{(x,z)\sim\mathcal{D},\ y\sim\pi_\theta(\cdot\mid x)} \left[\frac{1}{\sum_t M_t}\sum_{t=1}^{T} M_t\,D(p_t \Vert q_t)\right], \tag*{(3)}

where DD is a divergence between distributions, Mt∈{0,1}M_t \in\{0,1\} masks padding tokens, and no gradient flows through the teacher. In standard OPSD, zz is a reference solution (Zhao et al., [44]).

Existing OPSD methods, however, assume a curated problem set and typically use a reference solution as zz. The following sections describe how OPST generates both with the model itself.

Topic-Seeded Curriculum Generation

Instead of drawing problems from a curated distribution, OPST generates them from the topic descriptors T\mathcal{T}. Given a topic τ∈T\tau\in\mathcal{T}, a generator πgen\pi_{\mathrm{gen}} produces a query xx together with privileged information zz,

(x,z)∼πgen(⋅∣τ),z=(r,a),(4)(x,z) \sim\pi_{\mathrm{gen}}(\cdot\mid\tau), \qquad z = (r,a), \tag*{(4)}

where rr is a rationale and aa a candidate answer. Queries can be multiple-choice or open-ended, and we sample at temperature TgenT_{\mathrm{gen}} to obtain diverse problems. While the student receives the query xx alone, the teacher also receives zz as additional privileged information. We embed the topic name into a targeted data generation prompt, asking the model to pose problems on the given topic along with generated privileged information. In practice, the model generates partial or full answers to the posed problem. Where available, topics are data-domain category labels. The applied prompt templates are provided in Appendix § 8.7.

We instantiate πgen\pi_{\mathrm{gen}} either as the EMA teacher πϕ\pi_\phi (§ 3.5), so that data quality can improve as training progresses, or as a frozen copy of the initial model πθ0\pi_{\theta_0}. The student does not generate data, but its rollouts determine which samples are retained (§ 3.4).

Pseudo-Verification & Consensus Gate

Removing human supervision removes not only human data curation but also quality assurance. Generated samples can contain ill-posed questions or incorrect candidate answers, and training on them propagates these errors. At the same time, not every well-posed sample is worth training on: if the student already solves a problem reliably, the teacher conditioned on zz produces nearly the same next-token distribution as the student, and the sample contributes little beyond its generation and rollout cost.

We therefore filter any generated sample for two criteria: Pseudo-verification first estimates whether a sample is well-formed, and the consensus gate then approximates whether it is informative for the current student.

Pseudo-Verification. We evaluate if a given generated sample possesses two properties of a well-posed problem: its answer should be reproducible without the generator’s own intent as context, and it should not depend on the particular wording of the question.

To pseudo-verify the generated answer, we require the generator to recover it when solving the question in a fresh context, without access to zz and at a low temperature, which yields a^(x)=a\hat{a}(x)=a. This separates authoring from solving: when query and answer are generated in one pass, a flawed or ambiguous question can still receive a confident answer. To pseudo-verify the generated problem statement, we prompt the generator to paraphrase xx into x′∼πgen(⋅∣x)x^\prime\sim\pi_{\mathrm{gen}}(\cdot\mid x) with an instruction to preserve its meaning, solve x′x^\prime under the same conditions, and require the answer to be unchanged, a^(x′)=a\hat{a}(x^\prime)=a. A well-posed question has the same answer regardless of its wording, so this rejects questions whose answer depends on incidental phrasing, which typically indicates that the question admits more than one reading. A sample is retained only if: v(x,z)=I[a^(x)=a]⋅I[a^(x′)=a]v(x,z)=\mathbb{I}[\hat{a}(x)=a]\cdot\mathbb{I}[\hat{a}(x^\prime)=a]

Consensus Gate. We assume that the samples that carry the most signal are those the student cannot yet answer reliably: a question that is trivial for the student yields the same answer on every rollout, and the privileged teacher has little to add. We therefore measure answer agreement across sampled reasoning paths as in self-consistency [35]. For each verified sample, the student samples GG rollouts {yg}g=1G∼πθ(⋅∣x)\{y_g\}_{g=1}^{G}\sim\pi_\theta(\cdot\mid x), from which we extract answers {a^g}g=1G\{\hat{a}_g\}_{g=1}^{G}. Samples with full agreement with the candidate answer, c(x,z)=1G∑g=1GI[a^g=a]=1c(x,z)=\frac{1}{G}\sum_{g=1}^{G}\mathbb{I}[\hat{a}_g=a]=1, provide little new signal for the model and are discarded.

The two filters act in opposite directions by design. Pseudo-verification keeps samples that are stable enough for the generated answer to serve as privileged information, while the consensus gate discards samples the student already solves. The retained samples therefore lie at the student’s capability frontier, and since the gate depends on the current student, the difficulty of the curriculum evolves with the model.

Model Training

Given a pseudo-verified batch B={(xi,zi)}i=1B\mathcal{B}=\{(x_i,z_i)\}_{i=1}^{B}, we update the student with on-policy self-distillation (§ 3.2), where pi,tp_{i,t} and qi,tq_{i,t} denote the student and teacher distributions (Eqs. (1) and (2)) at step tt of the student’s rollout on sample ii.

Generalized Divergence. Following the generalized formulation of prior work [5, 44], we instantiate the divergence DD in Eq. (3) as a skewed Jensen–Shannon divergence. With the interpolated mixture distribution qˉi,t=(1−α)qi,t+αpi,t\bar{q}_{i,t}=(1-\alpha)q_{i,t}+\alpha p_{i,t} for α∈(0,1)\alpha\in(0,1), the per-token divergence is defined as

Dα(pi,t∥qi,t)=αDKL(qi,t∥qˉi,t)+(1−α)DKL(pi,t∥qˉi,t).(5)D_{\alpha}(p_{i,t}\Vert q_{i,t})=\alpha D_{\mathrm{KL}}(q_{i,t}\Vert\bar{q}_{i,t})+(1-\alpha)D_{\mathrm{KL}}(p_{i,t}\Vert\bar{q}_{i,t}). \tag*{(5)}

In the limits, Dα/α→DKL(qi,t∥pi,t)D_{\alpha}/\alpha\to D_{\mathrm{KL}}(q_{i,t}\Vert p_{i,t}) as α→0\alpha\to0, recovering the mode-covering forward KL divergence, and Dα/(1−α)→DKL(pi,t∥qi,t)D_{\alpha}/(1-\alpha)\to D_{\mathrm{KL}}(p_{i,t}\Vert q_{i,t}) as α→1\alpha\to1, recovering the mode-seeking reverse KL divergence. Unlike unbounded KL objectives, DαD_{\alpha} is upper-bounded by the binary entropy H(α)H(\alpha), ensuring that tokens to which one distribution assigns near-zero probability, such as localized artifacts introduced by an imperfect synthetic hint, are prevented from dominating the gradient update. We set α=0.8\alpha=0.8, which favors the mode-seeking regime. The final objective is

LOPST(θ;B)=1B∑i=1B1∑tMi,t∑t=1TMi,tDα(pi,t∥qi,t).(6)\mathcal{L}_{\mathrm{OPST}}(\theta;\mathcal{B})=\frac{1}{B}\sum_{i=1}^{B}\frac{1}{\sum_t M_{i,t}}\sum_{t=1}^{T}M_{i,t}D_{\alpha}(p_{i,t}\Vert q_{i,t}). \tag*{(6)}

EMA Teacher. The teacher is initialized from the student, ϕ0=θ0\phi_0=\theta_0, and updated after every gradient step as ϕk=(1−β)ϕk−1+βθk\phi_k=(1-\beta)\phi_{k-1}+\beta\theta_k. Since the teacher produces the dense supervision, it must remain stable. With β=1\beta=1, a single noisy batch immediately changes the targets for subsequent batches, and errors can reinforce themselves. The EMA instead averages out such fluctuations over roughly 1/β1/\beta steps. Averaging the student’s weights also yields a more accurate model than the student itself (Table 6d), consistent with momentum teachers in vision [6]. For this reason, the teacher is the model we evaluate.

rate β\betaMMLUsamples / roundMMLU
1.00 (no EMA)38.15743.8
0.2044.217147.2
0.0547.728547.7
0.0145.257046.9
rounds NNMMLUEval checkpointMMLU
0 (base IT)37.5Student (πθ\pi_\theta)45.9
343.1EMA teacher (πϕ\pi_\phi)47.7
747.7
1246.2
2046.2

Table 6. OPST component ablations on Gemma 4 E2B evaluated on MMLU. Default settings are highlighted in gray. (a) EMA update rate. (b) Dataset size per round. (c) Self-training rounds. (d) Evaluation target.

Iterative Self-Training

A single round of OPST is limited by the quality of the data and supervision that the initial model provides. We therefore repeat generation, verification, and training over NN rounds. The student and teacher weights carry over from one round to the next. For each additional round, πgen\pi_{\mathrm{gen}} generates a new training set, from which we remove problems that duplicate those of earlier rounds.

The iterative nature of our loop lets student and teacher co-evolve. Training on new data further improves the student, and through the EMA update, also the teacher. A stronger teacher, in turn, benefits the next round in three ways. It provides more accurate token-level targets on the student’s rollouts. As the generator, it produces better problems and privileged information. And as the verifier (§ 3.4), it judges its own data more reliably, so fewer incorrect samples reach training. Because the teacher is an average over many training steps, only improvements that persist across batches enter this cycle.

Visual OPST

OPST is modality-agnostic and can be extended to the multimodal setting by adding an unlabeled modality. We therefore explore this for vision-language tasks by adding images to the process. Visual OPST uses a collection of unlabeled images I\mathcal{I} and visual topic descriptors Tvis\mathcal{T}_{\mathrm{vis}}, such as “Spatial Relations”, “Object Counting”, or “Attribute Recognition”. Unlike the text setting, it requires unlabeled images as input, from which we generate the same data as detailed in § 3.3.

Visual Curriculum Generation. Given an image I∈II \in\mathcal{I} and a topic τ∈Tvis\tau\in\mathcal{T}_{\mathrm{vis}}, the generator produces a question about the image together with privileged information, (x,z)∼πgen(⋅∣I,τ)(x,z) \sim\pi_{\mathrm{gen}}(\cdot\mid I,\tau). We consider two forms of privileged information. In the first, zz is a detailed scene caption that the generator writes for II, describing its objects, attributes, and spatial arrangement. In the second, we prompt the model to detect the objects in the scene, from which we bootstrap bounding-box grounded spatial relations. The privileged caption informs the teacher rollout with explicit visual details that the student must extract from the image itself, while the detected spatial relations help ground the teacher with actual object positions. Both forms can be combined by conditioning the teacher on the caption and the spatial relations.

Visual Verification. Visual verification tests whether the candidate answer is supported by the image rather than by a hallucinated premise. The generator solves xx on KK augmented views Ak(I)\mathcal{A}_k(I), obtained with color jitter, rescaling, and random cropping, and we measure agreement with the candidate answer:

vaug(I,x,z)=1K∑k=1KI[a^(Ak(I),x)=a].(7)v_{\mathrm{aug}}(I,x,z)=\frac{1}{K}\sum_{k=1}^{K}\mathbb{I}\left[\hat{a}(\mathcal{A}_k(I),x)=a\right]. \tag*{(7)}

Horizontal flips are excluded because they change the answer to questions about left and right.

Model Training. Training follows Eq. (6) with the image added to both contexts: the student samples yi∼πθ(⋅∣Ii,xi)y_i \sim\pi_{\theta}(\cdot\mid I_i,x_i), and the teacher provides qi,t=πϕ(⋅∣Ii,xi,zi,yi,<t)q_{i,t}=\pi_{\phi}(\cdot\mid I_i,x_i,z_i,y_{i,<t}).

Experiments

Our experimental evaluation focuses on three aspects: in-domain self-improvement without human-curated data, knowledge retention on out-of-domain (OOD) benchmarks when mixing data generation domains, and transfer to multimodal inputs with Visual OPST.

Experimental Setup

Models and Datasets. As base models, we utilize the Gemma 4 E2B and E4B models [11]. Data Generation is seeded with topic descriptors covering the domains of four in-domain benchmarks: MMLU [15], GPQA [29], GSM8K [9], and HellaSwag [41]. To measure transfer and retention, we additionally evaluate on three OOD benchmarks that no topic descriptor targets: ARC-Challenge (Clark et al., 2018), MATH-500 [25, 16], and Winogrande [30]. For the multi-domain experiments (Table 3), GPQA is treated as out-of-domain.

Table 3: Multi-domain results. Performance comparison on in-domain and out-of-domain benchmarks for Gemma 4 E2B self-trained on combinations of topic descriptors. Single domain reports separate models, each self-trained only on the topics of the respective benchmark.

Training TopicsMMLUHellaSwagGSM8K*Avg.GPQAARC-CMATH-500*WinograndeAvg.
Gemma 4 E2B
Reference Base37.548.772.753.028.839.156.550.043.6
+ Single domain47.7+10.256.2+7.580.9+8.261.6+8.6
Mixtures + MMLU + HellaSwag46.9+9.455.6+6.974.3+1.658.9+5.930.3+1.544.9+5.848.0−8.551.8+1.843.8+0.2
+ HellaSwag + GSM8K42.6+5.154.6+5.977.0+4.358.1+5.131.3+2.543.0+3.955.8−0.751.8+1.845.5+1.9
+ MMLU + GSM8K48.8+11.350.8+2.178.5+5.859.4+6.428.8−0.044.5+5.464.1+7.652.7+2.747.5+3.9
+ All three48.4+10.955.4+6.777.9+5.260.6+7.629.3+0.545.3+6.251.6−4.951.2+1.244.4+0.8

Table 3. Multi-domain results. Performance comparison on in-domain and out-of-domain benchmarks for Gemma 4 E2B self-trained on combinations of topic descriptors. *Evaluated with thinking. All other benchmarks are evaluated without thinking.

Evaluation and Baselines. Table 1 compares training algorithms on identical self-generated data. SFT imitates the generated answer, Shao et al. [31] rewards agreement with it, and OPST supplies it as teacher context. Table 2 varies the level of supervision. Supervised SFT and Supervised OPSD train on curated questions with reference solutions, u-OPSD (Li et al. [24]) on curated questions with self-generated answers, and R-Zero (Huang et al. [17]) on self-generated problems with reinforcement learning. Base denotes the instruction-tuned model. In Table 2, methods above the light rule train on human-curated reference solutions, and bold marks the best result among the remaining methods. Elsewhere, bold and underline mark the best and second-best result. For OPST, we evaluate the EMA teacher (§ 3.5).

MethodSignalGen. AnswerMMLUGPQAGSM8K*HellaSwagAverage
Gemma 4 E2B
w/o Think. Base37.528.8–48.738.3
+ SFTDenseTarget38.3+0.838.3_{+0.8}30.8+2.030.8_{+2.0}–50.8+2.150.8_{+2.1}40.0+1.740.0_{+1.7}
+ GRPOSparseReward37.3−0.237.3_{-0.2}30.3+1.530.3_{+1.5}–50.3+1.650.3_{+1.6}39.3+1.039.3_{+1.0}
+ OPST (Ours)DenseContext47.7+10.247.7_{+10.2}32.8+4.032.8_{+4.0}–56.2+7.556.2_{+7.5}45.6+7.345.6_{+7.3}
w/ Think. Base69.927.872.763.158.4
+ SFTDenseTarget71.0+1.171.0_{+1.1}30.3+2.530.3_{+2.5}75.4+2.775.4_{+2.7}64.5+1.464.5_{+1.4}60.3+1.960.3_{+1.9}
+ GRPOSparseReward70.7+0.870.7_{+0.8}32.3+4.532.3_{+4.5}75.0+2.375.0_{+2.3}62.5−0.662.5_{-0.6}60.1+1.760.1_{+1.7}
+ OPST (Ours)DenseContext71.1+1.271.1_{+1.2}34.3+6.534.3_{+6.5}80.9+8.280.9_{+8.2}65.6+2.565.6_{+2.5}63.0+4.663.0_{+4.6}
Gemma 4 E4B
w/o Think. Base54.330.3–53.746.1
+ SFTDenseTarget58.1+3.858.1_{+3.8}35.9+5.635.9_{+5.6}–58.6+4.958.6_{+4.9}50.9+4.850.9_{+4.8}
+ GRPOSparseReward55.8+1.555.8_{+1.5}31.8+1.531.8_{+1.5}–54.3+0.654.3_{+0.6}47.3+1.247.3_{+1.2}
+ OPST (Ours)DenseContext61.9+7.661.9_{+7.6}36.4+6.136.4_{+6.1}–58.2+4.558.2_{+4.5}52.2+6.152.2_{+6.1}
w/ Think. Base76.043.984.480.971.3
+ SFTDenseTarget76.2+0.276.2_{+0.2}44.4+0.544.4_{+0.5}87.1+2.787.1_{+2.7}81.1+0.281.1_{+0.2}72.2+0.972.2_{+0.9}
+ GRPOSparseReward75.4−0.675.4_{-0.6}45.0+1.145.0_{+1.1}83.2−1.283.2_{-1.2}81.5+0.681.5_{+0.6}71.3±0.0071.3_{\pm0.00}
+ OPST (Ours)DenseContext80.1+4.180.1_{+4.1}48.0+4.148.0_{+4.1}93.0+8.693.0_{+8.6}83.0+2.183.0_{+2.1}76.0+4.776.0_{+4.7}

Table 1. Synthetic-data results. Performance comparison on language reasoning benchmarks for Gemma 4 models trained on self-generated data only. *Evaluated with thinking only.

Table 2: Results across various levels of supervision. Performance comparison on language reasoning benchmarks for Gemma 4 models across supervision levels. Methods above the light rule train on human-curated reference solutions. Bold marks the best result among the remaining methods.

MethodQuestionAnswerMMLUGPQAGSM8K*HellaSwagAverage
Gemma 4 E2B
Base37.528.8–48.738.3
+ Supervised SFT✓✓47.3+9.831.3+2.5–52.8+4.143.8+5.5
+ Supervised OPSD✓✓49.6+12.133.3+4.5–56.8+8.146.6+8.3
w/o Think. + u-OPSD‡✓×45.9+8.433.3+4.5–54.7+6.044.6+6.3
+ R-Zero‡××43.5+6.030.8+2.0–51.2+2.541.8+3.5
+ OPST (Ours)××47.7+10.232.8+4.0–56.2+7.545.6+7.3
Base69.927.872.763.158.4
+ Supervised SFT✓✓70.5+0.630.8+3.083.6+10.963.7+0.662.2+3.8
+ Supervised OPSD✓✓70.1+0.231.8+4.085.3+12.663.9+0.862.8+4.4
w/ Think. + u-OPSD‡✓×70.8+0.933.3+5.576.6+3.965.0+1.961.4+3.0
+ R-Zero‡××70.3+0.431.8+4.080.3+7.664.1+1.061.6+3.2
+ OPST (Ours)××71.1+1.234.3+6.580.9+8.265.6+2.563.0+4.6
Gemma 4 E4B
Base54.330.3–53.746.1
+ Supervised SFT✓✓62.4+8.134.3+4.0–60.3+6.652.3+6.2
+ Supervised OPSD✓✓63.4+9.136.9+6.6–62.6+8.954.3+8.2
w/o Think. + u-OPSD‡✓×60.0+5.732.8+2.5–56.5+2.849.8+3.7
+ R-Zero‡××54.7+0.431.3+1.0–54.9+1.247.0+0.9
+ OPST (Ours)××61.9+7.636.4+6.1–58.2+4.552.2+6.1
Base76.043.984.480.971.3
+ Supervised SFT✓✓78.5+2.543.5−0.493.2+8.881.5+0.674.2+2.9
+ Supervised OPSD✓✓77.7+1.747.5+3.693.4+9.082.4+1.575.3+4.0
w/ Think. + u-OPSD‡✓×76.9+0.944.3+0.488.9+4.582.8+1.973.2+1.9
+ R-Zero‡××75.4−0.644.0+0.190.2+5.882.1+1.272.9+1.6
+ OPST (Ours)××80.1+4.148.0+4.193.0+8.683.0+2.176.0+4.7

Table 2. Results across various levels of supervision. Performance comparison on language reasoning benchmarks for Gemma 4 models across supervision levels. Methods above the light rule train on human-curated reference solutions. Bold marks the best result among the remaining methods. *Evaluated with thinking only. ‡Our re-implementation.

Results

Learning from Self-Generated Data. On Gemma 4 E2B without thinking, OPST improves MMLU by +10.2 points, GPQA by +4.0, and HellaSwag by +7.5 (Table 1). On average, OPST exceeds SFT on the same data by 5.6/1.3 points (2B/4B) without thinking and by 2.7/3.8 points with thinking. The two methods differ only in whether the generated answer is the optimization target or the teacher’s context, so this gap isolates the effect of that choice. Only on 4B HellaSwag without thinking is SFT ahead (0.4 points). With thinking, the stronger base models leave less headroom on MMLU (+1.2 at 2B), whereas the gains on GPQA (+6.5) and GSM8K (+8.2) persist.

Comparison Across Supervision Levels. Table 2 compares OPST against methods that use human-curated data. Among methods without reference solutions, OPST attains the best average in all four settings, exceeding u-OPSD by 1.0 to 2.8 points and R-Zero by 1.4 to 5.2 points. As u-OPSD trains on curated questions and R-Zero operates at the same level of supervision as OPST, this suggests that a curated problem set is not necessary and that the improvements stem from our training approach rather than additional data. OPST also closes most of the gap to full supervision: at 2B without thinking, it recovers 88% of the supervised OPSD gain (+7.3 of +8.3), and with thinking, it matches supervised OPSD at both model sizes (+0.2 and +0.7 points).

Multi-Domain Curriculum & Domain Generalization. Table 3 evaluates whether OPST can self-train on several domains at once by combining the topic descriptors of two or all three in-domain benchmarks. Training on all three domains improves the in-domain average by 7.6 points, the largest gain among the mixtures, and comes within 1.0 point of single-domain training, indicating little interference between domains. Out-of-domain retention is mixed: the average improves by 0.8 points, but MATH-500 regresses by 4.9 points. Combining MMLU with GSM8K topics improves MMLU by 11.3 points, more than MMLU topics alone (+10.2), and yields the best out-of-domain average (+3.9), suggesting positive transfer from mathematical reasoning. Even without any MMLU topics, MMLU improves by 5.1 points, so part of the gain is not specific to the seeded domain.

Multimodal Self-Training: Visual OPST

To evaluate whether our self-training framework extends beyond text-only reasoning, we apply Visual OPST to Gemma 4 E2B and E4B vision-language models [11] using unlabeled images sampled from OpenImages V4 [22]. Table 4 reports performance on visual question answering (VQAv2 [12]) and object hallucination detection (POPE [23])

MethodCaptionSpatialAcc. (↑\uparrow)F1 (↑\uparrow)Acc. (↑\uparrow)Rec. (↑\uparrow)
Gemma 4 E2B
Base49.182.283.272.9
+ Visual OPST×\times✓\checkmark55.7 +6.684.7 +2.585.1 +1.977.4 +4.5
Ours + Visual OPST✓\checkmark×\times54.0 +4.983.0 +0.883.9 +0.774.0 +1.1
+ Visual OPST✓\checkmark✓\checkmark56.0 +6.984.6 +2.485.0 +1.877.4 +4.5
Gemma 4 E4B
Base55.984.885.278.0
+ Visual OPST×\times✓\checkmark59.6 +3.787.9 +3.187.4 +2.286.4 +8.5
Ours + Visual OPST✓\checkmark×\times58.0 +2.287.8 +2.987.2 +2.086.4 +8.5
+ Visual OPST✓\checkmark✓\checkmark59.2 +3.387.0 +2.186.6 +1.484.2 +6.2

Table 4. Multimodal results. Performance comparison on vision-language benchmarks for Gemma 4 models self-trained on unlabeled images only.

As shown in Table 4, Visual OPST transfers to vision-language models using unlabeled images alone. Both forms of privileged information improve VQAv2 accuracy at 2B, by 6.6 (spatial QA) and 4.9 (scene caption) points. At 2B, combining them yields the largest VQAv2 gain (+6.9), while its POPE F1 gain (+2.4) matches spatial QA alone (+2.5). At 4B, spatial QA alone performs best on VQAv2 and POPE, so the two forms of privileged information are not consistently complementary. On POPE, the improvement comes mainly from recall (+4.5 points at 2B and +6.2 at 4B).

Ablations

Quality Filtering. We ablate our quality filters by adding them incrementally, starting from OPST on all generated samples. Without filtering, OPST improves over the base model by +5.2 points (Table 5a). Pseudo-verification adds +2.7 points and the consensus gate a further +2.3.

Sample selectionMMLU
Base37.5
+ OPST w/o verification42.7
+ Verification45.4
+ Consensus gate47.7
Privileged infoMMLU
None40.6
Placebo40.2
Answer only44.6
Rationale only45.8
Full47.7
ppSFTOPST
0%38.842.7
25%38.142.1
50%35.441.4
75%33.140.8
Drop−5.7−1.9

Table 5. Ablations on Gemma 4 E2B on MMLU. Default settings are highlighted in gray. (a) Sample selection. (b) Privileged information. (c) Wrong candidate answers.

Privileged Information. In Table 5b, removing privileged information shrinks the gain over the base model from 10.2 to 3.1 points, and a placebo from a different problem on the same topic yields only 2.7. The rationale alone recovers 8.3 points and the answer alone 7.1. The gain therefore stems from sample-specific privileged information, with the rationale carrying more signal than the answer; the residual gain without it likely reflects self-distillation towards the EMA teacher.

Robustness to Wrong Answers. In Table 5c, we disable both filters and replace a fraction pp of the candidate answers with wrong ones. At p=75%p = 75\%, OPST loses only 1.9 points and remains 3.3 points above the base model, whereas SFT loses 5.7 points and falls 4.4 points below it. Conditioning the teacher on the generated answer, rather than imitating it, therefore limits the influence of incorrect generations. Further ablations are provided in Appendix A.2.

Conclusion

We introduced On-Policy Self-Training (OPST), which post-trains a model on self-generated problems and privileged information, seeded only by coarse topic descriptors. We showed that conditioning the teacher on the generated answer, rather than training on it, lets the teacher fill in omitted reasoning steps and limits the influence of potentially noisy data. We found that OPST outperforms SFT and GRPO on the same generated data and methods that rely on curated questions, and extends to multiple domains and vision-language models, demonstrating that models can self-improve from self-generated data alone.

Limitations. OPST can only amplify existing knowledge, and since pseudo-verification measures self-consistency rather than correctness, consistent errors still enter training. Our topic descriptors are hand-chosen, and our experiments are limited to small models. Promising directions include label-free stopping criteria and tool- or retrieval-grounded verification.

AI USE STATEMENT

In this work, we used generative AI tools for implementing training infrastructure and experiment scripts and for feedback on the experimental design. We have not used generative AI tools for generating core research ideas, formulating hypotheses, or modifying evaluation benchmark data, and tasks involving human-subject data are not applicable to this work. Additionally, we used generative AI tools for drafting and editing parts of the manuscript, improving readability, checking and formatting tables, plotting utilities, and debugging LaTeX compilation. Open-weight generative models (Gemma 4) serve as the primary object of study and generate the synthetic training data within our proposed self-training pipeline. We have reviewed all AI-assisted work: all generated code was manually audited and tested for correctness by the authors, all mathematical formulations and experimental numbers were verified against primary training logs, all AI-suggested interpretations were checked against the results, and all references were checked against original publications. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI.

Appendix

Self-Distillation Objectives on Generated Data.

Unlike a ground-truth reference solution, generated privileged information z=(r,a)z = (r, a) can be ambiguous: the rationale rr may miss important aspects, and the candidate answer aa may be partially or fully incorrect. Therefore, training a model on zz directly inherits these defects. SFT on the generated pairs maximizes

log⁡πθ(z∣x)=∑j=1∣z∣log⁡πθ(zj∣x,z<j),(8)\log\pi_\theta(z \mid x) = \sum_{j=1}^{|z|} \log\pi_\theta(z_j \mid x, z_{<j}), \tag*{(8)}

which supervises the model only on the tokens of zz. Critically, it treats every token as correct, including any potential omissions and errors. OPSD instead uses zz as context: at every state y<ty_{<t} the student visits, the teacher supplies a full next-token distribution, so supervision covers the entire student rollout rather than the ∣z∣|z| tokens of the generation. Crucially, the teacher produces this rollout from its own knowledge under the exposure to the generated privileged information. This leads to a more nuanced, higher-quality supervision, since the teacher reweights the model’s own next-token distribution by how consistent each continuation is with the privileged information. This has three consequences for generated data. First, zz need not spell out a step for the teacher to supervise it: tokens that move the rollout toward aa are upweighted, so the teacher completes reasoning that rr omits. Second, the target is expressed in the model’s own distribution rather than in the surface form of zz, so the student does not imitate the phrasing or formatting of the generation. Third, the prior limits the influence of a faulty zz: an error shifts the target only to the extent that the model finds it plausible, and where the prior favors a different step, the teacher retains that step instead of copying the error. The student therefore learns from what the model can infer given zz as a hint, not from zz itself. Consequently, conditioning the teacher on zz yields dense, prior-grounded token-level supervision that filters generation noise, making on-policy self-distillation well suited for learning from purely synthetic data.

Further Ablations

To isolate the contribution of each design choice in OPST, we conduct systematic component ablations on Gemma 4 E2B across four core axes: the EMA update rate β\beta, the synthetic dataset size generated per round, the number of outer self-training iterations NN, and the evaluation target (πϕ\pi_\phi vs. πθ\pi_\theta).

EMA Update. The EMA teacher is meant to provide stable targets despite noisy updates on generated data (§ 3.5). We vary its rate β\beta (Table 6a). Without an EMA (β=1\beta= 1), the teacher follows the student’s current weights, and OPST barely improves over the base model (38.1 vs. 37.5). Too much smoothing also slows progress: with β=0.01\beta= 0.01, the teacher incorporates the student’s improvements too slowly and reaches 45.2, whereas β=0.05\beta= 0.05 performs best (47.7).

Dataset Size per Round. The number of samples per round trades off the amount of data against how often the improving teacher regenerates it. We vary it from 57 to 570 samples (Table 6b). Accuracy increases up to 285 samples and then saturates, decreasing slightly to 46.9 at 570 samples. Larger rounds therefore add generation cost without improving results.

Number of Rounds. Iterating lets student and teacher improve each other, but errors in the generated data can also accumulate over rounds (§ 3.6). We evaluate the model after different numbers of rounds (Table 6c). Significant improvements emerge after the first three rounds (+5.6), and accuracy is highest after seven rounds (47.7). Further rounds bring no additional improvement, but performance remains stable at around 46, well above the base model.

Evaluation Target. Because the EMA teacher averages over the student’s recent updates, it may also be the better model to deploy. We compare the student and the EMA teacher at the end of training (Table 6d). The EMA teacher is 1.8 points more accurate, consistent with the student’s weights being affected by noise from recent batches that averaging removes.

Variance Across Seeds

To assess the stability of OPST, we repeat self-training on Gemma 4 E2B with three random seeds and report the mean and standard deviation in Table 7. The means correspond to the values reported in Table 1. Across both regimes, the standard deviation stays at or below 1.2 points and is therefore well below the gains of OPST over the base model. Without thinking, variance is low on all benchmarks. With thinking, it is highest on GPQA and GSM8K. On GPQA Diamond, which contains only 198 questions, a single question already corresponds to about 0.5 points. MMLU with thinking is the most stable benchmark (0.1), consistent with its large test set.

SettingMMLUGPQAGSM8KHellaSwagAverage
w/o Think.47.7 ± 0.832.8 ± 1.0–56.2 ± 0.845.6 ± 0.3
w/ Think.71.1 ± 0.134.3 ± 1.080.9 ± 1.265.6 ± 0.563.0 ± 0.4

Table 7. Variance across seeds. Mean and standard deviation of OPST on Gemma 4 E2B over three seeds. Means correspond to Table 1.

Generalization to Qwen

To test whether OPST is specific to the Gemma 4 model family, we additionally apply it to Qwen3-4B-Base (Yang et al. [38]). Unlike the instruction-tuned Gemma 4 models used in the main experiments, Qwen3-4B-Base is a pretrained model without instruction tuning. We use the same topic descriptors, prompt templates, and hyperparameters as Gemma 4 E2B and evaluate with the protocol described in Appendix § 8.§ 6. Table 8 reports the results. OPST improves all four benchmarks and raises the average accuracy by 6.0 points (66.7 →\rightarrow 72.7). The largest gain occurs on GSM8K (+13.3), followed by GPQA (+5.1), MMLU (+3.1), and HellaSwag (+2.5). Since the multiple-choice benchmarks are scored by log-likelihood over the answer options, the gains on MMLU, GPQA, and HellaSwag do not stem from improved answer formatting. OPST therefore transfers both to a different model family and to a model that has not undergone instruction tuning.

MethodMMLUGPQAGSM8KHellaSwagAverage
Qwen3-4B-Base
Base72.032.379.782.866.7
+ OPST (Ours)75.1+3.137.4+5.193.0+13.385.4+2.572.7+6.0

Table 8. Results on Qwen3-4B-Base. Performance of OPST applied to a pretrained Qwen3 base model, evaluated with the protocol described in Appendix A.6.

OPST Algorithm

Algorithm 1 summarizes the complete OPST training procedure. Both the student πθ\pi_{\theta} and the privileged teacher πϕ\pi_{\phi} are initialized from the same weights θ0\theta_0, so training requires no external teacher model. Each of the NN rounds consists of two stages: (1) a fresh curriculum of problems xx with privileged information zz is synthesized from the topic set T\mathcal{T} and filtered via pseudo-verification and the consensus gate (§ 3.3 and § 3.4), and (2) the student is trained on its own on-policy rollouts y∼πθ(⋅∣x)y \sim\pi_{\theta}(\cdot\mid x) by minimizing LOPST\mathcal{L}_{\mathrm{OPST}} (Eq. (6)) against the teacher conditioned on (x,z)(x,z). After every update, the teacher tracks the student through an exponential moving average with rate β\beta. The teacher therefore improves alongside the student while remaining a stable distillation target. After the final round, the EMA teacher πϕ\pi_{\phi} is returned as the final model.

Algorithm 1 On-Policy Self-Training (OPST)

Require: topics T\mathcal{T}, initial weights θ0\theta_0, rounds NN, EMA rate β\beta

  1. θ←θ0\theta\leftarrow\theta_0, ϕ←θ0\phi\leftarrow\theta_0

  2. for n=1,…,Nn = 1, \ldots, N do

  3. Generate (xi,zi)∼πgen(⋅∣τ)(x_i, z_i) \sim\pi_{\mathrm{gen}}(\cdot\mid\tau) for τ∈T\tau\in\mathcal{T} ▷ § 3.3

  4. Pseudo-verification and consensus gate; discard noisy, trivial, and duplicate samples ▷ § 3.4

  5. for each mini-batch B\mathcal{B} do

  6. Sample rollouts yi∼πθ(⋅∣xi)y_i \sim\pi_{\theta}(\cdot\mid x_i) for all samples in B\mathcal{B}

  7. Update θ\theta with ∇θLOPST(θ;B)\nabla_{\theta}\mathcal{L}_{\mathrm{OPST}}(\theta; \mathcal{B}) ▷ Eq. (6)

  8. ϕ←(1−β)ϕ+βθ\phi\leftarrow(1-\beta)\phi+\beta\theta

  9. end for

  10. end for

  11. return πϕ\pi_{\phi}

Evaluation Protocol

We follow the task formulations of the EleutherAI lm-evaluation-harness Gao et al., [10] and adapt them to the Gemma 4 chat template. Without thinking, we score all multiple-choice benchmarks, i.e., MMLU, GPQA Diamond, HellaSwag, ARC-Challenge, and Winogrande, by log-likelihood over the closed set of answer options. No answer is therefore parsed from generated text. We use the harness defaults, i.e., 5-shot prompting for MMLU, zero-shot prompting for the remaining multiple-choice benchmarks, and length-normalized accuracy where the harness specifies it. GSM8K is evaluated with 8-shot chain-of-thought prompting Wei et al., [37].

With thinking, multiple-choice answers are parsed from the final Answer: X line. GSM8K is graded by exact match of the final numerical answer, and MATH-500 by symbolic equivalence of the final \boxed{} expression using SymPy. For all models and benchmarks, both before and after self-training, the rate of parsing failures and invalid answers is 0%. Performance differences in the thinking regime therefore do not stem from changes in answer formatting. All remaining settings follow the harness defaults.

OPST Prompt Templates

We list the prompts used for problem synthesis, student rollouts, and teacher conditioning. Synthesis prompts are shown in gray, student prompts in blue, and teacher prompts in orange, with placeholders filled at runtime shown in {color}. With thinking, the system turn <|turn>system\n<|think|><|turn|> is prepended to both student and teacher prompts.

Problem Synthesis. The generator receives a topic descriptor, e.g., "college_mathematics" for MMLU or "rate, time, and distance word problems" for GSM8K. It returns a question xx together with a rationale rr and a candidate answer aa. We use a multiple-choice template for MMLU, GPQA, and HellaSwag, and an open-ended template for GSM8K.

Synthesis Prompt (Multiple-Choice: MMLU, GPQA, HellaSwag)

You are an expert exam writer creating challenging multiple-choice questions on the topic of {topic}.

Write ONE novel, self-contained, graduate-level multiple-choice question testing deep conceptual understanding or multi-step reasoning in {topic}. Provide exactly four plausible answer choices labeled (A), (B), (C), and (D), followed by a rigorous step-by-step solution and the final correct option letter.

Use the following exact format: Question: <question text> (A) <choice A> (B) <choice B> (C) <choice C> (D) <choice D> Solution: <step-by-step derivation explaining why the correct choice holds> Answer: <single letter A, B, C, or D>

Synthesis Prompt (Open-Ended: GSM8K)

You are an expert mathematics curriculum designer creating multi-step word problems on the topic of {topic}.

Write ONE novel, self-contained arithmetic or algebraic word problem about {topic} that requires 3 to 5 reasoning steps to solve. Ensure all numerical quantities are consistent and yield an exact numerical answer. After the question, provide a complete step-by-step worked solution and the final number.

Use the following exact format: Question: <word problem text> Solution: <step-by-step mathematical derivation> Answer: <final numerical value>

Student Prompts. The student receives only the question xx.

Student Prompt (Multiple-Choice)

Solve the following multiple-choice question step by step. After your reasoning, state your final choice on a new line in the exact format "Answer: X" where X is A, B, C, or D.

Question: {question} (A) {choice_A} (B) {choice_B} (C) {choice_C} (D) {choice_D}

Student Prompt (Open-Ended)

Solve the following math word problem step by step. Show all
intermediate
calculations and conclude with your final numerical result in the
exact format
"Answer: <number>".

Question: {question}

Teacher Prompts. The teacher receives the question together with the privileged information z=(r,a)z = (r, a) and is instructed to solve the problem without referring to it. If the generated rationale is incomplete, the teacher first completes it, conditioned on the question and the candidate answer.

Rationale Completion Prompt

Question: {question}
{choices_block}

The potentially correct answer to this problem is ({gen_answer}).
Write a clear, rigorous step-by-step derivation reasoning about
({gen_answer}).
Write the explanation as a natural, self-contained solution from first
principles. Do NOT mention that the correct answer was given to you
in advance.

Teacher Prompt

You are an expert tutor solving the problem below. You are provided
with a
potential reference solution and final answer as privileged
background context.
Use this privileged information to guide a clear, rigorous,
step-by-step
derivation of the problem from first principles. Do NOT mention the
reference
solution or state that the answer was provided to you; present the
reasoning as
a direct solution to the question.

### Privileged Reference Context:
Reference Rationale: {privileged_rationale}
Potential Answer: {gen_answer}

### Problem to Solve:
Question: {question}
{choices_block}

Provide your step-by-step reasoning and conclude with "Answer: ".

Visual OPST Prompt Templates

We list the prompts used in Visual OPST for visual question synthesis, privileged information extraction, visual verification, and student–teacher distillation. For all multimodal turns, the image token <|image|>\n is prepended to the user turn <|turn>user\n...\n<turn|>\n<|turn>model\n. Visual Problem Synthesis. Given an unlabeled image I∈II \in\mathcal{I} and a visual topic descriptor τ∈Tvis\tau\in\mathcal{T}_{\mathrm{vis}}, the generator synthesizes {n}\{n\} question–answer pairs. We use a dedicated prompt for object counting and a shared template for the remaining visual topics.

Visual Synthesis Prompt (Object Counting)

Look at this image carefully. Generate {n} counting questions about
objects in
the image. Include some objects that appear multiple times and some
that do not
appear at all (count 0).
Format EXACTLY as:
Q: <question>
A: <number>

Generate {n} different questions.

Visual Synthesis Prompt (Attributes and Spatial Relations)

Look at this image carefully. Generate {n} questions about
{topic_description}.
Format EXACTLY as:
Q: <question>
A: <short answer>

Generate {n} different questions.

where topic_description is instantiated per topic τ∈Tvis\tau\in\mathcal{T}_{\mathrm{vis}} as:

  • color_attributes: “colors, sizes, shapes, or visual attributes of objects”

  • spatial_relations: “spatial relationships between objects (left/right, above/below, in front/behind, near/far)”

Privileged Information. To construct the privileged information zz, the generator either writes a detailed scene caption for II or detects the objects in the scene with bounding boxes. From pairs of detected boxes, we derive grounded spatial relations between objects from their box centers ("left of", "right of", "above", "below").

Privileged Scene Caption Prompt

Describe this image in detail, including the main objects, their
attributes, and
their spatial arrangement.

Object Detection Prompt

Detect all objects in this image. For each object, provide the name
and bounding
box coordinates.
Format each detection as: name: [x1, y1, x2, y2]
Coordinates are normalized 0–1000.
List up to 10 objects.

Visual Verification. To compute vaug⁡(I,x,z)\operatorname{vaug}(I, x, z), the generator answers xx on each of the KK augmented views Ak(I)A_k(I) using a concise VQA prompt. Visual Verification Prompt

{question}
Answer concisely in 1–5 words.

Student and Teacher Prompts. During distillation, the student receives the image II and the question xx, while the teacher additionally receives the privileged information zz prepended to xx.

Visual Student Prompt

<|image|>
{question}

Visual Teacher Prompt

<|image|>
Image description: {privileged_caption} Objects present:
{detected_objects}
{question}

References

  1. [5]Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos Garea, S., Geist, M., & Bachem, O. (2024). On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (Vol. 2024, pp. 21246–21263).
  2. [6]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.arxiv.org/abs/2104.14294
  3. [7]Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Self-questioning language models. In ICLR 2026 Workshop on Multi-Agent Learning and Its Opportunities in the Era of Generative AI, 2026. URL https://openreview.net/forum?id=8OO1hDmpXL.openreview.net/forum?id=8OO1hDmpXL
  4. [9]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., & al., . et . (2021). Training verifiers to solve math word problems. In arXiv preprint. 2110.14168
  5. [10]Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPoFi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonnell, K., Muenchhoff, N., Ociepa, C., Pang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., … Zou, A. (2023). A framework for few-shot language model evaluation. <a href="https://doi.org/https://zenodo.org/records/10256836" rel="noopener noreferrer" target="_blank">https://doi.org/https://zenodo.org/records/10256836</a>
  6. [11]Team, G., El Abd, S., Aggarwal, V., Algayres, R., Andreev, A., Bachem, O., Ballantyne, I., Brick, C., Carbune, V., Casbon, M., & al., . et . (2026). Gemma 4 technical report. 2607.02770
  7. [12]Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017). Making the v in vqa matter: Elevating the role of image understanding in visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6904–6913.
  8. [13]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In International Conference on Learning Representations, volume 2024, pp. 32694–32717, 2024.DOI
  9. [14]Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., & al., . et . (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature. https://doi.org/10.1038/s41586-025-09422-z
  10. [15]Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). Measuring massive multitask language understanding. International Conference on Learning Representations. https://openreview.net/forum?id=d7KBjmI3GmQopenreview.net/forum?id=d7KBjmI3GmQ
  11. [16]Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. Advances in Neural Information Processing Systems Datasets and Benchmarks Track.
  12. [17]Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data. In International Conference on Learning Representations, volume 2026, pp. 130770–130790, 2026.arxiv.org/abs/2508.05004
  13. [18]Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 1051–1068, 2023.arxiv.org/abs/2210.11610
  14. [19]Hübotter, J., Lübeck, F., Behric, L., Baumann, A., Bagatella, M., Marta, D., Hakimi, I., Shenfeld, I., Kleine Buening, T., Guestrin, C., & al., . et . (2026). Reinforcement learning via self-distillation. In arXiv preprint. arXiv:2601.20802
  15. [20]Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, 2016. doi: 10.18653/v1/D16-1139.DOI
  16. [21]Köpf, A., Kilcher, Y., Von Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stanley, O., Nagyfi, R., & al., . et . (2023). Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 47669–47681.
  17. [22]Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Nikolenko, A., & al., . et . (2020). The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7), 1956–1981.
  18. [23]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 292–305, 2023.arxiv.org/abs/2305.10355
  19. [24]Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, and Nuno Vasconcelos. On-policy self-distillation without any supervision. arXiv preprint arXiv:2608.06296, 2026.arXiv
  20. [25]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pp. 39578–39601, 2024.arxiv.org/abs/2305.20050
  21. [26]Haoran Liu, Yuwei Zhang, Xiyao Li, Bohan Lyu, and Jingbo Shang. Hero: Hindsight-enhanced reflection from environment observations for agentic self-distillation. arXiv preprint arXiv:2606.11559, 2026.arXiv
  22. [27]Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., & al., . et . (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.
  23. [28]Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models. arXiv preprint arXiv:2602.04942, 2026.arXiv
  24. [29]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark. In First Conference on Language Modeling, 2024.arxiv.org/abs/2311.12022
  25. [30]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.arxiv.org/abs/1907.10641
  26. [31]Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., & al., . et . (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. In arXiv preprint. arXiv:2402.03300
  27. [32]Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026a.arXiv
  28. [33]Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RL’s razor: Why online reinforcement learning forgets less. In International Conference on Learning Representations, volume 2026, pp. 59839–59864, 2026b.arxiv.org/abs/2509.04259
  29. [34]Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A. T., Kumar, A., Alemi, A. A., Rizkowsky, A., Azova, N., Adlam, B., Bohnet, B., Elsayed, G. F., Sedghi, H., … Fiedel, N. (2024). Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research. https://openreview.net/forum?id=lNAyUngGFKopenreview.net/forum?id=lNAyUngGFK
  30. [35]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023a.arxiv.org/abs/2203.11171
  31. [36]Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pp. 13484–13508, 2023b.arxiv.org/abs/2212.10560
  32. [37]Wei, J., Wang, X., Schuurmans, D., Bosma, M., Lichter, B., Xia, F., Chi, E. H., Le, Q. V., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35.
  33. [38]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., & al., . et . (2025). Qwen3 technical report. In arXiv preprint. arXiv:2505.09388
  34. [39]Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr. arXiv preprint arXiv:2604.03128, 2026.arXiv
  35. [40]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488, 2022.arxiv.org/abs/2203.14465
  36. [41]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 4791–4800, 2019.arxiv.org/abs/1905.07830
  37. [42]Yaocheng Zhang, Yuanheng Zhu, Wenyue Chong, Songjun Tu, Qichao Zhang, Jiajun Chai, Xiaohan Wang, Wei Lin, Guojun Yin, and Dongbin Zhao. π-Play: Multi-agent self-play via privileged self-distillation without external data. arXiv preprint arXiv:2604.14054, 2026.arXiv
  38. [43]Andrew Zhao, Yiran Wu, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. Advances in Neural Information Processing Systems, 38:105816–105879, 2025.arxiv.org/abs/2505.03335
  39. [44]Zhao, S., Xie, Z., Liu, M., Huang, J., Pang, G., Chen, F., & Grover, A. (2026). Self-distilled reasoner: On-policy self-distillation for large language models. Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research.
  40. [45]Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. In International Conference on Learning Representations, volume 2026, pp. 2548–2581, 2026b.arxiv.org/abs/2505.19590
  41. [46]Zuo, Y., Zhang, K., Sheng, L., Qu, S., Cui, G., Zhu, X., Li, H., Long, X., Hua, E., Qi, B., & al., . et . (2025). Ttrl: Test-time reinforcement learning. Advances in Neural Information Processing Systems, 38, 131459–131483.

Paper details

vlnc:W_uebhwfvh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents