Self-Distillation for Supervised Fine-Tuning

Introduction

Large language models (LLMs) are typically trained once on a massive corpus and then deployed as static artifacts. In practice, however, the world does not hold still. New facts emerge, new tools appear, new domains demand attention—and the model must incorporate this knowledge without forgetting what it already knows. This is the post-training problem, and for modern LLMs it remains a major open challenge.

The standard approach, supervised fine-tuning (SFT), trains the model to imitate the expert demonstrations token by token. This procedure is simple, stable, and widely used. However, it also imposes a rigid supervision interface: every demonstrated token is treated as a one-hot target, and every non-demonstrated token is treated as equally wrong.

For language models, this supervision format can be unnecessarily sharp. A demonstration tells the model which response is correct, but it does not imply that every continuation supported by the model’s distribution should be driven to zero. This distinction is important in continual post-training, where a model must acquire new facts, tools, domains, or response formats without unnecessarily destroying the distributional structure that supports its existing capabilities. In this setting, the bottleneck is not only the amount or quality of post-training data, but also the interface through which demonstrations are converted into learning signals.

Recently, Shenfeld et al. [36] proposed Self-Distillation Fine-Tuning (SDFT), which exploits the in-context learning ability of LLMs for continual post-training. The core idea is that conditioning a model on the gold response in its prompt produces a useful supervision signal. The gold-response-aware model serves as a teacher and the gold-response-unaware model as a student; SDFT then samples on-policy rollouts from the student in real time and trains the student to match the teacher via KL divergence. However, SDFT requires the student to roll out a completion at every training step, which increases training time. Moreover, early in training or on unfamiliar domains these rollouts are often incorrect, and because the teacher and student see different contexts, the teacher’s distribution over an erroneous sequence may yield a noisy signal. More fundamentally, our experiments suggest that SDFT is not effective at learning a novel domain; like reinforcement learning in LLMs, it is better suited for domains where the model already has some prior capabilities ([61, 56]).

Inspired by SDFT’s core insight that in-context conditioning on the gold response elicits a usable soft target, we propose Gold-Conditioned Distillation (GCD), a simple modification that sidesteps the issues above by replacing the student’s rollout with the gold response from the dataset. The teacher conditions on a privileged prompt that exposes the gold response, while the student conditions only on the question; the student is then trained to match the teacher’s token-level distribution evaluated on the fixed gold response from the dataset via forward KL divergence loss.

We evaluate GCD under two LLM fine-tuning settings across different models, Llama-3.1-8B-Instruct ([8]) and Qwen2.5-3B-Instruct ([58]) as well as Qwen3-4B-Base and Qwen3-8B-Base ([57]). The first tests adaptation in partially known domains. The second training setting tests learning new knowledge from scratch in domains where the model has near-zero initial accuracy. Across both settings, GCD offers a favorable stability–plasticity trade-off: it often improves new-task acquisition over SFT and usually preserves general capabilities at a comparable or slightly better level.

Our contributions are summarized as:

• We propose Gold-Conditioned Distillation (GCD), a simple yet effective offline self-distillation method for learning from expert demonstrations, achieving strong average new-task learning performance while modestly improving or maintaining retention in most settings.

• We show that GCD improves held-out generalization in partially known domains and usually preserves general capabilities better than SFT. Token-level analysis shows that the gains come from preserving structured probability mass on model-plausible alternatives rather than from simple label smoothing.

Related Work

Catastrophic forgetting and continual learning. Catastrophic forgetting occurs when training on new data reduces a model’s performance on previously learned data or tasks ([16, 25, 29, 46, 17, 49]). Recent studies show that forgetting is a practical issue in LLMs. Continual instruction tuning can hurt domain knowledge, reasoning, reading comprehension, general ability, and instruction following ([23, 53, 66] (Li and Lee 2024)). Other work studies the causes of forgetting in LLM tuning and relates it to loss-landscape properties ([19, 33]). Continual learning aims to adapt models over time while preserving earlier capabilities. Classical methods are often grouped into replay-based approaches ([3, 32, 37]), regularization-based approaches ([16, 20, 63]), and architecture-based approaches ([35, 24]). These approaches are harder to apply directly to LLM fine-tuning: replay may require unavailable past data, regularization can add storage and computation costs, and architecture-based methods often require task-specific components. Recent methods also adapt replay and parameter-efficient tuning to LLMs, including synthetic rehearsal ([13, 54]), orthogonal LoRA subspaces ([31, 33, 52]), and analyses of why LoRA can forget less ([2]). These methods show progress, but they still involve trade-offs between learning new data, preserving old abilities, and adding extra data, computation, or model components.

Self-distillation. Knowledge distillation trains a student model to match the behavior of a teacher model ([34, 62, 48, 67, 60, 4, 55, 39, 5, 11, 26, 30, 41, 45, 50, 68]). The teacher signal can be a soft output distribution, intermediate features, attention maps, or contrastive relations ([7, 10, 28, 42, 44, 62, 51]). Self-distillation is a special case of knowledge distillation in which the teacher is the same model, a previous version of the model, or the same model under a different input condition. Recent LLM distillation methods study compression ([15, 22]), capability transfer ([14, 21, 27]), and self-improvement ([40, 59, 64]). Since LLMs generate sequences, both the distillation objective and the training distribution matter: MiniLLM studies reverse-KL distillation for LLMs ([9]), while Generalized Knowledge Distillation trains on student-generated sequences with teacher feedback to reduce the gap between training and inference ([1]). Closely related to our work, context distillation trains a model to internalize behavior that appears when useful context, examples, or reasoning steps are provided ([38]). Yang et al. [59] use model-generated data to reduce the distribution gap in LLM fine-tuning, and SDFT ([36]) uses the current model conditioned on a demonstration as its own teacher and generates on-policy completions for training. Our method keeps the same idea of a context-conditioned self-teacher, but uses the gold response as the completion, avoiding online generation and sampling noise.

Vector (PDF)

Method

We propose Gold-Conditioned Distillation (GCD), a token-level distillation method for learning from expert demonstrations. The goal is to replace the one-hot targets used in supervised fine-tuning with model-native targets produced by the model itself under gold response conditioning.

The student and teacher are initialized from the same model, but only the student receives gradient updates; the teacher is maintained as an EMA copy of the student throughout training. Thus, the teacher provides a slowly evolving self-distillation target. GCD is fully offline: neither the student nor the teacher autoregressively generates responses during training; both are evaluated only under teacher forcing along the gold response from the dataset.

Student and teacher contexts. Given an input $x$ and its gold response $y^* = (y^*_1,\ldots,y^*_T)$, at position $t$ the student conditions on the standard teacher-forced prefix, while the teacher conditions on a privileged prefix that additionally exposes the full gold response through a template $\phi(x,y^*)$:

where $[\,\cdot\,;\,\cdot\,]$ denotes token-sequence concatenation. The student and teacher next-token distributions are then obtained by evaluating each model on its own context:

where $\theta$ denotes the student parameters, and $\bar{\theta}$ denotes the EMA teacher parameters. The two models are initialized identically, $\bar{\theta}_0=\theta_0$, but diverge during training because only $\theta$ receives gradient updates.

A concrete instantiation of the privileged template is:

Both distributions are evaluated on the same gold prefix states $y^*_{<t}$; the only asymmetry is that the teacher uses an EMA-smoothed parameterization and observes an additional gold-revealing segment.

Objective. We train the student by matching the privileged teacher distribution with a token-level forward KL, averaged over positions and over the data distribution $\mathcal{D}$:

where $\operatorname{sg}[\cdot]$ denotes the stop-gradient operator. The teacher distribution is therefore treated as a fixed target for each update, and gradients are applied only to the student parameters $\theta$. We adopt the forward, mode-covering KL direction, so that the student is penalized for placing low probability on tokens that the teacher considers likely under the gold-revealed context. An ablation in Appendix § 11 confirms this choice: replacing it with reverse KL lowers accuracy on both LiveNews and Tooluse.

EMA teacher update. After each student update, the teacher parameters are updated using an exponential moving average of the student:

where $\alpha\in[0,1]$ controls the update rate. In this paper, we choose $\alpha= 0.01$ for the EMA update following SDFT (Shenfeld et al., [36]. We conduct an ablation study on $\alpha$ in Appendix § 10: GCD is stable for small update rates ($\alpha\le0.05$), while large rates that make the teacher track the student too closely cause training to collapse.

Experiments

Experiment Settings

Models. We evaluate four pretrained language models spanning model families, scales, and training regimes: Qwen2.5-3B-Instruct, Qwen3-4B-Base, Llama-3.1-8B-Instruct, and Qwen3-8B-Base. The base models are included to test whether GCD’s gold-conditioned teacher signal remains effective in the absence of post-training instruction-following priors.

Dataset and Evaluation. We conduct fine-tuning experiments under two learning settings, distinguished by the model’s initial capability in the target domain.

• Partial-knowledge setting. In this setting, the model already possesses partial ability in the target domain before fine-tuning. We fine-tune and evaluate on four datasets: Tooluse ([43]), which requires generating the correct API call and JSON arguments in response to user requests; and three scientific multiple-choice datasets, namely Chemistry, Physics & Material (physics and materials science), and Biology.

• Near-zero-knowledge setting. In this setting, the model has almost no prior knowledge of the target domain, with initial accuracy close to zero on LiveNews ([65]). We fine-tune the model on LiveNews to evaluate

its ability to continuously acquire new knowledge through updates. LiveNews consists of news question-answering instances collected from September 2025 and January 2026, which post-date the models’ training cutoff.

We evaluate models from two perspectives: (a) new-task performance, measured by accuracy on held-out evaluation splits of the corresponding fine-tuning datasets; and (b) previous capability retention, measured on five general benchmarks (MMLU, HellaSwag, Winogrande, TruthfulQA, and IFEval) using lm-evaluation-harness (settings in Appendix § 8.§ 3). These benchmarks collectively assess knowledge, commonsense reasoning, language understanding, factuality, and instruction-following ability. For the new tasks, all responses are generated with greedy decoding; LiveNews answers are graded by Qwen3.5-9B, also with greedy decoding. Please refer to Appendix § 7 for more details on the dataset setup.

Baselines. SFT: standard supervised fine-tuning. SDFT (Shenfeld et al., [36]): the original self-distillation fine-tuning method, in which the student generates an on-policy response and the gold-conditioned teacher provides the soft target along that generated response (see § 3). Following a known issue in the released implementation, we compute its KL in the forward direction (Appendix § 8).

Hyperparameters. For Qwen2.5-3B-Instruct and Qwen3-4B-Base, we conduct full fine-tuning using a learning rate of $5 \times10^{-5}$. For Llama-3.1-8B-Instruct and Qwen3-8B-Base, we adopt LoRA (Hu et al., [12]) with learning rates of $1 \times10^{-4}$ and $4 \times10^{-4}$, respectively. Across all three training methods (SFT, SDFT, and GCD), we keep the other variables like optimizer, learning-rate scheduler, batch size, sequence length, and number of training epochs the same for each (model, task) setting. Unless otherwise specified, all evaluation results are reported using the final checkpoint obtained at the end of training.

Results in the Partial-Knowledge Setting

Table 1 reports new-task accuracy on each of the four tasks for all four models. Overall, our method GCD consistently achieves the best average performance across all four models, demonstrating its strong generalization ability on new tasks. Table 2 shows that these new-task gains do not require a larger loss of general capabilities. On Tooluse fine-tuning, GCD improves new-task accuracy over SFT for all four models, while maintaining comparable average performance on previous-task benchmarks. We attribute these gains to the model-native soft labels produced by the gold-conditioned teacher. Compared with the one-hot targets used by SFT, these soft labels provide a less brittle supervision signal: the gold token still anchors the correct response, while the remaining probability mass preserves model-plausible alternatives. This encourages the student to learn a smoother local distribution around the expert demonstration, which can improve generalization to held-out examples.

Method	Tooluse	Chemistry	Physics & Material	Biology	Avg.
QWEN2.5-3B-INSTRUCT
Base	34.02	30.77	44.83	76.67	46.57
SFT	60.83	65.09	63.79	85.94	68.91
SDFT	55.67	63.31	52.30	83.53	66.38
GCD	66.30	66.67	63.22	86.45	70.66
QWEN3-4B-BASE
Base	36.08	38.46	56.32	84.56	53.86
SFT	62.89	70.02	68.40	86.11	71.86
SDFT	60.83	54.24	60.92	78.04	63.51
GCD	71.13	69.43	67.24	86.45	73.56
LLAMA-3.1-8B-INSTRUCT
Base	52.57	30.18	38.51	70.84	48.03
SFT	65.98	68.64	60.35	86.97	70.49
SDFT	35.05	39.65	47.13	43.91	41.44
GCD	67.01	68.84	62.64	85.94	71.11
QWEN3-8B-BASE
Base	55.67	39.25	55.75	83.36	58.51
SFT	62.89	69.82	64.94	86.96	71.15
SDFT	63.92	59.57	66.67	85.76	68.98
GCD	64.95	69.63	68.97	86.79	72.59

Model	Method	New Task	MMLU	HellaSwag	Winogrande	TruthfulQA	IFEval	Avg.
Qwen2.5-3B-Instruct	Base	34.02	65.45	74.85	69.69	58.63	53.33	64.39
	SFT	60.83	64.39	72.41	69.14	52.17	40.28	59.68
	SDFT	55.67	65.10	74.22	69.22	58.65	41.22	61.68
	GCD	66.30	<u>64.71</u>	<u>72.62</u>	69.93	<u>52.84</u>	38.49	<u>59.72</u>
Qwen3-4B-Base	Base	36.08	71.22	73.68	70.64	53.46	41.08	62.02
	SFT	62.89	68.79	69.54	68.43	49.22	29.16	57.03
	SDFT	60.83	70.62	72.84	71.51	53.64	33.41	60.40
	GCD	71.13	<u>70.05</u>	<u>71.91</u>	<u>69.85</u>	<u>50.95</u>	25.71	<u>57.69</u>
Llama-3.1-8B-Instruct	Base	52.57	68.27	79.52	73.64	54.52	53.96	65.98
	SFT	65.98	67.74	77.97	73.95	54.09	46.11	63.97
	SDFT	35.05	68.19	79.47	73.95	<u>54.71</u>	53.59	65.98
	GCD	67.01	<u>68.03</u>	<u>79.11</u>	73.95	55.02	<u>47.79</u>	<u>64.78</u>
Qwen3-8B-Base	Base	55.67	74.55	78.59	71.74	52.28	40.76	63.58
	SFT	62.89	72.98	76.37	70.48	<u>50.82</u>	33.08	60.75
	SDFT	<u>63.92</u>	73.86	77.70	73.09	53.67	17.12	59.09
	GCD	64.95	<u>73.22</u>	<u>77.67</u>	<u>72.45</u>	50.28	<u>32.16</u>	61.16

Results in the Near-Zero-Knowledge Setting

Learning new knowledge from scratch. Figure 2 reports evaluation accuracy across training epochs on LiveNews. In this setting, the base models have near-zero initial accuracy, indicating that the target knowledge is largely absent before fine-tuning. We observe that GCD learns the new task as effectively as, and sometimes better than, standard SFT. On LiveNews, GCD achieves higher final new-task accuracy than SFT on Qwen2.5-3B-Instruct and Qwen3-4B-Base, while SFT is slightly better on Llama-3.1-8B-Instruct. By contrast, SDFT performs poorly on new-task accuracy across all three models. This is expected in the from-scratch knowledge-injection setting: since SDFT relies on student-generated rollouts, the early trajectories are often incorrect when the model has little prior knowledge about the target domain. The teacher is then asked to provide supervision on low-quality trajectories, leading to weak acquisition of the new knowledge.

Vector (PDF)

Capability retention in knowledge injection. Table 3 reports retention on five general-purpose benchmarks after fine-tuning on LiveNews, and shows a clear trade-off between acquiring the new task and preserving previous capabilities. SDFT obtains the highest previous-task average on all three models, but only because it barely learns the injected knowledge (new-task accuracy below 11%), so high retention alone is not useful. GCD is more balanced: it matches or exceeds SFT on the new task on two of the three models and is slightly below it on Llama-3.1-8B-Instruct, while its previous-task average is higher than SFT’s on two models and marginally lower on Qwen3-4B-Base. The retention gains are modest, and GCD does not fully solve catastrophic forgetting; the takeaway is that GCD reaches SFT-level new-task learning while usually preserving general capabilities slightly better.

Model	Method	New Task	MMLU	HellaSwag	Winogrande	TruthfulQA	IFEval	Avg.
NEW TASK: LIVENEWS
Qwen2.5-3B-Instruct	Base	1.50	65.45	74.85	69.69	58.63	53.33	64.39
	SFT	36.75	64.86	74.37	67.80	36.78	43.41	57.44
	SDFT	10.25	65.12	74.80	69.69	53.94	35.12	59.73
	GCD	38.00	65.19	74.24	69.06	36.54	44.48	57.90
Qwen3-4B-Base	Base	1.00	71.22	73.68	70.64	53.46	41.08	62.02
	SFT	35.50	67.13	72.40	70.09	36.46	27.31	54.68
	SDFT	7.75	68.07	73.14	69.53	54.00	23.02	57.55
	GCD	38.75	66.83	72.45	70.72	36.83	25.00	54.36
Llama-3.1-8B-Instruct	Base	0.25	68.27	79.52	73.64	54.52	53.96	65.98
	SFT	48.25	66.27	77.99	71.90	38.48	47.78	60.48
	SDFT	5.75	67.81	79.63	74.74	53.44	50.10	65.14
	GCD	45.75	67.02	78.96	72.61	40.21	50.61	61.88

Analysis

What the GCD Soft Label Encodes and What the Student Inherits

GCD conditions the teacher on the gold response, asks it to reproduce that response token by token, and trains the student under $D_{\mathrm{KL}}(\pi_T \mathbin\Vert\pi_\theta)$. We ask two questions empirically: (i) what does the gold-conditioned target distribution $\pi_T$ encode beyond the SFT one-hot label, and (ii) does the trained student inherit that structure at inference time, when the gold response is no longer in context.

Setup. We use Qwen2.5-3B-Instruct on Tooluse dataset. We forward about 200 training prompts through the initial model. At every token position $t$ we record the teacher distribution $\pi_T(\cdot\mid c_t^{(T)})$ under the gold-conditioned context and, as a control, the distribution of the same model under the student context $c_t^{(S)}$, i.e., without the gold response; we refer to the latter as the base model. We focus on the low-confidence gold positions, selected by the threshold $\pi_T(y_t^*) \le0.85$ after discarding formatting, whitespace, and BPE-segmentation artifacts.

Context-conditional alternatives. Representative low-confidence positions place mass on paraphrase choices, semantically related action verbs, and reasoning-step discourse markers — ‘weight’ / ‘mass’, ‘create’ / ‘get’, ‘impacted’ / ‘influenced’, ‘Next’ / ‘Thought’. Crucially, the choice depends on the context. To quantify this, we define the assignment entropy of a gold token as the entropy of the distribution of the teacher’s top-1 alternative across the low-confidence instances of that token. Across 41 instances of ‘Next’, the top-1 alternative takes seven distinct values with an assignment entropy of 1.41 nats. Any context-independent regularizer (uniform or per-token-id label smoothing) has assignment entropy zero by construction.

Aligned with the base model’s natural continuations. The Teacher column of Table 4 compares the teacher’s top-5 alternatives with the base model’s top-5 at the same position: the average Jaccard overlap is 0.26, against a random baseline of about $1.9 \times10^{-5}$, and 46% of $\pi_T$’s residual non-gold mass lies on tokens that are also among the base model’s top-5. The gold-conditioned residual is therefore not a prompt-template artifact: it is concentrated on the branch points the base model would explore on its own, re-weighted toward the gold response.

	Base	SFT	GCD	Teacher $\pi_T$
$\bar{\pi}(\mathrm{gold})$	0.10	0.94	0.86	0.55
Mass on teacher top-5	0.62	0.05	0.13	0.45
Mass on base top-5	0.88	0.05	0.10	0.46
Top-5 Jaccard w/ base	0.99	0.14	0.27	0.26

Anti-saturation, alternative preservation, base-aligned shape. Table 4 also reports the output distributions of the SFT and GCD checkpoints at the same low-confidence positions under the student context. SFT pushes $\pi_\theta(\mathrm{gold})$ to 0.94, whereas GCD stops at 0.86. This is consistent with the direction predicted by the forward-KL objective, whose fixed point $\pi_\theta= \pi_T$ places only 0.55 gold at these positions; the student does not reach this value because it is trained against the EMA teacher, which moves toward the student during training, whereas Table 4 reports the target of the initial teacher. GCD retains 2.6× and 2.0× the mass of SFT on the teacher’s and the base model’s top alternatives, respectively, and at 10% of the positions it even ranks an alternative above gold, mirroring positions where $\pi_T$ itself does. The top-5 Jaccard overlap with the base model is 1.9× that of SFT (0.27 vs. 0.14).

Connection to forgetting. Because $\pi_T$’s residual mass falls on tokens the base model would naturally produce, the forward-KL fixed point $\pi_\theta= \pi_T$ pins the student close to the base distribution at these branch points, whereas the SFT one-hot loss keeps pushing alternative probability toward zero across epochs. We hypothesize that this base-aligned shape is a local signature of the retention differences measured in § 4; the gains there are modest, and establishing the link directly is left for future work.

Vector (PDF)

Vector (PDF)

Comparison with SFT+KL

To test whether the benefit of GCD can be reproduced by simply regularizing toward the base model, we compare it with SFT+KL(base), which augments the SFT objective with a KL penalty toward the base model distribution:

We sweep the regularization strength $\lambda\in\{0.1, 0.3, 0.5, 1.0\}$ and compare all variants with SFT and GCD on LiveNews with Qwen2.5-3B-Instruct.

As shown in Figure 3a, KL regularization toward the base model does not improve new-knowledge acquisition: $\lambda= 0.1$ behaves like SFT and still trails GCD at the end of training, while $\lambda\ge0.3$ increasingly constrains the model to the base distribution and falls substantially below SFT. GCD’s new-task gains therefore cannot be explained by simply keeping the fine-tuned model close to the base model; note that this comparison concerns new-task accuracy only.

Comparison with SFT+Label Smoothing

To test whether the structure of the GCD target matters, or whether any softened version of the one-hot target suffices, we compare against uniformly label-smoothed SFT (SFT+LS), which replaces the one-hot target at each position with

where $y^{*}_{t}$ is the gold token, $\epsilon$ is the smoothing coefficient, and $|\mathcal{V}|$ is the vocabulary size. This baseline controls for the effect of softening the target distribution while removing the model- and context-dependent structure present in GCD. We sweep $\epsilon\in\{0.03, 0.05, 0.10, 0.15, 0.20\}$ on Tooluse with Qwen2.5-3B-Instruct.

As shown in Figure 3b, label smoothing improves over SFT inconsistently and by at most about three points, with some coefficients falling below SFT, whereas GCD achieves the best held-out Tooluse accuracy. The gain of GCD therefore cannot be explained only by increasing the entropy of the target distribution.

Conclusion

In this paper, we propose GCD, a novel distillation framework for LLM fine-tuning. Unlike SFT, which relies on one-hot targets, a gold-conditioned teacher is introduced to provide soft, model-native supervision, enabling the student model to preserve plausible alternatives aligned with the model’s prior distribution while improving adaptation to new tasks. Unlike on-policy self-distillation methods such as SDFT, GCD performs distillation directly along the gold response, avoiding costly online rollout generation. Extensive experiments show that GCD achieves strong average new-task performance across multiple models and tasks. Compared with SFT, GCD often improves new-task accuracy and provides slightly better retention. Compared with SDFT, GCD substantially improves new-task acquisition in the settings we study.

Limitations

This work is an initial exploration of gold-conditioned supervision. In our experiments, we use a simple prompt that asks the teacher to repeat the provided gold response. This design may not fully exploit the model’s ability to interpret the demonstration. A better prompt could ask the teacher to compare the gold response with plausible alternatives, identify key decision points, or produce a more calibrated distribution over possible continuations. Such prompts may elicit richer and more informative soft targets than simple repetition. In addition, our token-level analysis is conducted on a single model and dataset, the retention gains over SFT are modest, the near-zero-knowledge setting covers three of the four models, and all models have at most 8B parameters; whether these findings transfer to larger models remains open.

Acknowledgement

The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST) - Center of Excellence for Generative AI, under award number 5940.

Dataset Details

We evaluate our method on five datasets. Table 5 provides an overview of the dataset; per-dataset descriptions follow.

Dataset	Domain	Answer format	#Train	#Eval
Tooluse	API tool calling	API name + JSON args	4,046	97
Chemistry	Mixed scientific QA (chemistry-heavy)	4-option MCQ	2,674	507
Physics & Material	Physics + materials science	4-option MCQ	1,257	174
Biology	School- and graduate-level biology	2/3/4-option MCQ	1,107	583
LiveNews	Post-training-cutoff news QA	Free-form answer	400	400

Tooluse. Same dataset used in Shenfeld et al. [36]. A function-calling benchmark in which the model receives a natural-language instruction together with the documentation of one or more API tools and must emit a structured Action:<tool_name> and Action Input:<JSON> sequence. Each instance contains the tool’s name, description, natural-language documentation, a user instruction, and a gold response consisting of one or more (Action, Action_Input) pairs.

Chemistry. Same dataset used in Shenfeld et al. [36]. A Chemistry QA corpus in which each example pairs a multiple-choice question with a chain-of-thought solution wrapped in <reasoning>...</reasoning> <answer>...</answer> tags.

Physics & Material. A merged physics + materials-science split derived from SciKnowEval [6]. Each prompt presents a quantitative question (e.g., Gauss-law fluxes, gravitational lensing, materials properties) together with the relevant physical constants, and each gold response is a worked solution followed by the chosen letter.

Biology. A union of (i) the biology portion of ScienceQA-style elementary/middle-school questions, and (ii) the biology subset of SciKnowEval, which contributes harder graduate-level questions. Both sources are normalized into a unified schema with a dynamic answer-format specification matching the actual number of options (2/3/4), so that no item is dropped for being non-quaternary. The combined dataset therefore contains a mixture of easy and hard biology MCQs.

LiveNews. A recent-news QA dataset built from the human-verified test splits of two LiveNewsBench [65] releases (September 2025 and January 2026). Each example consists of a free-form factual question about events that occurred after every model’s training cutoff, paired with a short human-written answer. To decouple “learning the fact” from “memorizing the surface phrasing,” we use GEMINI-3-FLASH to paraphrase both the question and the answer of every item, producing a rewritten variant for each original $(q, a)$. Training is performed on the rewritten $(q', a')$ pairs, while evaluation is performed on the original human-written $(q, a)$. A method that has truly absorbed the underlying news event must therefore answer a phrasing it has never been trained on, ruling out trivial token-level memorization. LiveNews thus measures whether a method can inject genuinely new world knowledge that cannot have leaked from previous training.

Construction of Physics & Material and Biology

Both training datasets are reorganized into the common JSONL schema {prompt, golden_response}. Every prompt is rendered as a self-contained exam item that bundles (i) a format specification, (ii) the question stem, and (iii) the option list, with the format string parameterized by the number of choices $k \in\{2, 3, 4\}$: the model is asked to wrap reasoning in <reasoning>...</reasoning> and to emit a single choice letter inside <answer>...</answer>. The corresponding golden_response follows the same wrapping.

The source of the reference response differs by dataset. ScienceQA-Biology already provides explanations, so we use them directly and convert the answer index to a letter. SciKnowEval provides only the final answer, so we synthesize reference responses with GPT-OSS-20B: for each item, we sample $N = 8$ candidates with $T = 0.7$, top-$p = 0.9$, and max_tokens=1024, discard candidates whose extracted answer letter does not match the gold label, and select the median-length remaining response. Items with no correct candidate are removed. We then form Biology by concatenating ScienceQA-Biology and SciKnowEval-Biology, and Physics & Material by concatenating the physics and materials subsets of SciKnowEval.

Experiment Details

Baseline Reproduction

SFT. For the SFT baseline we use the SFTTrainer from TRL (von Werra et al., [47]) with the cross-entropy loss masked to the gold-response tokens only; the prompt portion of each input is excluded from the loss support. This matches the loss support used by our GCD and SDFT.

SDFT. For SDFT we run the authors’ publicly released implementation¹. Following a known issue in that repository², we modify the distillation term so that the KL is computed in the forward direction $D_{\mathrm{KL}}(\pi_T \parallel\pi_\theta)$ rather than the reverse direction shipped in the original paper.

Hyperparameters and Shared Training Pipeline

All three methods (SFT, SDFT, and GCD) share the same training pipeline for every (model, task) combination, so that only the loss differs. We use AdamW with $(\beta_1, \beta_2) = (0.9, 0.999)$, $\epsilon= 10^{-8}$, and weight decay 0.01, clip the global $\ell_2$ gradient norm at 1.0, and use a linear warmup followed by a cosine decay of the learning rate to 0. Each run trains for 4 epochs in bf16 mixed precision on a single A100 80GB GPU, with a per-device batch size of 4 and 8 gradient accumulation steps (an effective batch size of 32) and a maximum sequence length of 2,048 tokens. We perform full fine-tuning on Qwen2.5-3B-Instruct and Qwen3-4B-Base. On Llama-3.1-8B-Instruct and Qwen3-8B-Base we use LoRA (Hu et al., [12]) with rank $r=64$ and scaling factor 128 on the attention and MLP projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), keeping all other layers frozen; the same LoRA configuration is used for all three methods on these models.

General Benchmark Evaluation

We evaluate previous capability retention with lm-evaluation-harness (v0.4.x) using identical settings for the base model and all fine-tuned checkpoints. All five benchmarks are evaluated zero-shot. We report accuracy for MMLU and Winogrande, length-normalized accuracy (acc_norm) for HellaSwag, MC2 for TruthfulQA, and, for IFEval, the mean of prompt-level and instruction-level strict accuracy. The multiple-choice benchmarks are scored by log-likelihood, while IFEval responses are generated with greedy decoding (temperature 0) and a maximum of 1,280 new tokens. All other options follow the harness defaults.

Learning-Rate Sensitivity of SDFT

A natural concern when comparing against SDFT (Shenfeld et al., [36]) is that its weaker performance might simply reflect a poorly tuned learning rate rather than a limitation of the method itself. To rule this out, we conduct a learning-rate sweep for SDFT and contrast every point in the sweep with a single GCD run that uses the same model, dataset, optimizer, batch size, schedule, and number of epochs.

For each (model, dataset) combination we train SDFT at five to six learning rates spanning roughly an order of magnitude—$2 \times10^{-5}$, $5 \times10^{-5}$, $8 \times10^{-5}$, $1 \times10^{-4}$, $2 \times10^{-4}$, and $3 \times10^{-4}$³. All other hyperparameters are inherited from the corresponding configuration in Appendix B; only the learning rate is varied. For GCD, we plot a single curve trained with the default reported in § 4.

EMA Update Rate Ablation

GCD maintains the teacher as an exponential moving average of the student, $\bar{\theta} \leftarrow(1-\alpha)\bar{\theta}+\alpha\theta$, with $\alpha= 0.01$ in all main experiments. To measure how sensitive GCD is to this choice, we vary $\alpha\in\{0, 0.01, 0.05, 0.2, 1.0\}$ on Qwen2.5-3B-Instruct for both LiveNews and Tooluse, keeping every other hyperparameter fixed at the setting used in the main experiments. Here $\alpha= 0$ freezes the teacher at the initial model, and $\alpha= 1.0$ copies the student into the teacher after every update. We report the accuracy of the final checkpoint.

Vector (PDF)

Table 6 shows that GCD is stable for small update rates: $\alpha\in\{0, 0.01, 0.05\}$ reach similar accuracy on both tasks, so GCD does not depend on a finely tuned $\alpha$. Large rates cause training to collapse: with $\alpha= 1.0$ the teacher becomes identical to the student at every step and accuracy drops sharply on both tasks, and at $\alpha= 0.2$ Tooluse first rises and then collapses late in training while LiveNews is unaffected. The teacher therefore needs to lag behind the student, but any small update rate works.

EMA rate $\alpha$	LiveNews	Tooluse
0 (frozen teacher)	34.25	64.95
0.01 (default)	38.00	66.30
0.05	35.75	62.89
0.2	36.25	16.49
1.0 (teacher = student)	20.50	0.00

KL Direction Ablation

GCD minimizes the forward KL divergence from the teacher to the student, $\mathrm{D}_{\mathrm{KL}}\left(\pi_{\bar{\theta}\,} \,\|\, \pi_\theta\right)$ (§ 3). To test this choice, we replace it with the reverse KL divergence, $\mathrm{D}_{\mathrm{KL}}\left(\pi_\theta\,\|\, \pi_{\bar{\theta}}\right)$, on Qwen2.5-3B-Instruct for both LiveNews and Tooluse, keeping every other hyperparameter fixed at the setting used in the main experiments. We report the accuracy of the final checkpoint.

Table 7 shows that forward KL outperforms reverse KL on both tasks. The gap is largest on LiveNews, where the target knowledge is absent before fine-tuning: with reverse KL the model barely learns the new facts, reaching only 11.00% compared with 38.00% for forward KL. On Tooluse, where the base model already has partial ability (34.02%), reverse KL still learns the task but trails forward KL by 10.63 points (55.67% vs. 66.30%).

KL direction	LiveNews	Tooluse
Forward (default)	38.00	66.30
Reverse	11.00	55.67

Multi-Seed Results

To check that the gains of GCD do not come from a single favorable run, we repeat the Tooluse experiments with five random seeds on three of the four models (Qwen2.5-3B-Instruct, Qwen3-4B-Base, and Llama-3.1-8B-Instruct), keeping every other hyperparameter fixed at the setting used in the main experiments. Given the training cost of multi-seed evaluation, we use Tooluse as a representative task. Table 8 reports the mean and standard deviation of the new-task accuracy over the five seeds.

Method	Qwen2.5-3B-Instruct	Qwen3-4B-Base	Llama-3.1-8B-Instruct
SFT	$61.1 \pm0.7$	$63.2 \pm0.6$	$66.0 \pm0.4$
SDFT	$56.4 \pm0.6$	$61.2 \pm0.8$	$35.6 \pm0.7$
GCD	$66.7 \pm0.6$	$69.6 \pm0.5$	$66.9 \pm0.4$

GCD achieves the highest mean accuracy on all three models: 66.7% on Qwen2.5-3B-Instruct, 69.6% on Qwen3-4B-Base, and 66.9% on Llama-3.1-8B-Instruct. The standard deviations of all methods are below one point, much smaller than the gaps between GCD and SDFT (10.3, 8.4, and 31.3 points) and between GCD and SFT on the two Qwen models (5.6 and 6.4 points). The improvements of GCD are therefore consistent across random seeds.