betaFeedback

WorksComputer Science

Effective Teacher Support for Reasoning Distillation

Abstract

Forward-KL language-model distillation normally computes dense teacher and student scores over the full vocabulary at every supervised position, making training and stored teacher targets expensive. We instead distill from the teacher's highest-probability vocabulary fraction, renormalizing the retained mass as we sweep support from 100% to 0.01%. Our main study distills Qwen3 4B -> 1.7B. We confirm the broader support-reduction finding on larger Qwen3 8B -> 4B and Nemotron Nano 12B -> 9B pairs and under answer-only supervision. We evaluate generation primarily on MMLU-Pro and secondarily on GPQA-Diamond. Across the Qwen3 settings, retaining 0.01% of the vocabulary causes no observed MMLU-Pro degradation relative to full-vocabulary distillation, although disagreement with the complete teacher distribution increases. Nemotron Nano matches full-vocabulary MMLU-Pro at 0.1%, but smaller fractions fail to emit the closing reasoning tag, showing that the reliable cutoff depends on the model family. At 0.01% support, caching the retained next-token scores from the teacher forward pass cuts training step time by 38%, while restricting student output computation to those tokens cuts peak memory by 24%; neither causes observed MMLU-Pro degradation.

Motivation

Knowledge distillation trains a student to match a teacher’s predictive distribution, so that a smaller model can inherit the generalization of a larger one [1]. For an autoregressive language model, this predictive distribution is the next-token softmax over the vocabulary. Token-level LLM distillation commonly matches this vocabulary-wide distribution and evaluates the resulting student by generation quality [2, 3, 4].

Standard next-token cross-entropy fine-tuning uses a one-hot target, whose support is one, at each supervised position, even though the student softmax remains normalized over the full vocabulary. This target sparsity motivates asking whether distillation likewise requires full-vocabulary teacher targets. This analogy is imperfect, however, because a teacher’s relative probabilities over alternative tokens can carry useful information.

Teacher probabilities can nevertheless be highly concentrated, making sparse targets appealing at LLM scale [5]. Yet high retained mass alone does not rule out degradation under a truncated objective [6]. How far support can be reduced without observed generation degradation is therefore an empirical question.

We study this question for distillation on reasoning traces [7]. We first measure the effective support of the frozen teacher, then train students with rank-kk truncated forward KL by retaining its kk highest-probability tokens at each supervised position. We compare these students with an otherwise identical full-vocabulary student on MMLU-Pro [8] and GPQA-Diamond [9]. To distinguish optimization of each truncated objective from agreement with the complete teacher distribution, we report both the kk-dependent training loss and a common full-vocabulary KL diagnostic.

Our study provides a controlled empirical account of retained teacher support in reasoning distillation. We characterize that support, test a rank-kk ladder against full-vocabulary KL, and relate the resulting loss diagnostics to generation. We also probe a second training setting with a separate answer-only experiment, confirm the support-size result on a larger Qwen3 teacher–student pair (Qwen3-4B from Qwen3-8B) [10], and test cross-family transfer on an auxiliary Nemotron Nano 12B→\to9B pair [11]. Finally, we quantify the systems consequences under matched configurations. Caching k=15k=15 targets (0.01% of the vocabulary) removes the teacher from the training loop and reduces step time by 38%, while compact kk-row output computation avoids materializing vocabulary-sized student logits and reduces peak allocated memory by 24% relative to cached full-head training. We report these systems effects separately from generation quality and verify that both cached variants show no observed MMLU-Pro degradation.

The remainder of the paper is organized as follows. Section 1 motivates effective teacher support, and Section 2 reviews the relevant distillation literature. Section 3 defines the retained-support objectives. Section 4 describes the data and experimental protocol, and Section 5 presents the results. Section 6 concludes the paper, and Appendices A and B decompose the retained-support objectives and derive the teacher-sampling construction.

Motivation

At each supervised position tt, forward-KL distillation compares a teacher distribution ptp_t and a student distribution qtq_t over the vocabulary of size VV:

Lt=∑v=1Vpt(v)(log⁡pt(v)−log⁡qt(v)).(1)\mathcal{L}_t = \sum_{v=1}^{V} p_t(v)\left(\log p_t(v) - \log q_t(v)\right). \tag*{(1)}

The softmax assigns positive probability to every token, so its mathematical support is all of VV, even when most of its mass is concentrated on a few tokens. If pt,(i)p_{t,(i)} is the ii-th largest teacher probability and 0<ρ≤10 < \rho\le1, define

kρ(t)=min⁡{k:∑i=1kpt,(i)≥ρ}.k_\rho(t) = \min\left\{k : \sum_{i=1}^{k} p_{t,(i)} \ge\rho\right\}.

This is the smallest rank cutoff that retains at least a fraction ρ\rho of the teacher mass at position tt. Its empirical distribution is over positions, not token identities or individual probabilities, and maps a target mass to a candidate truncation scale.

FIRST [5] shows that a few teacher tokens can retain most probability mass at LLM scale. Minitron and Sparse Logit Sampling show that aggressive top-kk objectives can nevertheless degrade the student relative to full-distribution distillation [6, 12]. We therefore use kρk_\rho to identify candidate support sizes, then test for observed generation degradation.

The distillation temperature τ\tau scales the teacher logits before the softmax; τ=1\tau= 1 leaves the native next-token distribution unchanged [1]. Figure 1 reports empirical CDFs of kρk_\rho for ρ∈{0.90,0.95,0.99}\rho\in\{0.90, 0.95, 0.99\} for frozen Qwen3-4B [10] at τ=1\tau= 1. At k=15k = 15, the retained tokens contain at least 90%, 95%, and 99% of the teacher mass at 99.0%, 97.9%, and 93.4% of later supervised positions, respectively. The first supervised position has kρ=1k_\rho= 1 in every example and is omitted. These CDFs use the native teacher ranking, before the Qwen3-specific tag correction applied during training in § 4.

Figure 1. Effective teacher support on reasoning traces. Empirical CDFs of kρk_\rho over the 905,193,844 supervised positions remaining after excluding the first supervised position in each example, for which kρ=1k_\rho= 1. The vertical line marks k=15k = 15.

The reasoning ladder extends to 0.01% of the 151,936-token vocabulary, corresponding to k=15k = 15. Figure 1 shows that this extreme truncation still retains most of the teacher mass at most positions, but retained mass alone does not establish that generation will be unaffected. Training optimizes a token-level objective under teacher-forced contexts, whereas benchmark evaluation requires the student to generate complete solutions autoregressively. We therefore compare rank-kk and full-vocabulary students on MMLU-Pro and GPQA-Diamond, and place their kk-dependent training losses alongside a common full-vocabulary KL diagnostic. A separate answer-only experiment examines a second training setting (§ 6.2).

For offline distillation, a dense teacher-target cache stores O(V)O(V) values per supervised position, whereas a rank-kk cache stores O(k)O(k) token identifiers and values. Caching removes the teacher from the training loop. For the renormalized objective, the loss also requires only the selected student logits, making restricted output computation possible (§ 6.4).

Related work

Classical knowledge distillation trains a student on a teacher’s softened predictive distribution, whose low-probability classes can encode useful information about the teacher’s generalization behavior. Temperature-based distillation flattens the teacher distribution and gives lower-probability classes greater influence [1]. Our findings point in a different direction in this setting: at τ=1\tau= 1, generation on the primary benchmark remains stable even when the native teacher tail is omitted. This suggests that further emphasizing the tail through a higher distillation temperature may be unnecessary here.

Modern LLM distillation changes either the divergence or the distribution of training sequences. MiniLLM optimizes reverse KL on student-generated responses [2], Generalized Knowledge Distillation combines fixed and student-generated sequences with several divergence choices [3], and DistiLLM introduces skew KL with adaptive off-policy training [4]. These methods alter the divergence or training contexts rather than selecting a fixed vocabulary subset for forward-KL supervision. Reasoning distillation commonly transfers teacher-generated rationales as sequence supervision [13]; our study instead examines the effective vocabulary support of the teacher distribution at each position within those traces. Complementary work reduces reasoning-distillation cost by truncating the supervised sequence rather than the vocabulary axis [14].

Several methods reduce the teacher output at LLM scale. SLIM stores the top 5% of teacher logits [15]. FIRST reports that five tokens can contain more than 95% of the probability mass and distills from recalibrated top-five targets [5]. A design-space study of pre-training distillation explores offline top-pp and top-kk targets for LLM pretraining [16]. High retained mass alone does not rule out degradation from a truncated objective. Minitron reports degradation for small top-kk objectives in post-pruning recovery [12], while Sparse Logit Sampling analyzes the bias introduced by truncation and proposes an importance-sampled estimator that preserves the full-KD gradient in expectation [6]. RLKD [17] is complementary: it aligns the ranking of teacher and student peaks rather than restricting forward KL to the teacher’s retained support.

Multi-axis selective KD [18] jointly selects positions, vocabulary classes, and training samples. It uses importance-sampled class selection and reports reductions in storage, wall time, and peak memory. A fused, chunked offline-KL implementation [19] directly caches partial-mass top-100 targets while retaining the full-vocabulary student normalizer. Our study instead isolates deterministic teacher-ranked support and selects kk through generation on reasoning traces. We then apply the same offline caching pattern at k=15k = 15. Our teacher-sampled robustness check has no importance correction and is therefore not an unbiased estimator of full KL [6]. The compact renormalized kk-row path is technically distinct because it removes the full-vocabulary student normalizer; we treat it as a secondary systems path.

Method

Retained-support objectives

Let ztT,ztS∈RVz_t^{\mathrm{T}}, z_t^{\mathrm{S}} \in\mathbb{R}^{V} denote the teacher and student logits at supervised position tt. At distillation temperature τ>0\tau> 0, define

ptτ=softmax⁡(ztT/τ),qtτ=softmax⁡(ztS/τ).p_t^{\tau} = \operatorname{softmax}(z_t^{\mathrm{T}}/\tau), \qquad q_t^{\tau} = \operatorname{softmax}(z_t^{\mathrm{S}}/\tau).

Equation (1) is the full-vocabulary forward KL at τ=1\tau= 1. For general τ\tau, we use ptτ,qtτp_t^{\tau}, q_t^{\tau} and multiply the loss by τ2\tau^2, following classical knowledge distillation [1]. We fix τ=1\tau= 1, measuring support on the teacher’s native next-token distribution; τ\tau is distinct from the decoding temperature used for generation.

At each position, let πt\pi_t be a permutation of the token identifiers that orders the teacher logits from largest to smallest, with ties ordered arbitrarily:

ztT(πt(1))≥⋯≥ztT(πt(V)),z_t^{\mathrm{T}}(\pi_t(1)) \ge\cdots\ge z_t^{\mathrm{T}}(\pi_t(V)),
St={πt(1),…,πt(k)},Mt=∑v∈StptT(v).S_t = \{\pi_t(1), \ldots, \pi_t(k)\}, \qquad M_t = \sum_{v \in S_t} p_t^{\mathrm{T}}(v).

Thus StS_t contains the kk most probable teacher tokens, since softmax preserves the logit ordering, and MtM_t is their total probability mass. The primary objective renormalizes both distributions within StS_t, yielding the conditional teacher and student distributions pˉt\bar{p}_t and qˉt\bar{q}_t:

pˉt(v)=ptT(v)Mt,qˉt(v)=exp⁡(ztS(v)/τ)∑u∈Stexp⁡(ztS(u)/τ)\bar{p}_t(v) = \frac{p_t^T(v)}{M_t}, \qquad\bar{q}_t(v) = \frac{\exp(z_t^S(v)/\tau)}{\sum_{u \in S_t} \exp(z_t^S(u)/\tau)}

The corresponding per-position loss, denoted ℓtrenorm\ell_t^{\mathrm{renorm}}, is the forward KL between these conditional distributions:

ℓtrenorm=τ2∑v∈Stpˉt(v)log⁡pˉt(v)qˉt(v).\ell_t^{\mathrm{renorm}} = \tau^2 \sum_{v \in S_t} \bar{p}_t(v) \log\frac{\bar{p}_t(v)}{\bar{q}_t(v)}.

This objective compares the relative probabilities within the selected set. It removes the teacher tail and the full-vocabulary student normalizer.

We also evaluate a partial-mass objective,

ℓtmass=τ2∑v∈StptT(v)log⁡ptT(v)qtT(v).\ell_t^{\mathrm{mass}} = \tau^2 \sum_{v \in S_t} p_t^T(v) \log\frac{p_t^T(v)}{q_t^T(v)}.

It retains the teacher’s original mass on StS_t and the full-vocabulary student normalizer [19]. For a proper unmodified subset StS_t, the retained teacher probabilities sum to Mt<1M_t < 1, so this truncated sum is not itself a KL divergence and need not be nonnegative. Both objectives recover full-vocabulary forward KL when k=Vk = V. Their different normalizations also expose the bias introduced by truncation [6]. We report renormalized KL as the primary method because it compares teacher and student distributions conditioned on the selected support and permits student computation restricted to the retained rows. An earlier auxiliary rank-kk sweep found similar generation behavior for the two objectives. We retain the partial-mass objective as an objective-design comparison. Appendix § 8.1 gives an exact decomposition and the corresponding student-logit gradients.

Support construction

Rank selection is deterministic and defines the main kk-ladder. As a robustness check, draw independent gv∼Gumbel⁡(0,1)g_v \sim\operatorname{Gumbel}(0,1) and let πtg\pi_t^g order the perturbed teacher logits from largest to smallest:

ztT(πtg(1))τ+gπtg(1)≥⋯≥ztT(πtg(V))τ+gπtg(V),Stsample={πtg(1),…,πtg(k)}.\frac{z_t^T(\pi_t^g(1))}{\tau} + g_{\pi_t^g(1)} \ge\cdots\ge\frac{z_t^T(\pi_t^g(V))}{\tau} + g_{\pi_t^g(V)}, \qquad S_t^{\mathrm{sample}} = \{\pi_t^g(1), \ldots, \pi_t^g(k)\}.

This is sampling without replacement from the teacher distribution [20]. We apply the same truncated objective without proposal or inclusion-probability weights, so teacher sampling is not an importance-sampling estimator of full KL. Uniform selection draws kk distinct vocabulary identifiers uniformly and serves as a negative control. Uniform selection produced a much worse training objective, and its partial-mass variant diverged. An earlier auxiliary sweep found no systematic MMLU-Pro advantage for teacher sampling over rank selection. Because this check was not repeated in the three-seed ladder, we treat it as qualitative evidence and exclude it from the main results.

Qwen3 uses <think> and </think> as structural generation tokens [10]. They mark entry to and exit from reasoning, so omitting either from a small support removes direct supervision for that transition. Preliminary runs without explicit retention of these tags often failed to emit </think>, leaving reasoning traces unclosed. Retaining both tags improved generation stability, though it did not prevent every failure. We therefore append missing tags, giving at most k+2k + 2 tokens. An inserted tag copies the weight of a tag already present; if neither is present, both receive the smallest retained weight. We rescale the corrected weights only when their total exceeds 1 and apply the same correction at every position and for every selection method. In the objectives above, StS_t, the teacher weights on that set, and MtM_t are replaced by the corrected set, weights, and their sum. The resulting target is not the original teacher distribution conditioned on the corrected set; Appendix § 8.1 formalizes this distinction. This empirical safeguard is Qwen3-specific; other model families may require different handling of thinking tokens, or none. The teacher-sampled set therefore follows the exact sampling law only before correction. Appendix § 8.2 derives this qualification and distinguishes the procedure from importance sampling.

Token weighting and diagnostics

For sequence ii, let CiC_i contain supervised tokens from <think> through </think>, inclusive, and let AiA_i contain subsequent supervised assistant tokens. Given a per-token objective ℓt\ell_t, when both regions are nonempty the sequence loss is

Li=(1−λ)1∣Ci∣∑t∈Ciℓt+λ1∣Ai∣∑t∈Aiℓt.(2)\mathcal{L}_i = (1-\lambda)\frac{1}{|C_i|}\sum_{t\in C_i}\ell_t+\lambda\frac{1}{|A_i|}\sum_{t\in A_i}\ell_t. \tag*{(2)}

A naive token average lets the longer reasoning region dominate the loss. Our companion study found that balancing the reasoning and answer regions improved downstream results over this unweighted objective [21]. Across our λ\lambda ablations, λ=0.5\lambda= 0.5 provided a stable common setting, so we use it throughout the reasoning-trace experiments. When one region is empty, the available region receives the full sequence weight. The batch objective averages equally over sequences. We add neither a hard-label language-modeling loss nor an intermediate-feature loss.

We report the mean training objective over the final 20 updates. For the primary ladder, we also evaluate full-vocabulary forward KL after training on a fixed sample, using all VV classes and the same sequence weighting. This provides a common diagnostic of disagreement with the complete teacher distribution; generation metrics determine model quality. Without shuffling the reasoning-trace training data, we apply the training tokenization and length filter and retain the first 256 surviving examples (430,657 supervised positions) for every primary-ladder checkpoint. The larger-pair confirmation uses the analogous diagnostic on eight examples.

Experimental setup

Data

Our primary corpus comprises the 545,844 English chat records marked reasoning=on in the Nemotron Post-Training Dataset v2 [22]. We preserve assistant reasoning traces and discard 428 records with unclosed think markup. This leaves 545,416 records before tokenization, including 545,160 with nonempty reasoning spans.

We use the Qwen3 tokenizer and a chat template that preserves assistant content verbatim. The loss mask includes assistant content and the end-of-turn token; system content, user content, and assistant role headers are excluded. Sequences longer than 8,192 tokens are dropped rather than truncated, removing 9,158 records and leaving 536,258 training examples with 905,730,102 supervised next-token positions.

The separate answer-only experiment uses the training split of Puzzle-KD-Nemotron-Post-Training-Dataset-v2 [23]. During preprocessing, closed reasoning spans are removed from each assistant turn, and the first bracketed answer is retained when present. Empty outputs and malformed think spans are discarded. After the 8,192-token length filter, 808,712 training examples remain. The resulting corpus contains chat, code, math, and stem examples, whereas the reasoning corpus contains only chat examples. We therefore treat Puzzle-KD as a separate training setting rather than a controlled trace-removal ablation.

In both settings, the corpus supplies training contexts and supervised assistant positions. The frozen Qwen3-4B model supplies the teacher distribution for the primary ladder and the answer-only experiment. The larger-pair confirmation uses a frozen Qwen3-8B teacher.

Training and evaluation

We perform full-parameter distillation from a frozen Qwen3-4B teacher to a trainable Qwen3-1.7B student [10]. The models share a vocabulary of 151,936 tokens. Training uses bfloat16 precision and PyTorch Fully Sharded Data Parallel v2 (FSDP2) on 64 NVIDIA GB200 GPUs (16 nodes) within a single NVLink domain.

All primary ladder runs use AdaGrad [24] with a constant learning rate of 10−510^{-5}, zero weight decay, no warmup, and global gradient clipping at 1.0. We selected this configuration in a preliminary sweep over AdaGrad learning rates and AdamW [25] hyperparameters. The best AdamW configuration favored β2=0.999\beta_2 = 0.999 without warmup, consistent with a long second-moment history, but produced slightly lower MMLU-Pro than AdaGrad. We hold the optimizer fixed across support sizes.

We train for one epoch (a two-epoch run did not improve MMLU-Pro). The per-device batch size is 2 and the global batch size is 128, giving 4,189 updates per epoch. All primary runs use τ=1\tau= 1, λ=0.5\lambda= 0.5, and KL-only distillation, with no hard-label cross-entropy loss. Rank-renormalized KL is the primary configuration. The partial-mass objective, teacher sampling, and uniform sampling provide supporting comparisons. We run the primary rank-kk ladder over three seeds. Full-vocabulary distillation is the primary reference, while the frozen Qwen3-1.7B checkpoint provides context for absolute model quality. The answer-only and additional-model checks follow the same one-epoch, three-seed protocol with AdaGrad configurations selected for their respective settings.

We also train Qwen3-4B students from a frozen Qwen3-8B teacher on the same reasoning corpus, AdaGrad learning rate, λ=0.5\lambda= 0.5, and three seeds, for one epoch at global batch size 144 on 18 nodes (3,724 updates). Rank-renormalized k=15k = 15 and k=152k = 152 use per-device batch size 2; the matched full-vocabulary students use per-device batch size 1 after the larger microbatch exhausted device memory. Peak allocation and step time are therefore not compared across those microbatches. The frozen Qwen3-4B checkpoint is an absolute reference for that pair, not the truncation control.

As an auxiliary cross-family and cross-architecture check, we distill NVIDIA-Nemotron-Nano-9B-v2 from frozen NVIDIA-Nemotron-Nano-12B-v2, hybrid Mamba–Transformer models [11]. We compare full-vocabulary training with rank-renormalized support at 0.1% (k=131k = 131), 0.02% (k=26k = 26), and 0.01% (k=13k = 13), evaluating generation with reasoning both enabled and disabled.

We select MMLU-Pro and GPQA-Diamond as the study concerns reasoning language models. MMLU-Pro provides broad coverage of knowledge-intensive questions with a stronger emphasis on reasoning than the original benchmark [8]. It contains 12,032 examples and is our primary benchmark. GPQA-Diamond complements it with 198 expert-validated graduate-level science questions [9] and serves as a smaller secondary benchmark.

The primary evaluation uses the Qwen3 chat template with thinking enabled, which we call think-on evaluation. We use greedy decoding, at most 8,192 generated tokens, and a 16,384-token model context. This template setting is specific to Qwen3 and is distinct from the reasoning=on metadata used to select training data. Absolute training time and milliseconds per step are reported; efficiency comparisons emphasize runs with a shared hardware and software configuration.

Results

We first evaluate whether retained-support distillation causes observed degradation in reasoning generation relative to full-vocabulary KL. We then examine answer-only supervision, additional model pairs, and the systems consequences of an offline retained-support pipeline. Training losses and full-vocabulary KL are reported as diagnostics rather than model-selection criteria.

Generation

All models in this subsection are evaluated with Qwen3 thinking enabled: the chat template permits a generated reasoning trace before the final answer.

We compare each rank-kk student with a full-vocabulary student trained under the same data, optimization, and update budget. Table 1 reports three-seed means and sample standard deviations for the final-20 training KL, train time, step time, MMLU-Pro, and GPQA-Diamond at each kk. Figure 2 plots the same generation scores in two panels and includes the frozen Qwen3-1.7B checkpoint as an absolute reference. The full-vocabulary student isolates the effect of truncation.

% of VVkkFinal-20 KLTrain (min)ms/stepMMLU-ProGPQA-Diamond
100151936 (full)0.424 ± 0.00159.5 ± 1.6853 ± 230.474 ± 0.0030.350 ± 0.023
50759680.424 ± 0.00194.8 ± 3.21358 ± 460.477 ± 0.0020.345 ± 0.021
25379840.424 ± 0.00170.1 ± 2.41005 ± 340.475 ± 0.0020.347 ± 0.013
10151940.423 ± 0.00162.6 ± 1.1897 ± 160.476 ± 0.0060.320 ± 0.034
575970.423 ± 0.00162.4 ± 1.5894 ± 220.477 ± 0.0040.352 ± 0.025
115190.422 ± 0.00158.3 ± 0.8835 ± 120.475 ± 0.0020.345 ± 0.031
0.11520.412 ± 0.00159.9 ± 2.5858 ± 350.476 ± 0.0030.323 ± 0.053
0.01150.375 ± 0.00158.7 ± 1.6840 ± 230.477 ± 0.0010.362 ± 0.018

Table 1. Rank-kk reasoning ladder. Three-seed means ± sample standard deviations. Final-20 KL averages the training objective over the last 20 updates. Times cover one epoch on the shared 16-node GB200 configuration. GPQA-Diamond contains 198 examples and is not used to rank nearby configurations.

Figure 2. Generation across retained support size. Three-seed MMLU-Pro (left) and GPQA-Diamond (right). Lines and large markers show means; light points show individual seeds. Horizontal lines mark the full-vocabulary student and frozen Qwen3-1.7B. Both panels use a log-scaled support axis but different vertical ranges.

No MMLU-Pro degradation is observed across the ladder: every mean is within 0.0033 of the full-vocabulary score of 0.474 ±\pm 0.003, and k=15k = 15 obtains 0.477 ±\pm 0.001. With three seeds, this is not a claim of statistical equivalence. All distilled students remain below the frozen Qwen3-1.7B reference on MMLU-Pro. This absolute drop reflects distribution drift from training on the distillation corpus; because the matched full-vocabulary student also exhibits it, it is not attributable to truncation. The experiment tests truncation relative to full-vocabulary distillation, not improvement over initialization. GPQA-Diamond is 0.350 ±\pm 0.023 at full VV and 0.362 ±\pm 0.018 at k=15k = 15. Given its 198 examples and seed variation, we do not rank nearby configurations on GPQA-Diamond.

Tokenization takes 67–72 minutes across the runs and has no systematic dependence on kk; Table 1 therefore reports training time separately. Truncation does not avoid the live teacher’s full-vocabulary computation, and top-kk selection adds overhead. Step time is therefore nonmonotonic: 840 ± 23 ms at k=15k = 15 versus 853 ± 23 ms for full-vocabulary KL. Truncating the objective alone provides no material speedup.

Figure 3 shows the optimized training objective throughout the epoch for one fixed seed across all support sizes. The full-vocabulary through 5% curves follow nearly the same trajectory, while the smallest supports converge to lower values of their renormalized objective. Because the objective changes with kk, these vertical differences show within-objective convergence rather than model quality and place the final-20 statistic in context.

Figure 3. Training KL throughout one epoch. Curves are 100-update moving averages of the optimized objective for one fixed seed, used across every kk. Table 1 and Figure 2 aggregate three seeds. Each kk defines a different renormalized objective, so vertical values are not a common measure of model quality.

Figure 2–Figure 4 distinguish the optimized conditional objective, agreement with the full teacher distribution, and generation quality. Renormalized KL matches only relative probabilities within the retained set. Shrinking kk removes constraints on the tail and on the total probability assigned to that set, so a lower training objective can coincide with greater disagreement with the full teacher distribution. Generation asks a different question. Under greedy decoding, it depends on whether the highest-probability choices along the generated trajectory change, not on whether the entire softmax is calibrated. For the concentrated teacher distributions studied here, this provides intuition for stable benchmark performance through k=15k = 15 despite that increased disagreement. Training loss and full-vocabulary KL are therefore useful diagnostics, but neither substitutes for generation.

Figure 4. Full-vocabulary KL across retained support size. Three-seed in-sample diagnostic on the same 256 training examples (430,657 supervised positions). The line shows the mean, light points show individual seeds, and the dashed line marks the full-VV mean.

Table 1 and Figure 3 report the kk-dependent optimized objective. To compare checkpoints under a common objective, Figure 4 evaluates an in-sample full-vocabulary KL diagnostic; Appendix § 8.1 identifies the terms retained and omitted by the truncated objective.

Answer-only supervision

To test whether very small support remains effective without reasoning traces in the training targets, we repeat the rank-renormalized study on the answer-only Puzzle-KD corpus [23]. It uses the same teacher and student but different source examples and categories.

We train a full-vocabulary reference and rank-kk students at 50%, 10%, 1%, and 0.01% of VV, over three seeds. The training loss is the mean rank-renormalized KL over supervised answer tokens. Puzzle-KD contains no reasoning region, so all sequence weight belongs on the answer region (λ=1\lambda= 1); there is no second region to balance. At evaluation, Qwen3 thinking disabled produces a direct answer, while thinking enabled permits a model-generated reasoning trace before the answer. These are Qwen3-specific template settings; reporting both separates the support-size result from the choice of generation mode.

Figure 5. Generation with answer-only supervision. Three-seed MMLU-Pro (left) and GPQA-Diamond (right) with Qwen3 thinking disabled and enabled. Lines and large markers show means; light points show individual seeds. The rightmost point is full-vocabulary distillation.

The support-size result from § 6.1 carries over to answer-only supervision: in both evaluation modes, every MMLU-Pro mean remains within 0.005 of its full-vocabulary reference through k=15k = 15. The separation between thinking enabled and disabled is much larger than the variation with kk, but the support-size conclusion is the same within each mode. Because Puzzle-KD supplies no reasoning traces, the generated reasoning in the enabled mode is not directly supervised by this corpus. GPQA-Diamond varies substantially across seeds and shows no consistent support-size trend, so we report it as a secondary result without ranking configurations.

Additional model pairs

Two additional teacher–student pairs test whether the retained-support result extends beyond the primary Qwen3 pair. For Qwen3-4B distilled from frozen Qwen3-8B, retaining 0.1% (k=152k = 152) or 0.01% (k=15k = 15) of the vocabulary produces no observed MMLU-Pro degradation relative to full-vocabulary training across three seeds; at the smaller support, the eight-example full-vocabulary KL diagnostic worsens even as the optimized objective falls, reinforcing the distinction between matching the teacher’s tail and preserving generation. For the auxiliary Nemotron pair, retaining 0.1% (k=131k = 131) keeps MMLU-Pro within 0.004 of full-vocabulary training in both reasoning modes, whereas both 0.01% (k=13k = 13) and 0.02% (k=26k = 26) fail specifically to emit the closing reasoning tag, preventing valid benchmark evaluation and showing that the smallest reliable support is model-family dependent.

Offline targets and compact output computation

The k=15k = 15 ladders in § 6.1 and § 6.2 show no observed MMLU-Pro degradation while retaining only 0.01% of the vocabulary. This small support makes the teacher practical to cache. Before student training, we run the frozen Qwen3-4B teacher once over the tokenized corpus. At each supervised position, we retain the top-kk token identifiers and corresponding teacher logits, together with the logits for both Qwen3 reasoning tags, and write them to disk. The resulting cache for all 536,258 reasoning examples requires approximately 108 GiB (116 GB); student training reads these targets instead of executing the teacher, removing the teacher forward pass from the training loop. With these targets cached, the renormalized objective also permits student output computation on only the cached rows. Sparse offline targets follow prior work [5, 6, 16, 19].

Table 2 shows that, across three matched seeds, caching reduces mean step time by 38%. Compact output reduces peak allocation by 24% relative to cached full-head training and by 46% relative to live-teacher training, but does not reduce step time further. All three final-20 objectives remain close. Tokenization remains 69–70 minutes.

ConfigurationFinal-20 lossTrain (min)ms/stepPeak alloc. (GB)
Live teacher + full output head0.3754 ± 0.001358.7 ± 1.6840 ± 23103.3 ± 0.2
Cached teacher + full output head0.3754 ± 0.001336.2 ± 1.8519 ± 2673.4 ± 0.2
Cached teacher + compact kk-row head0.3773 ± 0.001336.1 ± 2.5516 ± 3555.8 ± 0.3

Table 2. Matched full-epoch k=15k = 15 training. Means ± sample standard deviations over three seeds. Peak allocation is the maximum post-warmup CUDA allocation across GPUs.

On MMLU-Pro, live-teacher, cached full-head, and compact kk-row training obtain 0.477 ± 0.001, 0.478 ± 0.003, and 0.476 ± 0.002, respectively. Both cached means are within 0.0014 of live-teacher training, so no MMLU-Pro degradation is observed. The corresponding GPQA-Diamond scores are 0.362 ± 0.018, 0.337 ± 0.042, and 0.328 ± 0.022. Given the benchmark’s 198 examples and seed variation, we report these values without ranking configurations.

Cached full-head training still materializes student logits of shape (B,S,V)(B, S, V). Compact training instead projects the NN valid supervised positions onto only the cached rows, producing (N,k~)(N, \tilde{k}) logits with k~≤k+2\tilde{k} \leq k + 2 after the Qwen3 support correction. Appendix § 8.1 shows why this restriction is valid for renormalized KL but not for the partial-mass objective. Without a cache, the teacher must still score every vocabulary row to identify the retained set, so restricting only the student output does not remove the teacher-side output pass. After caching, the full student head uses one highly optimized dense GEMM, whereas compact output uses indexed weight-row gathers and batched narrow products. Their data movement and lower kernel efficiency offset the reduction in arithmetic, leaving step time unchanged while reducing peak allocation. Thus caching and compact output computation address wall time and memory, respectively, rather than compounding into the same speedup.

Conclusion

The native Qwen3-4B distribution is highly concentrated, and the three-seed rank-kk ladder shows no observed MMLU-Pro degradation relative to full-vocabulary distillation down to k=15k = 15, or 0.01% of the vocabulary. Distilling Qwen3-4B from Qwen3-8B reproduces the MMLU-Pro result at k=15k = 15 and k=152k = 152. The Nemotron check retains MMLU-Pro at 0.1% of the vocabulary but exposes a reasoning-termination failure at smaller supports, so the cutoff itself does not transfer universally across model families. Answer-only Puzzle-KD shows the same support-size trend under both Qwen3 evaluation modes, although it is a separate training setting rather than a matched ablation. Full-vocabulary KL nevertheless increases at the smallest supports, distinguishing the support needed to match the complete teacher distribution from that needed for benchmark generation.

This small effective support enables a practical offline pipeline. Caching the selected k=15k = 15 teacher identifiers and logits removes the teacher from training and reduces step time by 38%. Renormalized KL also permits compact kk-row student output computation, which avoids materializing vocabulary-sized logits and reduces peak allocated memory by 24% relative to cached full-head training, although it provides no additional speedup. Both cached variants show no observed MMLU-Pro degradation relative to live-teacher training. Caching and compact output computation therefore address distinct bottlenecks: training time and memory, respectively.

The primary conclusions are scoped to two Qwen3 teacher–student pairs (1.7B from 4B; 4B from 8B), with one auxiliary Nemotron family check, one training epoch, and the two corpora studied here. We do not locate the Qwen3 failure boundary below k=15k = 15. The structural-token correction and thinking-enabled evaluation are specific to Qwen3, while Puzzle-KD provides a separate answer-only setting rather than a controlled trace-removal ablation. The small GPQA-Diamond benchmark and in-sample full-vocabulary KL diagnostic provide supporting evidence, and the systems measurements depend on the hardware and implementation. More broadly, effective support should be measured for the teacher, data, and temperature of interest, then validated through generation. Once validated, it provides a direct path from concentrated teacher predictions to smaller offline targets and lower distillation cost.

Acknowledgments. We thank Saleh Ashkboos and Liana Mikaelyan for helpful discussions and reviews.

Appendix

What retained-support objectives optimize

We first decompose full-vocabulary KL at a fixed temperature, suppressing the common τ2\tau^2 factor, for a fixed, unmodified support SS. Let

M=∑v∈Spv,Q=∑v∈Sqv,M = \sum_{v \in S} p_v, \qquad Q = \sum_{v \in S} q_v,

and define the conditional distributions

pS(v)=pvM,qS(v)=qvQ(v∈S),p_S(v) = \frac{p_v}{M}, \qquad q_S(v) = \frac{q_v}{Q} \qquad(v \in S),

with pS‾p_{\overline{S}} and qS‾q_{\overline{S}} defined analogously on the complement. For a nonempty proper subset SS, softmax probabilities give 0<M,Q<10 < M,Q < 1. Splitting the KL sum over SS and S‾\overline{S} yields

∑v∈Spvlog⁡pvqv=MKL⁡(pS∥qS)+Mlog⁡MQ,\sum_{v \in S} p_v \log\frac{p_v}{q_v} = M \operatorname{KL}(p_S \| q_S) + M \log\frac{M}{Q},
∑v∉Spvlog⁡pvqv=(1−M)KL⁡(pS‾∥qS‾)+(1−M)log⁡1−M1−Q.\sum_{v \notin S} p_v \log\frac{p_v}{q_v} = (1-M) \operatorname{KL}(p_{\overline{S}} \| q_{\overline{S}}) + (1-M) \log\frac{1-M}{1-Q}.

The Bernoulli KL between the teacher and student masses assigned to SS is

KL⁡(Bern⁡(M)∥Bern⁡(Q))=Mlog⁡MQ+(1−M)log⁡1−M1−Q.\operatorname{KL}(\operatorname{Bern}(M) \| \operatorname{Bern}(Q)) = M \log\frac{M}{Q} + (1-M) \log\frac{1-M}{1-Q}.

Therefore,

KL⁡(p∥q)=MKL⁡(pS∥qS)+(1−M)KL⁡(pS‾∥qS‾)+KL⁡(Bern⁡(M)∥Bern⁡(Q)).\operatorname{KL}(p \| q) = M \operatorname{KL}(p_S \| q_S) + (1-M) \operatorname{KL}(p_{\overline{S}} \| q_{\overline{S}}) + \operatorname{KL}(\operatorname{Bern}(M) \| \operatorname{Bern}(Q)).

The identity extends to S=VS = V by continuity. Renormalized KL optimizes only KL⁡(pS∥qS)\operatorname{KL}(p_S \| q_S). It discards both the conditional tail term and the binary term that matches total mass on SS. The partial-mass objective is exactly

Lmass=MKL⁡(pS∥qS)+Mlog⁡MQ.\mathcal{L}_{\mathrm{mass}} = M \operatorname{KL}(p_S \| q_S) + M \log\frac{M}{Q}.

It retains pressure on the student’s total selected mass QQ, but omits the conditional tail and the complementary contribution (1−M)log⁡((1−M)/(1−Q))(1-M)\log((1-M)/(1-Q)).

Student-logit gradients. Let qτ=softmax⁡(zS/τ)q^\tau= \operatorname{softmax}(z^S/\tau), and treat the teacher distribution and selected set as fixed. Since

∂log⁡qvτ∂zjS=1τ(1{j=v}−qjτ),\frac{\partial\log q_v^\tau}{\partial z_j^S} = \frac{1}{\tau}\left(\mathbb{1}\{j=v\} - q_j^\tau\right),

the τ2\tau^2-scaled full-vocabulary objective has gradient

∂Lfull∂zjS=τ(qjτ−pjτ).\frac{\partial\mathcal{L}_{\mathrm{full}}}{\partial z_j^S} = \tau(q_j^\tau- p_j^\tau).

For renormalized KL,

∂Lrenorm∂zjS={τ(qˉj−pˉj),j∈S,0,j∉S.\frac{\partial\mathcal{L}_{\mathrm{renorm}}}{\partial z_j^S} = \begin{cases} \tau(\bar{q}_j-\bar{p}_j), & j \in S,\\ 0, & j \notin S. \end{cases}

For the partial-mass objective,

∂Lmass∂zjS=τ(Mqjτ−pj1{j∈S}),\frac{\partial\mathcal{L}_{\mathrm{mass}}}{\partial z_j^S} = \tau(Mq_j^\tau- p_j\mathbb{1}\{j \in S\}),

as also derived for cached partial-mass targets by Ryskulov et al. [19]. Thus renormalized KL has no student-logit gradient outside SS, whereas the partial-mass objective still acts on all VV logits through the full-vocabulary normalizer.

Why rank selection maximizes retained mass. Let π\pi be a permutation of the token identifiers such that

pπ(1)≥pπ(2)≥⋯≥pπ(V),p_{\pi(1)} \ge p_{\pi(2)} \ge\cdots\ge p_{\pi(V)},

and define Skrank={π(1),…,π(k)}S^{\mathrm{rank}}_k = \{\pi(1), \ldots, \pi(k)\}. Consider any set S⊆{1,…,V}S \subseteq\{1, \ldots, V\} with ∣S∣=k|S| = k. If S≠SkrankS \ne S^{\mathrm{rank}}_k, equal cardinality implies that some π(i)\pi(i) with i≤ki \le k is absent from SS, while some π(j)\pi(j) with j>kj > k is present. Replacing π(j)\pi(j) by π(i)\pi(i) changes the retained mass by

pπ(i)−pπ(j)≥0.p_{\pi(i)} - p_{\pi(j)} \ge0.

Repeating this exchange finitely many times transforms SS into SkrankS^{\mathrm{rank}}_k without decreasing its mass. Therefore,

Skrank∈arg⁡max⁡S⊆{1,…,V}∣S∣=k ∑v∈Spv.S^{\mathrm{rank}}_k \in\underset{\substack{S \subseteq\{1,\ldots,V\}\\ |S|=k}}{\arg\max}\ \sum_{v\in S} p_v.

and another size-kk set can attain the same mass only when teacher probabilities are tied across the selection boundary.

Qualification for the Qwen3 correction. The decomposition above is exact for a literal restriction of pp to SS. The Qwen3 correction enlarges the set and modifies inserted teacher weights. The gradient forms remain valid after replacing SS, the teacher weights, and their total mass by their corrected counterparts, but the result is no longer a decomposition of full KL for the original teacher distribution.

Gumbel-top-kk support sampling

This appendix derives the sampling law used by the teacher-sampled support check. For one supervised position, define

av=zvT/τ,wv=exp⁡(av),pv=wv∑uwu.a_v = z_v^{\mathsf{T}}/\tau,\qquad w_v = \exp(a_v),\qquad p_v = \frac{w_v}{\sum_u w_u}.

Let Ev∼Exp⁡(1)E_v \sim\operatorname{Exp}(1) independently and set gv=−log⁡Evg_v=-\log E_v. Since

Pr⁡(gv≤x)=Pr⁡(Ev≥e−x)=exp⁡(−e−x),\Pr(g_v \le x) = \Pr(E_v \ge e^{-x}) = \exp(-e^{-x}),

each gvg_v is a standard Gumbel random variable. The perturbed score can be written as

av+gv=log⁡wv−log⁡Ev=−log⁡(Evwv).a_v + g_v = \log w_v - \log E_v = -\log\left(\frac{E_v}{w_v}\right).

The random variable Tv=Ev/wvT_v=E_v/w_v is exponential with rate wvw_v. Sorting the perturbed scores from largest to smallest is therefore equivalent to sorting the independent arrival times TvT_v from smallest to largest. The probability that token ii arrives first is

Pr⁡(Ti=min⁡vTv)=∫0∞wie−wit∏v≠ie−wvt dt=wi∑vwv=pi.\Pr\left(T_i=\min_v T_v\right) = \int_0^\infty w_i e^{-w_i t}\prod_{v\ne i}e^{-w_v t}\,dt = \frac{w_i}{\sum_v w_v}=p_i.

After an arrival, the memoryless property of exponential random variables gives the same race over the remaining tokens. Hence the probability of an ordered sample (i1,…,ik)(i_1,\ldots,i_k) is

Pr⁡(i1,…,ik)=∏j=1kwij∑v∉{i1,…,ij−1}wv.\Pr(i_1,\ldots,i_k)=\prod_{j=1}^{k}\frac{w_{i_j}}{\sum_{v\notin\{i_1,\ldots,i_{j-1}\}}w_v}.

This is the Plackett–Luce distribution over rankings [26, 27]. Let πg\pi^g order the perturbed scores zvT/τ+gvz_v^{\mathsf{T}}/\tau+g_v from largest to smallest. Then, up to events of probability zero from ties, (πg(1),…,πg(k))(\pi^g(1),\ldots,\pi^g(k)) is an exact ordered sample of kk distinct tokens without replacement from the teacher distribution, with unordered support S={πg(1),…,πg(k)}S=\{\pi^g(1),\ldots,\pi^g(k)\} [20].

Qwen3 support correction. The exact sampling statement applies to the raw set SS. Our loss then adds each Qwen3 think-tag identifier when it is absent [10], producing a corrected set S~\widetilde{S} with kk, k+1k+1, or k+2k+2 elements. It also modifies the inserted teacher weights as described in § 4. Consequently, S~\widetilde{S} is not itself distributed according to the Gumbel-top-kk law.

Relation to importance sampling. Even before the support correction, sampling SS does not make the resulting truncated loss an unbiased estimator of full-vocabulary KL. For the partial-mass objective, let

hv=pvlog⁡pvqv,ιv=Pr⁡(v∈S).h_v = p_v \log\frac{p_v}{q_v}, \qquad\iota_v = \Pr(v \in S).

Then

ES[∑v∈Shv]=∑v=1Vιvhv\mathbb{E}_S\left[\sum_{v \in S} h_v\right] = \sum_{v=1}^{V} \iota_v h_v

which differs from ∑vhv\sum_v h_v unless the terms are corrected by their inclusion probabilities. The renormalized objective is further nonlinear in SS, because both teacher and student normalizers depend on the sampled set:

ES[KL⁡(p(⋅∣S)∥q(⋅∣S))]≠KL⁡(p∥q)\mathbb{E}_S\left[\operatorname{KL}\left(p(\cdot\mid S)\mathbin{\|}q(\cdot\mid S)\right)\right] \ne\operatorname{KL}(p\mathbin{\|}q)

in general. Random Sampling KD instead uses proposal corrections to preserve the full-KL gradient in expectation [6]. Our teacher-sampled variant uses no such correction and is only a robustness check on support construction.

References

  1. [1]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. 2015. URL https://arxiv.org/abs/1503. 02531.
  2. [2]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/8ac015d409635f196f9e3e9dcbf9a94e-Abstract-Conference.html.DOI
  3. [3]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2306.13649.
  4. [4]Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. DistillLM: Towards streamlined distillation for large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 24872–24895, 2024. URL https://proceedings.mlr.press/v235/ko24c.html.arxiv.org/abs/2402.03898
  5. [5]KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, and Muhammad Omer Raza. FIRST: Teach a reliable large language model through efficient trustworthy distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12646–12659, 2024. doi: 10.18653/v1/2024.emnlp-main.703. URL https://aclanthology.org/2024.emnlp-main.703/.DOI
  6. [6]Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, and Joohyun Lee. Sparse logit sampling: Accelerating knowledge distillation in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 18085–18108, 2025. doi: 10.18653/v1/2025.acl-long.885. URL https://aclanthology.org/2025.acl-long.885/.
  7. [7]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. URL https://proceedings.neurips.cc/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.arxiv.org/abs/2201.11903
  8. [8]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL https://arxiv.org/abs/2406.01574.
  9. [9]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. Conference on Language Modeling (COLM), 2024. URL https: //arxiv.org/abs/2311.12022.arxiv.org/abs/2311.12022
  10. [10]Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  11. [11]NVIDIA. NVIDIA Nemotron Nano 2: An accurate and efficient hybrid Mamba–Transformer reasoning model, 2025. URL https: //arxiv.org/abs/2508.14444.
  12. [12]Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2407.14679.
  13. [13]Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 14852–14882, 2023. doi: 10.18653/v1/2023.acl-long.830. URL https://aclanthology.org/2023.acl-long.830/.DOI
  14. [14]Wei-Rui Chen, Vignesh Kothapalli, Ata FatahiBaarzi, Hejian Sang, Shao Tang, Qingquan Song, Zhipeng Wang, and Muhammad Abdul-Mageed. Distilling the essence: Efficient reasoning distillation via sequence truncation. In Findings of the Association for Computational Linguistics: ACL 2026, pages 12092–12122, 2026. doi: 10.18653/v1/2026.findings-acl.587. URL https://aclanthology.org/2026.findings-acl.587/.DOI
  15. [15]Mrigank Raman, Pranav Mani, Davis Liang, and Zachary C. Lipton. For distillation, tokens are not all you need. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https://openreview.net/forum?id=2fc5GOPYip.
  16. [16]Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang, Lei Hou, and Juanzi Li. Pre-training distillation for large language models: A design space exploration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 3603–3618, 2025. doi: 10.18653/v1/2025.acl-long.181. URL https://aclanthology.org/2025.acl-long.181/.DOI
  17. [17]Tianyu Peng and Jiajun Zhang. Enhancing knowledge distillation of large language models through efficient multi-modal distribution alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2478–2496, 2025. URL https://aclanthology.org/2025.coling-main.169/.arxiv.org/abs/2409.12545
  18. [18]Almog Tavor, Itay Ebenspanger, Neil Cnaan, and Mor Geva. Rethinking selective knowledge distillation. 2026. URL https://arxiv.org/abs/2602.01395.
  19. [19]Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, and Román Orús. Efficient knowledge distillation for LLMs: Offline top-k logits and a fused chunked KL loss. 2026. URL https://arxiv.org/abs/2608.03796.
  20. [20]Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The Gumbel-top-k trick for sampling sequences without replacement. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 3499–3508, 2019. URL https://proceedings.mlr.press/v97/kool19a.html.
  21. [21]Lavinia Ghita, Dhruv Desai, and Ioana Boier. Scaling laws for task-specific LLM distillation. 2026. URL https://arxiv.org/abs/2606.24747.
  22. [22]NVIDIA. Nemotron post-training dataset v2. https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v2, 2025.
  23. [23]NVIDIA. Puzzle-KD-Nemotron-Post-Training-Dataset-v2. https://huggingface.co/datasets/nvidia/Puzzle-KD-Nemotron-Post-Training-Dataset-v2, 2026.
  24. [24]John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159, 2011.
  25. [25]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  26. [26]R. Duncan Luce. Individual Choice Behavior: A Theoretical Analysis. John Wiley & Sons, New York, 1959.
  27. [27]Robin L. Plackett. The analysis of permutations. Applied Statistics, 24(2):193–202, 1975. doi: 10.2307/2346567.DOI

Paper details

vlnc:W_atbpve65
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents