betaFeedback

WorksComputer Science

SilverSpoon-KD: A General-Purpose Toolkit for Knowledge Distillation

Abstract

Knowledge distillation transfers deep learning representations from a teacher model to a student model, typically reducing model size or inference cost. However, knowledge distillation tooling has not kept pace with the breadth of research methods: existing toolkits typically support only a limited subset of techniques and use cases. We present SilverSpoon-KD, an open-source PyTorch toolkit that unifies different distillation paradigms and architectural relationships across workloads and applications behind a single API. Moreover, SilverSpoon-KD offers the first general-purpose blockwise knowledge distillation implementation with unique scalability features. We validate its versatility across vision, masked language model, and causal language models on tasks spanning homogeneous, heterogeneous, and quantization distillation across paradigms. We also evaluate SilverSpoon-KD by comparing its performance, including throughput and GPU memory usage, with existing toolkits. Despite extensive support for convenience, metric calculation, and tracking, we have found SilverSpoon-KD's performance to be on par with or better than existing toolkits.

Introduction

Knowledge distillation (KD) [1] is a family of techniques for transferring learned representations from a teacher model (the trained source) to a student model (the target being trained, typically smaller), primarily to reduce inference cost (Figure 1a). KD workflows vary along three largely orthogonal axes (Figure 1b–Figure 1d): the workload (general task or training objective), the architectural relationship between teacher and student, and the distillation paradigm (how signals flow from teacher to student). Architectural relationships span homogeneous compression (same architecture family, smaller student) [2, 3], heterogeneous transfer (different architecture families with distinct internal operations) [4, 5], quantization-based distillation [6], and self-distillation [7, 8]. These techniques are central to deploying large models at scale, with successes spanning compressed CNN image classifiers [9], encoder language models like DistilBERT [2], and the distillation of modern LLMs [10].

Knowledge distillation at a glance

Figure 1. Knowledge distillation at a glance. KD spans diverse workloads, architectural relationships, and paradigms; existing tooling covers narrow slices of this combinatorial space, motivating a general-purpose toolkit. Panel (d) shows the four paradigms used throughout the paper: response-based (ResKD), feature-based (FeatKD), holistic (HKD), blockwise (BKD), and relation-based (RelKD).

The paradigm axis comprises four families. Response-based KD is the most fundamental approach, matching the teacher’s output distribution [1]. By contrast, feature-based KD aligns intermediate representations either through holistic KD1 [11] or blockwise KD [12], with isolated student blocks trained with teacher-forced inputs. Finally, relation-based KD [13] matches inter-sample relationships and can compose orthogonally with any of the other paradigms.

Despite the breadth and maturity of KD research, tooling has arguably not kept pace: response-based KD is well-served, but feature-based KD demands capturing intermediate tensors that vary significantly across architectures, often forcing researchers to build custom pipelines per project. A few dedicated toolkits support only holistic KD, and to our knowledge, no existing toolkit facilitates blockwise KD out of the box. KD tools have also lagged behind the broader deep learning ecosystem: the community increasingly expects scalable training infrastructure with built-in evaluation, logging, and multi-device support—as exemplified by Hugging Face’s Transformers [14]. However, current KD tooling often lacks these engineering conveniences.

This paper presents SilverSpoon-KD2, a toolkit designed to be a comprehensive solution for KD. Our key contributions are:

  • Multi-workload, multi-architecture, multi-paradigm coverage in one API: SilverSpoon-KD supports key workloads (vision, masked LM, causal LM), all architectural relationships, and all four KD paradigms behind a single API. A general-purpose abstraction is a non-trivial contribution (removing hard-coded assumptions while keeping the API ergonomic).

  • First general-purpose blockwise KD implementation: To our knowledge, SilverSpoon-KD is the first open-source toolkit to provide a general abstraction for blockwise KD. Prior implementations have been narrowly scoped and tied to specific use-cases, leaving users to build custom pipelines.

  • Practical engineering breadth with measured performance gains: SilverSpoon-KD interoperates with PyTorch [15], the HuggingFace Trainer, and the standard parallelism stack (DDP, FSDP, DeepSpeed). It offers flexible teacher placement, automatic teacher–student dimension matching via projectors, asynchronous teacher–student CUDA streams, and built-in evaluation. The resulting throughput and memory gains over existing toolkits are modest but consistently present across all three workloads; they accrue primarily from the sum of many parts rather than from any single optimization, which is notable given SilverSpoon-KD’s broader feature surface.

Related Work

Several open-source toolkits provide varying degrees of support for KD, though none currently offer the combination of workload, architecture coverage, paradigm breadth, and scalability features that

SilverSpoon-KD targets (see Table 1). The table records which features each toolkit supports, not how well each is implemented; performance benchmarks in § 4.3 address the latter.

FeatureSilverSpoon-KDtorchdistillTextBrewertorchtuneDistillKitEasyDistill
Distillation paradigms
Response-based KD✓✓✓✓✓✓
Holistic feature KD✓✓✓✓
Blockwise feature KD✓
Relation-based KD✓✓
Workload support
Encoder NLP (BERT, etc.)✓✓✓
Decoder LLM (GPT-2, Qwen, etc.)✓✓§✓✓✓
Vision (CNN, ViT)✓✓
Training features
Auto dimension projectors✓✓
Sub-module alignment (e.g. attention)✓✓
Cross-architecture distillation✓✓
Quantization-aware training✓*✓
Multi-teacher distillation✓✓
Teacher output caching✓✓✓
Built-in evaluation✓✓✓
Scalability
DDP (student)✓✓✓✓
FSDP (student)✓✓
DeepSpeed (student)✓✓✓
FSDP sharding (teacher)✓✓
Pipeline / tensor parallel (teacher)✓
Split-GPU placement (teacher)✓
Chunked KD losses✓✓
Fused kernel KD losses✓

Table 1. Feature comparison of knowledge distillation toolkits. SilverSpoon-KD provides the broadest feature surface across the five compared toolkits, including the only general-purpose blockwise KD implementation and FSDP-sharded teacher placement.

General-purpose KD toolkits. The five toolkits in Table 1 are the most directly comparable general-purpose KD libraries. torchdistill [16] is the broadest method library but does not target causal Large Language Models (LLMs) or blockwise training; TextBrewer [17] specializes in holistic KD for encoder NLP, with feature alignment restricted to higher-level module outputs; torchtune [18] and DistillKit [19] target decoder LLMs, the latter adding hidden-state matching with automatic projectors and logit-compressed teacher caching; and EasyDistill [20] emphasizes black-box pipelines (training on teacher-generated text) alongside multi-teacher aggregation and logit caching.

Adjacent toolkits. Several other toolkits are in an adjacent KD space rather than competing as general-purpose KD toolkits, and so are noted here but excluded from Table 1. RepDistiller [21] is a research benchmark, not a toolkit: it bundles 12 vision KD methods under shared infrastructure for reproducibility studies rather than as a substrate for building new pipelines, and its repository has been unmaintained for years. The other three are broader model-compression libraries that treat limited KD methods as one technique among several. KD-Lib [22] bundles vision KD with pruning (removing redundant weights or neurons), quantization, and integrated hyperparameter tuning, with no decoder LLM or blockwise KD support. NVIDIA ModelOpt [23] integrates LLM logit distillation into NeMo alongside quantization and pruning. Finally, Intel Neural Compressor [24] bundles KD with broader compression across PyTorch and TensorFlow. All three of the latter prioritize deployment optimization over KD methodology breadth, so a feature-by-feature comparison against SilverSpoon-KD’s KD-focused surface would mostly measure scope rather than capability.

Blockwise Knowledge Distillation. Blockwise KD trains student blocks in isolation against corresponding teacher blocks, decomposing a difficult end-to-end optimization problem into simpler per-block regression tasks. Wang et al. [12] and Koratana et al. [25] introduced the sequential and isolated per-block formulations; Blakeney et al. [26] and Jang et al. [27] extended these to parallel and pipelined variants; Shen et al. [28] explored progressive network grafting; and Kao et al. [29] and Yu et al. [30] addressed inter-block optimization entanglement and feature-level decoupling. Despite this active research, existing blockwise KD implementations remain tightly coupled to specific experimental setups; SilverSpoon-KD aims to provide a general-purpose blockwise KD toolkit that can accommodate these diverse strategies.

System Design and Usage

We first cover SilverSpoon-KD’s internal architecture (§ 3.1) and then its programmatic API (§ 3.2).

Design

SilverSpoon-KD is built on top of the HuggingFace Trainer, which lets the distiller inherit general-purpose deep learning boilerplate while adding KD-specific logic. Figure 2a traces the distiller’s training-step call flow, with nodes color-coded by provenance. The KD-specific surface area is intended to be concentrated to leverage the general training logic provided by Hugging Face. Beyond this scaffolding, three components enable effective KD, each described in turn below.

SilverSpoon-KD design overview showing the training flow, alignment mechanism, parallelism, and auto-projectors

Figure 2. SilverSpoon-KD design overview. (a) Training-flow spine, methods color-coded by provenance (inherited, added, extended). (b) Regex/list alignment hooks bind teacher–student modules. (c) Independent placement overlaps teacher/student forwards on disjoint GPUs. (d) Auto-projectors bridge hidden-dim mismatches.

(1) Alignments (Figure 2b) map which student modules learn from which teacher modules. SilverSpoon-KD attaches forward hooks that record module inputs and outputs without modifying the underlying models, and a pairing API (module-name list or regex, if more convenient) targets any granularity—whole layers or embedded sub-modules—even across heterogeneous teacher–student naming. (2) Parallelism (Figure 2c) places teacher and student under selectable, independent regimes across devices and uses separate CUDA streams so their forward passes overlap; this works because the two forwards have no data dependency until the loss step. (3) Auto-projectors (Figure 2d) detect teacher–student hidden-dimension mismatches at alignment time and auto-build learnable projectors trained jointly with the student. Among compared toolkits, this combination of arbitrary-granularity alignment and overlapping multi-placement teacher–student forwards is, to our knowledge, unique; DistillKit is the only other compared toolkit with automatic projectors.

The load-bearing distinction between holistic and blockwise feature-based KD is whether each student block receives the previous student block’s output (full end-to-end student forward pass; sequential gradients) or a detached teacher-forced input (per-block-independent gradients; lower peak memory; parallelizable across alignments). Appendix § 7 visualizes the gradient-flow topologies.

Usage

In contrast to most general-purpose KD toolkits (torchdistill, torchtune, DistillKit, EasyDistill), which expose their main entry point as a YAML or JSON configuration schema, SilverSpoon-KD’s API is intentionally programmatic Python. This keeps integration with bespoke training code direct. Users can embed a distiller inside an existing HuggingFace Trainer workflow, mix in custom callbacks or evaluation hooks, and avoid being locked into a fixed configuration format. The algorithmic difference between paradigms is hidden behind a single class swap (Figure 3): the same alignment objects, training arguments, dataset, and teacher/student models flow into either feature-based distiller, and response-based KD drops the alignments entirely. Paradigm-specific options are surfaced transparently through each constructor.

KD paradigms via class swap with Response-based KD, Holistic KD, and Blockwise KD code examples

Figure 3. KD paradigms via class swap. Switching between KD paradigms merely requires swapping the distiller class; alignments, training arguments, and dataset flow into all three. teacher, student, and train_dataset=ds assumed loaded; configurations are illustrative.

More information is available in the online documentation.5

# Logit matching only
distiller = ResponseBasedDistiller(
    teacher_model=teacher,
    student_model=student,
    soft_loss_fn=kl_divergence_loss(
        temperature=4.0,
    ),
    train_dataset=ds,
    args=TrainingArguments(alpha=0.5),
)
distiller.train()
# Hook layers + lm_head (final logits)
alignments = create_alignments(
    teacher, student,
    modules=r"^(layers\.\d+|lm_head)$",
)
for i, a in enumerate(alignments):
    a.loss_weight = i + 1
distiller = HolisticDistiller(
    teacher_model=teacher,
    alignments=alignments,
    train_dataset=ds,
    args=TrainingArguments(),
)
distiller.train()
# Hook all transformer layers
alignments = create_alignments(
    teacher, student,
    modules=r"^layers\.\d+$",
)
distiller = BlockwiseDistiller(
    teacher_model=teacher,
    alignments=alignments,
    train_dataset=ds,
    args=TrainingArguments(),
)
distiller.train()

Experiments

We evaluate SilverSpoon-KD across eight distillation-quality experiments and a ninth experiment benchmarking throughput and peak memory against four existing KD libraries. Figure 4 summarizes the axes each experiment stresses and the SilverSpoon-KD capabilities it exercises; full per-experiment results are in Appendix § 9.

Experimental coverage matrix

Figure 4. Experimental coverage matrix. The eight distillation-quality experiments above the dashed line span vision, encoder LM, and decoder LM workloads under homogeneous, heterogeneous, and quantization-based settings, exercising all four paradigms; the toolkit-performance benchmark below the line tests SilverSpoon-KD against four peers under matched workloads. Pills: Res response-based, H holistic, B blockwise, Rel relation-based KD.

Experimental Setup

This experimental program addresses three research questions tied to the contributions in the introduction: (RQ1) Does the unified API support non-trivial distillation across workloads, architectures, and paradigms? (RQ2) Does SilverSpoon-KD’s blockwise KD implementation deliver useful results across compression regimes, including in multi-stage runs? (RQ3) Does the toolkit’s engineering breadth translate into measured throughput and memory gains over existing libraries? Figure 4’s rightmost column flags which experiments speak to each question.

The eight distillation-quality experiments are positioned as workflow demonstrations rather than method-novelty studies: each follows a published recipe or typical KD setup, exercising the toolkit end-to-end through the same unified API to verify that it faithfully covers the workload–architecture–paradigm matrix of Figure 4, with head-to-head comparisons against established toolkit or reference papers where possible.

Most experiments were conducted on NVIDIA A40 GPUs (48GB) from the Alvis cluster at Chalmers University of Technology in Gothenburg, Sweden; the exception is in § 4.3, where short, hard-to-compute-saturate runs used NVIDIA A100s (40GB). Taken together, these experiments required approximately 2 000 GPU hours. Given the breadth of the workload, architecture, paradigm, and placement matrix, each reported configuration is a single-seed run, with per-experiment system metrics (peak memory, steps/s, wall time) reported alongside accuracy in Appendix § 9.

All the experiment code used for this paper can be found in a GitHub repository.6

WorkloadTeacherStudent(s)Architecture RelationParadigmsStagesRQ
§C.1 VGG CompressionCNNVGG-16VGG-16-DSHomogeneousRes H B✓1, 2
§C.2 VGG Relational KDCNNResNet-50VGG-11-BNHeterogeneousRes H Rel—1
§C.3 BERT Pre-TrainingLang / Enc.BERT-baseT6, T4-tinyHomogeneousRes H B—1, 2
§C.4 BERT DownstreamLang / Enc.BERT-baseT6, T4-tinyHomogeneousH B—1, 2
§C.5 GPT-2 CompressionLang / Dec.GPT-2 smallGPT-2 96MHomogeneousRes H B✓1, 2
§C.6 Cross-ArchitectureLang / Dec.GPT-2 smallDSV3 96MHeterogeneousRes H B✓1
§C.7 GPT-2 Attn. Linear.Lang / Dec.GPT-2 smallLoLCATsHeterogeneousRes H B✓1, 2
§C.8 Qwen3 QuantizationLang / LLMQwen3-1.7BINT4/5/6QuantizationRes H B—1, 2
§ 4.3 Toolkit Bench.mixedmixedmixedmixedRes H B—3

Table 4.

Distillation-Quality Results

Figure 5 summarizes the headline metric for each of the eight distillation-quality experiments. Full per-run results are reported in Appendix § 9.

Vision (Figure 5, panels 1–2). VGG Compression (Appendix § 9.§ 1) replicates Wang et al. [12] by distilling VGG-16 into a depthwise-separable VGG-16-DS on CIFAR-10 [31]; the BKD → ResKD pipeline reaches 90.42% (vs. teacher 87.26%, baseline 82.69%), with auto-projectors handling channel-dimension mismatches transparently. This demonstrates that the block abstraction generalizes to convolutional networks. VGG Relational KD (Appendix § 9.§ 2) replicates Park et al. [13] (ResNet-50 → VGG-11-BN on CIFAR-100), exercising SilverSpoon-KD’s pluggable loss-function architecture: ResKD leads at 73.65% and the relation-based variants (distance, angle, distance+angle) span 70.40–71.61%, all reachable through configuration-only changes.

Per-experiment results summary

Figure 5. Per-experiment results summary. Colored bars are SilverSpoon-KD runs: single-stage bars use paradigm colors (BKD, HKD, ResKD, RelKD), and multi-stage chains use green, with the chain (e.g., BKD→HKD→LoRA) shown in the row label. Gray bars are reference points (not SilverSpoon-KD): teacher (dark), no-distillation student baseline (light), and external comparison from a competing toolkit or reference paper (medium). FT = full fine-tuning; LoRA = low-rank adapters, base frozen.

BERT (Figure 5, panels 3–4). BERT Pre-Training (Appendix § 9.§ 3) distills BERT-base into bert.T6 (≈59% of teacher parameters) and bert.T4_tiny (≈13%) on Dolma; HKD wins at both compression levels but the gap widens dramatically with aggressiveness—the three paradigms cluster within ≈\approx1.3 percentage points (pp) at T6 (HKD +3.04pp MNLI [32]) and fan out to a 5.2pp spread at T4-tiny, where HKD remains strongest (+3.52pp) while BKD turns actively harmful, exposing a compression–paradigm interaction that the unified class-swap API made cheap to surface. BERT Downstream (Appendix § 9.§ 4) compares against TextBrewer on its own MNLI benchmark: HKD →\to fine-tune (FT) matches BERT-base within 0.12pp on T6 and beats TextBrewer by 3.38pp.

Decoder LLM, compression and transfer (Figure 5, panels 5–6). GPT-2 Compression (Appendix § 9.§ 5) aggressively shrinks an already-small 124M GPT-2 (n_inner 3,072 →\to 1,536, a 50% MLP-width reduction); standalone BKD diverges to perplexity (PPL) 245,281, but the three-stage BKD →\to HKD →\to LoRA pipeline rescues it (PPL 93.39, +28% LAMBADA [33] over baseline). Cross-Architecture (Appendix § 9.6) distills GPT-2 →\to DeepSeek-V3 across multi-head attention (MHA) vs. MLA, learned vs. RoPE, and GELU vs. SiLU; BKD was not feasible because per-block teacher-forced inputs land in a representation the student’s blocks cannot consume. The teacher–student performance gap is notable, drawing a practical limit on how heterogeneous KD can be.

Decoder LLM, capability showcases (Figure 5, panels 7–8). GPT-2 Attention Linearization (Appendix § 9.7) exercises a unique SilverSpoon-KD capability—granular sub-module targeting, where Blockwise Knowledge Distillation (BKD) aligns only the attention mechanism within each layer rather than the full layer output; BKD →\to FT (PPL 52.01) is the only configuration that beats the LoLCATs [34] no-KD floor of 53.49, and the experiment shows that comparable toolkits would require manual hook plumbing for this configuration. Qwen3 INT4 QAT (Appendix § 9.8) distills Qwen3-1.7B into a simulated quantized student: at INT4, HKD lifts MMLU [35] by 6.76pp toward the bf16 ceiling and delivers the lowest perplexity, ResKD edges out perplexity at INT5, and BKD provides distillation at zero memory overhead vs. QAT alone.

Toolkit Benchmarking

We benchmark SilverSpoon-KD against four comparison toolkits on a featured use case of each: torchdistill (vision, response-based), TextBrewer (encoder NLP, response-based and holistic), torchtune (decoder LLM, response-based+LoRA), and DistillKit (decoder LLM, holistic), matching each toolkit’s default setup as closely as possible; full per-toolkit configurations and additional placement sweeps are in Appendix § 9.9. The other toolkits in Table 1 are excluded for methodological reasons: EasyDistill’s pre-cached-teacher pipeline is incommensurable with our live-teacher comparison, and KD-Lib and RepDistiller target vision only.

Table 2: Head-to-head toolkit benchmark (fixed batch size, single GPU unless noted). SilverSpoon-KD wins peak memory in all 5 settings and wall time in 3/5. Δ\Delta columns are SilverSpoon-KD relative to comparison; negative is better. Bold = winner per metric.

SettingToolkitWall (s)Tput (samp/s)Peak Mem (MB)Δ\Delta WallΔ\Delta Mem
ResNet-50 →\to ResNet-18 / ImageNet (vision, Response-Based Knowledge Distillation (ResKD), DDP ×\times 2)torchdistill232.0551.71 341——
ResNet-50 →\to ResNet-18 / ImageNet (vision, Response-Based Knowledge Distillation (ResKD), DDP ×\times 2)SilverSpoon-KD219.6582.81 014−5.3%−24.4%
BERT-base →\to T6 / MNLI (encoder NLP, ResKD, DDP ×\times 2)TextBrewer130.1491.82 758——
BERT-base →\to T6 / MNLI (encoder NLP, ResKD, DDP ×\times 2)SilverSpoon-KD124.5514.12 747−4.3%−0.4%
BERT-base →\to T6 / MNLI (encoder NLP, Holistic Knowledge Distillation (HKD))TextBrewer (default)425.5300.86 360——
BERT-base →\to T6 / MNLI (encoder NLP, Holistic Knowledge Distillation (HKD))SilverSpoon-KD449.6284.76 355+5.7%−0.1%
Qwen3-4B →\to 0.6B / Dolma (decoder LLM, HKD, bf16)DistillKit (gradient ckpt)294.43.423 992——
Qwen3-4B →\to 0.6B / Dolma (decoder LLM, HKD, bf16)SilverSpoon-KD (Liger fused)288.63.520 750−2.0%−13.5%
Qwen3-4B →\to 0.6B / Dolma (decoder LLM, ResKD+LoRA, bf16)torchtune639.13.123 090——
Qwen3-4B →\to 0.6B / Dolma (decoder LLM, ResKD+LoRA, bf16)SilverSpoon-KD675.33.019 609+5.7%−15.1%

Table 2. Head-to-head toolkit benchmark.

SilverSpoon-KD is within ±\pm6% wall time of every comparison toolkit while using less GPU memory in all four (13–24.4% reductions). The capability frontier (Table 3) sweeps teacher size on 2×\timesA100: at modest scales (Qwen3-4B, 8B) all toolkits achieve comparable throughput, but at Qwen3-14B

TeacherToolkitBatchTput (tok/s)Peak Mem (MB)
Qwen3-4B (8 GB)DistillKit415 18437 967
TextBrewer515 78635 510
SilverSpoon-KD614 70037 333
Qwen3-8B (16 GB)DistillKit211 16133 027
TextBrewer311 51434 119
SilverSpoon-KD49 97534 842
Qwen3-14B (28 GB)DistillKitOOM
TextBrewer16 21737 379
SilverSpoon-KD36 81237 397

Table 3. Capability frontier as teacher size grows on 2×2 \times A100 (∼40\sim40 GB per-GPU), Qwen3-0.6B student, seq=1024. SilverSpoon-KD is the only setup that scales to Qwen3-14B with batch >1> 1; DistillKit OOMs at 14B. Bold = best per metric per teacher.

(28 GB bf16 weights), DistillKit fails to fit and TextBrewer collapses to bs=1, while SilverSpoon-KD’s FSDP-sharded teacher halves the per-GPU teacher footprint, fits 3×3 \times TextBrewer’s batch, and overtakes both on tokens/sec.

Discussion

Three cross-workload patterns emerge. First, multi-stage chaining wins consistently: essentially every workload’s headline result comes from a chain (BKD →\to ResKD on VGG, HKD →\to FT on BERT, BKD →\to HKD →\to LoRA on GPT-2), making SilverSpoon-KD’s easy setup a key composition mechanism rather than mere convenience. Second, paradigm–setting interactions emerge cheaply: BKD’s role shifts from competitive at mild BERT compression to actively harmful at aggressive compression, and which paradigm wins partitions across three axes under Qwen3 INT4 QAT, both surfaced through class-swap rather than re-implementation. Finally, the toolkit recovers workflows where distillation alone fails: standalone-BKD divergence on GPT-2 is rescued by chaining, and attention-linearization PPL inflation by granular sub-module targeting plus a fine-tune. Together with § 4.3, these address RQ1–RQ3 of § 4.

Limitations and Conclusion

SilverSpoon-KD assumes white-box access to the teacher—its hidden states and logits, computed live during student training—which rules out black-box regimes (where the student trains on teacher-generated text without logits) [36] and cached white-box pipelines that pre-compute teacher outputs offline; a run also currently trains a single student against a single teacher. QAT is supported via externally prepared quantized models, with native per-block precision reserved for future work. Methodological observations the evaluation surfaces (e.g., BKD →\to HKD pipelines recovering from standalone-blockwise divergence on autoregressive LLMs) open natural directions for focused study. Other promising extensions include heterogeneous teacher–student hardware, multi-teacher distillation, black-box pipelines for closed-weight teachers [20], teacher-output caching with logit compression, and a declarative (YAML/JSON) configuration layer.

SilverSpoon-KD unifies four KD paradigms behind a single API and provides, to our knowledge, the first general-purpose blockwise KD; with FSDP-sharded teacher placement, it fits 14B-parameter teachers on 2×2 \times A100 where compared toolkits run out of memory. The unified API also made several methodological observations cheap to surface: multi-stage chaining produced essentially every headline result; paradigm–setting interactions partitioned cleanly across compression and quantization regimes; and standalone-blockwise divergence on autoregressive LLMs was rescued by chaining—each obtained through configuration rather than re-implementation. SilverSpoon-KD enables systematic paradigm comparison that would otherwise require bespoke pipelines.

Acknowledgments and Disclosure of Funding

We thank Oscar Eriksson, Shubhra Mishra, Martin Orrje, Viktor Palmkvist, and John Wikman for insightful discussions that informed this work. This project was partially supported by the Digital Futures Research Center and by the Swedish Research Council (Vetenskapsrådet, grant no. 2024-05043). The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725.

Training Paradigm Visualizations

The following figures illustrate the forward and backward pass data flow for standard training, response-based KD, holistic KD, and blockwise KD, plus a multi-stage pipeline that chains paradigms by initializing each stage from the previous stage’s trained student. Black arrows denote the forward pass and are labeled with the block output hiSh_i^S (or hTjh_T^j) that they carry, where hiSh_i^S is the output of student block ii (equivalent to houtSh_{out}^S as used in the System Design algorithm boxes). Red arrows denote backpropagation and are labeled with the upstream activation gradient ∂L/∂hiS\partial\mathcal{L}/\partial h_i^S that they transmit during the chain-rule traversal. A small red annotation inside each student block shows the parameter gradient ∂L/∂θiS\partial\mathcal{L}/\partial\theta_i^S that is computed locally at that block from the arriving upstream gradient and subsequently used to update the block’s weights. Teacher parameters are frozen (no red arrows or in-block annotations).

Standard training from hard labels with a student model, input, blocks, loss, and student prediction

Figure 6. Standard training from hard labels (no teacher). The student’s learnable parameters θ\theta are updated via backpropagation to minimize the loss between its prediction and the ground-truth labels.

Holistic knowledge distillation diagram

Figure 8. holistic KD, a FeatKD strategy. The student performs a full end-to-end forward pass, and intermediate block outputs are compared with the corresponding teacher block outputs. Losses L1…N\mathcal{L}_{1\ldots N} all contribute gradients to θ1S\theta^{S}_{1} (since all subsequent outputs depend on Block 1), losses L2…N\mathcal{L}_{2\ldots N} update θ2S\theta^{S}_{2}, while θNS\theta^{S}_{N} only receives gradients from LN\mathcal{L}_{N}.

Multi-stage distillation pipeline showing blockwise KD, holistic KD, and LoRA fine-tuning

Figure 10. Multi-stage distillation pipeline (BKD → HKD → LoRA, the headline GPT-2 Compression configuration of Appendix § 9.§ 5). Each stage reuses the same frozen teacher TT and the trained student from the prior stage, switching distiller class to apply a different paradigm; the trained student weights θ(k)S\theta^{S}_{(k)} from stage kk become the initialization for stage k+1k+1. Stage 1 (BKD) trains all student blocks under per-block teacher-forced losses with no cross-block gradient flow; Stage 2 (HKD) keeps Stage 1’s per-layer alignments but propagates gradients end-to-end through the full student chain; Stage 3 freezes the post-Stage-2 student weights, attaches LoRA adapters ΔiS\Delta^{S}_{i} on top, and trains only the adapters under a single response-based loss (no feature alignments). The optimizer state is reset at every stage boundary. Per-paradigm internals (forward chains, parameter-gradient annotations, alignment-loss formulae) are detailed in Figure 7–Figure 9.

Response-based knowledge distillation with teacher and student models, soft-labels, loss, and student prediction

Figure 7. response-based KD: a teacher model produces soft-labels that the student learns to match. Only the student’s parameters θS\theta^{S} are updated; the teacher is frozen.

Blockwise knowledge distillation diagram

Figure 9. blockwise KD, a FeatKD strategy. Each student block receives the same input as the corresponding teacher block, and loss is calculated by comparing the resulting outputs. Since each student block is only used to calculate its own output and is not used for subsequent student blocks, only the configuration of student learnable parameters θiS\theta^{S}_{i} influences the loss Li\mathcal{L}_{i}, meaning backpropagation does not propagate to non-involved earlier student blocks.

Full Algorithms

This appendix gives the full formal algorithms referenced from § 3. These algorithms describe three distillation paradigms (each well-established in prior work) as they are exposed by SilverSpoon-KD’s unified API; the contribution of the toolkit is not the algorithms themselves but the shared infrastructure—CAPTUREFEATURES (Algorithm 2), learnable projectors, per-alignment optimizers, and the evaluation suite—that makes switching between them a single configuration change.

Algorithm 1 Response-Based Knowledge Distillation (ResKD)

Input: Teacher Model T\mathcal{T} with output logits zT=T(x)z^{T} = \mathcal{T}(x).

Input: Student Model S\mathcal{S} with output logits zS=S(x)z^{S} = \mathcal{S}(x).

Input: Training data X\mathcal{X}, ground-truth labels y\mathbf{y}.

Input: Soft loss Lsoft\mathcal{L}_{soft} (e.g., temperature-scaled KL divergence), hard-label weight α∈[0,1]\alpha\in[0,1].

Input: Optimizer Opt, max gradient norm gmaxg_{max}.

  1. procedure TRAINRESPONSEBASED(S,T,X,α)(\mathcal{S}, \mathcal{T}, \mathcal{X}, \alpha)

  2. Freeze all parameters of T\mathcal{T}.

  3. for each batch (x,y)∈X(x,\mathbf{y}) \in\mathcal{X} do

  4. zT←T(x)z^{T} \leftarrow\mathcal{T}(x) ▷ Teacher forward pass (no gradient)

  5. zS,Lhard←S(x,y)z^{S}, L_{hard} \leftarrow\mathcal{S}(x,\mathbf{y}) ▷ Student forward pass with hard loss

  6. Lsoft←Lsoft(zS,zT)L_{soft} \leftarrow\mathcal{L}_{soft}(z^{S},z^{T}) ▷ Soft distillation loss

  7. L←(1−α)⋅Lsoft+α⋅LhardL \leftarrow(1-\alpha)\cdot L_{soft}+\alpha\cdot L_{hard} ▷ α=0\alpha=0: pure distillation

  8. BACKPROPAGATE(L,θS)(L,\theta^{S})

  9. CLIPGRADIENTS(θS,gmax)(\theta^{S},g_{max})

  10. Opt.STEP

  11. end for

  12. return S\mathcal{S}

  13. end procedure

Algorithm 2 Block Capture Algorithm

Input: Model sequence B=(b1,…,bN)\mathcal{B}=(b_{1},\ldots,b_{N}) where bib_{i} implements fi(⋅;θi)f_{i}(\cdot;\theta_{i}).

Input: Input tensor xx.

Input: Set S⊂B\mathcal{S}\subset\mathcal{B} of blocks to capture, optional terminal block bstopb_{stop}.

Input: Boolean capture_inputscapture\_inputs: whether to record block inputs (used by blockwise KD).

Output: Dictionary D\mathcal{D} mapping blocks to captured tensors.

  1. procedure CAPTUREFEATURES(B,x,S,bstop,capture_inputs)(\mathcal{B},x,\mathcal{S},b_{stop},capture\_inputs)

  2. D←∅\mathcal{D}\leftarrow\varnothing

  3. h←x\mathbf{h}\leftarrow x ▷ Initialize hidden state with input

  4. for i←1i\leftarrow1 to NN do

  5. bcurr←bib_{curr}\leftarrow b_{i}

  6. if bcurr∈Sb_{curr}\in\mathcal{S} and capture_inputscapture\_inputs then

  7. D[bcurr]in←h\mathcal{D}[b_{curr}]_{in}\leftarrow\mathbf{h} ▷ Capture input to function fif_{i}

  8. end if

  9. h←fi(h;θi)\mathbf{h}\leftarrow f_{i}(\mathbf{h};\theta_{i}) ▷ Apply block function

  10. if bcurr∈Sb_{curr}\in\mathcal{S} then

  11. D[bcurr]out←h\mathcal{D}[b_{curr}]_{out}\leftarrow\mathbf{h} ▷ Capture output of function fif_{i}

  12. end if

  13. if bcurr=bstopb_{curr}=b_{stop} then

  14. break ▷ Early exit optimization

  15. end if

  16. end for

  17. return D\mathcal{D}

  18. end procedure

Algorithm 3 blockwise KD Algorithm
Input: Teacher Model $\mathcal{T}$ with blocks $\{f_j^{T}\}$.
Input: Student Model $\mathcal{S}$ with blocks $\{f_i^{S}\}$.
Input: Mapping $\mathcal{M} = \{(i,j)\}$ pairing Student block $i$ to Teacher block $j$.
Input: Output selectors $\sigma_i^{T},\sigma_i^{S}$ per alignment (extract relevant sub-tensors).
Input: Optional Projector sets $\Phi_{\mathrm{in}} = \{\phi_{\mathrm{in},i}\}$ and $\Phi_{\mathrm{out}} = \{\phi_{\mathrm{out},i}\}$ for each Student block $i$.
Input: Training data $\mathcal{X}$, Loss function $\mathcal{L}$, per-alignment Optimizers $\{\mathrm{Opt}_i\}$, max gradient norms $\{g_{\max,i}\}$.

1: procedure TRAINBLOCKWISE($\mathcal{S},\mathcal{T},\mathcal{M},\Phi,\mathcal{X}$)
2:   Freeze all parameters of $\mathcal{T}$.
3:   $\mathcal{B}_{\mathrm{req}} \leftarrow \{b_j^{T}\mid(i,j)\in\mathcal{M}\}$ ▷ Required Teacher blocks
4:   $b_{\mathrm{last}} \leftarrow \max_{\mathrm{order}}(\mathcal{B}_{\mathrm{req}})$ ▷ Last required Teacher block
5:   for each batch $\mathbf{x}\in\mathcal{X}$ do
6:     $\mathcal{D}^{T}\leftarrow\mathrm{CAPTUREFEATURES}(\mathcal{T},\mathbf{x},\mathcal{B}_{\mathrm{req}},b_{\mathrm{last}},\mathrm{true})$ ▷ Truncated teacher forward
7:     for each $(i,j)\in\mathcal{M}$ do ▷ Parallelizable: blocks are independent
8:       $\mathbf{h}_{\mathrm{in}}^{T}\leftarrow\mathcal{D}^{T}[b_j^{T}]_{\mathrm{in}}$ ▷ Teacher block input (detached)
9:       $\mathbf{h}_{\mathrm{out}}^{T}\leftarrow\sigma_i^{T}(\mathcal{D}^{T}[b_j^{T}]_{\mathrm{out}})$ ▷ Select from teacher output
10:      if $\phi_{\mathrm{in},i}\in\Phi_{\mathrm{in}}$ then ▷ Input projection: Teacher dim $\to$ Student dim
11:        $\mathbf{h}_{\mathrm{in}}^{S}\leftarrow\phi_{\mathrm{in},i}(\mathbf{h}_{\mathrm{in}}^{T})$
12:      else
13:        $\mathbf{h}_{\mathrm{in}}^{S}\leftarrow\mathbf{h}_{\mathrm{in}}^{T}$
14:      end if
15:      $\mathbf{h}_{\mathrm{out}}^{S}\leftarrow\sigma_i^{S}(f_i^{S}(\mathbf{h}_{\mathrm{in}}^{S};\theta_i^{S}))$ ▷ Student forward + select (teacher-forced input)
16:      if $\phi_{\mathrm{out},i}\in\Phi_{\mathrm{out}}$ then ▷ Output projection: Student dim $\to$ Teacher dim
17:        $\mathbf{h}_{\mathrm{pred}}\leftarrow\phi_{\mathrm{out},i}(\mathbf{h}_{\mathrm{out}}^{S})$
18:      else
19:        $\mathbf{h}_{\mathrm{pred}}\leftarrow\mathbf{h}_{\mathrm{out}}^{S}$
20:      end if
21:      $L_i\leftarrow\mathcal{L}(\mathbf{h}_{\mathrm{pred}},\mathbf{h}_{\mathrm{out}}^{T})$
22:      $\mathrm{BACKPROPAGATE}(L_i,\{\theta_i^{S},\phi_{\mathrm{in},i},\phi_{\mathrm{out},i}\})$ ▷ Per-block gradient, no cross-block flow
23:      $\mathrm{CLIPGRADIENTS}(\{\theta_i^{S},\phi_{\mathrm{in},i},\phi_{\mathrm{out},i}\},g_{\max,i})$
24:      $\mathrm{Opt}_i.\mathrm{STEP}$
25:    end for
26:  end for
27:  return $\mathcal{S},\Phi$
28: end procedure

Algorithm 4 holistic KD Algorithm

Input: Teacher Model T\mathcal{T} with blocks {fjT}\{f_j^\mathcal{T}\}. Input: Student Model S\mathcal{S} with blocks {fiS}\{f_i^\mathcal{S}\}. Input: Mapping M={(i,j)}\mathcal{M} = \{(i,j)\} pairing Student block ii to Teacher block jj. Input: Output selectors σiT,σiS\sigma_i^\mathcal{T}, \sigma_i^\mathcal{S} per alignment. Input: Optional Projector set Φout={ϕout,i}\Phi_{out} = \{\phi_{out,i}\} for each Student block ii. Input: Training data X\mathcal{X}, Loss function L\mathcal{L}, per-alignment loss weights {wi}\{w_i\}. Input: Optimizer Opt⁡\operatorname{Opt} over the full student (and projectors Φout\Phi_{out}), max gradient norm gmaxg_{max}. Input: Boolean magnitude_aware: normalize losses by magnitude before weighting.

  1. procedure TRAINHOLISTIC(S,T,M,Φout,X\mathcal{S}, \mathcal{T}, \mathcal{M}, \Phi_{out}, \mathcal{X})

  2. Freeze all parameters of T\mathcal{T}.

  3. BreqT←{bjT∣(i,j)∈M}\mathcal{B}_{req}^{\mathcal{T}} \gets\{b_j^\mathcal{T} \mid(i,j) \in\mathcal{M}\} ▷ Required Teacher blocks

  4. BreqS←{biS∣(i,j)∈M}\mathcal{B}_{req}^{\mathcal{S}} \gets\{b_i^\mathcal{S} \mid(i,j) \in\mathcal{M}\} ▷ Required Student blocks

  5. for each batch x∈Xx \in\mathcal{X} do

  6. DT←CAPTUREFEATURES⁡(T,x,BreqT,None,false)\mathcal{D}^{\mathcal{T}} \gets\operatorname{CAPTUREFEATURES}(\mathcal{T}, x, \mathcal{B}_{req}^{\mathcal{T}}, \text{None}, \text{false}) ▷ Teacher outputs only (detached)

  7. DS←CAPTUREFEATURES⁡(S,x,BreqS,None,false)\mathcal{D}^{\mathcal{S}} \gets\operatorname{CAPTUREFEATURES}(\mathcal{S}, x, \mathcal{B}_{req}^{\mathcal{S}}, \text{None}, \text{false}) ▷ Student outputs only (gradients retained)

  8. Ltotal←0L_{total} \gets0

  9. for each (i,j)∈M(i,j) \in\mathcal{M} do

  10. houtT←σiT(DT[bjT]out)h_{out}^{\mathcal{T}} \gets\sigma_i^\mathcal{T}(\mathcal{D}^{\mathcal{T}}[b_j^\mathcal{T}]_{out}) ▷ Select from teacher output (detached)

  11. houtS←σiS(DS[biS]out)h_{out}^{\mathcal{S}} \gets\sigma_i^\mathcal{S}(\mathcal{D}^{\mathcal{S}}[b_i^\mathcal{S}]_{out}) ▷ Select from student output (with gradient)

  12. if ϕout,i∈Φout\phi_{out,i} \in\Phi_{out} then ▷ Output projection: Student dim →\to Teacher dim

  13. hpred←ϕout,i(houtS)h_{pred} \gets\phi_{out,i}(h_{out}^{\mathcal{S}})

  14. else

  15. hpred←houtSh_{pred} \gets h_{out}^{\mathcal{S}}

  16. end if

  17. Li←L(hpred,houtT)L_i \gets\mathcal{L}(h_{pred}, h_{out}^{\mathcal{T}})

  18. if magnitude_aware then ▷ Normalize so gradients ∝wi\propto w_i

  19. Ltotal←Ltotal+wi⋅Li/∣Li∣detachL_{total} \gets L_{total} + w_i \cdot L_i / \lvert L_i \rvert_{\text{detach}}

  20. else

  21. Ltotal←Ltotal+wi⋅LiL_{total} \gets L_{total} + w_i \cdot L_i

  22. end if

  23. end for

  24. BACKPROPAGATE⁡(Ltotal,{θS,Φout})\operatorname{BACKPROPAGATE}(L_{total}, \{\theta^\mathcal{S}, \Phi_{out}\}) ▷ End-to-end gradient flow

  25. CLIPGRADIENTS⁡({θS,Φout},gmax)\operatorname{CLIPGRADIENTS}(\{\theta^\mathcal{S}, \Phi_{out}\}, g_{max}) ▷ Single global gradient clip

  26. Opt.STEP⁡\operatorname{Opt.STEP} ▷ Single optimizer step over the full student

  27. end for

  28. return S,Φout\mathcal{S}, \Phi_{out}

  29. end procedure

Detailed Experimental Results

This appendix presents the full per-experiment results for the eight distillation-quality experiments summarized in § 4.4. Each subsection corresponds to one experiment and contains the experimental setup, the full results table, and a discussion of the findings. Subsection ordering matches the canonical ordering of the companion experiments repository and the experimental coverage matrix (Figure 4). Throughout, model and dataset identifiers in monospace (e.g., Qwen/Qwen3-4B, timm/vgg16.tv_in1k, allenai/dolma3-pool) are HuggingFace Hub paths.

VGG Compression

This experiment validates that SilverSpoon-KD supports blockwise KD and works beyond transformer workflows by distilling a VGG-16 teacher (timm/vgg16.tv_in1k) into a VGG-16-DS student that replaces standard convolutions with depthwise-separable convolutions on CIFAR-10, replicating the setup of [12], one of the foundational papers for blockwise KD. Twelve cross-depth alignment points span the internal convolutional layers; SilverSpoon-KD automatically inserts input and output projectors to bridge channel-dimension mismatches, and BKD additionally uses copy_matching initialization to seed the student with the teacher’s classifier head and first convolution (frozen). We compare single-paradigm runs (blockwise KD; response-based KD with temperature T=2T=2 and soft/hard mixing α=0.5\alpha=0.5) and three two-stage pipelines (blockwise KD →\to fine-tune, blockwise KD →\to holistic KD, blockwise KD →\to response-based KD) against a no-teacher baseline and the VGG-16 teacher ceiling. All runs use AdamW (LR 10−310^{-3}, cosine schedule with 5% warmup, weight decay 0, max grad norm 1.0), batch size 128, and float32 precision; single-stage and stage-2 distillation runs train 100 epochs while stage-2 fine-tuning runs 60 epochs.

MethodTest Acc. (%)Peak Mem (MB)Steps/sTime (h)
Baseline (no teacher)82.6969229.090.37
BKD72.499 99132.600.33
BKD →\to Fine-tune88.9269228.450.23
BKD →\to HKD87.971 20014.900.73
ResKD84.3572020.160.54
BKD →\to ResKD90.421 04818.160.60
Teacher (VGG-16)87.2673481.540.13

Table 4 (PDF p. 20). VGG Compression on CIFAR-10. BKD →\to ResKD wins accuracy at 90.42% (+3.16pp over teacher). Bold = best per column among non-teacher rows.

Multi-stage chaining delivers gains beyond the teacher ceiling: the best result, blockwise KD →\to response-based KD (90.42%), exceeds the VGG-16 teacher (87.26%) by 3.16pp and the no-teacher baseline (82.69%) by 7.73pp. Standalone blockwise KD dips below baseline (72.49%), but the same blockwise initialization followed by fine-tuning (88.92%), holistic KD (87.97%), or response-based KD (90.42%) all yield strong final students—each chain recovers more than 15pp over standalone blockwise KD and outperforms standalone response-based KD (84.35%). Blockwise alignment behaves here as a useful initialization rather than a standalone training procedure, validating SilverSpoon-KD’s multi-stage pipeline as the central mechanism for harnessing blockwise KD in settings where it diverges in isolation. The result also demonstrates that the block abstraction generalizes beyond transformers (depthwise-separable CNNs are accommodated transparently) and that automatic projectors bridge the resulting channel-dimension mismatches without manual configuration.

VGG Relational KD

We replicate the setup of Park et al. [13] to validate SilverSpoon-KD’s relation-based KD support: a ResNet-50 teacher (77.81% on our CIFAR-100 split) and a VGG-11-BN student. All runs train for 200 epochs at batch size 128 in float32 with SGD (momentum 0.9, weight decay 5×10−45 \times10^{-4}), starting from learning rate 0.1 and decaying ×0.2\times0.2 at epochs 60, 120, and 160. We compare response-based KD (Hinton KD, temperature T=16T=16 with hard-label weight α≈0.06\alpha\approx0.06 to match Hinton’s λ=16\lambda=16 soft/hard mixing), holistic KD with MSE on global-pool features, and three relation-based KD configurations—distance loss, angle loss, and their combination—against Park et al.’s reported numbers and a no-teacher baseline. Feature alignment is placed on the global-pool layer (2048-d for ResNet-50, 512-d for VGG-11-BN); HKD with MSE bridges this dimension mismatch via SilverSpoon-KD’s auto-projectors, while the relation-based variants disable auto-projection because their losses (pairwise distances, triplet angles) are computed within each side and compared structurally, requiring no pointwise dimension matching. The relation-based KD variants are followed by a CE fine-tune of the classifier head, since the holistic KD stage trains feature layers only and leaves the head randomly initialized. This experiment exercises SilverSpoon-KD’s pluggable loss functions: switching between distance, angle, and combined relation-based KD losses requires only a configuration change.

All distillation variants improve substantially over the no-teacher baseline (65.16%), with response-based KD leading at 73.65%—within 0.61pp of Park et al.’s reported 74.26%, despite our weaker baseline (65.16% vs. 71.26%). Holistic KD with MSE on global-pool features reaches 72.78%, between response-based KD and the relation-based variants. The three relation-based KD configurations (distance, angle, distance+angle) range from 70.40% to 71.61%, modestly trailing both response-based KD and Park et al.’s reported relation-based KD numbers (72.27–72.97%) but confirming relation-based KD’s capacity to transfer relational structure across the cross-architecture

MethodOurs (%)Park et al. (%)Peak Mem (MB)Steps/sTime (h)
Teacher (ResNet-50)77.81—4 2127.892.75
Baseline (VGG-11-BN, no teacher)65.1671.2675032.200.67
ResKD (T=16T=16)73.6574.261 19013.201.64
HKD (MSE)72.78—1 19012.971.67
HKD (distance)70.4072.271 19212.751.70
HKD (angle)71.01—1 19212.331.76
HKD (dist. + angle)71.6172.971 19211.911.82

Table 5. VGG Relational KD on CIFAR-100 (ResNet-50 →\rightarrow VGG-11-BN). ResKD leads our results at 73.65%; relation-based variants span 70.40–71.61%. All paradigms reachable through configuration-only changes. Bold = best per column among non-teacher rows (within Ours column).

(ResNet-50 →\rightarrow VGG-11-BN) gap. Beyond the headline numbers, this experiment exercises SilverSpoon-KD’s pluggable loss-function architecture: switching between distance, angle, and combined relation-based KD losses requires only a configuration change, with no per-loss training-loop modifications.

BERT Pre-Training

We distill BERT-base-uncased (12 layers, 768 hidden, 110M parameters) during masked language model pre-training on Dolma [37] (allenai/dolma3 pool) into two students with different compression regimes: bert_T6 (6 layers, hidden 768, intermediate 3072, ≈\approx65M parameters, ≈\approx59% of teacher) and bert_T4_tiny (4 layers, hidden 312, intermediate 1200, ≈\approx14M parameters, ≈\approx13%). Both students are constructed by pruning BERT-base layers (T6 keeps layers {1,3,5,7,9,11}\{1, 3, 5, 7, 9, 11\}, T4-tiny keeps {2,5,8,11}\{2, 5, 8, 11\} and additionally narrows the hidden dimension), and cross-depth alignment maps each retained student layer to its pre-pruning teacher counterpart. T4-tiny’s narrower hidden dimension is bridged automatically by SilverSpoon-KD’s auto-projectors at every alignment point; T6 shares the teacher’s hidden size so no projector is needed. Stage-1 distillation runs for 20,000 steps in float32 at batch size 16 with sequence length 512, using AdamW (LR 2×10−42 \times10^{-4}, constant-with-warmup schedule, 10% warmup, weight decay 0, max grad norm 1.0). Every distilled checkpoint is then fine-tuned on MNLI in Stage 2 for 30 epochs at batch size 128 (AdamW, LR 1×10−41 \times10^{-4}, linear schedule, 10% warmup, weight decay 0.01, max grad norm 1.0) to measure downstream transfer. Beyond demonstrating SilverSpoon-KD’s support for encoder pre-training with cross-depth alignment, this experiment surfaces a paradigm–compression interaction discussed below.

All three paradigms improve MNLI over the no-teacher baseline at T6 compression (≈\approx59% of teacher parameters), and holistic KD leads (80.33%, +3.04pp over baseline 77.29%), followed by blockwise KD (79.13%, +1.84pp) and response-based KD (79.01%, +1.72pp). At the more aggressive T4-tiny compression (≈\approx13%), holistic KD’s lead widens substantially: 70.44% (+3.52pp over baseline 66.92%) while standalone blockwise KD actually hurts (−0.82pp) and response-based KD provides only modest gains (+0.91pp). The same paradigms that cluster within ≈\approx1.3pp of each other at T6 fan out to a 5.2pp spread at T4-tiny: as the teacher–student structural gap widens, end-to-end alignment (HKD) becomes increasingly indispensable while strict per-block matching (BKD) degrades into actively harmful supervision. The Stage-1 masked language modeling perplexity columns reflect the same gradient—both BKD and ResKD diverge to four-digit perplexities at T4-tiny while HKD’s Stage-1 PPL stays substantially closer to the no-teacher baseline. This compression–paradigm interaction is exactly the kind of empirical finding SilverSpoon-KD’s unified API is designed to surface, since switching paradigms across compression regimes requires only a class swap (Figure 3) rather than a re-implementation of the training loop.

BERT Downstream

We directly compare SilverSpoon-KD against TextBrewer on MNLI, one of TextBrewer’s primary benchmarks [17]. The teacher is BERT-base-cased fine-tuned on MNLI; the two students are again T6 (≈\approx65M params) and T4-tiny (≈\approx14M params), constructed by pruning teacher layers (T6: {1,3,5,7,9,11}\{1, 3, 5, 7, 9, 11\}; T4-tiny: {2,5,8,11}\{2, 5, 8, 11\}) with hidden-dim narrowing). All runs train for 30 epochs at batch size 128 with AdamW (LR 1×10−41 \times10^{-4}, linear schedule, 10% warmup, weight decay 0.01, max grad norm 1.0); TextBrewer additionally uses early stopping with patience 5, while SilverSpoon-KD’s runs train for the full epoch budget. TextBrewer’s GeneralDistiller uses hidden mse for per-layer intermediate alignment and CE-soft KD on logits at temperature T=8T=8, with linear projectors where T4-tiny’s hidden dimension differs from the teacher’s. SilverSpoon-KD’s holistic KD is configured as a single-stage run combining hidden-state alignment with a response-based loss at T=8T=8 (response weight 1/T21/T^2 to cancel SilverSpoon-KD’s internal T2T^2 scaling so the effective KD weight matches TextBrewer’s); SilverSpoon-KD’s blockwise KD is a two-stage pipeline—per-block alignment, then a standard fine-tune at LR 2×10−52 \times10^{-5}. Because TextBrewer cannot express blockwise KD, this experiment tests how a paradigm absent from existing toolkits fares against holistic methods, particularly under aggressive compression, on TextBrewer’s own turf—the only direct head-to-head comparison against an established encoder-distillation toolkit in this paper.

On the T6 student (≈59%\approx59\% of teacher parameters), SilverSpoon-KD’s holistic KD reaches 83.47% MNLI-m—outperforming TextBrewer T6 (80.09%) by 3.38pp and matching the BERT-base teacher (83.59%) within 0.12pp—on TextBrewer’s own benchmark and against a toolkit specifically optimized for the encoder-NLP domain. At T4-tiny (≈13%\approx13\%), TextBrewer narrowly leads SilverSpoon-KD’s holistic KD (81.20% vs. 80.87%), reflecting its mature encoder-specific tuning at extreme compression, while SilverSpoon-KD’s blockwise KD → fine-tune pipeline reaches only 66.01%, confirming that blockwise alignment alone struggles when the structural gap is too wide—consistent with the pre-training findings of Appendix § 9.§ 3. SilverSpoon-KD additionally enables blockwise KD on encoder tasks, a paradigm unavailable in TextBrewer; while blockwise KD does not outperform holistic KD here, its availability widens the experimental design space practitioners can explore.

StudentMethodAlign (Stage 1)Loss (Stage 1)MLM PPL (Stage 2)MNLI (%) (Stage 2)Peak Mem (MB)Steps/sTime (h)
T6Baseline—14.0877.296 5886.450.86
T6BKD0.077297.0279.1312 3343.311.68
T6HKD0.40840.5780.3313 7742.452.27
T6ResKD1.66136.5179.019 7461.942.86
T4-tinyBaseline—24.4266.925 45812.970.43
T4-tinyBKD0.3592545.7266.102 9625.181.07
T4-tinyHKD0.5482802.1070.443 9043.971.40
T4-tinyResKD2.1876413.9767.838 1602.722.04
Teacher (BERT-base)——83.59———

Table 6. BERT distillation (pre-training + downstream MNLI). HKD wins MNLI at both compression levels (T6: 80.33%, +3.04pp; T4-tiny: 70.44%, +3.52pp); the paradigm spread fans from ∼1.3pp at T6 to 5.2pp at T4-tiny, where standalone BKD turns harmful. Bold = best per column within each student-size group; Teacher row excluded from competition.

Table 7: BERT Downstream MNLI: SilverSpoon-KD vs. TextBrewer. HKD → FT matches the BERT-base teacher within 0.12pp at T6 (83.47% vs. 83.59%) and beats TextBrewer by 3.38pp; T4-tiny TextBrewer narrowly leads (81.20% vs. 80.87%). Reference-only rows (Teacher, baselines, TextBrewer) lack our system metrics. The T4-tiny HKD → FT row’s MNLI-mm cell is empty by design.7 Bold = best per column within each student-size group; Teacher and TextBrewer rows excluded from competition.

RunMNLI-m (%)MNLI-mm (%)Peak Mem (MB)Steps/sTime (h)
Teacher (BERT-base)83.5983.76———
T6 (≈59% of teacher)
Baseline (no distillation)78.4778.99———
TextBrewer80.0980.48———
SilverSpoon HKD → FT83.4783.936 1543.577.16
T4-tiny (≈13% of teacher)
Baseline (no distillation)31.8231.82———
TextBrewer81.2081.42———
SilverSpoon HKD → FT80.87—2 9604.615.54
SilverSpoon BKD → FT66.0165.793 78623.841.07

Table 7.

GPT-2 Compression

This experiment highlights multi-stage pipelines on a causal language model. The teacher is GPT-2 Small (124M, 12 layers, hidden 768, n_inner 3072); the student is a GPT-2 variant with reduced MLP width (96M parameters, 12 layers, hidden 768, n_inner 1536—a 50% MLP-width reduction), keeping the teacher’s hidden size so no projectors are required at any cross-depth alignment. All runs train on Dolma at context length 1024 with AdamW (LR 1×10−31 \times10^{-3}) in bf16, with step budgets sized to keep training tokens approximately constant across paradigms: standalone blockwise KD and the no-teacher baseline run 60,000 steps at batch size 32 (≈1.97\approx1.97B tokens), while standalone holistic KD, response-based KD, and the BKD →\to HKD continuation run 117,000 steps at batch size 16 (≈1.92\approx1.92B tokens). The final LoRA stage of the three-stage pipeline runs 60,000 steps at batch size 32 with rank 64, α=128\alpha= 128, and dropout 0.05 over the Conv1D projection modules (c_attn, c_proj, c_fc). Multi-stage pipelines chain paradigms by initializing each stage from the previous stage’s trained student checkpoint. GPT-2 Small was chosen for its combination of historical importance and small size, which kept the many iterations needed to approach a Chinchilla-style token-to-parameter ratio [38] computationally tractable.

MethodWiki PPLLAMBADA (%)HellaSwag (%)Peak Mem (MB)Steps/sTime (h)
Baseline (no teacher)107.1515.2527.2420 8481.789.36
BKD245,281.090.0025.4827 3044.533.68
HKD97.6917.7426.5923 2881.1528.21
ResKD133.996.4826.7425 5981.5221.32
BKD →\to HKD109.5616.9027.1223 2101.9516.68
BKD →\to HKD →\to LoRA93.3919.5627.2121 7201.759.52
Teacher (GPT-2 Small)37.3732.5631.14———

Table 8. GPT-2 Compression. The three-stage BKD →\to HKD →\to LoRA pipeline rescues standalone-BKD divergence (PPL 245k→\to93) and yields the best LAMBADA (19.56%, +28% over baseline). Benchmarks: LAMBADA [33], HellaSwag [39]. Bold = best per column among non-teacher rows.

Standalone blockwise KD catastrophically diverges (PPL 245,281, 0% LAMBADA), confirming that blockwise alignment in isolation is unstable for autoregressive language models that lack an end-to-end generation signal. Multi-stage chaining—which initializes each stage from the previous stage’s trained student—rescues this failure mode entirely: blockwise KD →\to holistic KD reaches PPL 109.56 (within 12 PPL of standalone holistic KD at 97.69), and the three-stage blockwise KD →\to holistic KD →\to LoRA pipeline becomes the strongest configuration overall (PPL 93.39, LAMBADA 19.56%—a 28% relative improvement over baseline LAMBADA at 15.25%). This BKD-divergence-then-recovery sequence is the central result for SilverSpoon-KD’s multi-stage capability and, to our knowledge, the first open report of blockwise KD →\to holistic KD chaining recovering from standalone-blockwise KD divergence on autoregressive LLMs. Standalone response-based KD reduces perplexity (133.99) but degrades LAMBADA from 15.25% to 6.48%, suggesting it transfers fluency without the task-specific knowledge that hidden-state alignment captures.

Ultimately, this experiment is one of the strongest demonstrations of SilverSpoon-KD’s multi-stage pipeline mechanism. The blockwise KD-alone divergence and subsequent multi-stage recovery is a result that was obtained entirely within the SilverSpoon-KD library, demonstrating a capability and outcome unique to this KD toolkit.

Cross-Architecture

This experiment pushes cross-architecture distillation within the context of FeatKD to its limit: teacher and student share no architectural lineage. The teacher is GPT-2 Small (124M, 12 layers, standard multi-head attention, learned positional embeddings, GELU activations); the student is a 96M-parameter model built on the DeepSeek-V3 architecture with Multi-head Latent Attention (kv_lora_rank 96, q_lora_rank 256), RoPE positional embeddings, SiLU activations, and dense (non-mixture-of-experts) MLPs. The student has 6 layers mapped to teacher layers {0, 2, 4, 6, 8, 10} via cross-depth alignment. Hidden size is intentionally shared (768) to isolate the architectural variable from projector effects, so all alignments are projector-free and the only difference between this and the same-family GPT-2 compression setting Appendix § 9.§ 5 is the layer architecture itself. Standalone-paradigm runs (HKD, ResKD) and the no-teacher baseline train for 60,000 steps at batch size 32 on Dolma at context length 1024 with AdamW (LR 1×10−31 \times10^{-3}) in bf16; stage-2 fine-tunes from HKD and ResKD start from those checkpoints and continue for 60,000 steps at LR 5×10−45 \times10^{-4} so the student can shake off any architecture-incompatible representations imposed by stage 1. Cross-architecture HKD additionally bumps the response-loss weight from the default (≈2\approx2 for 6 alignments) to 30 and uses temperature 4: the 6 feature-level MSE alignments otherwise dominate the single response alignment 3:1 and force the student’s hidden states into a GPT-2-shaped representation the DeepSeek-V3 block cannot naturally produce, so the optimization is shifted toward output-distribution matching while feature alignment becomes a weaker regularizer. We attempted blockwise KD as well, but per-block alignment is not viable in this regime: teacher-forced GPT-2 hidden-state inputs land in a representation space (MHA + learned positional encoding) that the DeepSeek-V3 block (MLA + RoPE) cannot consume compatibly, and the run did not yield reportable results.

MethodWiki PPLLAMBADA (%)HellaSwag (%)Peak Mem (MB)Steps/sTime (h)
Baseline (no teacher)117.7816.3427.4822 4842.576.48
HKD246.440.8726.2933 9787.292.29
ResKD127.146.0226.4739 8040.3843.82
HKD → Fine-tune118.4915.0627.6717 7062.606.41
ResKD → Fine-tune117.2814.9627.6522 4842.606.41
Teacher (GPT-2 Small)37.3732.5631.14———

Table 9. Cross-Architecture Distillation (GPT-2 → DeepSeek-V3, no shared architectural lineage). Standalone KD degrades LAMBADA below baseline; FT-recovery (HKD → FT, ResKD → FT) brings results back near baseline. Bold = best per column among non-teacher rows.

Cross-architecture distillation between teacher and student that share no architectural lineage (multi-head vs. multi-head latent attention, learned vs. RoPE positional embeddings, GELU vs. SiLU) is feasible but more delicate than the same-family setting. Standalone holistic KD diverges (PPL 246.44 vs. baseline 117.78, with LAMBADA collapsing from 16.34% to 0.87%) even with the rebalanced response-loss weight, and standalone response-based KD also degrades LAMBADA (16.34% → 6.02%) despite a smaller perplexity hit. Multi-stage recovery via fine-tuning brings both paradigms back to near-baseline: holistic KD → FT reaches PPL 118.49 with LAMBADA 15.06%, and response-based KD → FT reaches PPL 117.28 with LAMBADA 14.96%. As described above, blockwise KD is not reported because its teacher-forced inputs land in a hidden-state space the DeepSeek-V3 block cannot consume compatibly—a hard limit of feature-based KD when per-block input geometry diverges, one that no projector or alignment weight resolves at the per-block level. While the absolute gains over baseline are smaller than in the same-family GPT-2 setting (Appendix § 9.§ 5), the experiment establishes that SilverSpoon-KD’s cross-architecture layer-mapping and multi-stage chaining together make HKD- and ResKD-driven cross-family distillation tractable across fundamentally different attention mechanisms, positional encodings, and activation functions—a setting where the toolkit’s flexible alignment and multi-stage pipeline are jointly load-bearing.

This experiment extends SilverSpoon-KD’s applicability beyond model compression to cross-family knowledge transfer. It demonstrates that distillation has the potential to be viable even between fundamentally different architectures and outside of pure response-based KD, relevant for distilling from proprietary architectures into customized deployment models.

GPT-2 Attention Linearization

This experiment distills standard softmax attention into a linearized-attention variant of the same model family (LoLCATs [34]), a cross-architecture scenario that builds on prior softmax-to-subquadratic distillation work [40, 41]. The teacher is GPT-2 Small (124M, 12 layers, softmax attention); the student replaces each layer’s softmax attention with LoLCATs’ hybrid attention—sliding-window softmax over the most recent 64 tokens combined with low-rank linear attention over older tokens—while all other weights are copied from the teacher and frozen (freeze_copied=True), so only the new attention modules are trained. All runs train on Dolma at context length 1024 for 76,000 steps with AdamW in bf16. Standalone HKD and ResKD use batch size 16 with two-step gradient accumulation (effective batch 32) due to memory pressure from caching hidden states. across all 12 layers; BKD, baseline, and the stage-2 fine-tune use batch size 32 directly. Stage-1 distillation runs use LR 1×10−21 \times10^{-2} (the higher rate reflects training only the freshly initialized attention modules); stage-2 fine-tune and HKD/ResKD continuations use LR 1×10−31 \times10^{-3}. The benefits of attention linearization tend to come at much larger context lengths and model sizes than tested here (given the efficiency of quadratic attention kernels at the time of writing), but this experiment serves as a proof of application of SilverSpoon-KD to the domain. Beyond full-layer blockwise KD, we exercise SilverSpoon-KD’s granular sub-module targeting: a blockwise KD variant whose alignment pattern (^transformer.h.\d+.attn$) targets only the attention sub-module within each layer rather than the full layer output, isolating the swap-in linearization to its own alignment loss.

MethodWiki PPLLAMBADA (%)HellaSwag (%)Peak Mem (MB)Steps/sTime (h)
LoLCATs baseline (no KD)53.4930.5531.0025 4082.317.22
BKD235.3730.8931.2625 4282.197.61
HKD320.5530.9131.0342 6201.0116.44
BKD → HKD285.5330.8631.3442 6220.6412.94
ResKD441.3630.9530.8544 1303.624.61
BKD → ResKD267.2230.9531.2943 0660.5637.56
BKD → Fine-tune52.0131.0531.0825 4082.299.22
Teacher (GPT-2, softmax)37.3732.5631.14———

Table 10. Attention Linearization (softmax → LoLCATs hybrid attention). Only BKD → FT (PPL 52.01) beats the LoLCATs no-KD floor (53.49); standalone KDs push PPL above 230. Granular sub-module targeting makes this attention-only alignment accessible. Bold = best per column among non-teacher, non-LoLCATs-baseline rows.

In this linearization setting—where the student replaces softmax attention with a linearized variant while retaining the teacher’s other weights—distillation alone consistently degrades perplexity relative to the LoLCATs no-KD baseline (53.49). Standalone blockwise KD, holistic KD, and response-based KD all push PPL above 230, and even multi-stage chains without fine-tuning (blockwise KD → holistic KD, blockwise KD → response-based KD) remain in the 250–290 range. Only blockwise KD → fine-tune produces a net gain (PPL 52.01), narrowly beating the no-KD baseline. The setting nonetheless exercises a unique SilverSpoon-KD capability: comparable toolkits would require manual hook plumbing for the attention-only alignment used here, while SilverSpoon-KD exposes it through the same alignment configuration as full-layer blockwise KD—making the distinction one of pattern syntax rather than custom code, and showing that the toolkit is at least usable—for—if not yet decisive in—the practically motivated path from softmax to O(n)O(n) attention.

Qwen3 Quantization

This experiment combines quantization-aware training (QAT) with KD for a practical compression scenario. The teacher is Qwen3-1.7B [citep][42] at bf16 full precision; the student is the same architecture loaded from the same pretrained weights but with simulated low-precision integer weights through the straight-through estimator (STE). All runs train on Dolma for 20,000 steps with the 8-bit AdamW optimizer at LR 2×10−52 \times10^{-5} on a cosine schedule. The QAT baseline and blockwise KD use batch size 4 directly, while holistic KD uses batch size 2 with four-step gradient accumulation (effective batch 8) plus gradient checkpointing, and response-based KD uses batch size 2 with gradient checkpointing—both due to additional activation memory beyond the per-block-only gradients of BKD. Holistic KD additionally bumps the response-loss weight to 30 (mirroring the cross-architecture rebalance of Appendix § 9.6), and response-based KD uses pure soft distillation (α=0\alpha=0, T=1T=1) routed through a fused KD loss kernel from the Liger Kernel library [citep][43] on lm_head. We sweep INT4, INT5, and INT6: INT6 post-training quantization (PTQ) is already nearly lossless and serves as a practical upper bound for the sweep, while sub-INT6 precisions are becoming end-to-end viable as a new generation of kernels and hardware support matures—FlexQ [citep][44], for instance, delivers real memory and throughput gains for INT6 LLM serving, alongside emerging FP4 and MXFP microscaling formats. Simulated quantization at these bit-widths is therefore no longer purely academic: weights trained at these precisions can actually be deployed at them. The key question is how much accuracy each distillation paradigm recovers at each precision relative to its PTQ floor and the bf16 ceiling.

MethodMMLUARC-EARC-C(N)HS(N)Wiki PPLPeak MemSteps/sTime(h)
bf16 pretrained (ceiling)55.5572.2643.0960.3521.05———
INT4 PTQ (floor)40.7158.4632.4254.2637.20———
INT4 QAT (no teacher)41.5058.9632.8554.5336.1529 144 MB1.763.16
INT4 QAT + BKD42.6759.5133.8754.4133.7029 144 MB1.763.16
INT4 QAT + HKD48.2665.5736.6954.6629.9835 162 MB0.806.97
INT4 QAT + ResKD43.7563.8535.0753.9431.1132 436 MB0.926.02
INT5 PTQ (floor)53.3271.0940.7859.5425.89———
INT5 QAT (no teacher)53.2770.9641.4759.6825.3033 624 MB1.134.90
INT5 QAT + BKD54.9672.7742.4959.2322.3940 258 MB0.866.44
INT5 QAT + HKD54.8972.8142.4959.7322.0935 162 MB0.796.99
INT5 QAT + ResKD51.9970.8341.4759.7521.9631 972 MB0.866.43
INT6 PTQ (floor)54.6672.7743.2660.1821.26———
INT6 QAT (no teacher)55.3372.6943.6060.2920.7733 624 MB1.144.89
INT6 QAT + BKD54.3673.4042.4959.9121.4140 260 MB0.787.15
INT6 QAT + HKD55.5873.0243.8659.8921.6635 162 MB0.806.94
INT6 QAT + ResKD55.8373.4443.4360.2321.3631 972 MB0.866.47

Table 11. Qwen3-1.7B INT4/INT5/INT6 QAT with distillation. HKD dominates at INT4 (+6.76pp MMLU vs. QAT-alone, +9.42pp ARC-E); paradigms converge at INT5 and INT6 where PTQ already approaches the bf16 ceiling. Benchmarks: ARC-E/C [45], HS = HellaSwag, (N) = normalized accuracy. Bold = best per column within each precision group; bf16 ceiling and PTQ floor rows excluded from competition.

Distillation paradigms recover accuracy beyond QAT alone most clearly at the most aggressive precision: at INT4, holistic KD lifts MMLU from QAT-alone 41.50% to 48.26% (+6.76pp toward the bf16 ceiling of 55.55%) and simultaneously delivers the lowest perplexity (29.98 vs. 36.15 for QAT alone), while response-based KD lands in between on accuracy (43.75% MMLU, 31.11 PPL) and blockwise KD trails (42.67% MMLU, 33.70 PPL). At INT5, the three paradigms converge—all reach 51.99–54.96% MMLU within the noise of QAT-alone (53.27%)—and response-based KD edges out the others on perplexity (21.96 vs. 22.09 HKD, 22.39 BKD). At INT6, where PTQ is already nearly lossless (54.66% MMLU, 21.26 PPL), distillation contributes little, confirming INT6 as a practical upper bound for sweep-relevant quantization. Beyond accuracy recovery, the experiment exercises a unique SilverSpoon-KD differentiator: blockwise KD provides distillation at the same GPU memory cost as the QAT baseline (29 144 MB at INT4) because per-block gradients eliminate the need to materialize hidden states across the full network simultaneously, whereas holistic KD adds ≈6 GB and response-based KD adds ≈3 GB. For QAT on large models where GPU memory is the binding constraint, blockwise KD’s zero-overhead property is a unique practical advantage.

This experiment validates SilverSpoon-KD in a practical production scenario (INT4 quantization of a 1.7B-parameter model) and reveals clear paradigm-specific trade-offs—holistic KD for the largest accuracy gains at the most aggressive precision (INT4), response-based KD for slightly better perplexity at INT5, and blockwise KD wherever GPU memory is the binding constraint—that directly inform practitioner guidance for QAT-distillation pipelines. The setup also leaves a natural path open to non-uniform quantization research [46], where SilverSpoon-KD’s per-alignment configuration could compose with mixed-precision per-layer quantization.

Toolkit Benchmarking

This appendix documents the per-toolkit configurations behind the head-to-head benchmark (Table 2) and the capability-frontier sweep (Table 3) of § 4.3. Each comparison is driven by a self-contained benchmarking script under scripts/toolkit_comparison/ in the experiments repository that runs both toolkits against an identical teacher checkpoint, dataset, batch size, step count, and learning-rate schedule, with toolkit-specific defaults preserved wherever they are publicly documented—each setting therefore reflects the configuration a typical user of the comparison toolkit would write. Wall time, throughput, and peak GPU memory are recorded under torch.cuda.max_memory_allocated on a freshly initialized CUDA context per run.

Vision DDP (torchdistill, ResNet-50→\toResNet-18 / ImageNet [47], ResKD, 2×2\timesA40). Teacher and student are loaded from timm/resnet50.a1_in1k and timm/resnet18.a1_in1k respectively; both toolkits use AdamW (LR 2×10−42\times10^{-4}, batch size 32 per GPU, fp32, 2,000 steps, 10% linear warmup). KD is response-based at temperature T=2T=2 with soft/hard mixing α=0.5\alpha=0.5. Both runs use PyTorch DDP across two A40s, matching torchdistill’s recommended multi-GPU configuration.

Encoder NLP, ResKD (TextBrewer, BERT-base→\toT6 / MNLI, 2×2\timesA40). The teacher is our fine-tuned bert-base-cased MNLI checkpoint and the student is a 6-layer T6 initialized by copying teacher layers {1,3,5,7,9,11}\{1, 3, 5, 7, 9, 11\}. Both toolkits use AdamW (LR 1×10−41\times10^{-4}, batch 32 per GPU, max sequence length 128, 1,000 steps, 10% linear warmup) and response-based KD at temperature T=4T=4. TextBrewer uses its GeneralDistiller with soft-CE; SilverSpoon-KD uses its response-based distiller with KL-divergence loss.

Encoder NLP, HKD (TextBrewer, BERT-base→\toT6 / MNLI, 1×1\timesA40). Same teacher, student, and dataset as the ResKD setting; this configuration runs single-GPU and adds hidden-state alignment via TextBrewer’s hidden_mse on every layer pairing alongside response-based KD on logits at temperature T=4T=4. SilverSpoon-KD’s HKD configuration matches: per-layer MSE on aligned hidden states, response loss with T=4T=4, and the response weight set to 1/T21/T^2 to cancel SilverSpoon-KD’s internal T2T^2 scaling so the effective KD weight matches TextBrewer’s.

Decoder LLM, HKD (DistillKit, Qwen3-4B→\to0.6B / Dolma, 1×1\timesA100). Both toolkits load Qwen/Qwen3-4B as teacher and Qwen/Qwen3-0.6B as student in bf16 with gradient checkpointing. Training: 500 steps, batch size 2, sequence length 1024, AdamW with cosine schedule (LR 1×10−31\times10^{-3}). Both combine hidden-state alignment with response-based KD (wKL=0.5w_{\mathrm{KL}}=0.5, wCE=0.5w_{\mathrm{CE}}=0.5, T=2T=2). DistillKit’s HKD path uses linear projectors where dimensions mismatch (force_hidden_state_projection); SilverSpoon-KD’s auto-projectors handle the same alignment. SilverSpoon-KD additionally routes the response loss through Liger Kernel’s fused KD loss, while DistillKit uses an unfused implementation.

Decoder LLM, ResKD+LoRA (torchtune, Qwen3-4B→\to0.6B / Dolma, 1×1\timesA100). Same teacher and student as the DistillKit comparison, in bf16. Training: 1,000 steps, batch size 2, sequence length 1024, AdamW (LR 1×10−31\times10^{-3}). The student is wrapped in LoRA adapters on the q/v/output projections and MLP modules (rank 64, α=128\alpha=128, dropout 0.05); torchtune trains only the adapters via its ForwardKLLoss with kd_ratio 0.5, while SilverSpoon-KD applies the same LoRA configuration through its standard adapter integration with response-based KD. torchtune’s ForwardKLLoss does not expose a temperature parameter; SilverSpoon-KD’s reported numbers correspond to its toolkit-default T=2T=2, α=0.5\alpha=0.5 to keep the comparison representative of typical use rather than artificially matched.

Capability frontier (Qwen3-{4B, 8B, 14B}→\to0.6B / Dolma, 2×2\timesA100). For each (teacher, toolkit) pair the benchmarking script binary-searches the maximum batch size that fits on 2×2\timesA100 (40 GB per-GPU) at sequence length 1024, then measures throughput at that batch size in bf16 with AdamW. SilverSpoon-KD is configured with FSDP-sharded teacher placement (teacher_placement="fsdp"), splitting the 14B teacher’s ∼\sim28 GB of bf16 weights across the two GPUs at ∼\sim14 GB per GPU; DistillKit and TextBrewer support only replicated teachers (the full ∼\sim28 GB on each GPU), which leaves no headroom for student parameters, optimizer states, gradients, and activations on a 40 GB card and forces DistillKit to OOM and TextBrewer to bs=1. Reported peak memory and throughput are averaged over the post-warmup measurement window; incremental JSON saves between (toolkit, teacher) pairs let preempted jobs retain partial measurements.

Additional placement sweeps (not reported in § 4.3). The benchmarking driver supports three further sweeps beyond the head-to-head and teacher-scaling settings, exercised during development but kept out of the headline tables to preserve focus. (1) A teacher-placement comparison on 4×4\timesA40 contrasts replicated, FSDP-sharded, and split-GPU placement (teacher on GPUs 2–3, student DDP on GPUs 0–1) at fixed batch size; the FSDP-sharded and split-GPU variants are unavailable in DistillKit and TextBrewer. (2) An OOM-boundary probe extends the batch-size search at sequence lengths {1024,2048,4096}\{1024, 2048, 4096\} on 2×2\timesA40, surfacing each toolkit’s per-placement maximum batch. (3) A FSDP-vs.-tensor-parallel probe on 4×4\timesA40 substitutes tensor-parallel teacher inference for FSDP all-gather, trading a per-step weight gather for activation-communication cost; the trade-off is teacher-size-dependent. All three are reproducible from the experiments repository.

The configurations above establish that the headline numbers in Table 2 reflect each comparison toolkit’s default or recommended setup rather than a stripped-down version, and that the capability-frontier discrepancy at Qwen3-14B (Table 3) is driven by SilverSpoon-KD’s FSDP-sharded teacher placement rather than incidental engineering differences—the only configurable axis on which the compared toolkits genuinely diverge at scale.

References

  1. [1]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
  2. [2]Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.arxiv.org/abs/1910.01108
  3. [3]Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiaodan Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling bert for natural language understanding. In Findings of the association for computational linguistics: EMNLP 2020, pages 4163–4174, 2020.arxiv.org/abs/1909.10351
  4. [4]Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu. One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation. Advances in Neural Information Processing Systems, 36:79570–79582, 2023.arxiv.org/abs/2310.19444
  5. [5]Jhe-Hao Lin, Yi Yao, Chan-Feng Hsu, Hongxia Xie, Hong-Han Shuai, and Wen-Huang Cheng. Feature-based one-for-all: A universal framework for heterogeneous knowledge distillation. arXiv e-prints, pages arXiv–2501, 2025.
  6. [6]Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668, 2018.arxiv.org/abs/1802.05668
  7. [7]Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anand-kumar. Born again neural networks. In International conference on machine learning, pages 1607–1616. PMLR, 2018.arxiv.org/abs/1805.04770
  8. [8]Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3713–3722, 2019.arxiv.org/abs/1905.08094
  9. [9]Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10925–10934, 2022.arxiv.org/abs/2106.05237
  10. [10]Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws. arXiv preprint arXiv:2502.08606, 2025.
  11. [11]Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: hints for thin deep nets (2014). arXiv preprint arXiv:1412.6550, 3, 2014.
  12. [12]Hui Wang, Hanbin Zhao, Xi Li, and Xu Tan. Progressive blockwise knowledge distillation for neural network acceleration. In IJCAI, pages 2769–2775, 2018.DOI
  13. [13]Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3967–3976, 2019.arxiv.org/abs/1904.05068
  14. [14]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45, 2020.DOI
  15. [15]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1912.01703
  16. [16]Yoshitomo Matsubara. torchtistill: A modular, configuration-driven framework for knowledge distillation. In International Workshop on Reproducible Research in Pattern Recognition, pages 24–44. Springer, 2021.DOI
  17. [17]Ziqing Yang, Yiming Cui, Zhipeng Chen, Wanxiang Che, Ting Liu, Shijin Wang, and Guoping Hu. Textbrewer: An open-source knowledge distillation toolkit for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 9–16, 2020.DOI
  18. [18]torchtune maintainers and contributors. torchtune: PyTorch’s finetuning library. https://github.com/meta-pytorch/torchtune, 2024.
  19. [19]Arcee AI. Arcee AI DistillKit. https://github.com/arcee-ai/DistillKit, 2024.
  20. [20]Chengyu Wang, Junbing Yan, Wenrui Cai, Yuanhao Yue, and Jun Huang. Easydistill: A comprehensive toolkit for effective knowledge distillation of large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 787–795, 2025.arxiv.org/abs/2505.20888
  21. [21]Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10699, 2019.arxiv.org/abs/1910.10699
  22. [22]Het Shah, Avishree Khare, Neelay Shah, and Khizir Siddiqui. Kd-lib: A pytorch library for knowledge distillation, pruning and quantization. arXiv preprint arXiv:2011.14691, 2020.arxiv.org/abs/2011.14691
  23. [23]NVIDIA Corporation. NVIDIA Model Optimizer. https://github.com/NVIDIA/Model-Optimizer, 2024.
  24. [24]Intel Corporation. Intel Neural Compressor. https://github.com/intel/neural-compressor, 2022.
  25. [25]Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. Lit: Block-wise intermediate representation training for model compression. arXiv preprint arXiv:1810.01937, 2018.arxiv.org/abs/1810.01937
  26. [26]Cody Blakeney, Xiaomin Li, Yan Yan, and Ziliang Zong. Parallel blockwise knowledge distillation for deep neural network compression. IEEE Transactions on Parallel and Distributed Systems, 32(7):1765–1776, 2020.arxiv.org/abs/2012.03096
  27. [27]Hongsun Jang, Jaewon Jung, Jaeyong Song, Joonsang Yu, Youngsok Kim, and Jinho Lee. Pipe-bd: Pipelined parallel blockwise distillation. In 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 1–6. IEEE, 2023.arxiv.org/abs/2301.12443
  28. [28]Chengchao Shen, Xinchao Wang, Youtan Yin, Jie Song, Sihui Luo, and Mingli Song. Progressive network grafting for few-shot knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2541–2549, 2021.arxiv.org/abs/2012.04915
  29. [29]Shiu-hong Kao, Jierun Chen, and SH Chan. Stablekd: Breaking inter-block optimization entanglement for stable knowledge distillation. arXiv preprint arXiv:2312.13223, 2023.arxiv.org/abs/2312.13223
  30. [30]Chengting Yu, Fengzhao Zhang, Ruizhe Chen, Aili Wang, Zuozhu Liu, Shurun Tan, and Er-Ping Li. Decoupling dark knowledge via block-wise logit distillation for feature-level alignment. IEEE Transactions on Artificial Intelligence, 6(5):1143–1155, 2024.arxiv.org/abs/2411.01547
  31. [31]Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
  32. [32]Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, 2018.arxiv.org/abs/1704.05426
  33. [33]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lam-bada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers), pages 1525–1534, 2016.arxiv.org/abs/1606.06031
  34. [34]Michael Zhang, Simran Arora, Rahul Chalamala, Alan Wu, Benjamin Spector, Aaryan Sing-hal, Krithik Ramesh, and Christopher Ré. Lolcats: On low-rank linearizing of large language models. arXiv preprint arXiv:2410.10254, 2024.arxiv.org/abs/2410.10254
  35. [35]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  36. [36]Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024.arxiv.org/abs/2402.13116
  37. [37]Luca Soldaini, Rodney Kinney, Akshita B. Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, 2024.
  38. [38]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 10, 2022.arxiv.org/abs/2203.15556
  39. [39]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019.arxiv.org/abs/1905.07830
  40. [40]Aviv Bick, Kevin Li, Eric Xing, J Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. Advances in neural information processing systems, 37:31788–31812, 2024.arxiv.org/abs/2408.10189
  41. [41]Patrick Haller, Jonas Golde, and Alan Akbik. Empirical evaluation of knowledge distillation from transformers to subquadratic language models. arXiv preprint arXiv:2504.14366, 2025.
  42. [42]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Cheng'en Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  43. [43]Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for llm training. arXiv preprint arXiv:2410.10989, 2024.arxiv.org/abs/2410.10989
  44. [44]Hao Zhang, Aining Jia, Weifeng Bu, Yushu Cai, Kai Sheng, Hao Chen, and Xin He. Flexq: Efficient post-training int6 quantization for llm serving via algorithm-system co-design. arXiv preprint arXiv:2508.04405, 2025.arxiv.org/abs/2508.04405
  45. [45]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.arxiv.org/abs/1803.05457
  46. [46]Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023.arxiv.org/abs/2306.07629
  47. [47]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.

Paper details

vlnc:W_4sja2gnk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents