betaFeedback

WorksComputer Science

Angular Conditional Distillation: Overcoming Classifier Geometry Mismatch

Abstract

Classifier outputs provide a fundamental and univer- sally available knowledge interface for knowledge distil- lation (KD). However, standard logit objectives entangle transferable semantic relations with architecture-dependent parameterization artifacts, causing a classifier geometry mismatch that makes direct knowledge transfer unreliable. To address this, we propose Angular Conditional Distilla- tion (ACD). First, the Angular Coordinate Interface (ACI) reconstructs the student classifier using angular coordi- nates, suppressing magnitude and bias effects to provide a geometry-compatible space. Then, Scale-Invariant Condi- tional Distillation (SICD) standardizes outputs and exclu- sively transfers conditional non-target relations, filtering out untransferable absolute confidence. Together, ACI de- fines where to transfer, while SICD defines what to transfer. Extensive experiments show that ACD achieves state-of-the- art performance over existing KD methods. Moreover, as a universal drop-in replacement, it consistently improves existing distillation pipelines while requiring no feature adapters or additional inference-time cost. Code is avail- able at https://github.com/laniusnan/ACD.

Introduction

The continued scaling of neural networks has yielded remarkable advances across several tasks [17, 20, 31], yet their high computational costs hinder deployment on edge devices [13]. Knowledge distillation (KD) addresses this by transferring knowledge from a heavy teacher to a lightweight student. Over the past decade, KD has evolved from matching classifier outputs to leveraging intermediate features [18], sample relations [23], and structural knowledge [16]. Nevertheless, classifier outputs remain the most universally available supervision interface. Unlike feature- or relation-based methods that demand architectural[2] alignment or extra modules [23], logits offer a lightweight, architecture-agnostic endpoint. They capture rich interclass correlations and are highly flexible, serving either as

the primary objective in pure logit KD [30] or as a plugin auxiliary loss across diverse paradigms [2].

However, conventional logit matching implicitly assumes that teacher and student outputs reside in compatible coordinates[30]. We argue this assumption is fundamentally flawed. For a linear classifier, the logit of class kk can be written as zk=wkTh+bk=∥wk∥∥h∥cos⁡θk+bkz_k=\mathbf{w}_k^{\mathsf{T}}\mathbf{h}+b_k=\lVert\mathbf{w}_k\rVert\lVert\mathbf{h}\rVert\cos\theta_k+b_k, where h\mathbf{h} denotes the feature representation and θk\theta_k is its angle with the class weight wk\mathbf{w}_k. This decomposition reveals that a raw logit entangles transferable directional relations (θk\theta_k) with parameterization artifacts (feature magnitudes, weight norms, and biases). Consequently, classifiers with identical directions can yield entirely different supervisory signals. We refer to this discrepancy as classifier geometry mismatch.

To empirically verify this issue, we conduct a controlled probe (Figure 1). Across five heterogeneous pairs, merely restoring affine parameterization induces substantial distribution deformation and severe gradient shifts. Crucially, this mismatch is not merely a scalar confidence error. Even with oracle-calibration in § 3, that applying the opti­ mal positive scalar to student logits to eliminate confidence gaps, substantial residual mismatch remains. This confirms that traditional KD forces students to reproduce parameterization artifacts. This raises a central question: How can we transfer informative class relationships while minimizing dependence on parameterization?

Classifier geometry mismatch

Figure 1. Classifier Geometry Mismatch. (a) Empirical evidence: fixing teacher/student features but restoring parameterization causes substantial deformation in non-target distributions (x-axis) and gradient deviations (y-axis). (b) Local gradient directional disagreement under different readouts (VGG13→MobileNetV1). Darker overlays indicate higher disagreement 1−cos⁡(gp,gpang)1-\cos(g_p,g_p^{\mathrm{ang}}).

Recent logit-based approaches have improved output supervision via decoupling[30] or normalization[3]. However, they primarily modify how information is weighted after the classifier has defined its coordinates, failing to address whether the coordinates are compatible and which relations are transferable.

To bridge this gap, we propose Angular Conditional Distillation (ACD). We introduce the Angular Coordinate Interface (ACI), which reconstructs the student classifier using normalized angular coordinates (z~k=scos⁡θk\tilde{z}_k = s \cos\theta_k). By removing magnitudes and biases, ACI provides a geometry-compatible space. Furthermore, we initialize the classifier weights with an Equiangular Tight Frame (ETF) to provide an optimal angular prior. During training, this allows data-sensitive classes to refine their specific directions, while data-insensitive classes maintain their geometrically optimal, maximally separated angles, ensuring a robust and structured representation space.

However, establishing an angular space is insufficient. If standard KL is applied to ACI, it still forces the student to replicate the teacher’s absolute confidence, which is highly sensitive to the capacity gap. To address this, we propose Scale-Invariant Conditional Distillation (SICD). Specifically, SICD first applies dual-side standardization to both teacher and student outputs to eliminate residual scale and shift effects. Then, using a finite-temperature KL formulation, SICD mathematically decomposes the divergence into a target-confidence term and a conditional non-target relation term. By dropping the target-confidence term and exclusively optimizing the latter, SICD transfers only the true dark knowledge (relative class relations) without imposing untransferable absolute confidence. Furthermore, a gradient norm matching term is employed to stabilize the optimization scale.

Together, ACI defines the compatible geometric space (where to transfer), while SICD isolates and transfers the pure relational dark knowledge (what to transfer). In summary, our contributions are threefold:

  • Problem Identification. We identify classifier geometry mismatch as a fundamental obstacle in logit distillation, revealing how transferable directional class relations are deeply entangled with untransferable parameterization artifacts across diverse architectures.

  • Methodology. We propose ACD, a geometry-aware framework consisting of two synergistic modules. ACI establishes a well-conditioned angular coordinate space augmented by an ETF initialization, while SICD mathematically decomposes the KL objective to transfer exclusively the scale-invariant conditional non-target relations.

  • Empirical Validation. We demonstrate that ACD achieves state-of-the-art results across various settings. Notably, it serves as a universal replacement for conventional logit matching, consistently boosting existing KD pipelines while maintaining the extreme efficiency of logit-only methods.

Related Work

Beyond output matching, knowledge distillation has been explored through intermediate features [5, 18, 27], representation relations [11, 16, 23, 24], and structured or heterogeneous representation-transfer mechanisms [4, 8, 26].

Our work focuses on class-level output distillation. Classical KD matches temperature-softened teacher and student distributions [6]. Subsequent studies examine its connection to label smoothing regularization [27], reduce magnitude dependence through logit standardization [22], or introduce alternative discrepancy measures [12, 25]. DIST

Algorithm 1Per-Sample Oracle Calibration Analysis
Require:Frozen teacher ftf_t and student fsf_s
Require:Images {xi}i=1N\{x_i\}_{i=1}^{N}; temperature T>0T > 0
Ensure:Per-image scales and KL divergences before/after calibration
1:for i=1,…,Ni = 1, \ldots, N do
2:zit←ft(xi),zis←fs(xi)z_i^t \leftarrow f_t(x_i), z_i^s \leftarrow f_s(x_i)
3:pit←softmax⁡(zit/T)p_i^t \leftarrow\operatorname{softmax}(z_i^t/T)
4:Define pis(α)=softmax⁡(αzis/T)p_i^s(\alpha) = \operatorname{softmax}(\alpha z_i^s/T)
5:Define ℓi(α)=T2DKL(pit∥pis(α))\ell_i(\alpha) = T^2 D_{\mathrm{KL}}(p_i^t \Vert p_i^s(\alpha))
6:αi∗←arg⁡min⁡α≥0ℓi(α)\alpha_i^* \leftarrow\arg\min_{\alpha\ge0} \ell_i(\alpha)
7:eibefore←ℓi(1)e_i^{\mathrm{before}} \leftarrow\ell_i(1)
8:eiafter←ℓi(αi∗)e_i^{\mathrm{after}} \leftarrow\ell_i(\alpha_i^*)
9:end for
10:return {αi∗,eibefore,eiafter}i=1N\{ \alpha_i^*, e_i^{\mathrm{before}}, e_i^{\mathrm{after}} \}_{i=1}^{N}

Table 1.

Figure 3. Sensitivity to score parameterization. Panels show (a) non-target conditional total variation at T=4T = 4, (b) the fraction of images whose highest-scoring non-target class changes, and (c) KD feature-gradient direction deviation (1−cos⁡)(1-\cos), all relative to the original model.

transfers correlations among predictions [1], while DKD separates target-class and non-target-class supervision [30], and NKD investigates non-target normalization [3]. Recent studies in large language and multimodal models further explore confidence-aware, uncertainty-aware, and token-selective distillation [10, 19, 21].

In contrast, ACD establishes an angular student interface to separate directional class relations from magnitude and bias effects, and transfers scale-invariant non-target conditional relations without a separate target-confidence matching term.

Empirical Analysis

Is the mismatch merely a confidence-scale difference?

Figure 1 illustrates readout-dependent changes in non-target distributions and local KD gradient directions. To examine whether teacher–student output mismatch can be explained by overall confidence scale alone, we perform per-sample oracle calibration:

αi⋆=arg⁡min⁡α≥0  T2DKL(pit ∥ pis(α)).(1)\alpha_i^{\star} = \underset{\alpha\ge0}{\arg\min}\; T^2 D_{\mathrm{KL}}\left(p_i^t \,\|\, p_i^s(\alpha)\right). \tag*{(1)}

Why an angular interface?

A linear classifier entangles angular relations with magnitude- and bias-dependent modulation:

zk=∥h∥2∥wk∥2cos⁡θk+bk,(2)z_k = \lVert h \rVert_2 \lVert w_k \rVert_2 \cos\theta_k + b_k, \tag*{(2)}

where θk\theta_k is the angle between the student feature hh and the class-weight vector wkw_k. To isolate the effects of non-angular factors, we fix student features, class-weight directions, and teacher outputs, and separately vary the common logit scale, class-weight norm heterogeneity, and centered class biases. Despite unchanged angular relations, these interventions alter non-target conditional distributions (Figure 4a). Class-dependent factors can additionally change the highest-scoring non-target class, an effect that positive scalar calibration cannot reverse (Figure 4b). These changes also propagate to the direction of the local KD feature gradient (Figure 4c). Together, these observations motivate ACI: a normalized, bias-free student readout that separates angular relations from feature magnitudes, class-weight norms, and additive biases, while retaining a shared output scale.

Figure 4. Relation distribution. (a) Teacher and student have different target probabilities, but identical non-target conditional distributions. (b) Varying qyq_y while fixing these conditional distributions leaves the weighted conditional KL at zero. Full KL equals the target-mass term and remains positive unless qy=pyq_y = p_y.

where the teacher distribution pitp_i^t and the calibrated student distribution pis(α)p_i^s(\alpha) are defined as:

pit=softmax⁡(zit/T),p_i^t = \operatorname{softmax}(z_i^t/T),
pis(α)=softmax⁡(αzis/T).p_i^s(\alpha) = \operatorname{softmax}(\alpha z_i^s/T).

Both models remain frozen, and each image receives its own optimal scalar (Alg. 1), making this diagnostic more flexible than a single global temperature. Across all 10,000 CIFAR-100 test images, calibration reduces the mean T2T^2-scaled KL divergence from 1.2198 to 0.5575, leaving 45.7% of the original discrepancy (Figure 2(a)). The calibrated, centered class scores also remain dispersed around the identity line (Figure 2(b)). Thus, for this checkpoint, even image-specific scalar calibration cannot fully account for the teacher–student output mismatch, motivating a closer examination of class-dependent score parameterization.

Figure 2. Output mismatch beyond scalar confidence. (a) Empirical cumulative distributions of the weighted KL divergence before and after per-image oracle calibration, with a symmetric-log horizontal axis. (b) Density of image-class pairs, comparing centered teacher scores with calibrated, centered student scores, both divided by TT.

Figure 5. Overview of ACI and SICD. ACI parameterizes student logits using cosine similarities with a shared scale, removing explicit feature magnitude, class-specific weight norms, and biases. The teacher retains its original classifier. SICD standardizes teacher and student logits and matches their non-target conditional distributions, weighted by the teacher’s non-target probability mass. A detached gradient-strength matching factor calibrates the conditional objective against standardized full KL. The student is trained jointly with ground-truth cross-entropy supervision.

What should be distilled within the angular interface?

An angular interface specifies how class scores are parameterized, but not which aspects of the teacher distribution should be matched. Let pp and qq denote the teacher and student distributions. Their KL divergence decomposes as

DKL(p∥q)=DKLbin(py∥qy)⏟target-mass matching+(1−py)DKL(p−y∥q−y),(3)D_{\mathrm{KL}}(p\mathbin{\|}q)=\underbrace{D_{\mathrm{KL}}^{\mathrm{bin}}(p_y\mathbin{\|}q_y)}_{\text{target-mass matching}}+(1-p_y)D_{\mathrm{KL}}(p^{-y}\mathbin{\|}q^{-y}), \tag*{(3)}

where DKLbinD_{\mathrm{KL}}^{\mathrm{bin}} denotes binary KL, and p−yp^{-y} and q−yq^{-y} are the conditional distributions over non-target classes.

Figure 4 distinguishes relation transfer from confidence matching. Figure 4 (a) shows that teacher and student can share identical non-target conditional distributions, p−y=q−yp^{-y}=q^{-y}, while assigning different target probabilities. Figure 4 (b) varies only qyq_y: the weighted conditional term remains zero, yet full KL is positive whenever qy≠pyq_y\ne p_y. Thus, matching non-target relations does not remove full KL’s additional target-confidence supervision. This controlled probability-space example motivates SICD to retain the weighted conditional term while omitting explicit target-mass matching, with cross-entropy providing ground-truth supervision.

Method

In this section, we present Angular Conditional Distillation (ACD), a geometry-aware framework designed to bridge the classifier geometry mismatch in output-based KD. ACD consists of two core components: (1) an Angular Coordinate Interface (ACI) that reconstructs the student classifier into a pure angular space, eliminating parameterization artifacts like feature magnitudes and biases; and (2) a Scale-Invariant Conditional Distillation (SICD) objective that transfers only the scale-invariant non-target relations. We first revisit the standard KD formulation, then detail ACI and SICD respectively.

Preliminaries

Knowledge Distillation (KD) involves using a fixed teacher model fTf^T to improve a parameterized student model fSf^S. Given an input xx, the teacher and student produce logits zTz^T and zSz^S, which are converted into probability distributions p=σ(zT/τ)p=\sigma(z^T/\tau) and q=σ(zS/τ)q=\sigma(z^S/\tau) via a softmax function σ\sigma and a temperature τ\tau. The standard KD objective minimizes the divergence between these distributions:

LKD=τ2DKL(p∥q).(4)\mathcal{L}_{\mathrm{KD}}=\tau^2D_{\mathrm{KL}}(p\mathbin{\|}q). \tag*{(4)}

The final training loss is a combination of the cross-entropy loss LCE\mathcal{L}_{\mathrm{CE}} and the distillation loss:

Ltotal=LCE+λLKD,(5)\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{CE}}+\lambda\mathcal{L}_{\mathrm{KD}}, \tag*{(5)}

where λ\lambda is a balancing hyper-parameter.

As discussed, a raw linear logit zk=∥wk∥∥h∥cos⁡θk+bkz_k=\lVert w_k\rVert\lVert h\rVert\cos\theta_k+b_k deeply entangles transferable directions with untransferable magnitudes and biases.

Angular Coordinate Interface (ACI)

To provide a geometry-compatible space for knowledge transfer, we reconstruct the student’s classifier. Note that the teacher remains a standard linear classifier, ensuring our method requires no modification to the pre-trained teacher.

MethodResNet56ResNet110ResNet32x4WRN-40-2WRN-40-2VGG13
Teacher72.4074.3179.4275.5975.5974.64
StudentResNet20ResNet32ResNet8x4WRN-16-2WRN-40-1VGG8
65.7967.9272.4871.1269.5468.19
Feature
FitNet [29]69.6472.3575.2674.4673.2870.09
CRD67.2871.9774.1674.2272.4472.74
ReviewKD69.5471.1275.7173.9274.5072.29
CD70.7873.6674.4674.4571.3475.61
CoCaRS72.4074.4576.2475.2075.2274.56
Logit
KD71.7474.0174.7576.0474.5274.08
DKD71.1774.1276.5176.4175.3374.41
ABKD71.4273.8775.1276.3575.1375.33
WASB72.0174.0276.3476.4574.8874.45
SKD71.5474.7774.8676.2475.2475.75
ACD (Ours)72.9775.1478.3377.9476.0480.52

Table 1 (PDF p. 5). Comparison of Top-1 accuracy on CIFAR-100. We report the individual performance of the teacher and student models under homogeneous architecture settings.

Cosine Classifier

We replace the student’s linear classification head with a cosine classifier. The angular logit is computed as:

z~k=s⋅cos⁡θk=s⋅wkTh∥wk∥2∥h∥2(6)\widetilde{z}_k = s \cdot\cos\theta_k = s \cdot\frac{\mathbf{w}_k^{\mathsf{T}}\mathbf{h}}{\lVert\mathbf{w}_k\rVert_2\lVert\mathbf{h}\rVert_2} \tag*{(6)}

where ss is a global scale parameter controlling the sharpness of the distribution. By normalizing both the feature and class weights, ACI explicitly strips away the sample-dependent magnitudes (∥h∥\lVert\mathbf{h}\rVert) and class-specific weight norms (∥wk∥\lVert\mathbf{w}_k\rVert). Setting the bias bk=0b_k = 0 further removes class-specific preferences. Consequently, the student classifier is restricted to a pure angular space, isolating the transferable directional semantic relations.

Equiangular Tight Frame (ETF) Initialization

While ACI establishes a pure angular space, the initialization of the class directions significantly impacts the early training dynamics. To provide a symmetric and uniformly distributed geometric prior, we initialize the angular classifier weights using an Equiangular Tight Frame (ETF).

Specifically, given CC total classes and feature dimension dd (d≥Cd \ge C), the ETF prototype matrix WETF=[w1,…,wC]∈Rd×C\mathbf{W}_{\mathrm{ETF}} = [\mathbf{w}_1,\ldots,\mathbf{w}_C] \in\mathbb{R}^{d \times C} is constructed as:

WETF=CC−1U(IC−1C11T)(7)\mathbf{W}_{\mathrm{ETF}} = \sqrt{\frac{C}{C-1}}\mathbf{U}\left(\mathbf{I}_C-\frac{1}{C}\mathbf{1}\mathbf{1}^{\mathsf{T}}\right) \tag*{(7)}

where U∈Rd×C\mathbf{U} \in\mathbb{R}^{d \times C} is a semi-orthogonal matrix satisfying UTU=IC\mathbf{U}^{\mathsf{T}}\mathbf{U} = \mathbf{I}_C, IC\mathbf{I}_C is the identity matrix, and 1\mathbf{1} is the all-ones vector.

This initialization ensures that all class weight vectors have an equal l2l_2 norm and identical pairwise angles. Specifically, the inner product between any two class directions satisfies:

wiTwj={1,i=j,−1C−1,i≠j.(8)\mathbf{w}_i^{\mathsf{T}}\mathbf{w}_j = \begin{cases} 1, & i=j,\\ -\frac{1}{C-1}, & i\ne j. \end{cases} \tag*{(8)}

By initializing the ACI classifier with WETF\mathbf{W}_{\mathrm{ETF}}, the student model starts from a maximally separated angular configuration (a simplex). This geometric prior perfectly aligns with our SICD objective, which aims to transfer conditional non-target relations, providing a robust and well-conditioned space for subsequent distillation.

Scale-Invariant Conditional Distillation(SICD)

Dual-Standardization

To eliminate residual scale and shift effects, we apply Z-score standardization to the logits of both the teacher and the student before softmax:

z^k=zk−μ(z)σ(z)(9)\hat{z}_k = \frac{z_k-\mu(z)}{\sigma(z)} \tag*{(9)}

where μ(z)\mu(z) and σ(z)\sigma(z) are the mean and standard deviation across all classes for a single sample. This step ensures that the supervision focuses purely on the relative class rankings, ignoring the absolute output magnitudes.

Non-Target Conditional Distillation

To prevent the student from mimicking the teacher’s absolute confidence in the ground-truth class yy, we decompose the KL divergence into a binary confidence term and a conditional non-target relation term. We discard the former and distill only the latter:

LSICD=τ2(1−py)DKL(p−y∥q−y)(10)\mathcal{L}_{\mathrm{SICD}} = \tau^2(1-p_y)D_{\mathrm{KL}}(p^{-y}\mathbin{\|}q^{-y}) \tag*{(10)}

where p−yp^{-y} and q−yq^{-y} denote the teacher and student conditional distributions over non-target classes. For k≠yk \ne y, their entries are pk/(1−py)p_k/(1-p_y) and qk/(1−qy)q_k/(1-q_y), respectively. This objective transfers relative probabilities among non-target classes while omitting explicit target-versus-non-target probability mass matching, leaving cross-entropy to provide ground-truth supervision.

Gradient Norm Matching (Enhancement)

To stabilize the optimization, we introduce a gradient norm matching technique. The final SICD loss is scaled by a detached ratio rr between the gradient magnitude of the full KD loss and the SICD loss:

r=∥∇θLKD∥∥∇θLSICD∥,(11)r = \frac{\lVert\nabla_{\theta}\mathcal{L}_{\mathrm{KD}}\rVert}{\lVert\nabla_{\theta}\mathcal{L}_{\mathrm{SICD}}\rVert}, \tag*{(11)}

where θ\theta represents the student backbone parameters. This normalization ensures that the distillation gradients remain comparable in scale to standard KD, preventing the SICD loss from being overwhelmed by the LCE\mathcal{L}_{\mathrm{CE}} term during early training.

Overall Objective

By integrating the Angular Coordinate Interface (ACI) and the Scale-Invariant Conditional Distillation (SICD), the final training objective of our ACD framework is formulated as:

LACD=LCE+λ⋅r⋅LSICD,(12)\mathcal{L}_{\mathrm{ACD}} = \mathcal{L}_{\mathrm{CE}} + \lambda\cdot r \cdot\mathcal{L}_{\mathrm{SICD}}, \tag*{(12)}

where LCE\mathcal{L}_{\mathrm{CE}} is the standard cross-entropy loss for the classification task, λ\lambda is a balancing hyper-parameter, and rr is the detached gradient norm matching ratio defined in Eq. (11).

Experiments

Experimental Setup

To assess the effectiveness of our method, we consider two types of teacher-student architecture pairs: (1) homogeneous pairs using the same architectures (e.g., VGG16 →\rightarrow VGG8), and (2) heterogeneous pairs involving different architectures (e.g., ResNet50 →\rightarrow MobileNetV1). We included a range of architectures covering different capacity scales, such as ResNet, WideResNet (WRN), ShuffleNetV1/V2, MobileNetV1, and VGG. We compare ACD against logit-based methods (KD [6], DKD [2], ABKD [25], WASB [14], SKD [7]) and feature/relation-based methods (FitNet [18], CRD [23], Reviewkd [27], CD [9], CoCaRS [26]).

Training Details. Experiments run in FP32 on NVIDIA A100 80GB GPUs. For CIFAR-100, students are trained using SGD for 240 epochs with a batch size of 64, momentum of 0.9, and weight decay of 5×10−45 \times10^{-4}. For ImageNet-1K, students are trained for 100 epochs using SGD with a batch size of 256, an initial learning rate of 0.05, momentum of 0.9, and weight decay of 10−410^{-4}. The learning rate is reduced by a factor of 0.1 at epochs 30, 60, and 90.

Main Results

We evaluate ACD on CIFAR-100, ImageNet, and MS-COCO under homogeneous and heterogeneous teacher-student architecture settings. On CIFAR-100, Table 1 shows that ACD outperforms prior logit-based and feature-based methods across all six homogeneous pairs, improving over ordinary KD by 2.51 percentage points on average. Under heterogeneous settings (Table 2), ACD achieves the highest accuracy on all five pairs, exceeding the strongest competing methods by 4.19 points on average. On ImageNet (Table 3), ACD reaches 71.64% and 71.84% for ResNet34→ResNet18 and ResNet50→MobileNetV1, respectively, surpassing the strongest baselines by 0.51 and 0.31 points. For object detection on MS-COCO (Table 4), ACD remains competitive: it achieves the highest AP for

Comparison of Top-1 accuracy on CIFAR-100. We report the individual performance of the teacher and student models under heterogeneous architecture settings.

MethodResNet32x4WRN-40-2VGG13ResNet50ResNet32x4
Teacher79.4275.5974.6479.3379.42
StudentShuffleNetV1ShuffleNetV1MobileNetV1MobileNetV1ShuffleNetV2
69.4369.4360.0960.0973.5
FeatureFitNet [29]74.0973.4964.9864.0675.42
FeatureCRD75.5076.1566.5566.2874.66
FeatureReviewKD77.3977.6865.2662.5277.99
FeatureCD77.9577.8564.9867.9778.13
FeatureCoCaRS77.8576.0168.1268.9078.53
LogitKD [16]76.6477.2367.5768.4477.78
LogitDKD76.7575.9467.5867.5178.43
LogitABKD77.0175.1466.5667.8177.87
LogitWASB76.8977.1065.1365.3577.56
LogitSKD77.9177.5368.6068.4877.02
LogitACD (Ours)81.4681.6273.3073.3083.12

Table 2. Comparison of Top-1 accuracy on CIFAR-100. We report the individual performance of the teacher and student models under heterogeneous architecture settings.

TeacherStudentKD [6]DKD [2]AT [28]ReviewKD [27]RKD [15]SKD [7]CD [9]ACD (Ours)
R34 (73.30)R18 (69.76)69.11 ± 0.2170.88 ± 0.1670.54 ± 0.1570.51 ± 0.1869.94 ± 0.2071.13 ± 0.1271.01 ± 0.1071.64 ± 0.12
R50 (76.14)MV1 (66.51)67.81 ± 0.2470.58 ± 0.1870.90 ± 0.1666.77 ± 0.2371.22 ± 0.1371.53 ± 0.1171.23 ± 0.1571.84 ± 0.14

Table 3. Comparison of Top-1 accuracy on ImageNet with both homogeneous (e.g., R34 → R18) and heterogeneous (e.g., R50 → MV1) teacher-student architectures.

MethodTeacher
APAP50_{50}AP75_{75}APAP50_{50}AP75_{75}APAP50_{50}AP75_{75}
ResNet10142.0462.4845.8842.0462.4845.8840.2261.0243.81
StudentResNet1833.2653.6135.2637.9358.8441.0529.4748.8730.90
ResNet50
FeatureFitNet [18]34.1354.1636.7138.7659.6241.8030.2049.8031.69
ReviewKD [27]36.7556.7234.0040.3660.9744.0833.7153.1536.13
MDPD [29]36.1456.5138.3740.4461.2744.0431.7550.8932.34
LogitKD [6]33.9754.6636.6238.3559.4141.7130.1350.2831.35
DKD [2]35.0556.6037.5439.2560.9042.7332.3453.7734.01
SKD [7]36.7457.0339.7640.6261.6144.6134.2554.5136.29
ACD (Ours)36.9157.3639.5140.1761.6743.7933.9854.6536.22

Table 4. Object detection results on MS-COCO using Faster R-CNN-FPN.

ConfigurationV13 → V8V13 → MNV1
KD (Aff.)74.39 ± 0.4367.78 ± 0.40
CI75.10 ± 0.1966.60 ± 0.16
CI + ETF (ACI)75.80 ± 0.1568.10 ± 0.14
ACI + DS78.55 ± 0.0570.98 ± 0.02
ACI + SICD + GM80.52 ± 0.0273.30 ± 0.01

Table 5. Component ablation. Accuracy (%); best results are bold. Aff.: affine head; CI: cosine interface; ACI: CI with equiangular tight frame (ETF) initialization; DS: dual standardization with full KL; GM: gradient matching. V13/V8: VGG13/VGG8; MNV1: MobileNetV1.

ResNet101→ResNet18 and the highest AP50_{50} across all three pairs, while SKD retains stronger AP on the other two pairs. Together, these results demonstrate consistent classification improvements and competitive transfer to object detection

Ablation Studies

Table 5 evaluates the contributions of ACI, ETF initialization, logit standardization, and SICD on VGG13→VGG8 and VGG13→MobileNetV1. Replacing the affine head with ACI improves VGG8 by 0.71 percentage points but reduces MobileNetV1 accuracy by 1.18 points, indicating that an angular interface alone does not consistently improve distillation. ETF initialization subsequently increases accuracy by 0.70 and 1.50 points, while standardized full KL provides further gains of 2.75 and 2.88 points. The complete ACI+SICD configuration reaches 80.52% and 73.30%, improving over ordinary KD by 6.13 and 5.52 points, respectively. Relative to the preceding standardized-full configuration, the gains are 1.97 and 2.32 points. This final comparison evaluates conditional supervision together with gradient-strength matching, rather than isolating either component.

Further Analysis

Gradient pathways of the angular interface. For the temperature-scaled KL objective, the gradient with respect to student features h\mathbf{h} is

gh=JhTT(qT−pT),Jh=∂z∂h,(13)\mathbf{g}_{h} = J_{h}^{\mathsf{T}}T(q_{T}-p_{T}), \qquad J_{h} = \frac{\partial z}{\partial\mathbf{h}}, \tag*{(13)}

where pTp_{T} and qTq_{T} denote the teacher and student distributions at temperature TT. The angular interface can therefore affect both the probability error and the classifier Jacobian. At a fixed ImageNet CE-trained checkpoint, we cross the affine and angular Jacobians with the corresponding probability-error vectors. Relative to the original affine gradient, replacing only the probability error, only the Jacobian, or both yields mean direction deviations of 0.293, 0.062, and 0.283, respectively (Figure 7). The corresponding mean gradient-norm ratios are 1.247, 0.730, and 0.887. These fixed-state diagnostics expose two interacting paths through which the angular interface changes the local distillation signal; the crossed combinations are diagnostic constructions rather than trained variants.

Figure 7. Sources of KD feature-gradient changes.

Non-target relation agreement. We evaluate four completed ACI+SICD models on the same 1,024 CIFAR-100 test images. After independently standardizing teacher and student logits, we measure total variation between their non-target conditional distributions at T=4T = 4. Figure 6 presents the image-level discrepancy distributions and their summary statistics. Mean conditional TV ranges from 0.0578 to 0.0616, with similar distributions within the two ResNet pairs and within the two WRN pairs. These measurements quantify relation agreement at the trained endpoints, rather than improvements over matched baseline models.

Non-target relation discrepancies

Figure 6. Non-target relation discrepancies. (a) Empirical cumulative distributions of conditional TV. (b) Medians (circles), means (diamonds), and image interquartile ranges (segments).

Compatibility with feature-based distillation. We examine compatibility with two selected feature-distillation pipelines by retaining their feature losses and adapters while comparing affine + KL, cosine + KL, and cosine + SICD. As shown in Table 6, cosine + SICD achieves 78.97% with AT and 70.43% with ReviewKD-style, exceeding their affine + KL baselines by 1.67 and 1.95 percentage points, respectively. Under the same cosine interface, replacing KL with SICD yields gains of 1.37 and 2.95 percentage points. The cosine interface alone reduces accuracy in the ReviewKD-style pipeline, so its improvement cannot be attributed to the classifier change alone. These single-seed case studies support SICD as an output-loss replacement within cosine-head feature-distillation pipelines, while leaving the feature-transfer components unchanged; they do not establish universal gains across methods or architecture pairs.

MethodT → SKDACIACI + SICD∆
AT [28]R32×4 → SV177.3077.6078.97+1.67
ReviewKD [27]V13 → MNV168.4867.4870.43+1.95

Table 6. Compatibility with feature-based distillation on CIFAR-100. Top-1 accuracy (%). ∆: gain over KD in percentage points. Best results within each row are bold. T/S: teacher/student; R32×4: ResNet32×4; SV1: ShuffleNetV1; V13: VGG13; MNV1: MobileNetV1.

Visualizing spatial attention. We complement the output and gradient diagnostics with Grad-CAM++ visualizations on ImageNet. Figure 8 compares CE-trained and KD-trained MobileNetV2 students using identical inputs, ground-truth target classes, and the last convolutional block before global pooling. Both models highlight the foreground objects, while exhibiting different activation patterns, particularly around the dog’s face. These examples illustrate spatial differences between the two training objectives; they do not yet evaluate ACD or establish improved localization.

Grad-CAM++ visualizations for input, SKD, and Ours on ImageNet

Figure 8. Grad-CAM++ visualizations on ImageNet.

Conclusion

We presented Angular Conditional Distillation (ACD), which revisits knowledge distillation through the geometry of the classifier interface. ACI establishes a normalized, bias-free cosine readout for the student, while SICD transfers weighted non-target conditional relations through standardized logits, omitting explicit target-mass matching. Together, these components distinguish how class relations are represented from which aspects of the teacher distribution are distilled. Experiments demonstrate the effectiveness of this design on evaluated teacher–student pairs, while support its compatibility with existing feature-transfer objectives. Overall, our findings highlight classifier parameterization and supervision selection as complementary considerations in knowledge distillation. Future work will investigate broader settings and more efficient implementations of gradient matching.

LLM Usage Statement. Large language models were used to assist with language editing, code development, and figure preparation. All methodological decisions, experimental results, and scientific claims were reviewed by the authors, who take full responsibility for the final manuscript.

References

  1. [1]Lucas Beyer, Xiao Hua Zhai, Amelie Royer, Larisa Markeeva, and Rohan Anil Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. 3
  2. [2]Peng Guan Chen. Distilling knowledge via knowledge review. 1, 6, 7
  3. [3]Zhihao Chi, Tu Zheng, Heng Jia Li, Zheng Yang, Bo Xi Wu, Bin Bin Lin, and Deng Cai. NormKD: Normalized logits for knowledge distillation. 2023. 2, 3arxiv.org/abs/2308.00520
  4. [4]Hamza Etcibasi and Ramazan Gokberk Cinbis. You don’t need aligned representations: Knowledge distillation via random prototype spaces, 2026. 2
  5. [5]Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, No-jun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. 2
  6. [6]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, 2015. 2, 6, 7arxiv.org/abs/1503.02531
  7. [7]Hyeon-Jin Jeong, Han-Jin Lee, and Seok-Hwan Choi. Streamlined knowledge distillation. 6, 7
  8. [8]Dae Ung Jo, Jongin Lim, YoungJoon Yoo, and Daeho Um. Codebook-guided cross-modal knowledge distillation for structurally heterogeneous features, 2026. 2arxiv.org/abs/2609.37243
  9. [9]Seungku Kang and Songkuk Kim. What do students learn? A feature-level analysis of dark knowledge, 2026. 6, 7
  10. [10]Minsang Kim and Seung Jun Baek. Explain in your own words: Improving reasoning via token-selective dual knowledge distillation, 2026. 3arxiv.org/abs/2603.13260
  11. [11]Yu Fan Liu, Jia Jiong Cao, Bing Li, Chun Feng Yuan, Wei Ming Hu, Yang Xi Li, and Yun Qiang Duan. Knowledge distillation via instance relationship graph. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7089–7097, Long Beach, CA, USA, 2019. IEEE. 2DOI
  12. [12]Jia Ming Lv, Hao Yuan Yang, and Pei Hua Li. Wasserstein distance rivals kullback-leibler divergence for knowledge distillation, 2024. 2arxiv.org/abs/2412.08139
  13. [13]Amir M. Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei, Elaheh Badali Golezani, Zeynab Yasamani Ghamchi, Vida Ramezanian, Alireza Taherian, Kimia Dinashi, Amirali Miri, and Shohreh Kasaei. A comprehensive survey on knowledge distillation, 2025. 1arxiv.org/abs/2503.12067
  14. [14]Byeonghu Na, Donghyeok Shin, Yeongmin Kim, Mina Kang, and Il-Chul Moon. WASD: Wasserstein-based knowledge distillation for large language models, 2026. 6arxiv.org/abs/2610.07706
  15. [15]Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational Knowledge Distillation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3962–3971, Long Beach, CA, USA, 2019. IEEE. 7arxiv.org/abs/1904.05068
  16. [16]Bao Yun Peng, Xiao Jin, Jia Heng Liu, Dong Sheng Li, Yi Chao Wu, Yu Liu, Shun Feng Zhou, and Zhao Ning Zhang. Correlation congruence for knowledge distillation. 1, 2
  17. [17]Nikhila Ravi, Valentin Gabeur, Yuan Ting Hu, Rong Hang Hu, Chaitanya Ryali, Teng Yu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Jun Ting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos, 2024. 1arxiv.org/abs/2408.00714
  18. [18]Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for Thin Deep Nets, 2015. 1, 2, 6, 7DOI
  19. [19]Ayan Sengupta, Vaibhav Seth, and Tanmoy Chakraborty. Distilling what matters: Confidence-aware selective distillation for large language models, 2026. 3arxiv.org/abs/2609.36734
  20. [20]Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jian Yuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3, 2025. 1arxiv.org/abs/2508.10104
  21. [21]Jing Chen Sun, Shao Bo Han, Deep Patel, Wataru Kohno, Can Jin, and Chang You Chen. Uncertainty-aware knowledge distillation for multimodal large language models, 2026. 3arxiv.org/abs/2603.21426
  22. [22]Shang Quan Sun, Wen Qi Ren, Jing Zhi Li, Rui Wang, and Xiao Chun Cao. Logit standardization in knowledge distillation. 2024. 2arxiv.org/abs/2403.01427
  23. [23]Yong Long Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation, 2022. 1, 2, 6
  24. [24]Fred Tung and Greg Mori. Similarity-preserving knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1365–1374, Seoul, Korea (South), 2019. IEEE. 2arxiv.org/abs/1907.09682
  25. [25]Guang Hui Wang, Zhi Yong Yang, Zi Tai Wang, Shi Wang, Qian Qian Xu, and Qing Ming Huang. ABKD: Pursuing a proper allocation of the probability mass in knowledge distillation via Sα$-$β$-divergence, 2025. 2, 6arxiv.org/abs/2505.04560
  26. [26]Feng Ming Yu, Hai Wei Pan, Ke Jia Zhang, Chun Ling Chen, Jian Guan, and Bao Ying Ma. CoCaRS: Correlation calibration-based redundancy suppression for heterogeneous knowledge distillation, 2026. 2, 6arxiv.org/abs/2607.27054
  27. [27]Li Yuan, Francis EH Tay, Gui Lin Li, Tao Wang, and Jia Shi Feng. Rectifying knowledge distillation via label smoothing regularization. 2020. 2, 6, 7arxiv.org/abs/1909.11723
  28. [28]Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer, 2017. 7
  29. [29]Yu Tong Zhang, Jia Xin Chen, Hong Lin Chen, Kai Qi Zheng, Sheng Cai Liao, Han Wen Zhong, Wei Xin Li, and Yun Hong Wang. Memory-efficient transfer learning with fading side networks via masked dual path distillation, 2026. 7arxiv.org/abs/2604.09088
  30. [30]Bo Rui Zhao, Quan Cui, Ren Jie Song, Yi Yu Qiu, and Jia Jun Liang. Decoupled knowledge distillation, 2022. 1, 2, 3arxiv.org/abs/2203.08679
  31. [31]Yi An Zhao, Wen Yu Lv, Shang Liang Xu, Jin Man Wei, Guan Zhong Wang, Qing Qing Dang, Yi Liu, and Jie Chen. DETRs Beat YOLOs on Real-time Object Detection. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16965–16974, Seattle, WA, USA, 2024. IEEE. 1arxiv.org/abs/2304.08069

Paper details

vlnc:W_jr57ewqy
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents